[gouyt13]

Summary

Remove duplicated logic in four areas. All changes are internal: no public C++ API, Python API, saved-index format or numerical result changes. One commit per area:

• OpenMP error capture and team sizing
  • Add index_detail::omp_for_dynamic. It replaces 8 copies of the dynamically scheduled ParallelException capture/rethrow loop: 5 in the QG builder, 2 in IVF, 1 in PiPNN.
  • Use ParallelException in the 4 clustering parallel regions and in the OpenMP branch of parallel_for, instead of per-worker exception_ptr vectors.
    • After a failure, the remaining iterations are now skipped, matching the std::thread branch.
    • When several workers fail, the first captured exception is rethrown rather than the one from the lowest worker index. No test or document relied on that order.
  • Add detail::openmp_team_size, shared by compute_centroid, exact_nn and QuantizedGraph batch search.
  • New tests cover both helpers.
• Core
  • Share the direct-code rescale between the one-bit encoder and xy code_factors.
  • Share sign-bit unpacking between the v1 API and HNSW reconstruction.
• Python bindings
  • Derive IvfIndex/SymqgIndex build state from the native index; this also removes a try/catch that only resynchronized a flag.
  • Remove the unused rotator_from_string.
  • Share NumPy rank conversion and the prefilled search outputs.
  • Drop C++ default arguments that duplicated the py::arg defaults; only the py::arg defaults ever took effect.
  • Let pybind11 copy k-means results instead of a manual memcpy.
• HNSW
  • Raw construction and quantized add share one greedy upper-layer descent and one ef_construction layer search. They are private member templates over the distance function and the construction-data prefetch.

Left out on purpose:

• Sharing the estimator formulas in core/estimator.hpp and the query corrections in core/query.hpp. Both are inlined into the HNSW search kernels and changed their code generation. Pinned measurements could not rule out a 1–4% search QPS loss.
• Merging the HNSW get_bin_est_direct/get_full_est_direct helpers, for the same reason.

With these left out, the HNSW search kernels compile to identical machine code (generic, AVX2, AVX-512 core and AVX-512 VPOPCNTDQ).

Testing

Linux x86-64 host: Xeon Gold 6418H, 2 sockets; GCC 14.3; RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.

• CTest (--no-tests=error):
  • Release: 412 passed, 1 skipped (the placement test, which requires OpenMP binding).
  • ASan/UBSan Debug, with asserts enabled: 412 passed, 1 skipped.
  • Core-only RABITQ_USE_OPENMP=OFF: 167 passed.
  • With OMP_PROC_BIND=true, the ThreadCount tests (11), including the bound parallel_for path, pass in both builds.
• check-format.sh, full check-tidy.sh build-tidy (clang-tidy 23, including bindings) and git diff --check pass.
• Python: the extension was built and installed into a temporary prefix, and the imported module path was checked.
  • pytest tests/python: 794 passed, 2 skipped (wheel-only tests).
  • Signatures, defaults and docstrings of every public class and function (2382 lines) are identical to a build of the pre-change commit.
• Bitwise parity with the pre-change build (GCC): a harness writes about 397 KB of raw outputs and they match byte for byte. It covers:
  • core split and xy encoders and estimators, with 0–8 extra bits, L2 and IP, including a zero residual;
  • FastScan query corrections;
  • the v1 API: encode, convert_codes and compute;
  • single-threaded HNSW construct/add/search and IVF search on a shared saved index.
• HNSW construction equivalence: single-threaded builds of 50k×128 synthetic clustered vectors (45k construct plus 5k add) write byte-identical index files for L2/IP with 4 bits and L2 with 1 and 9 bits.

Benchmarks

The process was pinned to one idle core and its NUMA node (numactl --physcpubind=20 --membind=0), with baseline and candidate runs alternated. Shared host: load average 3–7 during the runs; CPU frequency was not locked.

HNSW, synthetic data. 50k×128 vectors in 64 Gaussian clusters, M=16, efConstruction=200, single thread. Search uses 1000 queries, k=10, with recall measured against exact top-10.

Config	Build time (pair ratio new/base, median)	Add time	Search QPS ef=50/100/200	Recall@10
L2, 4 bits (10 pairs)	1.011	1.007	0.995 / 0.973 / 0.986	0.8889 (unchanged)
L2, 9 bits (10 pairs)	0.993	0.996	1.009 / 0.999 / 1.007	0.9786 (unchanged)

The search code is byte-identical, so its spread (individual pair ratios between 0.95 and 1.13) shows the host noise. Construction and add stay within it. No speedup is claimed.

Stable core benchmark (rabitq_core_benchmark 128 {4,9} 4096 64 7, 6 alternating runs): the median per-run ratio is between 0.944 and 1.038 for encode_packed, prepare_query, compute_packed and convert_to_unpacked. Checksums are identical.

Not run locally: NEON, Windows and macOS builds, clang parity for this exact combination (the core changes were checked with clang separately), and real-dataset benchmarks.