[gouyt13]

Summary

Remove duplicated logic in the index and clustering code. Results are unchanged, and there are no public API, Python API or saved-format changes. One commit per area:

• HNSW
  • The level-0 record layout is derived once (set_record_layout); load validates against the same derivation instead of recomputing it.
  • Label and cluster-ID writes go through set_external_label and a new set_raw_clusterid. The two raw-pointer accessors used only for writes are removed.
  • The default ef_search of 10 is a constant applied once in search_knn_sorted_direct, replacing a member read by five ISA wrappers.
• QG, PiPNN and IVF
  • QuantizedGraph sets raw_dist_func_ in initialize(), which the constructor and load share.
  • QGBuilder::write_row uses one lambda for its three ways of producing a rotated row: cached, rotated raw vector, or reconstructed codes.
  • PiPNN copies ID-selected rows through one helper and finishes merged small buckets through one lambda.
  • IVF uses one constant for the removal marker in remove and load.
• Shared helpers
  • mem_prefetch_l1 and mem_prefetch_l2 share one unrolled template.
  • k-means computes double-precision squared norms and L2 distances with two helpers, keeping the same index order.
  • The v1 API checks UnpackedU16 codes with one helper; error messages are unchanged.

Left out on purpose:

• The rotator byte counts in the IVF and QG loaders: each sits inside its own overflow-checked file accounting.
• The two-line batch-size checks: their messages differ.
• The serial branch of IVF search_batch: removing it would put single-query batches inside an OpenMP region.

Testing

Linux x86-64 host: Xeon Gold 6418H, 2 sockets; GCC 14.3; RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.

• CTest (--no-tests=error):
  • Release: 424 passed, 1 skipped (the placement test, which requires OpenMP binding).
  • ASan/UBSan Debug: 424 passed, 1 skipped.
  • Core-only RABITQ_USE_OPENMP=OFF: 173 passed.
  • Each commit builds on its own.
• Python: 810 passed, 2 skipped (wheel-only tests).
• Static checks: check-format.sh, full check-tidy.sh build-tidy (clang-tidy 23) and git diff --check pass.
• Equivalence:
  • Single-threaded HNSW builds of 50k×128 synthetic vectors (45k construct plus 5k add) write byte-identical index files for L2/IP with 4 bits and L2 with 1 and 9 bits.
  • Recall, k-means objectives, exact assignments and core benchmark checksums are identical.

Benchmarks

Single thread, pinned to one idle core and its NUMA node (numactl --physcpubind=20 --membind=0). Baseline (main 0734ec2) and candidate runs were alternated, 6–10 pairs per row. Removing an HNSW member shifts later field offsets, so object code cannot be compared directly; these runs check performance instead.

Ratio is candidate/baseline (median).

Workload	Ratio
HNSW 50k×128, M=16, efC=200: build / add, 4 bits	1.009 / 0.997
HNSW search QPS ef 50 / 100 / 200, 4 bits	1.014 / 1.033 / 1.016
HNSW search QPS ef 50 / 100 / 200, 9 bits	0.997 / 0.982 / 1.005
SymphonyQG 100k×128 search QPS, ef 50 / 100 / 200	1.022 / 0.998 / 1.015
exact_assign 262144×128, k=1024	0.984
exact_assign 65536×960, k=1024	0.999 (10 pairs; a first 6-pair run gave 1.046)
QGKMeans 30k×768, k=256, 5 iterations	1.011
rabitq_core_benchmark 128 dims, 4 and 9 bits: all operations	0.96–1.08

Every result is within this host's noise (single-pair ratios vary by about ±5–10%), and no speedup is claimed. Shared host: load average 3–11 during the runs; CPU frequency was not locked.

Not run locally: NEON, Windows and macOS builds.