[gouyt13]

Summary

Four performance changes. Results are unchanged: identical LUT bytes, codes, assignments, objectives and search results. No public API or saved-format changes.

• FastScan LUT preparation.
  • pack_lut now also returns the table's minimum and maximum (generic, AVX2, AVX-512 and NEON kernels), instead of a second data_range pass.
  • Finite float LUTs are then quantized by a non-negative uint8 kernel: truncation plus a half test. LUT entries are at least their minimum, so this equals the existing round-half-away quantizer. It shares the core AVX2/AVX-512 quantizer body through a NonNegative template parameter (scalar_quantize_nonnegative_uint8_*).
  • NaN, infinite and zero bounds fall back to the original reduction (keeping signed-zero behavior), and Lut<double> is unchanged.
  • The ISA translation units use compile-time constants and plain comparisons, so unoptimized builds export no additional ISA-compiled std:: functions.
• v1 API input check. check_input tests the IEEE exponent bits with an integer reduction that compilers vectorize. It rejects the same inputs with the same message.
• QGKMeans query-sum cache. It is now enabled from 512 padded dimensions instead of 1024.
• Hash visited set. The direct table is twice as large, so fewer collisions spill into the unordered_set. The cost is twice the table memory and clear time.

A larger exact_assign point block (128 → 256) was also tried and dropped: at 960 dimensions it was 16% slower (0 of 6 pairs faster), and it showed no gain at 128 dimensions.

Testing

Linux x86-64 host: Xeon Gold 6418H, 2 sockets; GCC 14.3; RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.

• CTest (--no-tests=error):
  • Release: 418 passed, 1 skipped (the placement test, which requires OpenMP binding).
  • ASan/UBSan Debug: 418 passed, 1 skipped.
  • Core-only RABITQ_USE_OPENMP=OFF: 173 passed.
• New tests:
  • lut_prepare_test compares fused tables, extrema and uint8 codes byte for byte against the original path on every available backend, together with floating-point exception flags, in all four rounding modes. It also checks half-way boundaries against std::round in the documented round-to-nearest mode, plus signed zeros, denormals, NaN and infinities, and the double LUT.
  • Under Clang, the scalar tails of the existing AVX2/AVX-512 quantizers round half-way values down in downward and toward-zero modes, because the inlined std::round uses a mode-dependent addition. This was already the case on main, and those modes are outside the documented contract, so the std::round comparison runs only in round-to-nearest.
  • The v1 API rejects non-finite values at every input position.
  • QGKMeans cached and uncached sums are compared at 511, 512, 960 and 1024 dimensions.
• Python: 806 passed, 2 skipped (wheel-only tests).
• Static checks: check-format.sh, full check-tidy.sh build-tidy (clang-tidy 23) and git diff --check pass.
• NEON: a clang --target=aarch64-linux-gnu syntax and semantic check passes. It was not built into object code or run.
• Code generation:
  • With -O0, the AVX2/AVX-512 FastScan objects export the same set of weak std:: symbols as main.
  • The existing scalar_quantize_uint8_avx512 compiles to identical machine code; the AVX2 version differs only by a register swap.

Benchmarks

The process was pinned to one idle core and its NUMA node (numactl --physcpubind=20 --membind=0), single-threaded. Baseline (main) and candidate runs were alternated, 6–8 pairs per row. Synthetic clustered Gaussian data with fixed seeds.

Ratio is candidate/baseline (median, with the range); "wins" counts the pairs in which the candidate was better. Quality was identical in every row.

Change	Workload	Ratio	Wins
LUT	SplitBatchQuery::reset, d=128, standard / HACC	0.55 / 0.64	16/16
LUT	SplitBatchQuery::reset, d=960, standard / HACC	0.40 / 0.56	16/16
LUT	QGKMeans training, 30–50k points, k=256, 5 iterations, d=512 / 768 / 1024	0.87 / 0.82 / 0.86	18/18
LUT	IVF search, 200k×128, 1024 lists, nprobe 8 / 32	0.96–1.02	within noise
Input check	rabitq_core_benchmark prepare_query, 4 and 9 bits	0.68	12/12
Input check	encode_packed	0.98–0.99	within noise
Query-sum cache	QGKMeans d=768 (this change only)	0.97 (0.93–1.00)	5/6
Visited set	HNSW 50k×128, M=16: build / search ef 50–200 (this change only)	0.96 / 1.08–1.10 QPS	8/8 / 23/24
Visited set	SymphonyQG 100k×128 search, ef 50–200 (this change only)	0.955–0.993 QPS	3/8 per ef; within the ±5% noise

The LUT gain is in query preparation. It is visible end to end where preparation is a large share of the work (QGKMeans assignment), but not in IVF search.

Limits:

• Single-threaded only.
• Indexes of 50k–200k points; the visited-set clear cost grows with the table, and million-point indexes were not measured.
• Synthetic data only.
• Shared host: load average 3–13 during the runs; CPU frequency was not locked.

Not run locally: NEON execution, Windows and macOS builds.