[gouyt13]

Summary

Fix parallel HNSW centroid construction and routing, and define zero-range scalar quantization consistently across backends. Also retain the reviewed HNSW worker reuse and PiPNN scheduling/local-neighbor changes.

• Draw hnswlib insertion levels under the registration lock and read the entry point while holding the global lock. Concurrent IVF routing now passes ef per search instead of changing shared state under a whole-search mutex.
• Reuse OpenMP workers for top-level HNSW query batches while preserving existing graph-insertion and nested-call scheduling.
• Size PiPNN root/sketch tiles and teams by available work, keep Eigen parallelism local to each worker, and select the two nearest local neighbors in one triangular-distance traversal with (distance, ID) ordering.
• Return zero codes for delta == 0 in scalar, generic, AVX2, AVX-512, and NEON quantization. Add regressions for reused LUTs, backend/tail parity, concurrent routing, insertion RNG state, persistence, and PiPNN reference results.

No public API, saved-index format, or dependency changes. Formatting follows clang-format 23; vendored files retain their existing style.

Testing

Linux x86-64, Intel Xeon Gold 6418H, Clang 23.1.4, portable builds (RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF), existing Python 3.13 environment:

• Full C++ CTest suite: 405 passed, 1 conditional skip (406 registered). The separate OpenMPBinding test passes. Additional bound-mode initializer/PiPNN/thread tests: 23 passed, 1 expected affinity-test skip.
• Rebuilt and reinstalled the wheel; verified installed extension bytes against that wheel. Full Python suite, including FAISS comparisons and required C++ examples: 717 passed, 2 platform-specific wheel skips.
• OpenMP-disabled core/FastScan: 168/168 passed. Installed CMake consumer: 3/3 passed.
• Full production/bindings clang-tidy and focused analysis of all modified C++ tests pass. C++ format, Python format/lint, and git diff --check pass.
• Both new zero-range/LUT regression tests fail on unchanged main and pass with the fix. Generic, AVX2 and AVX-512 quantizers execute on this host; ARM64/NEON and Windows/macOS remain for CI.
• Focused HNSW initializer TSan: 10/10 passed with detect_deadlocks=0; the unchanged baseline reproduces a data race. Default TSan separately reports a vendored hnswlib lock-order warning on both baseline and this branch, so this is not a clean unrestricted TSan result.
• Full ASan/UBSan: 392 passed, 13 failed, 1 skipped. All failures report the same existing null-pointer-offset operation in vendored Eigen AVX512/GemmKernel.h:815. All 12 failing tests that exist on unchanged main reproduce there with identical sanitizer flags; the remaining failure is the new PiPNN reference test reaching that same kernel. No vendor patch or sanitizer suppression is included.

The initial clang-tidy invocation also included tests/examples, which CI excludes; it reported existing negative-test diagnostics. The new concurrency test's thread-vector reservation was fixed, and the changed-test analysis subsequently passes.

Matched performance checks

These are bounded shared-host measurements, not a general performance guarantee. Baseline is 4c1dfdb. Both variants use Clang 23 Release, portable runtime dispatch (AVX-512 available), CPU affinity 16–23, eight workers, and alternating baseline/current run order. Input loading is excluded. Exact Recall@10 uses exhaustive float64 L2 ground truth.

HNSW uses one shared persisted 4-bit index over 4,096 SIFT centroids (128 dimensions), M=16, construction/search ef=64, and the same queries. Median batch latency over ten samples, each averaging 30 searches:

Batch	Baseline, 8 threads	Current, 8 threads	Recall@10, both
8	193.1 us	47.0 us	0.9125
32	351.0 us	197.8 us	0.859375
128	870.9 us	727.5 us	0.880469

ID checksums match across variants/thread counts, and repeated searches compare IDs and distances exactly within each run. Single-thread medians were 270.8/272.9 us, 1216.8/1350.7 us, and 4394.5/4504.3 us (baseline/current); no single-thread speedup is claimed.

For raw SymphonyQG with PiPNN initialization, 4,096 real centroids, degree 32, build ef=64, seed 42, eight build threads and 128 queries, six measured builds per variant after warmup:

Dataset	Build ms, baseline/current	Query ms at ef=64, baseline/current	Recall@10 at ef=16 / 64 / 128, both
SIFT, d=128	50.63 / 43.74	1.897 / 2.070	0.885938 / 1 / 1
GIST, d=960	124.73 / 104.10	3.307 / 3.263	0.922656 / 0.995313 / 0.998438

Build timings include the complete QG construction; query timings use one thread. Recall matches in every measured repetition; query timings do not improve uniformly.

An additional synthetic routing check (8,192 centroids, d=128, 256 queries, nprobe=16, ef=768) measures 123.83 ms -> 16.13 ms at eight threads, with matching candidate ID checksums and no serial/parallel mismatches.