Repository navigation
fix: bound IVF initializers, QG degree sums, HNSW capacity errors and ISA helper linkage - #192
Merged
Merged
Conversation
QGBuilder::avg_degree passed 0U to std::accumulate over uint32_t degrees, so the sum was an unsigned int and wrapped once the total degree exceeded 2^32 - 1, for example about 67 million points at degree 64. Start from size_t{0}. The new test sets 65 degrees of 2^31 and fails before the fix.
FlatInitializer::centroids_distances partially sorted to nprobe and copied nprobe entries into the caller's vector without checking either bound, so nprobe above the cluster count or an undersized output read and wrote out of range. HNSWInitializer also wrote into the output without sizing it. Both now clamp nprobe to the cluster count and resize the output, as FlatRaBitQInitializer already did, and the Initializer interface documents that contract. IVF always passed a clamped, presized vector, so index search is unchanged. The new test fails before the fix with a segmentation fault.
Out-of-memory in construct and add threw std::runtime_error, while load and resize threw std::bad_alloc, so Python saw RuntimeError or MemoryError for the same failure. Throw std::bad_alloc for the link-list allocations, and drop the null checks after huge_page_allocate, which throws instead of returning null. A zero capacity reached the allocator, which returns null for zero bytes: the constructor reported "Not enough memory" and resize(0) on an empty index threw std::bad_alloc. Reject both with std::invalid_argument (Python ValueError) before allocating, and document the minimum. Also correct the entry-point promotion comment in add_point.
Headers compiled into generic, AVX2 and AVX-512 translation units defined external-linkage inline functions and templates (pack_4bit_block, load_excode_tail, bitops::load_word and countr_zero32, contribute_ip*, BoundedRescaleState's constructor). When the compiler does not inline them, each object emits a weak copy and the linker keeps one for every caller, so a lower-ISA path could run code compiled for a higher ISA. docs/development.md already requires internal linkage for these headers. Use anonymous namespaces or static inline. Release object code is unchanged; at -O0 the shared weak symbols drop from 10 (GCC) and 11 (Clang) to 0.
- docs/development.md: HNSW has kDefaultEfSearch, not default_ef_search_, and query search inserts removed points into search_pool, not candidate_set. - HNSW: the quantized-construction, live-node and bounded-queue comments described old behavior or the wrong declaration. - IVF: the save/load format comments omitted the initializer payload written first, and ex_bits_ is 0 with raw reranking. - FastScan: kPos, the 1-bit code offsets and the HACC LUT layout comments were misleading or stale. - Core: unpack_sign_bits accepts any 8-coordinate tail, not only 32. - The k-means overflow message named size_t although the check uses ptrdiff_t, and an AVX-512 static_assert named a constant that does not exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five fixes found while auditing the library. Each commit has its own fix and, where behavior changes, a regression test that fails before the fix.
avg_degreeaccumulateduint32_tdegrees into anunsigned int(initial value0U), so the total wrapped above 2^32 − 1 (for example about 67 million points at degree 64). It now sums insize_t.FlatInitializer::centroids_distancespartially sorted tonprobeandmemcpy'dnprobeentries into the caller's vector without checking either bound;HNSWInitializeralso wrote without sizing the output. Both now clampnprobeto the cluster count and resize the output, asFlatRaBitQInitializeralready did.IVFalways passed a clamped, presized vector, so index search is unchanged; direct callers of the public initializers could read or write out of range.construct/addthrewstd::runtime_errorwhileload/resizethrewstd::bad_alloc; all now throwstd::bad_alloc. A zero capacity reached the allocator, which returns null for zero bytes; the constructor andresize(0)now reject it withstd::invalid_argument. Null checks afterhuge_page_allocate, which throws instead of returning null, are removed.docs/development.mdalready requires internal linkage here.static_assert, plus two stale references indocs/development.md.Python-visible changes:
HnswIndex(max_elements=0, ...)now raisesValueError("HNSW max_elements must be greater than zero")instead ofRuntimeError("Not enough memory"), and HNSW allocation failure during build/add raisesMemoryErrorinstead ofRuntimeError. The k-means input-overflow message now saysptrdiff_t. No other API, Python keyword or saved-index format changes.Testing
Linux x86-64 (Xeon Gold 6418H), GCC 14.3 and Clang 23,
RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF.--no-tests=error): 427 passed, 1 skipped (the OpenMP placement test). Three new tests:QGConstructionTest.AverageDegreeDoesNotWrapAboveUint32Range,IvfInitializerTest.CentroidSearchSizesOutputAndClampsProbeCount,HnswResizeTest.RejectsZeroCapacity.src/simdsource at-O0with its CMake ISA flags, weakrabitqlibsymbols defined under more than one ISA flag set drop from 10 (GCC) and 11 (Clang) to 0. In Release, all 26 SIMD and FastScan objects keep identical.textsizes.check-format.sh, fullcheck-tidy.shwith clang-tidy 23 (bindings on),git diff --check.check-python.sh,pytest tests/python795 passed, 18 skipped (the C++ clustering-example tests, which need a local example build, and wheel-only tests);tests/test_pyright.pyandtests/test_mypy.pypass.