Repository navigation
fix: harden IVF file handling, filtered test runs, and formatter scripts from the audit - #196
Merged
Merged
Conversation
IVF::load accepted any bytes after the ID section while the HNSW and SymphonyQG readers reject a payload that is not exactly the expected length. IVF::write_file set the stream exception mask before checking that the file opened, so an unwritable destination surfaced as the generic basic_ios::clear message. FlatInitializer::save/load now check the stream like FlatRaBitQInitializer; through IVF the stream already carried an exception mask, but the class is public.
A zero search window inserted no candidates, so every row was completed by random fill without any error. The Python wrapper already rejected the value and the docs call it required positive.
…tion count niter is a documented maximum with early stopping; reserving it up front allocated 32 bytes per possible iteration after centroids and assignments had already been resized.
…format scripts format-changed.sh read the helper's output through process substitution, so a failing helper formatted nothing and exited 0; it now captures the list and checks the helper's status like check-format.sh. The scripts also used xargs --no-run-if-empty and realpath --relative-to, which BSD userlands lack although macOS is a documented host.
…les per process GoogleTest exits 0 when a --gtest_filter matches no test, so a renamed suite silently removed the OpenMPBinding check and parts of the Intel SDE coverage; the CTest entry now requires its one expected test and the SDE step checks each filter pattern lists a test. The two consumer ctest runs without --no-tests=error gain it. Test temp files carry the process ID so concurrent suites in one temp directory cannot delete each other's files, and the no-SIMD HACC test asserts the generic backend ran.
MSVC's OpenMP runtime has no thread placement, so the test skips itself there; the CTest gate only needs to prove the filter selected it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #195: the audit's Suggestion-level items that need no contract decision. Five independent commits.
Library
fix(ivf)—IVF::loadaccepted any bytes after the ID section (HNSW and SymphonyQG reject a payload that is not exactly the expected length); it now throwsIVF index file has unexpected trailing dataand leaves the target index unchanged.IVF::write_filechecked nothing before setting the stream exception mask, so an unwritable destination surfaced asbasic_ios::clear: iostream error; it now throwsIVF cannot open index file for writinglike the other writers.FlatInitializer::save/loadcheck the stream asFlatRaBitQInitializeralready did. Regression tests for all three; the frozen v0.5.2 and pre-per-cluster fixtures still load.fix(symqg)— the C++QGBuilderacceptedef_construction == 0and silently built a graph from reverse edges and random fill; it now throwsstd::invalid_argument(the Python wrapper already did).fix(clustering)—iteration_stats.reserve(niter)allocated for the documented maximum iteration count (32 GB forniter = 1e9) aftercentroidsandassignmentshad been resized; the reserve is gone.Scripts
build—format-changed.shread the file-list helper through process substitution, so a failing helper formatted nothing and exited 0 (the same defectcheck-format.shwas already fixed for); it now captures the list and checks the helper's status. The three formatter scripts no longer usexargs --no-run-if-emptyorrealpath --relative-to, which BSD userlands lack although macOS is a documented host;xargs -ris accepted by GNU and BSD, and absolute paths are normalized withcd && pwd -P.Tests and CI
test— GoogleTest exits 0 when a--gtest_filtermatches nothing, so a renamed suite could silently remove theOpenMPBindingCTest entry and parts of the Intel SDE coverage. The CTest entry now requires its one expected test viaPASS_REGULAR_EXPRESSION; the SDE step verifies that every filter pattern lists at least one test before running. The two consumerctestcalls intests/core_consumer/run.cmakegain--no-tests=erroras AGENTS.md requires. Test temp files (TempFileand the historical-persistence outputs) carry the process ID so concurrent suites sharing a temp directory cannot delete each other's files.fastscan_no_simd_test's HACC case asserts that the generic backend ran, instead of relying on a sum the AVX2 kernel produces identically.Verification
check-format.sh, ShellCheck on the four scripts,git diff --check;.github/scriptsunittest suite 60/60; actionlint ontest.yaml(only the pre-existing runner-label notice).ctest --no-tests=error437/437 (three new tests); Clang 23 ASan/UBSan build: 437/437, no diagnostics.tests/core_consumer/run.cmake(OpenMP ON, all integrations) passes with the stricter ctest flags.pytest tests/python837 passed, 2 skipped.ivf.cpp,ivf_initializer.cpp,qg_builder.cppand the binding TUs that include the changed headers.format-changed.shwith a failing helper exits 1, with an empty list exits 0;clang-format-files.shresolves/repo/src/core/../core/api.cppand skips/etc/hostname.Not verified here: macOS execution of the scripts (static only), the SDE step on a hosted runner (its CI run on this PR is the check).