Skip to content

fix: harden IVF file handling, filtered test runs, and formatter scripts from the audit - #196

Merged
gouyt13 merged 6 commits into
mainfrom
fix/audit-suggestions
Oct 31, 2026
Merged

gouyt13 merged 6 commits into
mainfrom
fix/audit-suggestions

Conversation

@gouyt13

@gouyt13 gouyt13 commented Oct 31, 2026

Copy link
Copy Markdown
Member

Follow-up to #195: the audit's Suggestion-level items that need no contract decision. Five independent commits.

Library

  • fix(ivf) — IVF::load accepted any bytes after the ID section (HNSW and SymphonyQG reject a payload that is not exactly the expected length); it now throws IVF index file has unexpected trailing data and leaves the target index unchanged. IVF::write_file checked nothing before setting the stream exception mask, so an unwritable destination surfaced as basic_ios::clear: iostream error; it now throws IVF cannot open index file for writing like the other writers. FlatInitializer::save/load check the stream as FlatRaBitQInitializer already did. Regression tests for all three; the frozen v0.5.2 and pre-per-cluster fixtures still load.
  • fix(symqg) — the C++ QGBuilder accepted ef_construction == 0 and silently built a graph from reverse edges and random fill; it now throws std::invalid_argument (the Python wrapper already did).
  • fix(clustering) — iteration_stats.reserve(niter) allocated for the documented maximum iteration count (32 GB for niter = 1e9) after centroids and assignments had been resized; the reserve is gone.

Scripts

  • build — format-changed.sh read the file-list helper through process substitution, so a failing helper formatted nothing and exited 0 (the same defect check-format.sh was already fixed for); it now captures the list and checks the helper's status. The three formatter scripts no longer use xargs --no-run-if-empty or realpath --relative-to, which BSD userlands lack although macOS is a documented host; xargs -r is accepted by GNU and BSD, and absolute paths are normalized with cd && pwd -P.

Tests and CI

  • test — GoogleTest exits 0 when a --gtest_filter matches nothing, so a renamed suite could silently remove the OpenMPBinding CTest entry and parts of the Intel SDE coverage. The CTest entry now requires its one expected test via PASS_REGULAR_EXPRESSION; the SDE step verifies that every filter pattern lists at least one test before running. The two consumer ctest calls in tests/core_consumer/run.cmake gain --no-tests=error as AGENTS.md requires. Test temp files (TempFile and the historical-persistence outputs) carry the process ID so concurrent suites sharing a temp directory cannot delete each other's files. fastscan_no_simd_test's HACC case asserts that the generic backend ran, instead of relying on a sum the AVX2 kernel produces identically.

Verification

  • check-format.sh, ShellCheck on the four scripts, git diff --check; .github/scripts unittest suite 60/60; actionlint on test.yaml (only the pre-existing runner-label notice).
  • GCC 14 portable build: ctest --no-tests=error 437/437 (three new tests); Clang 23 ASan/UBSan build: 437/437, no diagnostics.
  • tests/core_consumer/run.cmake (OpenMP ON, all integrations) passes with the stricter ctest flags.
  • Reinstalled package: pytest tests/python 837 passed, 2 skipped.
  • Focused clang-tidy on ivf.cpp, ivf_initializer.cpp, qg_builder.cpp and the binding TUs that include the changed headers.
  • Script harness: format-changed.sh with a failing helper exits 1, with an empty list exits 0; clang-format-files.sh resolves /repo/src/core/../core/api.cpp and skips /etc/hostname.

Not verified here: macOS execution of the scripts (static only), the SDE step on a hosted runner (its CI run on this PR is the check).

IVF::load accepted any bytes after the ID section while the HNSW and SymphonyQG readers reject a payload that is not exactly the expected length. IVF::write_file set the stream exception mask before checking that the file opened, so an unwritable destination surfaced as the generic basic_ios::clear message. FlatInitializer::save/load now check the stream like FlatRaBitQInitializer; through IVF the stream already carried an exception mask, but the class is public.
A zero search window inserted no candidates, so every row was completed by random fill without any error. The Python wrapper already rejected the value and the docs call it required positive.
…tion count

niter is a documented maximum with early stopping; reserving it up front allocated 32 bytes per possible iteration after centroids and assignments had already been resized.
…format scripts

format-changed.sh read the helper's output through process substitution, so a failing helper formatted nothing and exited 0; it now captures the list and checks the helper's status like check-format.sh. The scripts also used xargs --no-run-if-empty and realpath --relative-to, which BSD userlands lack although macOS is a documented host.
…les per process

GoogleTest exits 0 when a --gtest_filter matches no test, so a renamed suite silently removed the OpenMPBinding check and parts of the Intel SDE coverage; the CTest entry now requires its one expected test and the SDE step checks each filter pattern lists a test. The two consumer ctest runs without --no-tests=error gain it. Test temp files carry the process ID so concurrent suites in one temp directory cannot delete each other's files, and the no-SIMD HACC test asserts the generic backend ran.
@github-actions github-actions Bot added the release:fixed Release category managed from the PR title label Oct 31, 2026
MSVC's OpenMP runtime has no thread placement, so the test skips itself there; the CTest gate only needs to prove the filter selected it.
@gouyt13
gouyt13 merged commit 87ba863 into main Oct 31, 2026
48 checks passed
@gouyt13
gouyt13 deleted the fix/audit-suggestions branch October 31, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:fixed Release category managed from the PR title

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant