Skip to content

fix: align diverged edge cases in FHT, HNSW, xy encoding and PiPNN - #185

Merged
gouyt13 merged 4 commits into
mainfrom
fix/audit-edge-cases
Oct 30, 2026
Merged

gouyt13 merged 4 commits into
mainfrom
fix/audit-edge-cases

Conversation

@gouyt13

@gouyt13 gouyt13 commented Oct 30, 2026

Copy link
Copy Markdown
Member

Summary

Fix four edge cases where duplicated code had diverged between backends or callers. Each commit has its own fix and regression test.

  • AVX FHT rotation with overlapping buffers. Rotators may be called in place. The generic and NEON backends copy the input with memmove, but the shared AVX2/AVX-512 template used memcpy, which is undefined for overlapping ranges. It now uses memmove. The existing overlap test only ran on aarch64; a new test covers every available backend.
  • Uninitialized HNSW. On a default-constructed HierarchicalNSW, search and save dereferenced a null rotator. Both now throw std::logic_error using the initialization check resize already has. save throws before opening the output file.
  • Zero-residual xy codes. When the residual norm is zero, quantize_xy_single wrote magnitude codes 2^ex_bits - 1, while the full, split and v1 encoders write 0. It now writes 0 as well. All factors are zero on this path, so distance estimates do not change; only these degenerate code bytes do.
  • PiPNN point bound. QuantizedGraph and IVF accept up to kSearchBufferMaxPointCount points, but the PiPNN seed rejected a count equal to it. It now uses the same bound.

No public API, Python API or saved-index format changes.

Testing

  • The new FHT, HNSW and xy tests fail before the fixes under ASan/UBSan: memcpy-param-overlap in fht_rotate_avx512, a null-pointer SEGV in HierarchicalNSW::search, and xy code 7 vs. full code 0. They pass after the fixes.
  • Full portable CTest (RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF, --no-tests=error) on an AVX-512 Linux host:
    • Release: 410 passed, 1 skipped.
    • ASan/UBSan Debug: 410 passed, 1 skipped.
    • The skipped test is the placement test, which requires OpenMP binding.
  • check-format.sh, full check-tidy.sh build-tidy and git diff --check passed.
  • The PiPNN bound has no regression test, because reaching it needs 2^31 points; the existing PiPNN tests pass.
  • Not run locally: NEON, Windows and macOS builds.

Rotators may be called in place, and the generic and NEON backends copy the input with memmove. The shared AVX2/AVX-512 template used memcpy, which is undefined for overlapping ranges. Use memmove and test overlapping calls on every available backend.
A default-constructed HierarchicalNSW has no rotator, so search and save dereferenced a null pointer. Reuse the initialization check from resize and throw std::logic_error before touching the rotator or opening the output file.
For a residual with zero norm, ex_bits_code_with_norm zeroed the magnitude codes and then inverted them, so quantize_xy_single wrote 2^ex_bits - 1 while the full, split and v1 encoders write 0. Return before the inversion. All factors are zero on this path, so estimates are unchanged.
QuantizedGraph and IVF accept up to kSearchBufferMaxPointCount points, but the PiPNN seed rejected a count equal to it. Use the same bound.
@github-actions github-actions Bot added the release:fixed Release category managed from the PR title label Oct 30, 2026
@gouyt13
gouyt13 merged commit e172f5f into main Oct 30, 2026
49 checks passed
@gouyt13
gouyt13 deleted the fix/audit-edge-cases branch October 30, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release:fixed Release category managed from the PR title

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant