Skip to content

Check index input coordinates in one pass #217

Open

Description

@gouyt13

Problem

Since 0.8.0 (#195), indexes reject non-finite and out-of-range coordinates with
check_coordinates in include/rabitqlib/utils/input_checks.hpp. It reads the input twice
in serial: all_finite, then a range check. On 1M × 960 float32 vectors the check alone took
about 0.70 s, which accounts for most of the 0.6–0.7 s that IVF construction gained between
0.7.1 and 0.8.0 in our benchmark.

The range check !(std::fabs(v) <= bound) is already true for NaN and ±Inf, so one pass can
detect every bad value; the finite check is only needed to choose the error message.

Task

Make check_coordinates read the data once in the common (valid) case:

  1. Count out-of-range values in one pass.
  2. Only if that count is non-zero, call all_finite to choose between the two existing
    messages.

Done when

  • Error types and messages are unchanged, including which one is raised when the input has
    both NaN and large finite values (non-finite wins today).
  • Existing input tests pass: tests/python/test_input_boundaries.py and the C++ unit tests.
  • The PR reports the check's time before and after on a large array (for example 1M × 960
    random float32), with the CPU, compiler and build type.

Notes

  • This header is included by ISA-specific translation units (src/fastscan/estimator_avx2.cpp
    and src/fastscan/estimator_avx512.cpp). Besides
    ./scripts/check-format.sh and the C++ tests, run ./scripts/check-isa-symbols.sh on a
    build with RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF (see
    verification by change type).
  • Parallelizing the scan is a possible follow-up; keep this PR to the single pass.

Setup: contribution guide, C++ changes.

Activity

  1. added this to the 0.8.x milestone on Nov 2, 2026
  2. nicklesternl commented on Nov 3, 2026

    @nicklesternl

    Hi! I'd like to work on this issue as my first contribution. I'll investigate the current implementation and submit a PR if nobody is already working on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions