fix(data): search and compare closest_index_sorted exactly

[janickm]

Summary

closest_index_sorted located the query with np.searchsorted and then compared the two neighbouring elements with numpy arithmetic. Both steps lose exactness on uint64 timestamps, with one failure mode per numpy major version — both found by fuzzing against the intended algorithm evaluated in python ints, and both returning a silently wrong index:

• numpy 1 compares a python int against an integer array by converting both to float64, which is lossy above 2^53, so searchsorted lands next to the true insertion point and the wrong neighbour wins.
• numpy 2 searches exactly (NEP 50) but, for the same reason, keeps the neighbour subtraction in the array dtype. Where the old code computed sorted_array[idx] - value with a larger value on the right, the uint64 result wrapped to ~1.8e19 instead of going negative, so the nearer neighbour lost the comparison. numpy 1 had hidden that wrap by promoting to float64. numpy 2 also raises OverflowError for a negative query.

Fuzzing 80,000 uint64 queries (mixed magnitudes, ~40% with duplicate elements) against the reference:

	wrong index	differs from old
old, numpy 1.26.4	1889	—
old, numpy 2.3.5	2242	—
new, both	0	only where old was wrong

What changed

Bound the query by the array's own range first. Out-of-range queries are answered by the end elements, and what remains is representable in the array's dtype — so the search can run in that dtype and stay exact. That also guarantees both neighbours exist, which removes the two index guards the old code needed. The neighbours are then compared as python scalars: .item() yields an arbitrary-precision int for integer dtypes, so the differences cannot overflow or round.

The net result is shorter than the original:

if value <= sorted_array[0].item():
    return 0
if value > sorted_array[-1].item():
    return len(sorted_array) - 1

idx = int(np.searchsorted(sorted_array, sorted_array.dtype.type(value), side="left"))

if abs(value - sorted_array[idx - 1].item()) < abs(sorted_array[idx].item() - value):
    return idx - 1
return idx

The query is normalised with int() up front so the bounds check and the dtype conversion see the same value. No dtype branching, no type introspection, and float arrays are unaffected — nothing is truncated to an integer.

Why the int(value) normalisation earns its line

Every caller today already passes a python int (ncore's own via get_frame_timestamp_us, NRE's timestamp sampler via .item()), so this is a safety net rather than a functional requirement. It is worth having: without it, an accidentally-passed numpy scalar is compared under numpy's own promotion rules, and a float rounds differently in the bounds check than in the dtype conversion.

Fuzzing 20,000 high-magnitude queries per value type:

query type	without int()	with int()
python int (the contract)	0 wrong	0 wrong
np.uint64	4 wrong (numpy 1)	0 wrong
np.int64	0 wrong	0 wrong
np.float64 / python float	4 wrong + 1 IndexError	3 wrong, no crash

The IndexError is the sharp edge: the float passes the upper-bound check but converts up to the last element, so the search runs off the end. The 3 remaining float misses are unfixable — a float64 cannot hold these timestamps, and is already off by 112–220 before the call; the function answers correctly for the value it was actually given.

The upper bound is deliberately strict: a query landing exactly on a repeated last element still goes through the search and resolves to the start of that run, as before. A non-strict bound silently changed that; test_duplicate_elements pins it.

Affected callers — SequenceProtocol.get_closest_frame_index via compat.py:407 and tools/ncore_vis/components/camera.py:1027 — are otherwise unchanged for in-range microsecond timestamps, which sit below 2^53 today.

The existing regression test passed a python list rather than a uint64 array, so it could not observe any of this. Input is now normalised with np.asarray, so the dtype is well defined either way.

Validation

check	result
new tests vs main	fail on numpy 1.26.4 and 2.3.5 (one case per version)
new tests with fix	pass on both
80,000-query fuzz vs reference	0 wrong on both; differs from old only where old was wrong
value-type fuzz (int / np.uint64 / np.int64 / float)	exact for every integer type on both versions
//ncore/impl/data:all, //ncore/impl/sensors:all, //tools/...	19 pass, python 3.8 (numpy 1.19.5, torch 1.12.1) and 3.11
numpy 1.26.4 + torch 2.7.0 / numpy 2.3.5 + torch 2.13.0	all suites pass on both
bazel run format, ty	clean

Downstream

nre/datasets/samplers/timestamp.py:72 is the only NRE caller of this symbol, and NRE has no test covering that sampler. So its call path was replayed directly: an int64 frame-timestamp tensor (per RigTrajectory.cameras_frame_timestamps_us), rng.choice(...).item() as the query, over 4000 trials including duplicated end timestamps and magnitudes above 2^53.

Identical indices before and after on both numpy versions, no overflow warnings, and every sampled timestamp still resolves to a frame carrying exactly that timestamp.

NRE bazel tests against this branch via --test_env=PYTHONPATH (sentinel-verified the override was live): //nre/datasets/samplers:all, //libs/vren:lidars_test, //nre/models/gaussians:gsplat_lidar_test — 8 pass.

Also audited the wider surface: every production caller reaches closest_index_sorted through get_frame_timestamp_us, which already wraps in int() — ncore's ncore_vis components and tools, and NRE's nre/viewer/ncore_dataset_interface.py and internal/scripts/ncore_vis/loader.py. (nre/nrm/datasets/samplers.py defines its own unrelated get_closest_frame_index.)

[janickm]

how can the input be int | np.integer if it's declared as int? do we not trust the inputs? - could we use value = np.asarray(value) or so? why should it be bool?

[janickm]

same question on input type - it should be a form of int