Repository navigation
Add HNSW removal with a deleted mark - #127
Conversation
|
The only failing job, Windows x86-64 (VS 2026), stopped before any test ran: CTest timed out while listing the tests ( |
remove sets the high bit of each point's stored cluster ID. construct and load keep the cluster count at or below 2^31, so a real cluster ID never has that bit, and the getter masks it for every existing reader. Search keeps marked points out of boundedKNN but still adds them to candidate_set, so it walks through them as before. add still links new points to them. The arguments are external labels; every one is checked before any point is marked, as in IVF::remove. load masks the bit before its range check.
Cover the API, the deleted mark in the file layout, scaling ef after removals as a caller choice, and which older releases refuse files with removed points.
add can leave a point unreachable from the entry point, with or without removals, so a test that expects every point to be found fails at random across rotators. Compare each search with the same index before the removal, or with a copy that removes nothing, which is the property remove has to keep.
|
Rebased on main now that #125 is merged. The new run passed every job, Windows x86-64 (VS 2026) included, so the re-run I asked for above is no longer needed. |
|
Currently in Thank you. |
|
I ran the experiment. Short answer: I'd keep the current implementation. Refining removed neighbors recovers some recall at the same ef, but it costs more than raising ef does, and the docs already tell callers to raise ef after removals. What I compared. A is the PR as it stands: a removed neighbor keeps its 1-bit estimate in Recall at the same ef. M 16, ef_construction 200, inner product, recall@10 on the live points against exact ground truth, 1,000 queries, 3 builds per cell. Data: 69.5k HotpotQA passages (Qwen3-Embedding-4B, 384 dimensions) and 100k synthetic vectors with 384 dimensions. Numbers are B minus A:
B is better at low ef, more so with more removed points, and the gain is gone by ef 100 to 200. Over the 33 cells with nbits above 1 and 30% or more removed, the mean at ef 10 is positive in all 33 and no cell is significantly negative. I also ran L2 on the synthetic data (2 builds) and saw the same pattern. Cost. B does one extra full estimate for every removed neighbor that looks promising. At ef 10 with 4 bits on HotpotQA that is +31 full estimates per query on top of 74 with 30% removed, and +62 on top of 63 with 50% removed. On one pinned core (i7-12700KF, GCC 13.3, AVX2, 100k x 384 synthetic, 12 alternating rounds), B's QPS is 2 to 6% lower than A's at the same ef, and B was slower in 11 or 12 of 12 rounds in every cell. Same cost. Since ef is the caller's knob, I also compared them at equal QPS (4 bits, 50% removed, ef 10):
With 8 bits, B at ef 10 gives 0.658 at 13655 QPS, and A at ef 14 gives 0.726 at 13710 QPS. Interpolating A's measured curve at B's QPS (4 and 8 bits, 30% and 50% removed, ef 10 to 200; 16 of the 20 cells fall inside A's range), A is better by more than 0.0005 in 10 cells and B in 1. The difference is largest at ef 10 to 20. The documented On the invariant you raised. You are right that marking a node changes the score search uses to traverse it, and B removes that difference. The data say it is a small effect that a slightly larger ef recovers more cheaply. A removed entry point already gets the full estimate, because search computes it before the removed check, and I left that as is. If you would still rather have B for the invariance, it is a change of a few lines in |
Summary
Closes #126. This builds on #125 and is rebased on main now that #125 is merged.
It adds
removeto the HNSW index with the design from #126: a deleted mark, ef left to the caller.How it works
removesets the high bit of a point's stored cluster ID.constructandloadalready keepnum_cluster_at or below 2^31, so a real cluster ID never has that bit, and astatic_assertties the two together.get_clusterid_by_internalidmasks the bit, so the existing readers (the search estimators,reconstruct_rotated,get_quant_dist,cluster_id_of) see the real cluster.boundedKNN, so it is never returned. It still enterscandidate_setwith its usual estimate, so the search walks through it as before. A removed entry point is treated the same way. The upper layers do not change.addfrom Add HNSW insertion after construction #125 does not change. Removed points stay valid neighbours for new points, as in hnswlib.searchdoes not change ef. The docs describe searching withef * num_points() / live pointsas a tuning option for callers.The rules follow
IVF::remove. The arguments are external labels, the IDs thataddandsearchreturn, translated throughlabel_lookup_. Every label is checked before anything changes, so an unknown label throwsstd::invalid_argumentand removes nothing. The return value counts newly removed points: a repeated label counts once, and removing it again returns 0. Removed points stay innum_points()and cannot be restored. The marks survive save, load and resize. Likeadd, it is not safe to call while another thread searches or adds.The Python binding takes any 1-D integer array and raises
ValueError, with nothing removed, for an ID outside[0, num_points).File compatibility
resizeandadd, and every search result matches.loadmasks the bit before its range check, so a cluster ID out of range is still refused when it carries the mark.Testing
Tests
HnswRemoveTest, over L2 and IP and nbits 1, 2, 4, 8 and 9: survivors keep their order and distances, the entry point can be removed, search reaches live points with 90% removed, labels that differ from internal IDs, save/load/resize and later adds, bad input removes nothing, removing every point, andloadrefusing an out-of-range cluster ID that carries the mark.HnswRemoveAllocationTest.FailureRemovesNothingin the allocation failure harness from Add HNSW insertion after construction #125.HnswRemoveTest.*is added to the SDE filter in CI.The tests compare each result with the same index before the removal, or with a copy that removes nothing, instead of expecting every point to be found. While testing I saw that
addcan leave a point unreachable from the entry point. With 120 built points, 120 added points, M 8 and L2, this happened in 2 to 9 of 100 runs (nbits 1, 2 and 4), with or without removals and in the same runs. With IP it never happened. In every run, removal only hid the removed points from the set of reachable points. A first version of the Python test assumed full reachability and failed on Windows and macOS in my fork's CI, so all the remove tests now use this comparison. The C++ tests passed 300 repeated runs and the Python ones 60.Each of these changes to the code makes at least one test fail: returning a removed entry point, returning removed neighbours, not walking through removed points, reading the cluster ID without the mask,
loadwithout the mask, marking before validating, using labels as internal IDs, and counting a repeated label twice.Checks run locally on an i7-12700KF (Ubuntu 24.04 under WSL, GCC 13.3):
-hsw-iclscripts/check-tidy.shscripts/check-format.sh(clang-format 15)scripts/check-python.sh(ruff)mkdocs build --strictI did not run NEON. The change is in the search template that every SIMD backend shares.
Recall after remove
Same setup as #126: M 16, ef_construction 200, 4·√n KMeans clusters, inner product on normalized vectors, 100k synthetic vectors with 384 dimensions, and 69.5k HotpotQA passages embedded with Qwen3-Embedding-4B and truncated to 384 dimensions (1,000 held-out queries). Each cell is the change in recall@10 on the live points against an index rebuilt without the removed points. The last column searches with the scaled ef and compares it with the rebuild at ef 10.
No search returned a removed point, and every search filled all k results.
Search speed without removals
This measures the cost of the mask and the extra check for users who never remove anything. Setup: 100k synthetic 384-d vectors, inner product, M 16, 2,000 queries, k 10, one thread pinned to one core of an otherwise idle machine. Both builds are portable Python wheels (native optimization off, AVX2 dispatch at run time) built with GCC 13.3, and both load the same saved index files. There are 12 rounds, and the order of the two builds alternates each round. Each round reports the median of 5 timed passes.
Every per-round range includes 1.0, but the rounds lean slightly below it: at nbits 1 and ef 10 the median per-round ratio is 0.973 and 9 of 12 rounds were slower. So I'd put the cost at 1 to 3% at most, which is within the noise of this machine.