perf: software prefetch bitmask filter K=8 steps ahead in iterate_to_… - #1270
Merged
Conversation
Sanhaoji2
reviewed
Jul 27, 2026
| assert(id < _max_points); | ||
|
|
||
| // Prefetch bitmask K steps ahead (sliding window) | ||
| if (use_filter && i + BITMASK_PREFETCH_K < nbrs_count) |
Contributor
There was a problem hiding this comment.
we can move the prefetch part after visited check?
cave-g-f
force-pushed
the
deyu/perf
branch
2 times, most recently
from
July 27, 2026 08:10
8316e9a to
bebff14
Compare
…fixed_point
Reduces contain_filtered_label self-cost from 49.2% to 14.3% by hiding
DRAM latency on the random-access node bitmask load. Measured on
DANNMemoryTable with 21M-node index, 189 filter labels, 12 threads:
QPS 7,705 -> 12,435 (+61.4%), P99 3012us -> 1656us (-45%).
Changes:
- filter_match_proxy: add prefetch_bitmask(id) virtual method that
issues _mm_prefetch on the node's bitmask address. No-op for the
integer_label path.
- iterate_to_fixed_point: pre-prefetch first K=8 neighbors' bitmasks
before the loop, then sliding-window prefetch K steps ahead inside
the loop. Guarded by use_filter to avoid overhead on unfiltered
queries. Applied to both dynamic and static graph paths.
- Use NeighborList::data() for indexed access (no operator[]).
- Also brings AVX2 buffer padding from downstream d9583a32:
* bitmask_filter_match ctor: query_bitmask_buf resized to at least
4 words for safe 256-bit loads.
* build_bitmask_index: node bitmask buffer padded with 4 extra
uint64 words at end for safe AVX2 reads on the last node.
The prefetch instruction itself accounts for 11% self-cost in the post-
optimization profile, which is the expected steady-state cost of hiding
the DRAM stall on 44 billion filter checks (4.6M queries x 200 search
list x ~48 out-edges).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…fixed_point
Reduces contain_filtered_label self-cost from 49.2% to 14.3% by hiding DRAM latency on the random-access node bitmask load. Measured on DANNMemoryTable with 21M-node index, 189 filter labels, 12 threads: QPS 7,705 -> 12,435 (+61.4%), P99 3012us -> 1656us (-45%).
Changes:
The prefetch instruction itself accounts for 11% self-cost in the post- optimization profile, which is the expected steady-state cost of hiding the DRAM stall on 44 billion filter checks (4.6M queries x 200 search list x ~48 out-edges).
Reference Issues/PRs
What does this implement/fix? Briefly explain your changes.
Any other comments?