perf(fts): bound classic conjunction scoring in dense windows - #9033
Open
BubbleCal wants to merge 1 commit into
Open
perf(fts): bound classic conjunction scoring in dense windows#9033BubbleCal wants to merge 1 commit into
BubbleCal wants to merge 1 commit into
Conversation
This was referenced Sep 7, 2026
BubbleCal
added a commit
that referenced
this pull request
Sep 8, 2026
Before a competitive floor exists, AND queries still decoded frequencies and prepared bounds while looking for intersections. This PR keeps that work lazy, particularly when the final intersection has fewer than `k` matches. - Split DocID and frequency decoding for full posting blocks and tails, retaining the existing encoded format and combined decoding entry points. - Advance and intersect DocIDs before decoding frequencies; initialize score bounds when a positive competitive floor makes them useful. - Avoid loading document lengths for partitions whose dictionaries cannot satisfy all required tokens. - Use SIMD catch-up for the existing classic 4/5-clause path, with unsigned comparisons covering the full `u32` DocID range and a binary-search fallback without AVX2. Tests cover empty/underfilled intersections, delayed frequency and bound activation, block/tail transitions, required-token misses, and scalar/SIMD boundary behavior. Code and its regression tests stay together in this PR. No isolated `main` A/B measurement is claimed for this layer. The later bulk-intersection measurements belong to [#9030](#9030). ## Validation - Local inverted-index suite: 530 passed. - Required all-workspace Clippy: passed. - CI: pending. Stack, 1/4: **this PR** → [#9032](#9032) → [#9033](#9033) → [#9030](#9030). Base: `main`. Merge in stack order; retarget/restack the remaining PRs after their parent merges.
BubbleCal
force-pushed
the
yang/fts-stack-certified-and
branch
from
September 8, 2026 06:25
0311000 to
f694a58
Compare
Certify dense posting ranges before progressive scoring, retain conservative suffix bounds and exact query-order scores, and reuse zero-floor windows and lazy norm lookup. Preserve sparse, phrase, and compound-cursor behavior. Share test fixtures without removing assertions or parameter coverage.
BubbleCal
added a commit
that referenced
this pull request
Sep 8, 2026
Fully prewarmed FTS queries still pay asynchronous document-frequency aggregation overhead, and segment searches clone an already prepared canonical scorer. This PR reuses the resident metadata and scorer. - Aggregate corpus statistics and posting lengths synchronously only when every required value is already loaded. The probe never starts I/O; incomplete residency uses the existing asynchronous path. - Reuse the canonical prepared `Arc<MemBM25Scorer>` across modern segments. Legacy and externally supplied scorers retain their existing fallback behavior. - Check statistic shapes and integer overflow while merging global counts. - Provide `LANCE_FTS_SYNC_DF=off` and `LANCE_FTS_REUSE_PREPARED_SCORER=off` switches. The accompanying tests cover cold/partial/full residency, multiple segments, missing terms, overflow, injected scorers, fuzzy preparation, and exact result parity. Shared fixtures remove repetition while retaining the relevant execution modes and assertions. This layer does not change the query API or persistent format. No isolated `main` A/B performance result is claimed. ## Validation - Local inverted-index suite: 542 passed. - Required all-workspace Clippy and formatting: passed. - PR metadata/gatekeeper checks: pending. Full hosted test workflows only run for `main`/`release/**` bases; this intermediate layer has been tested and linted locally. The full hosted matrix will run after its base is retargeted to `main` following the parent merge. Stack, 2/4: [#9031](#9031) → **this PR** → [#9033](#9033) → [#9030](#9030). Base: [#9031](#9031); this diff contains only resident-statistic and prepared-scorer changes with their tests. Merge in stack order; retarget/restack the remaining PRs after their parent merges.
BubbleCal
force-pushed
the
yang/fts-stack-certified-and
branch
from
September 8, 2026 16:05
f694a58 to
50c5d9b
Compare
Contributor
There was a problem hiding this comment.
The rebase onto main leaves the score-first implementation unchanged. Its strict eligibility, conservative suffix bounds, canonical score reconstruction, and focused parity/transition coverage continue to support correctness.
The non-blocking risk remains performance: no isolated latency/CPU comparison establishes the net benefit or possible short-tail overhead. Keep the scope narrow until representative classic 4/5-term workloads confirm the trade-off; if the incremental gain is small, progressive scoring after full DocID alignment is the simpler alternative.
LuQQiu
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Classic 4/5-term AND evaluation can decode and score every follower even when a partial exact score already proves the candidate cannot compete. Applying score-first evaluation universally is costly when posting overlap is low. This PR enables it only inside a metadata-certified dense range.
f32scores and inclusive tie behavior.The classic path remains used by compound
WandCursorand explicit classic searches after #9030 enables standalone wide-query bulk routing.Most of this diff is regression coverage: approximately 327 production-section additions and 1,334 test additions. Shared fixtures retain the distinct dense/sparse, malformed-metadata, short-tail, high-DocID, score-bit, tie, norm-fallback, seek/shallow, and floor-transition cases. No isolated
mainA/B result is claimed for this layer.Validation
main/release/**bases; this intermediate layer has been tested and linted locally. The full hosted matrix will run after its base is retargeted tomainfollowing the parent merge.Stack, 3/4: #9031 → #9032 → this PR → #9030. Base: #9032. Uses the lazy posting-cursor support introduced in #9031.
Merge in stack order; retarget/restack the remaining PRs after their parent merges.