Skip to content

perf(fts): bound classic conjunction scoring in dense windows - #9033

Open
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-stack-certified-and
Open

perf(fts): bound classic conjunction scoring in dense windows#9033
BubbleCal wants to merge 1 commit into
mainfrom
yang/fts-stack-certified-and

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Classic 4/5-term AND evaluation can decode and score every follower even when a partial exact score already proves the candidate cannot compete. Applying score-first evaluation universally is costly when posting overlap is low. This PR enables it only inside a metadata-certified dense range.

  • Prove continuous coverage from each current posting block's first/last/count metadata, while retaining actual follower membership checks.
  • Build conservative suffix bounds only for eligible live windows and lazily cache exact BM25 norm addends.
  • Preserve canonical query-order f32 scores and inclusive tie behavior.
  • Reuse an unbounded zero-floor window and invalidate it when local, shared, or compound floors become positive; clear eligibility before phrase evaluation.

The classic path remains used by compound WandCursor and explicit classic searches after #9030 enables standalone wide-query bulk routing.

Most of this diff is regression coverage: approximately 327 production-section additions and 1,334 test additions. Shared fixtures retain the distinct dense/sparse, malformed-metadata, short-tail, high-DocID, score-bit, tie, norm-fallback, seek/shallow, and floor-transition cases. No isolated main A/B result is claimed for this layer.

Validation

  • Per-layer inverted-index suite: 603 passed.
  • Required all-workspace Clippy and formatting: passed.
  • PR metadata/gatekeeper checks: pending. Full hosted test workflows only run for main/release/** bases; this intermediate layer has been tested and linted locally. The full hosted matrix will run after its base is retargeted to main following the parent merge.

Stack, 3/4: #9031#9032this PR#9030. Base: #9032. Uses the lazy posting-cursor support introduced in #9031.

Merge in stack order; retarget/restack the remaining PRs after their parent merges.

lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 7, 2026
BubbleCal added a commit that referenced this pull request Sep 8, 2026
Before a competitive floor exists, AND queries still decoded frequencies
and prepared bounds while looking for intersections. This PR keeps that
work lazy, particularly when the final intersection has fewer than `k`
matches.

- Split DocID and frequency decoding for full posting blocks and tails,
retaining the existing encoded format and combined decoding entry
points.
- Advance and intersect DocIDs before decoding frequencies; initialize
score bounds when a positive competitive floor makes them useful.
- Avoid loading document lengths for partitions whose dictionaries
cannot satisfy all required tokens.
- Use SIMD catch-up for the existing classic 4/5-clause path, with
unsigned comparisons covering the full `u32` DocID range and a
binary-search fallback without AVX2.

Tests cover empty/underfilled intersections, delayed frequency and bound
activation, block/tail transitions, required-token misses, and
scalar/SIMD boundary behavior. Code and its regression tests stay
together in this PR.

No isolated `main` A/B measurement is claimed for this layer. The later
bulk-intersection measurements belong to
[#9030](#9030).

## Validation

- Local inverted-index suite: 530 passed.
- Required all-workspace Clippy: passed.
- CI: pending.

Stack, 1/4: **this PR** →
[#9032](#9032) →
[#9033](#9033) →
[#9030](#9030). Base: `main`.

Merge in stack order; retarget/restack the remaining PRs after their
parent merges.
@BubbleCal
BubbleCal force-pushed the yang/fts-stack-certified-and branch from 0311000 to f694a58 Compare September 8, 2026 06:25
Base automatically changed from yang/fts-stack-prepared-scoring to main September 8, 2026 16:05
Certify dense posting ranges before progressive scoring, retain conservative
suffix bounds and exact query-order scores, and reuse zero-floor windows
and lazy norm lookup. Preserve sparse, phrase, and compound-cursor behavior.

Share test fixtures without removing assertions or parameter coverage.
BubbleCal added a commit that referenced this pull request Sep 8, 2026
Fully prewarmed FTS queries still pay asynchronous document-frequency
aggregation overhead, and segment searches clone an already prepared
canonical scorer. This PR reuses the resident metadata and scorer.

- Aggregate corpus statistics and posting lengths synchronously only
when every required value is already loaded. The probe never starts I/O;
incomplete residency uses the existing asynchronous path.
- Reuse the canonical prepared `Arc<MemBM25Scorer>` across modern
segments. Legacy and externally supplied scorers retain their existing
fallback behavior.
- Check statistic shapes and integer overflow while merging global
counts.
- Provide `LANCE_FTS_SYNC_DF=off` and
`LANCE_FTS_REUSE_PREPARED_SCORER=off` switches.

The accompanying tests cover cold/partial/full residency, multiple
segments, missing terms, overflow, injected scorers, fuzzy preparation,
and exact result parity. Shared fixtures remove repetition while
retaining the relevant execution modes and assertions.

This layer does not change the query API or persistent format. No
isolated `main` A/B performance result is claimed.

## Validation

- Local inverted-index suite: 542 passed.
- Required all-workspace Clippy and formatting: passed.
- PR metadata/gatekeeper checks: pending. Full hosted test workflows
only run for `main`/`release/**` bases; this intermediate layer has been
tested and linted locally. The full hosted matrix will run after its
base is retargeted to `main` following the parent merge.

Stack, 2/4: [#9031](#9031) →
**this PR** → [#9033](#9033) →
[#9030](#9030). Base:
[#9031](#9031); this diff
contains only resident-statistic and prepared-scorer changes with their
tests.

Merge in stack order; retarget/restack the remaining PRs after their
parent merges.
@BubbleCal
BubbleCal force-pushed the yang/fts-stack-certified-and branch from f694a58 to 50c5d9b Compare September 8, 2026 16:05
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 8, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The rebase onto main leaves the score-first implementation unchanged. Its strict eligibility, conservative suffix bounds, canonical score reconstruction, and focused parity/transition coverage continue to support correctness.

The non-blocking risk remains performance: no isolated latency/CPU comparison establishes the net benefit or possible short-tail overhead. Keep the scope narrow until representative classic 4/5-term workloads confirm the trade-off; if the incremental gain is small, progressive scoring after full DocID alignment is the simpler alternative.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants