Skip to content

fix(fts): prevent overflow for unordered token positions - #9017

Open
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-9016-1
Open

fix(fts): prevent overflow for unordered token positions#9017
lance-gatefixer[bot] wants to merge 1 commit into
mainfrom
gatekeeper/fix-9016-1

Conversation

@lance-gatefixer

Copy link
Copy Markdown
Contributor

Summary

  • Rebase FTS query token positions from the minimum emitted position so overlapping Jieba terms can arrive out of position order safely.
  • Add a regression test using the reported mixed Latin/CJK query and a custom Jieba dictionary.

Root cause

Jieba search mode can emit overlapping dictionary terms out of position order. Query token collection subtracted the first emitted position from every later position, so a later smaller position underflowed u32. In release builds that wrapped near u32::MAX, and the modern BM25 candidate path used it to size a token-position vector, producing the reported allocation.

Validation

  • cargo test -p lance-index --features tokenizer-jieba --lib
  • cargo test -p lance-index --features tokenizer-jieba --lib scalar::inverted::query::tests
  • cargo clippy -p lance-index --features tokenizer-jieba --tests --benches -- -D warnings
  • cargo clippy --all --tests --benches -- -D warnings
  • cargo fmt --all

Fixes #9016

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Sep 5, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

This fixes the reported allocation crash at the query-token boundary: minimum-position rebasing preserves every relative gap, leaves ordered token streams unchanged, and prevents out-of-order Jieba terms from becoming near-u32::MAX allocation indices. The feature-gated regression test exercises the reported mixed Latin/CJK emission order.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 5, 2026
@lance-gatefixer

Copy link
Copy Markdown
Contributor Author

Blocked: PR #9017 remains blocked on head 4ef7121 by non-attributable Rust linux-arm and Python windows Check failures.

The linux-arm log shows dataset::mem_wal::index::btree::tests::test_resident_bytes_covers_keys_already_published finished before its timing sampler observed an intermediate insert state; that test passed locally, in the current-head linux-build job, and in ARM CI on the exact base SHA. The windows log shows test_write_hf_dataset failed because Hugging Face returned HTTP 429 maximum queue size reached; the same test passed in the current-head Linux Python job. The repair changes only FTS query token normalization. Failed-job reruns were attempted, but this App lacks Actions rerun permission.

A maintainer can unblock the PR by rerunning the failed linux-arm and windows jobs with Actions write permission. Alternatively, the next legitimate branch update can trigger fresh checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: BM25 FTS query attempts ~96 GiB allocation and aborts in InvertedIndex::bm25_search_modern_candidates (regression vs lance 10)

0 participants