fix(fts): prevent overflow for unordered token positions - #9017
fix(fts): prevent overflow for unordered token positions#9017lance-gatefixer[bot] wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
✅ Gate recommendation: approve.
This fixes the reported allocation crash at the query-token boundary: minimum-position rebasing preserves every relative gap, leaves ordered token streams unchanged, and prevents out-of-order Jieba terms from becoming near-u32::MAX allocation indices. The feature-gated regression test exercises the reported mixed Latin/CJK emission order.
|
Blocked: PR #9017 remains blocked on head 4ef7121 by non-attributable Rust linux-arm and Python windows Check failures. The linux-arm log shows dataset::mem_wal::index::btree::tests::test_resident_bytes_covers_keys_already_published finished before its timing sampler observed an intermediate insert state; that test passed locally, in the current-head linux-build job, and in ARM CI on the exact base SHA. The windows log shows test_write_hf_dataset failed because Hugging Face returned HTTP 429 maximum queue size reached; the same test passed in the current-head Linux Python job. The repair changes only FTS query token normalization. Failed-job reruns were attempted, but this App lacks Actions rerun permission. A maintainer can unblock the PR by rerunning the failed linux-arm and windows jobs with Actions write permission. Alternatively, the next legitimate branch update can trigger fresh checks. |
Summary
Root cause
Jieba search mode can emit overlapping dictionary terms out of position order. Query token collection subtracted the first emitted position from every later position, so a later smaller position underflowed u32. In release builds that wrapped near u32::MAX, and the modern BM25 candidate path used it to size a token-position vector, producing the reported allocation.
Validation
Fixes #9016