feat(index): share IVF partition scans across batch vector queries - #2
Open
sezruby wants to merge 727 commits into
Open
feat(index): share IVF partition scans across batch vector queries#2sezruby wants to merge 727 commits into
sezruby wants to merge 727 commits into
Conversation
sezruby
force-pushed
the
knn-batch-6822
branch
2 times, most recently
from
June 22, 2026 15:18
dd01c55 to
35e21ad
Compare
Hi team, `lance-datafusion` currently includes `lance-datagen` as a production dependency, even though it is only used by test-data generation utilities. This PR makes the dependency optional and enables it only for tests, keeping it out of normal production builds.
…lance-format#8539) initialize_mem_wal validates schema-dependent state -- sharding fields and maintained indexes -- so a Merge that commits concurrently invalidates what the install validated, yet both interleavings committed: check_create_index_txn accepted any Merge and check_merge_txn accepted any CreateIndex. Both arms now conflict when the CreateIndex carries the MemWAL index; ordinary column-index builds stay compatible with Merge.
…8548) Picks up the namespace spec computed column surface (AddColumnsEntry.computed, backfill_column) for downstream SDKs. Every response model gained an optional context map, so the directory namespace initializers fill remaining fields with Default::default(); the new num_inserted_rows and version on InsertIntoTableResponse stay unpopulated there for now.
…-format#8526) `MemTableStats` already tracks `frozen_count` and `frozen_bytes`, but the Python binding dropped them on the way out. This adds both keys to the dict returned by `ShardWriter.memtable_stats()`. - `frozen_count` — frozen memtables in the read view: sealed-awaiting-flush, plus flushed ones still inside `frozen_memtable_grace`. - `frozen_bytes` — heap bytes still owed to flush. Together with the active memtable's `estimated_size_bytes`, this approximates what backpressure meters against `max_unflushed_memtable_bytes`. ## Testing Extends the closed-writer stats assertions in `test_mem_wal.py` to cover both keys. Not run locally — no built extension in this worktree, so CI is the first execution. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
External `Operation::Merge` commits install a caller-supplied schema. Without validating that schema against the current manifest, a renumbered or reused field ID can silently rebind a live column to data from another column. The same risk exists when a shared field ID changes its logical type, nullability, storage encoding, or dictionary while old base or overlay files remain. Validate field bindings before accepting a merge commit. Existing field IDs must keep the same field path, new IDs must be greater than `Manifest::max_field_id()`, and semantic binding changes are rejected whenever any old field-bearing file is retained. Complete physical rewrites remain supported: a binding change is allowed only when every old base and overlay file carrying the field is replaced and every proposed fragment materializes the field in a base data file. Field drops and metadata-only updates remain legal. Fixes lance-format#7700 --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary - Allow PyLance BTree, Bitmap, and ZoneMap indices on `LargeBinary`, `Decimal128`, `Decimal256`, and duration fields. - Coerce typed LargeBinary, decimal, and duration literals in Rust so eligible predicates are planned as scalar index queries. - Add Decimal128/Decimal256 extrema support for ZoneMap planning. - Extend the existing scalar-index type test to cover index planning, query results, and uncommitted segment creation. ## Root cause Python validation was narrower than the Core index capabilities. In addition, some accepted types could create an index but their typed query literals were not coerced by the planner, causing filters to fall back to `LanceRead`. ## Non-goals `Decimal32` and `Decimal64` are not included because Lance does not yet support them as dataset types. Their support is tracked separately in lance-format#5174. ## Testing - `cargo test -p lance-datafusion expr::tests` - `cargo test -p lance-arrow-stats test_rstest_primitives` - `uv run --python 3.11 pytest python/tests/test_scalar_index.py::test_scalar_index_types` - `uv run make lint` - `cargo fmt --all --check` - `cargo clippy --all --tests --benches -- -D warnings`
…#7714) Adds lance-format#5021 In line with the [suggestion from this issue](lance-format#5021), this added support for full-zip encoding for fixed-length structs. Please note - this is a change from current behavior (per-value Fixed-length packed structs would error out prior to this). Please let me know if that warrants marking this feat as a breaking change. Will note - I'm fairly new to rust, so any and all feedback is appreciated 😉
…ormat#7777) ## Problem For IVF indexes with cosine distance, vectors are already normalized by the IVF transform pipeline (NormalizeTransformer), so cosine distance is mathematically equivalent to dot distance over normalized vectors. However, cosine still computes the L2 norm of each vector during distance calculation — redundant work that dot avoids. ## Solution Convert Cosine to Dot in `FlatFloatStorage::try_from_batch`, consistent with how PQ and RQ already convert Cosine to L2 for the same reason. ## Benchmark scripts below [my_ann_bench.py](https://github.com/user-attachments/files/29961863/my_ann_bench.py) | Metric | Origin | Optimized | |---------------|----------|-----------| | avg latency | 4.1298 | 2.3223 | | min latency | 3.9978 | 2.1293 | | max latency | 4.2227 | 2.4728 | | QPS | 242.1435 | 430.5993 | ## Tests - `test_try_from_batch_converts_cosine_to_dot` — verifies Cosine → Dot - `test_try_from_batch_keeps_non_cosine_distance_types` — verifies L2/Dot unchanged <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved vector distance handling by treating cosine distance as dot-product distance for normalized vectors. * Preserved existing behavior for L2, dot-product, and Hamming distance types. * Updated validation to confirm equivalent cosine and dot-product results on normalized vectors. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: clearlvli <clearlvli@tencent.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…rmat#8532) ## Summary - add `excluded_fragment_ids` to compaction planning options - treat excluded fragments as hard planning boundaries and skip collecting their metrics - expose the option consistently through Rust, Python (including `dataset.optimize.compact_files`), and Java/JNI - preserve Java deserialization compatibility for older serialized compaction options ## Semantics Excluded fragments remain unchanged. Fragments on opposite sides of an excluded fragment are not combined into the same compaction task. Duplicate and unknown IDs are ignored. ## Validation - `cargo test -p lance dataset::optimize::tests` (123 passed) - `cargo clippy -p lance --all-targets -- -D warnings` - `cargo fmt --all -- --check` - `uv run pytest python/tests/test_optimize.py` (22 passed) - `uv run make lint` - `./mvnw -Dtest=CompactionTest test` (7 Java tests and 17 JNI Rust tests passed) - `./mvnw spotless:apply` - `cargo clippy --tests --manifest-path ./lance-jni/Cargo.toml` --------- Co-authored-by: wangzheyan <wangzheyan@bytedance.com>
…e-format#8449) `tracked_files` walks every present manifest with a four-stage pipeline: a lister that enumerates manifest locations and applies `min_version`, a reader that fetches them with bounded parallelism under a memory budget, an emitter that turns each manifest into file rows, and an index lister that materializes index directories. Only the last two are about the rows it emits. The first two are about walking manifests, and a second consumer needs exactly them. That consumer is `Dataset::referenced_files` in lance-format#8097, the keep-set an external orphan-cleanup driver uses to decide what it may delete. In the discussion there the suggestion was to factor out the reusable part before rebasing that PR onto it, which is what this does. Nothing in this PR depends on lance-format#8097; `tracked_files` is the only caller here and its behavior is unchanged. ## What moved The lister and reader now live in `dataset::files::scan`, which yields a `ScannedManifest` per present manifest: the manifest, its own path, and the index metadata read alongside it. `tracked_files` keeps its emitter and index lister and consumes that stream. Channel capacities, the `can_launch` predicate, the `biased` select ordering, and the `min_version` filter are carried over unchanged. ## Why the budget accounting changed shape Previously the reader charged bytes before sending and the emitter released them after processing. That worked because the emitter was the only consumer and sat in the same file, so the charge was bounded by the reader's in-flight reads plus two channel slots. A shared walk cannot rely on that: a second consumer that forgets to release would silently stall the reader. The charge now lives in a `MemoryPermit` held by `ScannedManifest` and released on drop, so backpressure follows the manifest's lifetime rather than a convention. Field order is load-bearing and commented: the permit drops after the manifest it accounts for. The bound is on the reader's prefetch, not on what a consumer retains. One read is always allowed when nothing is in flight, which is what keeps a manifest larger than the whole budget from deadlocking the walk, so a consumer that holds every manifest gets serial reads rather than a stall. That escape hatch is unchanged from before, and the module doc now states this rather than promising a bound it does not provide. ## Tests Six cases in `scan::tests`, covering what the previous arrangement had no way to observe: - the budget returns to zero once every manifest is dropped, and stays charged while a consumer holds them; - `min_version` really does skip manifests, which is why a keep-set must leave it unset; - `total` counts every manifest the walk yields; - a failed manifest read surfaces one `Err` per manifest rather than being skipped, asserted as `errors == 3` because `errors > 0` would also pass on a reader that stopped at the first failure or on a listing failure; - dropping the stream early releases every in-flight permit. Each fails against the corresponding mistake: a leaked permit, a bypassed filter, a reader that aborts on first error. `cargo clippy -p lance --all-targets -- -D warnings`, `RUSTDOCFLAGS="-D warnings" cargo doc -p lance --no-deps`, and `cargo fmt --all --check` are clean; `dataset::files` (18) and `dataset::cleanup` (41) pass. The full suite is left to CI. ## Reviewing this The second commit is the result of reviewing the first, so the two are worth reading separately. It removes an unreachable error branch the extraction left behind, moves the index fan-out after the row batches so a full index channel cannot block row output while holding budget, makes the test-only budget accessor private, and corrects the module doc described above.
) ## Summary - resolve source and target join-key columns independently when classifying indexed merge rows - preserve matched updates when a partial source uses a different field order than the dataset - add a regression test that verifies both update statistics and the stored value ## Root cause The indexed join keeps target columns in dataset-schema order, but Merger::extract_selections derived target key positions by offsetting source key positions. A reordered source could therefore inspect a nullable payload column instead of target_<key> and classify a matched row as absent. ## Validation - cargo test -p lance test_indexed_partial_merge_with_reordered_source -- --test-threads=1 - cargo test -p lance test_repro_3515_partial_schema_fully_indexed -- --test-threads=1 - cargo fmt --all - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#8280 <!-- lance-gatekeeper-fix:v1 agent=b8066ab256924d14b5f518570326d33c generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
…mat#8587) ## Problem After a scalar index returns its row-ID mask, `FilteredReadExec::get_or_create_plan_impl` still loads deletion vectors, row counts, and row-ID metadata for every fragment before applying that mask. An exact index miss therefore performs an O(fragment count) metadata pass even though no fragment can contribute a row. This is a follow-up to lance-format#7792. That PR removed the second all-fragment metadata load during stream construction; the first planning-time pass remained. Related to lance-format#4189, which tracks broader filtered-read planning costs. ## Change Use the index result's upper bound to decide whether a fragment can contribute before loading its full metadata: - skip every covered fragment for an exact empty allow-list; - use the fragment portion of address-style row IDs for direct pruning; - for non-empty stable row-ID masks, use each fragment's row-ID sequence to route candidates, then load deletion vectors and row counts only for candidate fragments; - carry the routed stable row-ID sequence and upper offset ranges into final planning, so exact and refined/at-most results do not map the same IDs twice; - cap retained stable-ID range payload at 16 MiB per plan and drain retained vectors fragment-by-fragment; over-budget fragments reuse the loaded row-ID sequence but recompute ranges during final planning; - keep uncovered fragments unless `only_indexed_fragments` is enabled; - conservatively keep block-list upper bounds and every fragment needed to calculate a pre-filter scan range. Candidate fragments still load and apply their deletion vectors, so stale deleted index hits remain excluded. Pruning uses the upper bound of refined index results to avoid false negatives. ## Benchmark Lower is better for every metric below. | Scenario / metric | Baseline (`e958adfdf`) | This PR (`150e000f0`) | Benefit | | --- | ---: | ---: | ---: | | Exact-empty plan latency | 1,630.20 ms/query | 5.14 ms/query | 316.9x speedup | | Indexed zero-hit query latency | 2,067.66 ms/query | 84.73 ms/query | 24.4x speedup | | Indexed one-hit query latency | 2,081.91 ms/query | 163.69 ms/query | 12.7x speedup | | Exact-empty plan S3 reads | 2,600 reads/query | 0 reads/query | 2,600 reads/query eliminated | | Indexed zero-hit S3 reads | 2,603 reads/query | 3 reads/query | 867.7x fewer reads | | Indexed one-hit S3 reads | 2,606 reads/query | 7 reads/query | 372.3x fewer reads | The review follow-up also measures cache-hot dense stable-row-ID planning, where all 5,200 physical row IDs route to all 2,600 fragments: | Scenario / metric | Before follow-up (`824551d44`) | This PR (`150e000f0`) | Benefit | | --- | ---: | ---: | ---: | | Dense stable-ID plan latency | 9.17 ms/plan | 7.86 ms/plan | 1.17x speedup | The main-branch control was 7.13 ms/plan. The current implementation is 0.72 ms/plan (1.10x) above that control because it performs the stable-ID routing pass needed for pruning, but it no longer repeats the same mapping during final planning. All cache-hot measured plans performed 0 S3 reads. Environment and methodology: - AWS `m7i.4xlarge` in `us-east-1a`, reading S3 in the same region. - One dataset with 2,600 stable-row-ID fragments, 2 rows per fragment, one real deletion vector per fragment, 14 payload columns, and BTree indices on `org_id` and `repo_id`. - Separate main, pre-follow-up, and current PR binaries built with Rust 1.97.1, `release-with-debug`, `--no-default-features --features aws`; their SHA256 hashes were checked before measurement. - Three serialized trials per case; base/PR order alternated by trial. Each trial used a new process and fresh Lance `Session`; the table reports medians. - Dataset-open time and I/O were excluded. Query/planning I/O uses incremental `IOTracker` statistics after open. - The dense case used three processes per revision. Each process ran one excluded S3 warm-up plan followed by 21 measured plans; the table reports the median of the three process medians. Index-result serialization and exec construction were outside the timed region. - This measures the metadata path against real S3, not the Plan Executor persistent-disk cache. Measured bytes followed the same pattern: exact-empty planning dropped from 1,814,800 B/query to 0 B/query; the zero-hit query dropped from 1,882,874 B/query to 68,074 B/query; the one-hit query dropped from 1,965,927 B/query to 151,825 B/query. ## Correctness and limitations The regression coverage includes stable and address-style row IDs, exact-empty and sparse non-empty masks, refined upper bounds, partially indexed datasets, and `only_indexed_fragments` behavior. A deterministic dense stable-ID test counts `mask_to_offset_ranges` spans: four candidate fragments must produce exactly four mappings; the pre-follow-up implementation produced eight. A separate boundary test verifies that retained range payload cannot exceed the per-plan budget. For a non-empty stable row-ID mask, this change still visits each covered fragment's row-ID sequence to discover candidate fragments. The benchmark fixture stores those sequences inline, so it removes the S3 deletion-vector/row-count reads but not the O(fragment count) routing walk. Datasets with external row-ID metadata can still perform O(fragment count) row-ID metadata reads. Eliminating that remaining cost requires carrying physical candidate-fragment information from scalar-index execution or persisted row-ID routing metadata. ## Validation - `cargo fmt --all` - `cargo test -p lance io::exec::filtered_read::tests -- --test-threads=1` (75 passed) - `cargo clippy --all --tests --benches -- -D warnings` - `RUSTFLAGS='-D warnings' cargo +nightly-2026-07-13 check -p lance --tests` - AWS S3 benchmark described above
Setting IVF shuffle buffer names only copies strings; invalid or missing files are reported by the later fallible I/O operations and cannot violate memory safety. Remove the misleading unsafe contract and unnecessary unsafe blocks, and cover the missing-buffer error path through the safe setter.
Blob reads repeatedly used `transmute` to reinterpret Rust byte slices as JNI signed-byte slices. Keep the necessary zero-copy representation cast in one narrowly scoped helper with explicit size, alignment, bit-pattern, and lifetime invariants, and reuse it across all blob read paths.
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> Follow-up to lance-format#8357 after it was merged before Weston's review comments landed. ## Changes Apply the wording suggestions from [@westonpace](https://github.com/westonpace) on lance-format#8357: - Prefer "generate" over "mint" when describing how commits assign row ids - Updated `docs/src/guide/distributed_write.md` and `RowIdSequence` docs in `fragment.pyi` Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ack (lance-format#8934) When a single top-level row carries more rep/def levels than one mini-block chunk can hold, the primitive encoder falls back to full-zip after a pre-check that the value block is something full-zip can serialize. That pre-check rejected 1-bit booleans as non-byte-aligned, although `encode_full_zip` widens them to bytes before compressing (lance-format#6723). A sparse `List<List<Boolean>>` row therefore failed with "Mini-block cannot encode N rep/def levels in one top-level row" even though it encodes fine, and even when the user explicitly requested `structural_encoding=fullzip`. Before lance-format#6787 the same row was written through full-zip. The pre-check now lets 1-bit fixed-width blocks through, matching what `encode_full_zip` accepts. The boolean test that asserted the error now asserts a full-zip round trip alongside the existing string case.
## Summary - enforce equal input lengths at the scalar and dispatched accumulation boundaries before selecting a SIMD backend - retain debug-only assertions inside the unsafe SIMD kernels to match the sibling u8 distance implementations - cover both shorter and longer right-hand inputs in the regression test ## Root cause The safe u8 cosine dispatcher selected a backend whose only length guard was a debug assertion. Release builds removed that guard, while SIMD loops bounded loads by the left slice length and loaded from the right slice unchecked. The scalar path also silently truncated mismatched inputs through iterator zipping. ## Validation - `cargo fmt --all -- --check` - `cargo test -p lance-linalg --lib distance::cosine_u8` (8 passed) - `cargo test -p lance-linalg --lib` (250 passed, 1 ignored) - `cargo test --release -p lance-linalg --lib distance::cosine_u8::tests::rejects_mismatched_lengths` (2 passed) - `cargo clippy --all --tests --benches -- -D warnings` - `cargo doc -p lance-linalg --no-deps` Fixes lance-format#8638 <!-- lance-gatekeeper-fix:v1 agent=b706f8f1584a805146e88dcfe3fede58 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…ns (lance-format#8933) Serialize the scanner's `batch_size_bytes` budget into `FilteredReadOptionsProto` so remote execution no longer drops the byte cap when `file_reader_options` is reconstructed from the wire. ## Summary Fixes lance-format#8857 (follow-up to lance-format#8927): carry `batch_size_bytes` through distributed FilteredReadOptions This is a follow-up to lance-format#8927 ("feat(io): apply byte-sized batch budget to blob materialization"), which applies the scanner's `batch_size_bytes` budget a second time after blob v2 payloads are materialized. The gatekeeper review on lance-format#8927 flagged that the budget is silently dropped under distributed execution, because `FilteredReadOptionsProto` does not serialize `file_reader_options`. This PR lands that missing wire field. ## Problem lance-format#8927 forwards the byte budget into the row-stream `FilteredReadExec`, but its distributed codec does not serialize `file_reader_options`; `FilteredReadOptionsProto` has no corresponding field and decode reconstructs `None`. Remote execution therefore silently drops the byte cap. ## Fix Add an optional `batch_size_bytes` field to `FilteredReadOptionsProto` and round-trip it through the proto codec. ### Changes - `protos/filtered_read.proto` - Add `optional uint64 batch_size_bytes = 14` to `FilteredReadOptionsProto`, documented as the scanner-level byte budget corresponding to `FileReaderOptions.batch_size_bytes`. - `rust/lance/src/io/exec/filtered_read_proto.rs` - `fr_options_to_proto`: serialize `options.file_reader_options.batch_size_bytes` into the new field. - `fr_options_from_proto`: reconstruct `file_reader_options` with `batch_size_bytes` when present. - Extend `test_options_roundtrip_basic` to set `batch_size_bytes: Some(4096)` and assert it survives the round-trip. ### Tests - `cargo test -p lance --features substrait --lib filtered_read_proto::tests::test_options_roundtrip_basic` — verifies `batch_size_bytes` round-trips through the proto codec (sender `Some(4096)` → decoded `Some(4096)`). - `cargo check -p lance --features substrait` — compiles cleanly. ## Relationship to lance-format#8927 lance-format#8927 implements the core blob byte-budget feature and is still open. This PR is a focused follow-up that addresses one of its three gatekeeper findings (distributed execution dropping the budget). The other two findings — skewed-row chunk sizing in `split_batch_by_bytes` and chunk boundaries being re-merged by `RowStreamRead::read_batch` — are tracked separately and not addressed here.
## What changed - Change the Rust IVF_RQ build defaults from 1 bit to 5 bits. - Preserve explicit namespace `num_bits` values with checked conversion and 1..=9 validation while keeping omitted values at the 5-bit default. - Keep the Python model helper, type stub, documentation, and Java builder aligned with the Rust default. - Update operational sizing guidance for the 5-bit layout and document the 1-bit storage and search trade-off. - Add regression coverage for Rust, Python, Java, and namespace default handling while preserving explicit 1-bit coverage. ## Why Implicit IVF_RQ index creation should consistently use `num_bits=5` across supported language surfaces, while explicit `num_bits` values must remain unchanged. Capacity guidance must also reflect the larger multi-bit layout so users can choose the 1-bit opt-out when appropriate. ## Validation Static checks and formatting completed locally: - `cargo fmt --all -- --check` - `cargo clippy -p lance-index --tests -- -D warnings` - `cargo clippy -p lance-namespace-impls --tests -- -D warnings` - pre-commit `ruff`, `ruff-format`, `fmt`, and `typos` The complete test suite is delegated to CI. Java validation was not run locally because this host does not have a JDK.
…d up (lance-format#8935) ## Problem When two transactions commit concurrently and one of them needs to be rebased, `initial_fragments_for_rebase` calls `checkout_version(transaction.read_version)` and `.unwrap()`s the result: ```rust dataset .checkout_version(transaction.read_version) .await .unwrap(), ``` If a concurrent cleanup_old_versions (e.g. triggered by VACUUM) removes that version's manifest between the conflicting commit and the rebase, checkout_version returns DatasetNotFound and the unwrap() panics. In builds compiled with panic = "abort" (common when Lance is embedded via FFI) this aborts the entire host process instead of failing just the commit. Observed in production as: ``` thread '<unnamed>' panicked at .../lance/src/io/commit/conflict_resolver.rs: called `Result::unwrap()` on an `Err` value: DatasetNotFound { path: ".../_versions/85.manifest", ... } ``` ## Fix Return a Result from initial_fragments_for_rebase and propagate the error instead of unwrapping. A commit whose read version has been garbage-collected now fails gracefully with DatasetNotFound, allowing the caller to retry, rather than panicking. All 5 call sites are updated to use ?. This is a behavior change only for the previously-panicking path; the success path is unchanged. ## Test Added test_rebase_errors_when_read_version_was_cleaned_up, which: 1. writes two versions of a dataset, 2. builds a transaction pinned to version 1, 3. deletes version 1's manifest to simulate concurrent cleanup, 4. asserts TransactionRebase::try_new returns DatasetNotFound instead of panicking. Verified the test fails (panics) without the fix and passes with it. ## Verification - cargo check -p lance — clean - cargo test -p lance --lib io::commit::conflict_resolver — 59 passed (58 existing + 1 new) Co-authored-by: qiuyuhang <14160990+qiuyuhang@users.noreply.github.com>
Blob v2 schema normalization must distinguish logical writer input from the prepared writer intermediate. Rebuilding an already-logical field can collapse the complete `data, uri, position, size` shape to the minimal form and lose schema properties. Prepared child IDs must also follow the semantic `data` and `uri` fields instead of their positions in the prepared layout. Logical minimal and complete schemas now pass through normalization unchanged, while prepared input alone normalizes to the minimal logical shape with IDs matched by child name. Descriptor and malformed layouts remain explicit errors. The contract is exercised through create, append, merge-insert, external-range, and nested Rust/Python paths and is documented as public behavior. Blob v2 is beta, so this enforces the complete invariant directly without compatibility handling for intermediate beta schemas. A mutation check that routes logical input through the prepared normalization branch makes all logical identity matrix cases fail; restoring the intended branch makes the full matrix pass.
## Problem Callers need to encode independent row ranges once and assemble them in caller-supplied order without decoding and re-encoding. Lance should provide encoded-part validation and final ordinary data-file construction while leaving part storage and orchestration to the caller. ## Behavior DataFileTarget is a runtime-only value for one live assembly operation. It creates the same canonical random file name used by ordinary Lance writes without creating, reserving, or registering an object. Lance does not serialize or restore this value and does not define coordinator restart or cross-process recovery semantics. Each DataFilePart is a runtime view of an ordinary, complete Lance file. Callers choose part paths and order, retain the live target for the operation, and own part storage, cleanup, and commit fencing. Blob v2 writers receive disjoint ID leases and write managed payloads directly beneath the sidecar directory selected by the final target. BlobTargetId only rejects mixing parts assigned to different final targets within one assembly operation. It is not a dataset, base, or object-store identity. The caller must use the same dataset and resolved base for every part write and final assembly. ## Format and ownership No new Lance file, manifest, transaction, target, or part format is introduced. The completed output is an ordinary DataFile committed through the existing transaction path; readers cannot distinguish it from a normally written file. The implementation uses the current file format and existing encoded page-relocation machinery. Callers own target lifetime, part storage, dataset/base association, cleanup, and commit fencing. Lance owns target name generation, part encoding and intrinsic validation, runtime target-identity checks, and final data-file construction. Follow-up to lance-format#8660 and Discussion lance-format#8615.
## Summary Add dense and near-dense bitmap decode paths. This follows lance-format#8713 and targets `RangeWithBitmap` segments after the sequential cursor has removed repeated prefix scans. The decoder now: - emits a full `0xff` byte as one contiguous eight-value range; - expands full bytes with at least six set bits as contiguous runs; - selects the specialized dense stream once for a single bitmap segment; - seeds the cursor with the cardinality used for that decision, avoiding a second bitmap scan; - keeps multi-segment sequences on the existing sparse path because one stream-wide decoder cannot safely assume that every segment has the same density. The sparse cursor and bitmap loop remain separate and source-equivalent to `main`. This avoids the measurable fallback regression caused by performing adaptive dispatch in every batch. `Bitmap.data` and `Bitmap.len` remain publicly accessible for source compatibility. A proposed popcount cache was removed because direct mutation of the public byte vector could otherwise make the cached cardinality stale. The on-disk encoding remains byte-for-byte unchanged. ## Performance Measured on Linux x86_64 with `release-with-debug`, 1,000,000 output rows, batch size 1,024, 10 Criterion samples, 1 second warm-up, and 3 seconds measurement. Both binaries used the same benchmark source. Baseline was current `main` at `d57d0fb42`; candidate was `fbde7d600`. | Shape | Payload | `main` | This PR | Change | |---|---:|---:|---:|---:| | 50% density (`holes_2`) | no | 2.243 ms | 2.274 ms | +0.96% (within Criterion noise threshold) | | 50% density (`holes_2`) | yes | 2.600 ms | 2.594 ms | -0.22% (no significant change) | | ~94% density (`holes_17`) | no | 2.484 ms | 1.600 ms | **-35.58%** | | ~94% density (`holes_17`) | yes | 2.592 ms | 1.677 ms | **-35.14%** | Linux `perf` attributes the dense-shape improvement to the intended decoder change: on `main`, `SegmentCursorState::extend_range` accounts for 68.17% of CPU samples; this PR moves that work to `SegmentCursorState::extend_dense_range` (48.61% of samples) while reducing end-to-end time by 35.58%. For the 50%-density fallback, both `main` and this PR remain in `SegmentCursorState::extend_range` (66.83% and 70.13% respectively); no adaptive-dispatch helper appears in the hot path. ## Validation - `cargo test -p lance-table --lib` (365 passed) - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all -- --check` - `git diff --check` - full-width-byte regression coverage, including the previous shift-by-8 panic - dense sequential cursor coverage across batch and byte boundaries, tail reads, past-end reads, and rewind - sparse and multi-segment fallback coverage - concurrent multi-batch stable row-ID coverage with deletions - unsorted stable row-ID index coverage - direct public-byte mutation cardinality coverage - public-field source-compatibility coverage through `U64Segment::RangeWithBitmap` - byte-exact serde coverage Dependency lance-format#8713 is merged. This PR targets `main` directly and contains only the bitmap follow-up. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
…paths (lance-format#8752) `vector/residual.rs` had no tests, unlike its siblings `sq/transform.rs`, `flat/transform.rs` and `vector/transform.rs`. It runs on the IVF_PQ build path and computes what PQ actually encodes: each vector minus the centroid of its own partition. Two behaviours had no coverage. `compute_residual` dispatches on the centroid/vector type pair, and the `(Float32, Int8)` arm widens the vectors, so the residual comes back wider than the input. `ResidualTransform` then has to rewrite the field type for that case, or the schema would claim Int8 while the data is Float32. Adds 13 tests: exact subtraction across interleaved partitions, all four dispatch arms, both mismatch errors, and the three transform error paths. Verified non-vacuous: pinning `part_id` to 0, dropping the dimension guard, and forcing the in-place column replacement each fail the matching test (5 of 13). Co-authored-by: Xuanwo <github@xuanwo.io>
…lance-format#8765) `ivf/transform.rs` had no tests, unlike its siblings `sq/transform.rs` and `flat/transform.rs`. Both types in it fail silently: a vector assigned to the wrong partition is never searched in the right one, and a `PartitionFilter` that keeps the wrong rows drops vectors from a sharded build without an error. The subtle part is the guard at the top of `PartitionTransformer::transform`. Partitions already present means skip — except when `with_distance` is set and the distance column is missing, where it has to drop both and recompute, or it returns a batch without the column it promised. Adds 11 tests: nearest-centroid assignment, the loss metadata `v3/shuffler.rs` reads back, opt-in distances, both skip and recompute branches, the two column errors, and range filtering including the keep-nothing case. Verified non-vacuous: zeroing the assignment, zeroing the loss, ignoring `with_distance`, and making the filter keep everything fail 6 of the 11. Co-authored-by: Xuanwo <github@xuanwo.io>
…rmat#7942) ## Summary `AimdConfig::validate` checks its rate fields with sign and ordering comparisons (`initial_rate <= 0.0`, `min_rate > max_rate`, `decrease_factor >= 1.0`, and so on). Every one of those comparisons is `false` for `NaN`, so a `NaN` value passes validation untouched; `+inf` likewise slips through on any field that has no opposing bound (`max_rate`, `additive_increment`). These configs are user-reachable. The `LANCE_AIMD_*` environment variables and the equivalent `storage_options` keys are parsed with `f64::parse`, which accepts `"nan"`, `"inf"`, and `"infinity"` case-insensitively, and the parsed value flows straight into `AimdConfig` and then `AimdController::new` → `validate`. ## Failure mode A non-finite rate does not fail loudly, which is what makes it worth guarding. With the default burst capacity, a `NaN` rate makes the token bucket refill to full on every acquire — `(tokens + elapsed * NaN).min(burst)` returns `burst` because `f64::min` drops the NaN — so throttling is silently disabled and the only visible symptom is a `NaN` leaking into rate metrics and logs. Only when the burst capacity is zero does the code reach `Duration::from_secs_f64(NaN)` and panic. ## Change Reject all six `f64` fields up front when they are not finite, with an error naming the offending field, before the existing range checks run. This is the common chokepoint for all three ingress paths (env vars, storage options, and direct construction of the public `AimdConfig`), so it is more complete than validating at the parse layer. ## Note for reviewers This also rejects `max_rate = f64::INFINITY`, which previously passed validation and acted as an undocumented "no ceiling" alias. The documented sentinel for no ceiling is `max_rate = 0.0`, which is finite and unaffected. No code, test, or configuration in the repository sets any of these fields to a non-finite value. ## Test plan Extended the `test_config_validation_rejects_invalid` table with a `NaN` case for each of the six fields plus `+inf` on `initial_rate` and `max_rate`. `cargo test -p lance-core --lib utils::aimd` and `cargo test -p lance-io --lib object_store::throttle` pass; `cargo fmt --all` and `cargo clippy -p lance-core --all-targets -- -D warnings` are clean.
…nce-format#7740) Opening a dataset eagerly decodes the inline manifest transaction section (added in v1.0) to warm the session cache. If the transaction was written by a newer version of Lance with an operation type this version cannot decode, the decode error fails the whole `load_manifest` call — making the dataset version unopenable even though the transaction contents are not needed to read data. This already applies to recently added operation types (e.g. `UpdateBases`, `Clone`, `UpdateMemWalState`) read by older 1.x releases, and would apply to any operation added in the future. Since this decode is purely an opportunistic cache warm-up, tolerate the failure: log a warning and skip caching. Paths that actually need the transaction contents (`read_transaction`, conflict resolution) still read and surface errors at their call sites. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved dataset opening compatibility when transaction data contains unsupported or newer operation types. * Corrupted or unrecognized inline transaction data no longer prevents datasets from opening; it is deferred until the transaction details are needed. * **Tests** * Added coverage for unknown operations, corrupted transaction data, and valid transaction decoding. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
… columns (lance-format#7494) ## What `RecordBatchExt::merge` produced a `StructArray` with a **duplicated field** when both batches had a `List<Struct>` column of identical type. The equal-types branch pushed the left field/column, then fell through (a missing `else`) and pushed the merged column again — so the merged batch gained a phantom extra column. Guarding the merge path with `else` makes an identical `List<Struct>` column be taken from the left exactly once. ```rust // Before: the if body ran, then execution fell through and pushed a second time. if left_list.data_type() == right_list.data_type() { fields.push(left_field.as_ref().clone()); columns.push(left_column.clone()); } // This ran even when the types were identical, duplicating the field: let merged_sub_array = merge_list_struct(&left_column, &right_column); ``` ## Tests - `test_merge_list_struct_identical_schema` — top-level identical `List<Struct>` merge. Left and right use distinct values so the equality assertion proves the *left* column is kept; asserts a single column equal to the input. - `test_merge_nested_list_struct_identical_schema` — the same case reached through a recursive `merge` (a `List<Struct>` nested inside a struct), guarding the recursive path. Both tests fail on `main` (the double-push yields two columns) and pass with the fix. `cargo test -p lance-arrow`, `cargo clippy -p lance-arrow --tests -- -D warnings`, and `cargo fmt -- --check` are green. ## Out of scope (separate follow-up) While fixing this I noticed the `List<Struct>` arm of `merge` (both the identical-type and the structural-merge paths) does not call `adjust_child_validity`, unlike the sibling non-list arms — so a null parent-struct row is not propagated into a `List<Struct>` child during a recursive merge. This is a pre-existing gap that affects the whole arm, and doing it correctly on the merge path is non-trivial (which validity to propagate into a *merged* array). It is intentionally left out of this focused fix and tracked separately.
Adds a slice(start, end, columns=None) convenience method, equivalent to take(list(range(start, end))) but implemented as a thin wrapper over scanner(offset=start, limit=end-start), reusing the existing offset/limit scan pushdown instead of materializing an index list. Fixes lance-format#1808 Co-authored-by: Sapnil Basnet <vbb23@txstate.edu>
…ance-format#8772) ## What this fixes `MergeInsertBuilder.analyze_plan(data)` coerced every input to a one-shot stream, so it reported the streaming plan even when `execute(data)` on the same input would run a different one (lance-format#8771). Which side of the hash join gets collected, and the join type, both follow the statistics the source reports. `execute` sends a materialized source through an in-memory table that reports an exact row count and byte size, and DataFusion's `JoinSelection` picks the collected side from that. A stream reports nothing. `analyze_plan` reported the wrapping it chose rather than the one `execute` would choose, so anyone profiling a merge read metrics off the wrong side of the join. `analyze_plan` now dispatches on `_is_materialized` exactly as `execute` does. ## What the diagnostic printed, and what it prints now The docstring example in `dataset.py` passes a `pa.table`. Before, for that input: ``` HashJoinExec: mode=CollectLeft, join_type=Right, ... LanceRead: ... RepartitionExec: ... ProjectionExec: expr=[..., true as __merge_source_sentinel] StreamingTableExec: ... ``` After: ``` RepartitionExec: ... HashJoinExec: mode=CollectLeft, join_type=Left, ... ProjectionExec: expr=[..., true as __merge_source_sentinel] DataSourceExec: ... LanceRead: ... ``` The second one is what `execute` has been running all along. The doctest asserted the first. ## Rust surface `MergeInsertJob` gains `analyze_plan_batches` and `analyze_plan_provider`, mirroring the existing `execute_batches` and `execute_provider`. `analyze_plan(stream)` keeps its signature and delegates to the provider entry, so external Rust callers still compile and a stream is still reported as a stream. Two doc corrections came out of reviewing this. `explain_plan` now says outright that it only ever reports the streaming shape, because it receives a schema rather than data and so cannot know how the source would be wrapped; it also points at `analyze_plan` while noting that `analyze_plan` runs the merge and may write data files, which `explain_plan` does not. And `analyze_plan_batches` documents the two cases where it reports the streaming shape anyway: `SourceDedupeBehavior::FirstSeen` re-wraps the source in a stream ahead of the join, and an empty batch list carries no schema so the provider falls back to the dataset's. ## What this does not change No execution behaviour. `execute` already routed materialized sources through the in-memory table; only the diagnostic was out of step with it. A materialized `analyze_plan` now collects the reader into memory in Rust before planning, where it used to stream. The inputs `_is_materialized` accepts are already fully in memory, so the extra copy is bounded by data the caller holds, and it is the same copy `execute` has always made. The source types that could report statistics but do not are untouched. `lance.LanceDataset`, `pa.dataset.Dataset`, and `pa.dataset.Scanner` all arrive as a bare reader through `_coerce_reader` even though each knows its row count and can be scanned again, and the default streaming path drains the whole source into a spill before reporting no statistics at all. Both are remaining bullets on lance-format#4583, and this change is what makes their effect visible from Python. One pre-existing gap this touches without fixing: `batches_to_provider` falls back to the dataset's schema when the batch list is empty, so a zero-batch materialized source is validated against the target's columns rather than its own. `execute_batches` and `execute_uncommitted_batches` have always done this, and closing it changes `execute`'s public behaviour from a silent no-op to an error, which needs its own change and its own tests. One drive-by, disclosed rather than hidden: `explain_plan`'s not-supported message said only full-schema sources are supported. `can_use_create_plan` accepts a subset schema and, for a delete-only merge, the join keys alone, and its own doc comment lists all three. Rewriting the sibling message on the `analyze_plan` path made the two contradict each other, so both now name the two real reasons instead. The `does not support explain_plan` prefix that four tests match on is unchanged. ## Test plan - New `test_merge_insert_analyze_plan_matches_execute_routing`: a `pa.Table` source must report `DataSourceExec` and `join_type=Left`, a `RecordBatchReader` must report `StreamingTableExec` and `join_type=Right`. The first assertion fails without the dispatch change. - New `test_analyze_plan_reports_the_given_source_shape` covers the three Rust entries, including `analyze_plan_provider` directly. - New `test_plan_join_build_side_follows_source_statistics` pins which side the join collects at both of DataFusion's decision points: past `hash_join_single_partition_threshold_rows` where only the source can be collected, and below it where the smaller side wins. - `cargo test -p lance --lib merge_insert -- --test-threads=1`: 220 pass. - `uv run pytest python/tests/test_dataset.py -k merge_insert`: 26 pass. - `uv run pytest --doctest-modules python/lance/dataset.py -k "explain_plan or analyze_plan"`: 2 pass. - `cargo fmt --all`, `cargo clippy --all --tests --benches -- -D warnings`, `uv run make lint` from `python/`. --------- Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary - allocate random primitive bytes in an Arrow cache-line-aligned mutable buffer - cover empty and nonempty Float16, Decimal128, and Decimal256 generation ## Root cause RandomBytesGenerator filled a Vec<u8> and reinterpreted it as wider Arrow native values. Vec<u8> only guarantees byte alignment, and its empty dangling pointer is deterministically misaligned for these types. ## Fix Use Arrow MutableBuffer storage, which remains cache-line aligned for both empty and allocated buffers, before constructing the typed ScalarBuffer. ## Validation - cargo test -p lance-datagen - cargo fmt --all -- --check - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#7911 <!-- lance-gatekeeper-fix:v1 agent=dd598c88e2de9746a6afdf74f916a612 generation=1 --> Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary - make the NGRAM tokenizer disable stemming and stop-word removal by default so valid substring tokens are not discarded - preserve explicit filter overrides and analyzer settings persisted by existing indexes - document the conditional defaults and extend the existing multi-fragment Python FTS regression test ## Root cause Selecting `base_tokenizer="ngram"` changed the lexical tokenizer but inherited the text analyzer defaults, which enable word-oriented stemming and stop-word removal. Both indexed content and queries therefore dropped valid NGRAM tokens such as `the`. Existing NGRAM indexes retain their persisted analyzer behavior and must be rebuilt to adopt the corrected defaults. ## Validation - `cargo test -p lance-index scalar::inverted::tokenizer::tests` - `uv run make build` - `uv run pytest python/tests/test_scalar_index.py::test_fts_ngram_tokenizer -q` - `uv run make lint` - `cargo fmt --all` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8777 <!-- lance-gatekeeper-fix:v1 agent=2f76efc34dbba322cbd817f75555de94 generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…lance-format#8915) Merge insert (upsert) through the namespace API only accepted a single column as the match key, so there was no way to upsert on a composite key — even though Lance core has always supported it (`MergeInsertBuilder::try_new` takes a list of join columns). The limitation was purely in the request and transport layer. `MergeInsertIntoTableRequest.on` is now a list of Lance field paths, and both implementations pass it straight through to the builder. An empty list or a repeated column is rejected with `InvalidInput`. This is the implementation half of lance-format/lance-namespace#363, which made the same change in the spec and the generated clients. ## Example ``` POST /v1/table/orders/merge_insert?on=customer_id&on=order_date&when_matched_update_all=true ``` ## Breaking changes `MergeInsertIntoTableRequest.on` changes from `Option<String>` to `Option<Vec<String>>`. Rust callers passing a single column need to wrap it in a list. The HTTP wire format is unchanged for single-column callers: the query parameter uses `style: form, explode: true`, so a one-element list still serializes to `?on=id`. An older client keeps working against a server built from this change. Moving from one match column to several also changes how NULL keys behave, because core switches NULL join semantics on the arity of the key: a single-column key treats NULL as equal to NULL, while a composite key uses standard SQL equality, under which a NULL key matches nothing — not even a byte-identical NULL. That is pre-existing core behavior, but composite keys are reachable through the namespace API for the first time here, so it is newly visible. `test_merge_insert_composite_key_never_matches_a_null_key_column` pins it down. Java and Python SDK requests reach the Rust implementations as JSON across JNI and pyo3, so a jar or wheel predating this change sends `"on": "id"` where the model now expects `"on": ["id"]`. Those four bridge sites deserialize through `LenientMergeInsertIntoTableRequest`, which promotes a scalar to a one-element list, so mismatched SDK builds keep working. This is inbound only — a Java namespace implementation called *from* Rust still needs a matching jar. ## Not included The Java and Python `lance-namespace` pins stay on 0.11, so their generated models still send `on` as a bare string and rely on the promotion described above. Moving those pins to 0.12 and adding binding-level merge-insert coverage is a follow-up. The LanceDB Enterprise namespace server needs to accept the repeated `on` query parameter separately; that change is backward compatible on its own and does not depend on this one. Part of ENT-2084. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t stores (lance-format#8800) Conflict detection only sees versions newer than a handle's, so a dataset dropped and recreated at the same path accepts a stale writer's commit, and no check made before the commit can prevent it. The store that reserves versions has to decide. `ExternalManifestStore::put_if_predecessor` reserves a version only if the store's record for the predecessor still carries the identity the writer observed; `CommitHandler::commit_after` publishes on that condition and refuses with `PrerequisiteFailed`. The manifest is written once, at a staging path that listing never discovers, and the reservation records it as final: such a store is the dataset's history (`list_versions` backs the conflict scan and cleanup, and cleanup retires each record it removes through `forget_version`), and the canonical path a recreated dataset would share is never written. A write cancelled before its reservation leaves an orphan, as a retained staging manifest does today. No built-in store implements the contract; nothing changes for unconditioned commits. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…mat#8827) ## Summary - normalize out-of-range physical dictionary keys in null slots before structural validity extraction - preserve the zero-copy fast path for already-safe dictionaries - cover both hand-built and Arrow-concatenated reproductions ## Root cause Arrow permits arbitrary physical keys in null dictionary slots. The structural encoder records the key validity in rep-def and then removes the Arrow null buffer, which makes those previously meaningless keys appear valid and triggers dictionary bounds validation. ## Fix When a non-empty dictionary has an out-of-range key in a null slot, rebuild only its keys from the logical iterator. This writes key zero into null slots while preserving logical nullness and dictionary values. Dictionaries without the defect remain untouched. ## Validation - cargo test -p lance-encoding test_dictionary_out_of_range_null_keys_round_trip -- --nocapture - cargo test -p lance-encoding - cargo fmt --all -- --check - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#8826 <!-- lance-gatekeeper-fix:v1 agent=5c888a884a27ca24f9a7a9348ba6bb9c generation=1 --> Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>
…ance-format#8832) Fixes lance-format#8833. Also fixes lance-format#2514, which is the same bug reached through DuckDB and was closed without a fix. ## Problem A PyArrow filter on a timestamp column returned the wrong rows and raised nothing. `pc.field("ts") > pa.scalar(datetime(2024, 1, 3, 2), pa.timestamp("us"))` matched 0 of 100 rows where 49 was correct; the same filter written as SQL was fine. Timezone-aware columns failed the scan instead of answering wrongly. ## Cause PyArrow encodes the literal as Substrait's deprecated `Literal.timestamp`, defined by the spec as microseconds since the epoch, and leaves `type_variation_reference` at 0. DataFusion's consumer takes that field's unit from `type_variation_reference` and maps 0 to seconds, so the literal arrived a million times too large. On `timestamp[us]` the following cast overflows to null, which is why `>` and `<` both returned nothing. The deprecated `Literal.timestamp_tz` has no consumer branch at all. ## Change Before handing the expression to DataFusion, rewrite both deprecated literals into `precision_timestamp` / `precision_timestamp_tz`, which state the unit rather than implying it. Only the default reference changes meaning: references 1, 2 and 3 keep the milli/micro/nano units DataFusion gives them, and any other reference is left alone so DataFusion still reports it. Lance's own encode path uses the current DataFusion producer, which emits `precision_timestamp`, so it is unaffected. `remap_expr_references` is renamed to `normalize_expr` because it now does more than remap field references. PyArrow could also be changed to emit the newer encoding; this handles the plans it produces today. ## Tests - `rust/lance-datafusion/src/substrait.rs`: parses the deprecated literal at each variation reference and asserts the resulting unit, plus the tz form that used to fail. - `python/python/tests/test_filter.py`: `>`, `<` and `==` against `timestamp[s|ms|us]`, naive and with a timezone, checked against PyArrow's own answer. The DuckDB query from lance-format#2514 returns the matching row on this branch and an empty frame on pylance 10.0.0. Co-authored-by: Xuanwo <github@xuanwo.io>
## Summary - cast projected arrays against the projected target schema so nested struct children retain their hierarchy - supersede rewritten ancestor field entries in legacy files to keep fragment metadata valid - cover nested `int32` to `int64` casts across multiple fragments and both storage formats ## Root cause The cast mapper looked up each field by its leaf name in a batch whose projected nested column remained under its parent struct. That lookup panicked, and legacy files also needed their duplicated ancestor field IDs tombstoned when the child was rewritten. ## Validation - `cargo test -p lance test_cast_nested_column -- --nocapture` - `cargo test -p lance dataset::schema_evolution::test::test_cast_column -- --nocapture` - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#6926 <!-- lance-gatekeeper-fix:v1 agent=f2dd51b621a7ad313300457a6a44927b generation=1 --> --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>
Extend batch vector search (lance-format#6821) to the indexed/ANN path so a single multi-query request reads each IVF partition's storage once and scores every query that probes it, instead of re-running a full single-query plan per vector and unioning the results (which re-opens the index and rebuilds the prefilter for each query). - Add `VectorIndex::search_partitions_batch` + `supports_batch_partition_search` (defaulted so non-IVF indices stay explicitly unsupported). - Implement them for `IVFIndex` with a flat-style sub-index (IVF_FLAT/PQ/SQ/RQ): load each distinct partition once and accumulate one top-k heap per query, sharing the prefilter across the whole batch. - Add `ANNIvfBatchExec`, which ranks every query against the centroids, runs the shared-scan batch search, merges per-query top-k across deltas, and emits `query_index`-tagged results; route to it from `Scanner::batch_indexed_vector_search` when the gate below holds. - Normalize each query vector independently for cosine (`normalize_batch_query_for_index`): normalizing the concatenated batch key with one global norm would scale each vector by a batch-composition-dependent factor and break equivalence with single-query search. The shared-scan fast path is gated to cases that are provably equivalent to repeated single-query search: fixed nprobes (`minimum_nprobes == maximum_nprobes`), no refine step, an IVF flat-style index, and fully-indexed fragments. With adaptive nprobes the single-query path applies an `early_pruning` floor and late-search expansion that the batch path does not, so those queries fall back to the per-query loop, which stays exact. HNSW, refine, and mixed indexed/unindexed scans also fall back. Tests: plan shape; exact batch-vs-repeated-single equivalence (nprobes pinned); cosine regression; shared prefilter; multi-delta cross-delta merge; and fallbacks for refine and adaptive nprobes. Python parametrized over L2 + cosine; a batch-vs-repeated-single ANN benchmark. Closes lance-format#6822 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ministic ties, safe column access - Fall back from the batch path for any refine_factor (not just > 1); refine(1) still reranks and refine(0) errors on the single-query path - Fall back when stale-row overlays are present (gated before fast_search) - Stable partition scan order so tie truncation is deterministic across runs - Access result columns by name + validate per-query batch count Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback on the shared-scan batch IVF path. Memory: `search_partitions_batch` collected every loaded partition into a `Vec` before scoring, so peak memory scaled with the batch width — a wide batch probes up to `min(query_count * nprobes, num_partitions)` distinct partitions, i.e. potentially the whole index. Stream the loaded partitions through scoring in `STREAMING_SEARCH_BATCH_SIZE` chunks and drop each chunk once scored, bounding resident partition storage to the load window plus one chunk. `buffered` preserves the sorted (by part_id) load order, so the across-partition tie-break at the k-th distance stays deterministic, and each chunk is scored in a single `spawn_cpu` dispatch with the partition-loading `await`s kept in async code so no CPU-pool thread parks on I/O (lance-format#7642). nprobes(0): `min == max == 0` slipped past the fixed-nprobes gate. The single-query path probes nothing and returns an empty result, whereas the batch node clamped up to one partition — a silent divergence. Gate on `minimum_nprobes == 0` so the per-query loop defines the semantics, and drop the `.max(1)` clamp in favor of a `debug_assert!` documenting the invariant. Tests: add a nprobes(0) fallback test and a wide-batch test that spans multiple streaming chunks (exercising heap-threading across chunk boundaries), both pinned to repeated single-query search. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the streaming memory fix. `spawn_cpu` dispatches work to the CPU pool eagerly and its docs recommend pairing it with `StreamExt::buffered()`, but the chunk loop awaited each scoring dispatch inline, so `loaded_chunks` was not polled during scoring and partition loading paused on every chunk. Prefetch the next chunk and `join!` it with the current chunk's `spawn_cpu` scoring, so partition I/O stays in flight while the CPU pool scores. Scoring remains sequential across chunks (each mutates the same per-query heaps), so a step now costs about max(load, score) instead of their sum. Memory stays bounded: a scored chunk's storage is dropped before the next is scored. No behavior change; the 19 batch-knn tests still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hunk `test_batch_knn_indexed_streams_multiple_chunks` relies on the 20-partition index probing more distinct partitions than one `STREAMING_SEARCH_BATCH_SIZE` chunk (16) holds. That premise was implicit: if the default chunk size were raised past the partition count, the test would silently collapse to a single chunk and stop covering the heap-threading seam it is named for, while still passing. Assert `num_partitions > chunk_size` so that regression fails loudly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rd partitions_searched Three follow-ups on the shared-scan batch IVF path: - Gate eligibility on whether the *selected* index_segments cover every requested fragment, not whether the whole logical index does. A subset selected via with_index_segments could otherwise let the batch node search only the selected segments and silently drop a fragment covered solely by an unselected segment. Extracted the coverage check into fragments_missing_from_index_segments, shared by the gate and knn_combined so eligibility and fallback stay in lockstep. - Run the per-query centroid ranking on the dedicated CPU runtime (find_partitions_batch_on_cpu) instead of the async worker: the ranking is pure CPU and batch width multiplies it, so a wide batch over a large centroid set could monopolize a Tokio worker. - Record partitions_searched on ANNIvfBatchExec. It built the metric but never incremented it, so EXPLAIN ANALYZE reported 0 for every batch query. Report the distinct partitions read -- the shared I/O this node exists to save -- mirroring the single-query ANNIvfSubIndexExec. Tests: partial-segment fallback, CPU-runtime ranking, and a partitions_searched=2 assertion (distinct union, not the per-query sum of 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sezruby
force-pushed
the
knn-batch-6822
branch
from
September 3, 2026 00:58
a1060cd to
8fc0e0c
Compare
… is set The shared-scan batch path builds one prefilter across the batch and does not carry a caller-supplied external row-address mask (`with_row_addr_prefilter`), whereas the per-query path threads it into each query via `with_external_mask`. After the merge with the external-mask feature, an otherwise batch-eligible query with a mask would have selected the batch node and silently dropped the mask, returning masked-out rows. Disqualify the batch path in `batch_index_search_supported` whenever an external mask is present so the query falls back to the per-query indexed loop, which honors the mask. Matches the existing fallback pattern (refine, adaptive/zero nprobes, HNSW, overlay, partial coverage). Adds `test_batch_knn_indexed_external_mask_falls_back`: the same query is batch-eligible without a mask, falls back to ANNSubIndex with one, and every returned row is in the allowlist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch vector search (#6821, PR lance-format#6828) made indexed multi-query search work by looping the full single-query plan once per query vector (re-opening the index and rebuilding the prefilter each time) and unioning the results. This PR makes the indexed/ANN path share index-level state across the batch: it reads each probed IVF partition's storage once and scores every query that probes it, with the prefilter built once and shared.
Approach
VectorIndextrait (lance-index): defaultedsupports_batch_partition_search()+search_partitions_batch(...)(default returnsnot_supported), so non-IVF indices are explicitly unsupported.IVFIndex(ivf/v2.rs): batch search for flat-style sub-indices (IVF_FLAT/PQ/SQ/RQ). Invert per-query partition lists, load each distinct partition once, accumulate one top-k heap per query, reusingaccumulate_prepared_partition_search/global_heap_to_batch.ANNIvfBatchExec(io/exec/knn.rs): ranks each query against the centroids, runs the shared-scan batch search per delta, merges per-query top-k across deltas, emits{query_index, _distance, _rowid}. Prefilter wiring shared with the single-query node viabuild_dataset_prefilter.normalize_batch_query_for_index).Design notes (pre-empting review questions)
KNNVectorDistanceExec/ the two ANN nodes? The two-node single-query pipeline streams one partition-list per delta through a per-query top-k. Sharing the scan requires inverting queries onto partitions and keeping one heap per query in a single pass — a different dataflow. The new node still reuses the underlying primitives (partition load,build_dataset_prefilter, and the index's per-partition accumulate), and the single-query nodes are untouched. Happy to fold it in differently if you'd prefer.supports_batch_partition_search()? The gate is a planning-time decision and the single-query path likewise doesn't open the index there;derive_vector_index_typereads metadata with no I/O. The opened index re-checks the trait as a defensive invariant.minimum_nprobespartitions/query. The single-query path is adaptive (early_pruningfloor + late-search expansion), so it only matches when nprobes is fixed. The fast path is therefore gated tominimum_nprobes == maximum_nprobes; adaptive nprobes falls back to the per-query loop (verified: an unpinned batch diverged on every query before the gate; 0 divergence after). Open question for you: fixed-nprobes-first with batched early/late as a follow-up, or the full adaptive path in one PR?search_partitions), widened to the batch's partition union. Per-delta output is k-bounded, so cross-delta accumulation is O(deltas × k), not O(nprobes × rows).Fallback matrix (no regression)
refine_factor/ IVF_HNSW_* / mixed indexed+unindexedTest plan
cargo test -p lance --lib test_batch_knn— 15 tests: plan shape, exact batch-vs-repeated-single equivalence (nprobes pinned), cosine regression, shared prefilter, multi-delta cross-delta merge, and explicit fallbacks for refine, adaptive nprobes, and IVF_HNSW (acceptance: "unsupported index types have explicit behavior and tests").cargo test -p lance --lib dataset::scanner::test::test_knn(29) — no single-query regression (exercises the sharedbuild_dataset_prefilter).cargo fmt --all&&cargo clippy -p lance -p lance-index --tests --benches -- -D warnings.pytest -k batch(L2 + cosine × three/single queries);ruffclean;pyrightclean on changed lines.benchmarks/test_search.py): batch vs repeated-single ANN; standalone timing (50k rows, dim 128, IVF_PQ 64 partitions, m=32, k=10, nprobes=10) → 2.48× speedup.Closes lance-format#6822