feat: support atomic insert_overwrite writes - #62
Merged
Merged
Conversation
Adds mode="overwrite_where" to daft_lance.write_lance: one Lance commit deletes the rows matching a SQL predicate and adds the new data, so readers see either the whole replacement or none of it. The delete runs distributed, one task per fragment, reusing the DatasetOpenContext pattern the merge and compaction paths already use. When a scalar index covers the predicate, a row-address lookup prunes the fragment list first; without one, every fragment goes to the workers rather than paying for a serialized scan on the driver. Input rows are checked against the predicate by default: overwrite_where appends the input verbatim, so a row outside the predicate would survive the next run of the same write instead of being replaced. validate_predicate=False opts out. The new mode writes exactly like an append and is normalized to it in the constructor, so the existing schema, blob-column and storage-version checks cover it -- adding a fourth mode string to those checks would have skipped them silently. Known gap, documented in write_lance and the README: Lance does not treat a concurrent append as conflicting with the Update this commits, so rows another writer adds during the overwrite survive it. test_concurrent_append_survives _the_overwrite pins that behavior. Co-authored-by: fanng <“fanng@apache.org”> Claude-Session: https://claude.ai/code/session_017b8TVnEmyN8wbXFTHJwYip
Review of the previous commit found two ways the input-row check could hurt
rather than help, both confirmed against real datasets.
The check evaluates the predicate with Daft while the delete runs with
Lance/DataFusion, and the two engines disagree on a decimal literal compared
against a narrow float column: Daft widens 0.1f32 to 0.10000000149 and calls
"score > 0.1" true, Lance narrows the literal and calls it false. A row like
that passed validation and was then never deleted, so re-running the same
write piled up duplicates -- exactly what the check exists to prevent. A
predicate reading a float32/float16 column is now refused up front.
The driver-side check also only parsed the Daft expression, never evaluated
it, so a Lance-valid predicate Daft types differently ("t >= TIMESTAMP '...'"
against a naive timestamp column, the shape the docstring advertises) failed
mid-write with a raw DaftTypeError and no pointer to validate_predicate=False.
It is now evaluated against a zero-row input typed like the table.
Also: from_pylist produces a single partition, so the per-fragment delete ran
as one task on a distributed runner. It is now repartitioned by fragment id,
except on the native runner where repartition is a no-op that only warns. The
dead concurrency parameter is gone and the docstrings that claimed the work
spread across the cluster now match what it does.
Tests: the concurrency and idempotence tests asserted so little that they
passed with the overwrite stubbed out to a no-op, and no test drove the
index-pruning path end to end. Both now assert whole row sets, and a new test
overwrites twice through an index on the predicate column -- the second run
only replaces the row the first one appended if pruning still finds that
fragment. Stubbing the overwrite to a no-op now fails 13 of 26 tests instead
of 7 of 22, and a pruning miss fails 2.
Co-authored-by: fanng <“fanng@apache.org”>
Claude-Session: https://claude.ai/code/session_017b8TVnEmyN8wbXFTHJwYip
FANNG1
changed the title
FANNG1
marked this pull request as ready for review
September 10, 2026 08:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mode="insert_overwrite"with a requiredoverwrite_whereSQL filter todaft_lance.write_lance: one Lance commit deletes matching rows and adds the DataFrame rowsoverwrite_whereselects existing rows to removeappend, retaining existing schema, Blob-column, and storage-version validationNo Lance-side change: this composes
LanceFragment.delete()withLanceOperation.Update.Closes #61
Semantics
start()pins the dataset version. Workers write input into new fragments asappenddoes;finalizeappliesoverwrite_whereto that pinned snapshot and commits oneLanceOperation.Update(updated_fragments, removed_fragment_ids, new_fragments, fields_modified=[]). Deletes write deletion files rather than rewriting data, so existing row addresses and indexes remain valid. New fragments are unindexed and are scanned flat. If no rows match and no data is written, no version is created.Fragment pruning is opportunistic: when
explain_planshowsScalarIndexQuery, a streamed_rowaddrscan derives the exact matching fragment IDs; otherwise every fragment is visited, avoiding an additional full scan merely to plan pruning.The input is appended verbatim. Rows outside
overwrite_whereare not replaced by a later re-run and can accumulate, so callers needing idempotent replacement should filter the input first.Concurrency behavior
Lance does not currently treat a concurrent
Appendas conflicting with thisUpdate: rows another writer adds during the overwrite can survive even if they matchoverwrite_where.README.mddocuments this behavior, andtest_concurrent_append_survives_the_overwritepins it for future Lance changes.Test plan
ruff format --checkandruff checkdaft_lance/and the focused test fileShuffleRead -> UDFProjectdelete stageFocused coverage in
tests/io/lancedb/test_insert_overwrite.pyincludes multi-fragment atomic replacement, idempotent re-runs, empty/no-match cases, full-fragment removal, argument and schema validation, index pruning, scalar and IVF_PQ index behavior, namespace tables, and concurrent append behavior.Mutation checks verify that a no-op overwrite or empty pruning result makes behavioral tests fail.