Expose version-range Lance changes as lazy Daft DataFrames

[hello-peter-tang]

Problem and business value

Incremental enrichment and downstream search-index synchronization need to process changed rows between two Lance versions and remove entries whose rows were deleted. Reading complete snapshots or materializing a Lance delta reader on the driver adds avoidable orchestration and memory costs.

At main 1b1fa3c, read_lance reads one snapshot; there is no public version-range/change-kind reader or use of Lance's delta-reader APIs. Users can currently compose snapshot joins or manually wrap PyLance readers, but there is no supported lazy, worker-side delta source.

Upstream capabilities

PyLance already had DatasetDelta.get_inserted_rows() and get_updated_rows(). Lance v11 adds get_deleted_row_ids(), completing the deletion-ID side of this integration:
lance-format/lance#8589

This is a read-side feature, separate from writing updates in #57 / #58 or checkpointed append commits in #16.

Proposed MVP

• Provide an explicit begin/end-version reader with separate inserted-row, updated-row, and deleted-row-ID modes; each produces a Daft DataFrame using the upstream result schema.
• Support URI and namespace addressing. Resolve and pin both endpoints, and open the dataset used by the reader at the requested end version rather than an unrelated latest snapshot.
• Construct and consume the Arrow RecordBatchReader on a worker when the DataFrame executes, without collecting the entire result on the driver.
• Start with one streaming source task per requested mode and allow downstream Daft processing to parallelize. Do not claim distributed delta computation or split the temporal range unless equivalence is proven.
• Define supported-version, stable-row-ID, empty-result, and schema behavior explicitly. The current PyLance dependency floor permits versions lacking the new API.

Semantic boundaries

These are version-range results, not an ordered CDC event log or a source of all intermediate before/after images. Deleted results contain only _rowid: IDs live at the beginning and absent at the end. Consumers needing business keys must retain a mapping or read the beginning snapshot.

Compaction alone must not look like deletion. Concatenating arbitrary subrange deletions is incorrect when rows are inserted, deleted, or restored within the overall interval. Automatic checkpoint management, exactly-once downstream delivery, and schema-evolution reconciliation are outside this MVP.

Acceptance criteria

• Match PyLance results for append, repeated update, row/whole-fragment deletion, and empty results.
• Verify deletion IDs against begin-live minus end-live IDs, including insert-then-delete, delete-then-restore, and compaction cases.
• Cover equal/reversed/unavailable endpoints, stable-ID prerequisites, and explicit schema handling.
• Demonstrate lazy worker execution, bounded batch consumption, and correct namespace context reconstruction.
• A newer commit after planning does not alter either endpoint or the results.

[hello-peter-tang]

To clarify, this is an integration gap: daft-lance reads individual snapshots but does not expose version-range changes as a lazy DataFrame source. PyLance's inserted/updated readers predate v11; v11 adds deleted-row IDs. Users currently need custom reader integration or snapshot-comparison workflows.

The practical value is incremental processing and downstream index synchronization: process new/updated rows and remove deleted entries rather than reprocessing unchanged data. A worker-side streaming reader also avoids collecting the complete delta on the driver. Deleted results contain row IDs only, so downstream business-key synchronization needs a mapping; this is not a full CDC event log.

[hello-peter-tang]

PyLance version requirement

The proposed MVP, including deleted-row IDs, requires pylance>=11.0.0 because DatasetDelta.get_deleted_row_ids() was added in v11. Inserted/updated-row readers predate v11; the v11 requirement should not be attributed to those APIs.

The current pylance>=8.0.0 floor is insufficient for the complete MVP. Its implementation should explicitly enforce/document the requirement for both driver and workers and test the minimum supported version.

This is a runtime-library requirement, not dataset snapshot version=11 or a requirement that data was written with v11. Dataset prerequisites, including stable-row-ID support and retained endpoint versions, must be checked separately. Upgrading PyLance alone does not provide the proposed lazy daft-lance reader.