chore: prepare 0.5.0 release

[FANNG1]

Summary

Bump the version to 0.5.0 so a release can be cut from main. 0.4.0 went out on 2026-06-05 and main has 11 unreleased commits since.

Same shape as the 0.4.0 prep (deeb798): the version in pyproject.toml and in the daft-lance entry of uv.lock. No other changes.

Note on #66

#66 (pylance floor to >=11.0.0) is deliberately not included. Releasing it together would make this release require a Lance upgrade for anyone on pylance 8–10. This PR keeps pylance>=8.0.0 so the backlog ships on its own, and #66 can go into the next release. Happy to reorder if you would rather have it in 0.5.0.

Draft release notes

GitHub's auto-generated notes are just PR titles, and several of these (for example "support segmented bitmap indexes") don't tell a reader what changed for them. Draft below to paste into the Release instead — please edit freely.

---

Breaking changes

• create_scalar_index(..., replace=...) now defaults to False — it was True in 0.4.0 (Add segmented index workflow for INVERTED and FTS #28). Code that re-ran index creation under the same name to refresh an index, relying on the default, now fails with ValueError: Index with name '<name>' already exists. Set replace=True to replace it. Pass replace=True explicitly to keep the old behavior.
• segmented=True now refuses to build under an existing index name (Add segmented index workflow for INVERTED and FTS #28), including with replace=True. Lance's public commit_existing_index_segments(...) exposes no atomic replacement, and 0.4.0 did not check: committing segments under a name that already existed silently left the dataset with two indexes of that name (verified on pylance 8 — the original entry disappeared and two same-named ones took its place). 0.5.0 rejects it up front: ValueError: Index with name '<name>' already exists and cannot atomically replace existing index with Lance's public segmented index API. Drop the index first, or use a different name. Non-segmented BITMAP with replace=True still replaces through Lance's scalar path (Segment index bitmap #31). This newly affects INVERTED / FTS callers: in 0.4.0 segmented=True was ignored for those types, so the call went down the legacy path, where replace=True did replace the index. The flag now takes effect, and with it this restriction.
• Requires pylance>=8.0.0 (0.4.0 allowed >=7.0.0). The distributed scalar-index path now uses Lance's public segment-index API, which needs Lance 8 (Add segmented index workflow for INVERTED and FTS #28).

Features

New daft_lance.write_lance(df, ...) (#35) — a module-level write entry point alongside read_lance, taking a DataFrame plus mode="create" | "append" | "overwrite" | "insert_overwrite", an optional explicit schema, and either a URI or a namespace-addressed table. 0.4.0 exported read_lance only.

Lance Namespace support across table operations (#35) — Tables can now be addressed through a Lance Namespace (for example a Gravitino Lance REST server) instead of only by URI. daft_lance.read_lance, write_lance, LanceDataSink, and the merge / index / compaction operations all accept namespace-addressed tables, resolve them through a shared resolver, and merge namespace-vended storage credentials with caller-supplied options. Distributed workers reopen the dataset from a serializable DatasetOpenContext rather than receiving a live LanceDataset, so maintenance tasks keep their namespace identity and vended credentials on the worker side.

Atomic insert overwrite (#62) — write_lance(..., mode="insert_overwrite", overwrite_where="<SQL filter>") replaces the rows matching a filter with the DataFrame's rows in a single Lance commit: matching rows are deleted and the new rows added, with no window where a reader sees one without the other. The delete runs as a distributed Daft job and uses scalar-index pruning when the filter plan supports it, so it visits only the fragments that can contain matches. Deletes write deletion files instead of rewriting data, so existing row addresses and indexes stay valid. If nothing matches and no data is written, no new version is created.

Segmented index building for INVERTED, FTS, and BITMAP (#28, #31) — segmented=True now builds these index types the way BTREE already did: each worker builds a fully independent index segment over its own fragments through Lance's public create_index_uncommitted(...), and the coordinator commits them together with commit_existing_index_segments(...). In 0.4.0 the flag only took effect for BTREE — passing segmented=True with INVERTED or FTS was silently ignored and fell back to the legacy partitioned workflow, and BITMAP always fell back to single-threaded creation inside Lance. Bitmap segments carry a shard id so independently built segments stay distinguishable before the merge.

Two consequences worth knowing: indexes built this way carry real index details, so describe_indices() works on them (the legacy partitioned path leaves those details empty and describe_indices() still raises on indexes it produced); and the legacy path was incompatible with Lance 8 for BTREE, which is why 0.5.0 requires pylance>=8.0.0. See the breaking changes above for the replace changes that come with this.

Fixes

merge_columns_df fast path delegated to PyLance (#60, closes #59 and #20) — daft-lance had its own implementation of Lance's merge-column write path (file writing, field-ID allocation, fragment metadata, file URI composition), which drifted from Lance itself. Two user-visible bugs came out of that: merges against datasets with deletion vectors could commit a version that could not be read back, and merges on object-store datasets (s3://) crashed because the writer called os.path.getsize() on a URI. Both are structurally gone now that PyLance's fragment.merge_columns(...) owns file creation and metadata.

Untouched fragments preserved in slow merges (#56) — When merge_columns_df took the slow path with input covering only some fragments, the commit succeeded but the new version contained only the touched fragments. Untouched fragments silently disappeared from the manifest, reducing visible row and fragment counts. The slow path now completes the fragment list from the driver snapshot before committing, sharing that logic with the fast path.

Correct field IDs for new columns (#46, fixes #41) — The fast-path writer derived the next field ID from top-level schema fields only. Nested fields (struct children) and field IDs left behind by schema evolution have their own IDs, so a new column could be committed with an ID colliding with an existing nested or historical field, and reads of the new column could return nulls or wrong values. The writer now uses the manifest's max_field_id high-water mark.

Arrow types preserved for merged columns (#44, fixes #40) — The fast-path writer rebuilt new columns through pa.array(series.to_pylist()), which drops Arrow type information: Python floats become float64 and lists lose their fixed-size constraint. A fixed_size_list<float32>[128] embedding column from a UDF was silently widened to list<double> and committed with the wrong schema type. The writer now goes through to_arrow() and keeps the type the UDF declared.

Blob reads no longer exhausted by the fast-path check (#45, fixes #42) — _can_use_fast_path counted rows with len(df.collect()), which populates Daft's result cache. With take_blobs() in the pipeline, that cache holds one-shot BlobFile streams; the subsequent merge reused those exhausted objects, so downstream UDFs calling blob.read() got empty bytes and produced nulls or wrong output. The check now uses count_rows(), which leaves the cache unset.

Internal

• uv.lock excluded from the pretty-format-toml pre-commit hook, which had been failing the style job on every PR (chore: exclude uv.lock from pretty-format-toml hook #48, fixes CI style job fails on every PR: uv.lock conflicts with pretty-format-toml hook #47)
• segmented=False honored again in create_scalar_index (fix: respect segmented scalar index flag #49). No effect for anyone upgrading from 0.4.0: this fixes a regression introduced by Add segmented index workflow for INVERTED and FTS #28, and both landed after 0.4.0 shipped.

---

https://claude.ai/code/session_017ZCLiXSVbAbtEV7z8smf8D