create_scalar_index commits indexes with empty index_details, so Lance reports their type as Unknown

[FANNG1]

Summary

create_scalar_index() commits indexes whose index_details field is never populated, so Lance reports their type as Unknown. This affects the default path — segmented=False — so every caller who does not explicitly opt into segmented=True gets an index with missing type metadata.

The index data itself is correct: full-text search returns the expected rows and stats.index_stats() reports the real type. Only the manifest-level type marker is lost.

Reproduction

import tempfile, os
import pyarrow as pa, lance
from daft_lance import create_scalar_index

uri = os.path.join(tempfile.mkdtemp(), "t.lance")
tbl = pa.table({"text": ["hello world", "foo bar", "lance daft"] * 20})
lance.write_dataset(tbl, uri, max_rows_per_file=20)   # 3 fragments

create_scalar_index(uri=uri, column="text", index_type="INVERTED")   # segmented=False is the default

ds = lance.dataset(uri)
print("list_indices     :", ds.list_indices()[0]["type"])
print("describe_indices :", repr(ds.describe_indices()[0].type_url))
print("index_stats      :", ds.stats.index_stats("text_inverted_idx")["index_type"])
print("FTS still works  :", ds.to_table(full_text_query="lance").num_rows, "rows")

Output:

list_indices     : Unknown
describe_indices : ''
index_stats      : Inverted
FTS still works  : 20 rows

Comparison

Path	list_indices()["type"]	describe_indices().type_url
create_scalar_index(..., segmented=False) — default	Unknown	''
create_scalar_index(..., segmented=True)	Inverted	/lance.table.InvertedIndexDetails
pylance native ds.create_scalar_index(...)	Inverted	/lance.table.InvertedIndexDetails

Root cause

This is not a pylance regression, and not a serialization problem. I ruled both out:

• Driving pylance's create_index_uncommitted() + commit_existing_index_segments() directly, with one uncommitted segment per fragment and no daft-lance involved, produces Inverted.
• Pickling and unpickling the Index objects between those two calls — what the distributed path does to ship segments back to the coordinator — still produces Inverted, and no attribute of the object changes across the round trip.

The loss happens in _create_partitioned_index() in daft_lance/lance_scalar_index.py, the legacy merged-metadata path that segmented=False selects. It hand-builds the index metadata and commits it through a manual transaction:

index = lance.Index(
    uuid=index_id,
    name=name,
    fields=[field_id],
    dataset_version=lance_ds.version,
    fragment_ids=set(fragment_ids_to_use),
    index_version=0,
)
...
create_index_op = lance.LanceOperation.CreateIndex(new_indices=[index], removed_indices=removed_indices)

lance.Index exposes no way to set index_details, so the committed manifest entry has that field empty and Lance falls back to reporting Unknown. _create_segmented_index() avoids this because commit_existing_index_segments() lets Lance populate index_details itself.

The code already carries a note acknowledging this:

# NOTE: kept on list_indices() until the distributed-index commit path is
# rewritten to populate index_details (e.g. via commit_existing_index_segments).
# describe_indices() raises on indices produced by this flow because their
# index_details field is empty.

Two things about that note are worth updating: the problem is not confined to replace=True (it applies to every index this path commits), and under pylance 11 describe_indices() no longer raises — it returns an empty type_url, which is quieter and therefore easier to miss.

Impact

Queries are unaffected, so this is metadata correctness rather than data loss. What breaks is anything that introspects index type through the documented APIs. Concretely, it surfaced while upgrading Daft to daft-lance 0.5.0: assertions of the form list_indices()[...]["type"] == "Inverted" started failing, and had to be rewritten against stats.index_stats() to pass — see Eventual-Inc/Daft#7512.

stats.index_stats() is a workaround for reading the type back, but it does not repair the manifest, and indexes already committed by this path stay Unknown.

Suggested fix

Migrate _create_partitioned_index() to commit_existing_index_segments(), the same public API _create_segmented_index() already uses successfully. That is the direction the note points at, and it overlaps with the INVERTED / FTS rows of #27 — though this issue is narrower: it is a correctness bug in the path that ships today by default, independent of the broader segment-index migration.

If the migration is not near-term, defaulting segmented=True for MERGED_SEGMENTED_INDEX_TYPES would avoid the bad path for BITMAP and INVERTED in the meantime.

Environment

daft-lance       0.5.0
pylance          11.0.0
lance-namespace  0.8.6
daft             0.3.0.dev0 (main)
python           3.11
platform         macOS arm64