feat(mem_wal): record what a flushed SSTable holds by xuanyu-z · Pull Request #8981 · lance-format/lance

xuanyu-z · 2026-09-04T02:26:33Z

A shard manifest entry records only where an SSTable is, not what it contains. Anything a reader wants to know before opening one — how much payload it holds, how many rows, how much of that is primary key — costs a read of the generation to find out, which is precisely the read such a reader is deciding whether to make.

Record three optional fields on SsTable, all accounted on the writer's MemTable at flush:

field	meaning	cost to populate
`in_memory_bytes`	payload size of the rows	a counter the batch store already keeps — it is what the flush threshold is measured against
`physical_rows`	rows held, counting the older duplicates of a primary key that the deletion vector masks	a counter the batch store already keeps
`primary_key_bytes`	total payload size of the primary-key columns over those rows	one size estimate per key column per batch, accumulated where the bloom filter already holds the key columns — O(columns) on a path already O(rows)

Flat scalars on the entry, matching how the format already carries per-object stats (DataFragment.physical_rows, DataFile.file_size_bytes, DeletionFile.num_deleted_rows) rather than grouping them into a nested message.

They are estimates of payload, not bounds on the cost of reading an SSTable. They exclude the per-array structure a reader materializes, and primary_key_bytes is not the size of any encoded form of the key — a comparable byte-ordered encoding built for sorting or deduplication is a different and larger thing. A consumer budgeting memory adds its own headroom. The proto and docs/src/format/table/mem_wal.md both say so, so the contract does not promise more than the numbers support.

primary_key_bytes is a total, not a per-row size: a variable-length key differs from row to row, and a consumer wanting a mean divides by physical_rows. Totals are the composable primitive — they sum over a set of generations, where averages do not.

All three are optional rather than following the older zero-means-unknown convention of DataFile.file_size_bytes, so presence is a type a reader must handle: an entry written before these fields existed records none of them, and primary_key_bytes is also absent on a table with no primary key. A zero measurement records as absent for the same reason — an empty MemTable is refused before a flush, so a flushed generation always holds rows. SsTable::unmeasured builds an unmeasured entry.

Read off the MemTable while it is frozen, which is load-bearing: BatchStore::append bumps these counters before it publishes committed_len, and a scan is bounded by committed_len, so only a sealed store agrees with what a scan of it will see. FlushedSize carries the three together so both flush paths share one measurement and neither the manifest write nor the FlushResult can transpose them.

Note for downstream: SsTable gains three fields, so code constructing it with a struct literal needs updating.

Testing. Flush tests read the entry back through read_version, not latest — the store caches the manifest it just wrote, so latest returns that same Rust value and a test asserting through it passes even for a field that never reached the protobuf. Verified by dropping primary_key_bytes from the encode path: the naive shape passed, this one fails. Two conversion tests cover the round trip and an older payload decoding as unmeasured rather than as zero.

FlushFixture carries the store/shard/flusher setup that all thirteen flush tests in the module repeat; the new tests use it, and converting the other eleven is a mechanical follow-up.

663 mem_wal and 386 lance-table tests pass; ci/check_proto_comments.py and fmt clean.

github-actions · 2026-09-04T02:26:53Z

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals


Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Wed 2026-09-09 02:26 UTC (Tue 19:26 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

lance-gatekeeper

❌ Gate recommendation: request changes.

The outer shard-manifest is a reasonable place for pre-read generation metadata, but the byte value needs a defensible admission contract and the persisted format must be reviewed independently of writer behavior. A viable sequence is a focused protobuf plus format-spec proposal first, followed by the flush implementation and its storage-roundtrip and decoded-memory tests after that contract lands.

Please mark this PR with the breaking-change label.

lance-gatekeeper · 2026-09-04T02:39:48Z

+  // could not measure the memtable. It is the size of the rows written, so it
+  // covers a read that materializes all of them; a read applying the
+  // generation's deletion vector materializes less.
+  optional uint64 in_memory_bytes = 3;


SsTable is embedded in the persisted shard manifest, so these fields are a format-specification change. The repository contract requires a proposal containing the protobuf, matching docs/src/format/ text, and only the minimum compilation edits; writer behavior belongs in a follow-up. This PR changes both flush paths while docs/src/format/table/mem_wal.md still says an entry records only generation and path, leaving the durable estimate/presence semantics outside the standalone contract voters review. Please split the flush implementation and test into a dependent PR and document those semantics in the format proposal.

The specification and format documentation are now complete, but a563daf1 still adds the writer behavior in this PR. The repository format process requires that implementation to land in a follow-up, so the split remains necessary.

A shard manifest entry records only where an SSTable is, not what it contains. Anything a reader wants to know before opening one -- how much payload it holds, how many rows, how much of that is primary key -- costs a read of the generation to find out, which is the read such a reader is deciding whether to make. Add three optional fields to `SsTable`, all accounted on the writer's MemTable at flush: - `in_memory_bytes`: payload size of the rows. - `physical_rows`: rows held, counting the older duplicates of a primary key that the generation's deletion vector masks. - `primary_key_bytes`: total payload size of the primary-key columns over those rows -- a total, not a per-row size, which varies for a variable-length key. All three are estimates of payload rather than bounds on the cost of reading the SSTable: they exclude the per-array structure a reader materializes, and `primary_key_bytes` is not the size of any encoded form of the key. A consumer budgeting memory adds its own headroom. Documented as such, so the contract does not promise more than the numbers support. All three are optional, so an entry written before they existed records none of them and a reader must not treat absence as zero; `primary_key_bytes` is also absent on a table with no primary key. `SsTable::unmeasured` builds such an entry. Flush writes unmeasured entries for now, which is the minimum edit that keeps the library compiling. Populating the fields is a follow-up, per the change process in `protos/AGENTS.md`.

The manifest entry has fields for an SSTable's payload size, row count and primary-key size; flush wrote none of them. Populate all three from the MemTable being written, which is where the numbers exist and where two of them are already maintained. `in_memory_bytes` and `physical_rows` are counters the batch store keeps -- the first is what the flush threshold itself is measured against. `primary_key_bytes` is accumulated where the bloom filter is updated, which already holds the key columns: one size estimate per key column per batch, against a loop that already walks every row. Read off the MemTable while it is frozen, which is load-bearing: `BatchStore::append` bumps these counters before it publishes `committed_len`, and a scan is bounded by `committed_len`, so only a sealed store agrees with what a scan of it will see. `FlushedSize` carries the three together so both flush paths share one measurement and neither the manifest write nor the result can transpose them. A zero reads as absent, which is also how a table with no primary key reports: an empty MemTable is refused before a flush, so a flushed generation always holds rows. Tests read the entry back through `read_version` rather than `latest`. The store caches the manifest it just wrote, so `latest` returns that same Rust value and a test asserting through it passes even for a field that never reached the protobuf -- verified by dropping `primary_key_bytes` from the encode path, which these tests now catch. `FlushFixture` carries the setup all thirteen flush tests repeat.

lance-gatekeeper

❌ Gate recommendation: request changes.

2 fixed / 1 remains. The payload/headroom contract and persisted read-back tests now address the two technical findings. The remaining blocker is the repository format process: this PR still combines the specification with the writer implementation.

Keep the c9c5303b protobuf, format documentation, and minimum compilation edits here, then land the a563daf1 population work separately after the format proposal. The closed #9002 already identifies that follow-up.

Please mark this PR with the breaking-change label.

jackye1995

looks good! this is experimental feature, a vote is not needed

github-actions Bot added A-format On-disk format: protos and format spec docs format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). labels Sep 4, 2026

github-actions Bot added the enhancement New feature or request label Sep 4, 2026

lance-gatekeeper Bot reviewed Sep 4, 2026

View reviewed changes