fix(mem-wal): resolve writer shard spec identity

[u70b3]

Summary

• resolve the default dataset-writer shard_spec_id (0) to the MemWAL index's sole automatic sharding spec before creating or claiming a shard manifest
• accept an explicit id only when it matches that sole spec, and require manually sharded indexes to keep id 0
• reject metadata with multiple automatic specs because the current writer model supports at most one
• cover the original fresh-shard bug at the public Dataset::mem_wal_writer boundary plus invalid manual, mismatched, and multiple-spec inputs

Scope

This PR fixes the current writer path only. MemWAL initialization already creates either no spec (manual sharding) or one automatic spec with id 1; the writer must record that same identity instead of its unresolved default 0.

It does not add monotonic allocation, spec revisions, activation semantics, metadata validation for hypothetical producers, or a file-format change. Those require a separate end-to-end design if a re-shard API is introduced.

This PR complements but is not a prerequisite for #7949: this PR resolves the table identity before a fresh manifest is created, while #7949 defensively rejects changing an identity already stored in a manifest.

Testing

• cargo fmt --all -- --check
• cargo test -p lance --lib test_writer_shard_spec_resolution_rejects_invalid_identity -- --nocapture
• cargo test -p lance --lib test_mem_wal_writer_uses_automatic_sharding_spec -- --nocapture
• cargo clippy --all --tests --benches -- -D warnings

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[u70b3]

Scope reset completed and force-pushed on latest main.

The PR is now one commit / one file (+96) instead of 8 files (+541/-20). It fixes only the current public writer path: initialization records automatic spec 1, while ShardWriterConfig::default() supplies unresolved 0; Dataset::mem_wal_writer now resolves that default to the sole stored spec before creating the manifest and rejects explicit mismatches.

Removed: monotonic allocation, multi-revision/activation semantics, metadata-boundary validation, format docs, and all format-change claims. The original boundary regression failed with manifest spec 0 vs expected 1 before the fix and now passes. Full workspace clippy and the focused cases pass after the latest-main rebase.

This is complementary to, but not a prerequisite for, #7949. Please re-review the reduced current-path fix.

[u70b3]

CI note: the current failures are runner-environment flakes unrelated to this change (which touches only mem_wal/api.rs): uring::tests::test_read_range died with io_uring ENOMEM ("no io_uring worker threads are available: Cannot allocate memory"), and test_torch_index_with_nans[Legacy] hit StopIteration — main's own Python workflow is red with vector-test flakes at the same time. Retriggering via close/reopen.

[u70b3]

The remaining CI failures here are tracked in #8789 — systematic main-level test failures (not caused by this PR). Will retrigger once main's own CI recovers.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The writer resolves shard identity before any claim side effect, and the success-path cases cover manual identity, automatic default resolution, and an explicit matching ID. Future spec revisions and activation semantics remain cleanly out of scope.