[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #628 (part of #624).

Stacked on #394 (base branch feat/backup-format-streaming): SnapshotInfo is #394's SnapshotDescriptor and the orderability test is its KeyOrder::resolve. The diff shows this PR's single commit on top of #394; GitHub retargets it to main when #394 merges.

What

The one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code in #384 keeps its own copy of this logic; the engine-side backup job (next PR) is the first caller.

Trait (crates/storage/src/snapshot.rs):

pub struct TableSnapshot {
    pub description: TableDescription,   // the storage-level description at the snapshot instant (not the API response: restore_summary may be set, table_throughput_mode_summary is not, and the counts are the stats pass's)
    pub info: SnapshotInfo,              // = extenddb_core::backup_format::SnapshotDescriptor { kind, marker }
    pub key_ordered: bool,               // items arrive in base-table key order
    pub items: BoxStream<'static, Result<Item, StorageError>>,  // owns the transaction/session
}
pub trait SnapshotEngine: Send + Sync {
    fn open_table_snapshot(&self, account_id: &str, table_name: &str)
        -> BoxFuture<'_, Result<TableSnapshot, StorageError>>;
}

Dropping the stream releases the backend transaction or session. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. key_schema_is_orderable is KeyOrder::resolve(..).is_ok(), so a stream a backend marks ordered is one the manifest parser accepts, and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. The trait doc lists each backend's errors (Cassandra answers Unsupported before it resolves the table). StorageError::SnapshotExpired is new, for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient. snapshot.rs's module doc gains one sentence (the design overview states the rules the trait's callers honour; this file's and each backend's snapshot_engine.rs module docs are the record of what the snapshots do); DEFAULT_FETCH_BATCH's doc now says why the size is a constant (over a 1 ms link the larger fetch would save about a second per million rows); the PostgreSQL module doc's pre-#361 paragraph carries the sort's cost (7 MB on disk for 100k rows at work_mem 4 MB).

Indexes. The description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since #401 UpdateTable creates the data table first, so this is a guard). A restore recreates the indexes in the description and builds them from the items, so a CREATING index comes back complete; an index left out would not come back at all.

PostgreSQL. On one shared database the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. With a separate data database the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both layouts. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, and an open over the bound is LimitExceeded at once, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key; a fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit), and the byte budget is the backup job PR's. A table created before #361 declared the key columns COLLATE "C" keeps the database collation; ordering it bytewise would sort the whole table before the first row while the lock, connection and permit are held, so such a table is streamed unordered with a warning. Only a collation named "C" or "POSIX" counts as bytewise; "default" on a C database streams unordered, which is complete. SQLSTATE 40001 (a recovery conflict on a standby) and 72000 (snapshot_too_old on a primary, PostgreSQL 16 and earlier) map to SnapshotExpired. The stream holds ACCESS SHARE until its last fetch, so a DROP TABLE waits behind it; DeleteTable takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that). On both layouts the synchronous path takes every drop's lock before the catalog removal commits. It deletes the table's pending queue rows first, waiting for in-flight claims with no timeout, as #401 intends. It then drops the base, GSI, LSI and vector-index data tables, base first, within one 1 s budget for the group (SYNC_DROP_LOCK_TIMEOUT; before each DROP, lock_timeout is the time left). On the shared layout that runs in the catalog transaction behind a savepoint. With a separate data database it runs in a data transaction opened first, and a task the request awaits commits the catalog and then that transaction, so a cancelled caller cannot stop between the two. A lock wait that fails (55P03, a deadlock victim 40P01, or a statement_timeout's 57014) rolls the drops back and takes the asynchronous path with the row still held. A write transaction, a SHARE holder, an index build or a reader that starts after the probe therefore costs the request at most the 1 s, and no data table is left without its row. With a separate data database a data commit that fails after the catalog commit has the drop retried on a fresh connection for up to 60 s, so a failover, a dropped connection or a network blip no longer orphans the data tables; only a process that dies, or a data database unreachable for that whole window, between the catalog commit and the last retry leaves them without a row, and then they stay until an operator drops the relations the error log names (docs/troubleshooting.md, "Data tables without a catalog row"; catalog-check --fix cannot reclaim them on this schema). The base table goes first because a write holds it before it writes an LSI or a synchronous GSI: with the index tables first, a write in progress waited about 1 s on the drop, and the drop went asynchronous. On the shared layout a DeleteTable probing between an open's snapshot and its LOCK does not see the open; if the LOCK is granted first, the drop times out after 1 s, is rolled back with the removal, and DeleteTable returns DELETING (in 80 unaided open-and-delete pairs, the delete first in half of them, 0 left a data table without a row on either layout; 39 to 40 deletes ran synchronously and 40 to 41 asynchronously). The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred, also after a restart: startup recovery runs the same pass and wakes the poller when it deferred a drop. A drop is deferred when the probe finds another session holding any lock on the table's relations, or when the drop fails in a lock wait (55P03, 40P01 or 57014); any other failure is logged and left to the normal window and the 60 s idle pass. Each pass probes a held table and skips it without waiting, and drops a free one within the same 1 s budget, so every held table is retried about once a second however many are held, and a CreateTable that is due is activated on the same cadence (1.1 s with four drops deferred). A drop that DeleteTable deferred is due at once, not after control_plane_delay_seconds. The active loop waits on the wake as well as on its timer, so a snapshot's or an in-catalog backup's release has the drop retried at once, and any holder's release is followed within about a second (the maintainer measured 51.8 s before for a snapshot stream, and round 6 57 to 60 s for the in-catalog backup and a plain reader; now 108 to 166 ms after a snapshot or backup release or a plain reader's, 0.94 to 1.08 s after a release that spans a restart (the reader is released only once the poller is past its window), and 108 to 164 ms for the first of four held tables, in the tests). The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the default (0.25 on every backend), the bound and the retry cadence, the same-name CreateTable refusal until then, and a two-step query, over every relation of the table, that finds any session holding a lock on them and its lock mode. A DeleteTable of a table that is already DELETING returns its description and changes nothing.

SQLite. One deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened. A TRUNCATE checkpoint meanwhile waits out the busy timeout and reports busy = 1; the module doc gives the measured reply. build_table_description_on reads the definition, including #384's restore_summary, on that transaction. The stream holds a slot from #384's backup_server_slots until its read transaction ends (the short page that shows the end, or the first error) or it is dropped (the pool arithmetic that keeps half the pool free counts snapshots); with none free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them; the pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index; a page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it has yielded its last item or is dropped, so the slots bound how many decoded copies exist. This is the shape #384's create_backup_single_connection uses, so CreateBackup keeps working on sqlite-memory. SQLite DeleteTable of a DELETING table (control_plane_delay_seconds of 1 or more) returns its description too.

MongoDB. A session started with snapshot: true reads the tables document, the indexes documents and the base collection at one cluster time, which is the marker. Every read sets readConcern: snapshot explicitly, because the driver only fills it in when neither the operation nor its database carries one, and the production catalog handle carries readConcern: local (as any handle does under a URI with readConcernLevel=); on that handle a read without the level fails: the first reports no snapshot time, which the open refuses naming the read concern, and later ones are refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported; that refusal and the no-snapshot-time refusal each name their own cause. The index read in the session maps 239/246 like the others. Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes; CursorNotFound maps to Internal). key_ordered is false: a composite key's _id is a netstring of the key text, not the format's order.

Cassandra. open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog path stays its backup path until the design settles it (#396 section 3).

Test hooks. A fetch-size knob, a round-trip counter and a pause at two points of the open (AfterDefinition, BeforeItems) live in extenddb_storage::snapshot::paging; every backend reads the fetch size there and calls the pause. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with extenddb-storage's test-hooks feature, which each backend crate's tests enable through a dev-dependency on itself (on MongoDB through a new snapshot-test-hooks, so the crate's existing coordination test-hooks stays off in its unit tests, as on main); production builds carry none of them. PostgreSQL also gains test-hooks entries spawn_control_plane_poller_for_tests and spawn_control_plane_poller_with_active_poll_for_tests (the production poller, the second with its active retry interval as a parameter), recover_control_plane_at_startup_for_tests, control_plane_drop_attempts_for_tests (a per-engine count of drop attempts, a held table the probe skipped counted) and arm_delete_table_pause (holds DeleteTable between its probe and its drops), and re-exports, under test-hooks, the data relation names and apply_all_migrations (the three extenddb migrate runners), so tests/table_snapshot.rs binds the crate's lists instead of copying them. The two pauses (the open's and DeleteTable's) are one generic type, extenddb_storage::test_pause::Pause, and snapshot_conformance::PageSize is the one page-size guard the three backends' tests use. tests/table_snapshot.rs fails instead of skipping without a connection string when EXTENDDB_TEST_PG_REQUIRED is set, which integration.yml sets. The trait doc says why SnapshotEngine has no default body and StorageError is not #[non_exhaustive]: both breaks are meant. devtools/run-mongodb-tests publishes its container on 127.0.0.1 only.

Why

#396 section 3. An earlier draft of the trait returned items only and left the definition to a separate DescribeTable; a GSI added or the table deleted between the two calls produces a manifest that does not match its data files. Returning both from one operation is the only way the backup job can promise the manifest describes the data, and it has to be a backend obligation because the mechanism differs per backend.

Four adversarial review rounds on this PR found, and this revision fixes: the conformance check was inert at the production page size, then depended on the backend test setting the page size, then could not observe whether the knob had taken effect (it now sets the shared knob and asserts from the counter that a second fetch happened); the MongoDB engine inherited the client's read concern and the constant for SnapshotTooOld was SnapshotUnavailable's; an ACTIVE-only index filter dropped a GSI mid-backfill; the PostgreSQL ORDER BY lacked COLLATE "C", so a pre-#361 table streamed in locale order under a key_ordered flag and the design's restore would have refused a valid backup; the PostgreSQL open took two catalog connections and could deadlock a full pool; the MongoDB tests dropped fixed database names, so a URI in the environment could destroy another instance's data; the SQLite path held the writer lock across a pool acquire and bypassed the slot accounting; a missing sort-key attribute definition made the stream page by hash key alone and lose rows; SnapshotInfo and the orderability rule were second copies of #394's definitions. The fourth round: the MongoDB counter echoed the batch size the stream asked for (a cursor without batch_size passed); a synchronous DeleteTable during a stream could orphan the data table; a half-created GSI left an unused attribute definition CreateTable refuses; a shared catalog/data pool could deadlock concurrent opens; the Cassandra test created no table; the harness's key-order oracle was a third statement of the order. An earlier review by the maintainer then found: nothing checked that the definition and the items come from one instant or that the stream read after the writer committed (READ COMMITTED, a missing FOR SHARE, a MongoDB read outside the session, and a SQLite stream that read everything during the open all passed); only one key shape was tested; with a separate data database the bound was the whole pool, and ten snapshots made GetItem fail after 30 s; DeleteTable probed only the base data table, so a reader on a GSI table left the catalog row gone and the drop blocked; the collation clause turned a legacy table's read into a full sort; a MongoDB decode error did not end the stream and a standalone server reached the client as a 500; the SQLite refusal blamed backups when snapshots held the slots; and sqlite-memory lost a working CreateBackup. The fifth round found: on the shared layout DeleteTable took a second connection from the pool it already held one from, and twenty concurrent deletes on a ten-connection pool timed out; the open took two connections from one pool on the shared layout; four release paths (a drained stream, a fetch error, an open that failed or was cancelled) did not wake the poller; the one-instant scenario passed with the pauses outside the read window; the SQLite rowid page was untested and the MongoDB counter counted the GSI backfill's reads; a foreign test's open could take the armed pause; N range keys were not among the key shapes; in-memory SQLite copies were not bounded by the slots. Round 6 found that the one-second DeleteTable bound held only for readers that woke the poller (a snapshot stream), while a plain read, pg_dump or the in-catalog backup left the row DELETING until the 60 s idle pass; the poller now stays active while a drop is deferred, the backup wakes it too, and the conformance check asserts the branch it depends on, compares the description with DescribeTable, and bounds every drain. The maintainer's review of 2026-10-28 found: on one shared database an open's window between its snapshot and its LOCK let a synchronous DeleteTable commit the removal and wait in DROP TABLE with no timeout (6 of 80 unaided races left the data tables without a row); the release wake was inert because the active poller slept on its timer alone; startup recovery discarded the deferred count, so a reader spanning a restart left the row DELETING until the 60 s idle pass; every drop failure kept the poller in its one-second loop; the troubleshooting query matched only the base table; and the conformance check cleared restore_summary. Round 7 found: with a separate data database a DeleteTable that met a write transaction, a SHARE holder, an index build or a queue claim waited with no bound after the catalog commit, and a caller cancelled or timed out in that wait left the data tables without a row; a lock timeout per DROP let a table with n indexes stall its base table n seconds; the poller retried deferred drops one after another at 3 s each, delaying every other transition by 3 s per held table; the troubleshooting query found only readers; DeleteTable of a DELETING table returned ResourceNotFoundException; a deferred drop waited out control_plane_delay_seconds; a deadlock victim or a statement timeout in the drop was an InternalServerError; and SQLite kept a finished or failed stream's backup slot. Round 8 found: with a separate data database a data commit that failed after the catalog commit (a terminated backend, a failover) left the data tables orphaned with the row gone, the client told the delete had failed, and nothing to reclaim them.

Testing done

extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared fetch size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open, then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. It asserts 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, that the counter advanced after the writer committed, key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare when ordered, and that a second snapshot after the writer stops shows the final round. Then, once per pause point and on a fresh table, it holds the open while it commits a write W1, an UpdateTable adding a GSI and a write W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts, the snapshot's description must equal the storage layer's describe_table with the running counts and table_throughput_mode_summary cleared; restore_summary is compared, and each backend's suite also compares a table given a restore record (assert_snapshot_describes_the_table). Every drain is under a 120 s deadline naming the stage and the items read, and a test waiting for the hooks lock gives up when one holder has kept it 600 s, naming the wait. Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3, with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. These mutants fail it: a fresh transaction per page or every page read during the open on SQLite, and >= in its hash-only or binary range keyset; OFFSET/LIMIT in READ COMMITTED, READ COMMITTED with the cursor, and no FOR SHARE on PostgreSQL; the data cursor or the index read outside the session on MongoDB; the pauses moved outside the read window on SQLite or MongoDB; the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion); a description built from a second code path (the DescribeTable comparison); a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. 27 hand mutants were built for round 4; 25 fail on the assertion they target, and the two that remove COLLATE "C" from the PostgreSQL clause pass because, with legacy tables streamed unordered, the clause is equivalent on every table that is ordered (kept as defence in depth and marked so). Round 5: 18 more mutants (DeleteTable's probe on a second pooled connection, the barrier shape on the shared layout, each wake site removed, the 42P01 mapping, the fetch-error path keeping its slot, no LOCK TABLE, the last page keeping its transaction, the barrier released before the pause, the pauses moved outside the read window on SQLite and MongoDB, rowid >=, the in-memory copy dropping its slot, the MongoDB counter counting every find); 17 fail on the assertion they target, and the barrier released between the pause and the pin passes, because no statement sits between them and UpdateTable takes over a second after the barrier goes. Round 6: the poller breaking at its deadline regardless fails the plain-reader test on both layouts (two earlier shapes of that test, an 8 s and a 12 s hold with the poller started before the DeleteTable, did not catch it: the window's last drop attempt was still waiting out its lock timeout, and the DeleteTable's own wake chained a second window); the SQLite pauses moved before the transaction with a 2.5 s UpdateTable fail the branch assertion; a SQLite open blanking the ARN, id and summaries fails the description comparison; a SQLite keyset stream without its end branch fails the drain deadline. For the maintainer's review of 2026-10-28: 9 mutants (the shared-layout drops after the commit; the notify removed from release and from finish; the active loop ignoring the notify; startup not waking the poller; every drop failure counted; the snapshot dropping restore_summary on PostgreSQL, SQLite and MongoDB), each failing on the assertion it targets. Round 7: 17 mutants (the separate layout's commit-then-drop order; a lock timeout on the pending-row delete; a timeout per DROP; index tables dropped first; the poller's probe skipped; activation after the drops; the DELETING filter on PostgreSQL and on SQLite; the deferred row scheduled after the delay; 40P01 or 57014 left out of the classifier; the poller idle at its window's end; DeleteTable's probe never finding a reader; the SQLite slot kept on the short page, on the error and in the in-memory copy; description_diff keeping commas). 16 fail on the assertion they target. Activation after the drops passes, since with the probe the drops no longer take time. Round 8: 2 mutants (the failed data commit returned instead of retried; the retry run inline instead of in the awaited task), each failing on the assertion it targets (the relations stay and the request errors; a cancelled caller strands the data tables).

Passes locally on PostgreSQL 16 (both pool layouts), SQLite (the shared check on a file-backed database; the in-memory copy has its own tests, since a stream that reads every page during the open fails the check's fetch-after-commit assertion by design), a MongoDB 7 replica set (devtools/run-mongodb-tests --mongo-only) and a standalone mongod, and Cassandra 4.1:

• PostgreSQL: crates/storage-postgres/tests/table_snapshot.rs, 81 tests (and two ignored race measurements): 36 run on both layouts (on_both_layouts!): consistency; marker and key order; legacy-collation tables streaming unordered and complete on a scratch database checked not to be C-collated, beside an ordered one; DeleteTable with only a GSI or a vector-index data table locked taking the asynchronous path; an unparseable row ending the stream and freeing its permit; a GSI without its data table left out with its attribute; DeleteTable during a snapshot returning DELETING; twenty concurrent DeleteTables on a ten-connection pool; an open held while nine DeleteTables queue; an open completing with nine of ten connections held; the production poller at a 10 s active poll woken during its pause, and the table gone within 3 s, by a dropped stream, a drained stream, a fetch error, an open cancelled at BeforeItems and the in-catalog backup; a reader across a restart; a drop failing on a dependent view kept to the normal window; a restored table's restore_summary; a plain reader holding ACCESS SHARE for 10 s past the poller's window, sending no wake, with the table gone within 3 s of its release; a fetch error freeing the permit; DeleteTable at BeforeItems taking the asynchronous path; a drained stream out of its transaction before its end is polled; a ROW EXCLUSIVE and a SHARE holder sending DeleteTable down the asynchronous path within about 1 s with no data table left without its row; a DeleteTable cancelled during that wait leaving the table ACTIVE with its data; write holders on the base and three GSI tables released 0.8 s apart taking it there at about 1 s; a write holding the base and then writing the GSI table finishing first, with the table removed in the request; a claimed queue row holding DeleteTable until it ends; a probed reader sending it asynchronous in milliseconds; a statement timeout and a deadlock victim taking the asynchronous path; DeleteTable of a DELETING table returning its description and changing nothing; a deferred drop due at once; four held tables probed with no lock wait, the first released gone within 2 s and a CreateTable ACTIVE within 2 s. Plus five held snapshots on a ten-connection pool on each layout, with GetItem and DescribeTable unblocked, the sixth LimitExceeded and an open succeeding after one is dropped; the SQLSTATE mapping raised on the live server; and, on the shared layout, a snapshot open or a GSI-only reader locking after DeleteTable's probe sending it down the asynchronous path, and a table dropped before the open's LOCK being TableNotFound; and, on the separate layout, a data commit that fails after the catalog commit (its backend terminated at a pause between the two) retried until the drop goes through, with the request returning DELETING, the row gone and no relation left; a data database refusing connections past a bound shortened to 500 ms returning Internal naming every leaked relation, with the row gone; and a caller cancelled between the two commits, with the detached task completing the drop and no orphan at any step. The two ignored race measurements delay the delete or the open in alternate pairs and assert both outcomes occurred. Every DeleteTable runs under a caller timeout (300 ms in the cancellation test). Added to the direct-storage step in integration.yml. Scratch databases are dropped by a Drop guard on success or panic. The full storage-postgres suite (259 tests), fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401's GSI build tests included, passes.
• SQLite: 11 in-module tests (consistency; key order; the in-memory copy releasing its connection; in-memory copies holding their backup slot; a damaged attribute definition streaming unordered and complete at a page of seven with no duplicates and at least six fetches; slots shared with CreateBackup and its refusal text; the WAL checkpoint reply while a snapshot is held; a restored table compared on a file-backed and an in-memory engine; a drained stream and a stream that hit an error freeing their slot while still held; an in-memory copy freeing its slot with its last item), run by cargo test --workspace; scratch files under the test binary's target directory, removed by a Drop guard.
• MongoDB: 11 in-module tests: live tests on per-test scratch databases (snap<uuid>_cat/_data, built the way fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's live tests build theirs with the production catalog options, with the GSI backfill worker running and the command monitor feeding the counter, dropped by a guard on success or panic), gated on EXTENDDB_TEST_MONGODB_URI and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, holding the snapshot hooks lock (consistency; a restored table's restore fields compared; readConcernLevel=majority in the URI; an undecodable document ending the stream; reads without the snapshot level failing with code 72, with the error text checked; the round-trip counter counting only the snapshot stream, so a GSI backfill leaves it unchanged; a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod it starts, and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it); and unit tests pinning the error-code mapping from reply documents. A new integration-mongodb.yml job (in the fan-in) starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod, both on 127.0.0.1 only, and runs them (storage-postgres: UpdateTable awaits the GSI backfill inside the request, so large tables time out at the client #654).
• Cassandra: a direct test asserting Unsupported naming the backend on an existing table and a missing one.
• Storage: unit tests for drop_deleting_indexes (GSIs and vector indexes).
• Engine: SnapshotExpired maps to ServiceUnavailable (storage_error_mapping_tests).
• Wire: tests/test_table_operations.py's DeleteTable of a DELETING table returning its description, against PostgreSQL and SQLite servers (the state written into the catalog); it skips on MongoDB and Cassandra, which never leave a table DELETING.

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy ... -- -W clippy::pedantic                                             # 0 findings on the lines this PR adds and in the files it adds (storage crates and extenddb-engine)
cargo check --workspace --all-targets                                                # with test-hooks via dev-deps
cargo check -p extenddb-storage -p extenddb-storage-postgres -p extenddb-storage-sqlite -p extenddb-storage-mongodb -p extenddb-storage-cassandra --lib   # without
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo test -p extenddb-storage -p extenddb-engine -p extenddb-server                 # 132, 179, 202 passed
cargo test -p extenddb-storage-sqlite --lib snapshot_engine                          # 11 passed
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres --test table_snapshot   # 81 passed, 2 ignored
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres                         # 259 passed
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 EXTENDDB_TEST_MONGODB_STANDALONE_URI=... cargo test -p extenddb-storage-mongodb --lib snapshot_engine   # 11 passed
EXTENDDB_TEST_CASSANDRA=required cargo test -p extenddb-storage-cassandra --test direct_integration snapshot_engine   # 1 passed

Not changed here

• The design's max_concurrent_jobs is the caller-side bound, in the catalog PR; this PR's per-server bounds (SQLite's slot, PostgreSQL's permit) only keep snapshots from exhausting a pool.
• The in-catalog backup path in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 does not call this trait.
• The PostgreSQL and SQLite fetches are bounded by rows, not bytes; the byte budget is the backup job PR's (design section 7).
• DeleteTable's returned description omits VectorIndexes (both the ACTIVE and the new DELETING branch reuse build_table_description_from_row, which never populated them); this predates the PR (present on main) and VectorIndexes is an ExtendDB extension absent from DynamoDB, so no client answer disagrees with the service (storage-postgres: DeleteTable's response omits VectorIndexes that DescribeTable reports #656).
• The drops take the base table before the index tables, not after: a write holds the base before it writes an LSI or a synchronous GSI, and the other order made such a write wait about 1 s on the drop.
• Four issues round 7 found outside this diff are present on main (cf321f0) and out of scope here:
  • extenddb catalog-check --fix expects data relations named _ddb_{account}_{table} and _ddb_{account}_{table}__gsi__{index}, while main names them _ddb_{table_id} and _ddb_{index_id}, so it reports every live data table as orphaned and --fix drops it; it does not see _ddb_vec_ tables either.
  • On the shared layout the catalog migration's stream_shards.table_id references tables(table_id), and CreateTable inserts the shards on its own connection before its catalog transaction commits, so a CreateTable with a stream specification fails the foreign key with an InternalServerError (extenddb init does not produce that layout).
  • UpdateTable that creates a GSI awaits the backfill inside the request, so on a large table the client's read timeout expires first (boto3's 60 s against 20,000 items); the build itself continues server-side.
  • Query and Scan with IndexName resolve the table and then the index in separate statements, so a DeleteTable committing between them yields ValidationException ("The table does not have the specified index") instead of ResourceNotFoundException.
• Attribute pruning for a left-out GSI is not shared between the snapshot and the in-catalog backup (they prune different types); backup_engine::abort_restore builds its own drop list in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's code.
• cargo clippy on the local 1.98 toolchain did not fire the chunks_exact lint in storage-sqlite/src/vector_search.rs seen in an earlier round; CI runs stable and 1.88.0 (test.yml).
• docs/design/04-component-storage.md (DeleteTable described as always asynchronous) and docs/differences-from-dynamodb.md (a 5 s default delay) are pre-existing inaccuracies not edited here.
• The CodeQL alert on this PR (cleartext-storage-database, storage-sqlite/src/backup.rs) is in a fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 test on main that this PR does not touch.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic) for the files this PR adds
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, section 3. Storage trait: SnapshotEngine is added to the StorageEngine supertrait bound and implemented by all four in-tree backends; StorageError gains SnapshotExpired.

Breaking changes

• Out-of-tree storage backends must implement SnapshotEngine and handle the new StorageError::SnapshotExpired variant in any exhaustive match.
• PostgreSQL DeleteTable returns DELETING and finishes in the background, instead of synchronously, when another session holds a lock on the table's data or index tables that its drops cannot get within about 1 s, on both layouts: at once for a reader (a table snapshot, the in-catalog backup, a Scan or Query, pg_dump, a plain read), after the 1 s for an open write transaction, an index build or VACUUM, and also when the drop is a deadlock victim or meets a statement_timeout. The table is gone within about a second of that session's release (at once when a table snapshot or in-catalog backup ends; also when the server restarted in between). Until then DescribeTable reports DELETING, ListTables lists it, a CreateTable of the same name is refused with ResourceInUseException, and a DeleteTable of it returns its description, which on PostgreSQL and SQLite used to be ResourceNotFoundException. The response shape is unchanged. No wire, CLI or on-disk change.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.