[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework · this PR: the catalog index · next: configuration and the store identity file (PR 7), the CreateBackup job and worker (PR 8), the restore job (PR 9a).
Closes #669 (part of #624).

Base: main at 94b5f68 (which carries #725, #798, #806, #776, #764, #535, #513 and #773 over e8ca7df; the move onto it conflicted in one file, crates/storage-cassandra/tests/direct/backup_engine.rs, where #806 rewrote test_restore_heartbeat_runs_during_target_ddl to check order, the number of target tables that exist at the first heartbeat, in place of wall-clock time: the merged test keeps #806's order check and this PR's WRITETIME(restore_heartbeat_at) reading of the beat). Main at e8ca7df carries #666's migration runner, #681 (Cassandra and MongoDB on that runner) and #335 (PITR answered as unsupported, migration 5). This PR adds migration 6, backup_catalog_index, to the MigrationId registry, with a body on every backend that the runner applies (Kind::Breaking, convergent). The rebase onto main moved the migration from id 5 to 6, turned Cassandra's V006 file and MongoDB's versioned step into migration 6 code bodies on #681's runner, and dropped the 0.0.5 catalog version, this PR's own serve-time version checks (the runner's status check refuses the catalog on every backend) and the PITR methods #335 removed. The move from 03baa32 to e8ca7df (#463, #433, #782, #785, #786, #416, #791, #457, #424, #509, #718, #734) changed one thing in the code: #416 replaced ServerComponentsOptions' derived Default with a hand-written one, which now sets this PR's max_concurrent_jobs to 0; the differences document's backup rows keep #509's text beside this PR's. Where this head meets what main brought in: StorageEngine requires BackupCatalog + LegacyBackupReader + SnapshotEngine (#395); SQLite's backup_server_slot_count slots are shared by backups and #395's table snapshots, with the formula stated once on SqliteConfig::pool_size and one refusal text; the PostgreSQL control-plane pass keeps #395's drop_unless_held and loses only sweep_abandoned_restores; MongoDB's factory registers and checks the catalog before it builds the engine or starts #473's startup key repair, so a server that refuses has written nothing.

What

The catalog index of design section 4 on every backend, the engine's Describe, List and Delete over it, the reconciler, and the legacy in-catalog paths kept beside it as a frozen reader. No request or response shape changes; the one visible API change is that ListBackups accepts a table ARN in TableName, as CreateBackup already does.

Traits (crates/storage/src/backup_catalog.rs). BackupEngine is replaced by BackupCatalog and LegacyBackupReader, both supertraits of StorageEngine, implemented by all four backends. BackupCatalog addresses every backup row by BackupKey (account id, backup id, ARN) and every restore target by RestoreTargetKey (account id, table name, table id); no method takes a bare id, and read_backup_row returns a row only when its account, table name and ARN equal the request's. Its methods: catalog_id, claim_backup (the row is inserted CREATING under a fresh owner_generation; a second claim is AlreadyExists), list_backup_rows (by store identity and status) and list_account_backup_rows (by account, table and time bounds), both ordered (created_at, backup_arn), read_backup_row, start_backup_delete (the in-use check and the AVAILABLE-to-DELETING write in one transaction under the row's lock; Deleting, InUse(table) or Status), set_backup_deleted, remove_backup_row (DELETED or FAILED only), backup_in_use, remove_restore_target (conditional on a CREATING restore target with this table id, so it and the ACTIVE flip exclude each other), list_dead_restore_jobs(stale_after) (RUNNING records whose heartbeat is older than the window on the database's clock, and FAILED records whose table is still a CREATING restore target, or on Cassandra a DELETING one with its registration, once the record is that old), fail_restore_job(key, owner_generation, heartbeat_at, reason) (RUNNING to FAILED, conditioned on RUNNING and on the record still carrying the owner_generation and heartbeat_at the listing read, written with a fresh generation, so a heartbeat that landed after the listing makes it match nothing: design 4(e)), check_maintenance_claim. JobClaim is a job's handle on its row: generation, heartbeat, set_status(from, to, fields) conditioned on the generation and from (Matched, NotMatched, or Unknown when the connection failed and the write may have landed), read. LegacyBackupReader is #384's create, describe, list, delete, restore-source and restore, frozen apart from the restore's registration, which writes the job fields; create and restore take the job's JobPermit and hold it for the job's life, and abandoned_creating_backups(stale_after) reports a CREATING row whose create died. BackupRow derives BackupDescription and BackupSummary from source_table_details_json (the manifest's source_table; the summary reads only the table id and ARN, so a later build's required member does not break a listing); BackupStatus reports DELETING as DELETED and hides DELETED and FAILED; SourceTableFeatureDetails is always present, {} when the source recorded no index, stream, TTL or SSE, as the service returns it. The in-use text ("Backup is being used to restore another table: ", the service's, with no table name; the name goes to a log line), the not-found signal and the LimitExceededException text (limit_exceeded_message(JobBound), naming the bound met: [backup] max_concurrent_jobs, the [storage.sqlite] pool_size snapshot-slot bound, or the PostgreSQL server's connection slots) are one implementation every backend and the engine bind, as is JOB_HEARTBEAT_INTERVAL (15 s), the period every backend's jobs heartbeat at. StorageError::AlreadyExists is new.

Config (crates/config/src/backup.rs). [backup] with data_file_target_bytes (64 MiB, minimum 32 MiB), heartbeat_interval_secs (15), stale_job_after_secs (300, at least three times the period the jobs heartbeat at, max(heartbeat_interval_secs, 15): 45 at the least), failed_row_retention_secs (7 days), reconcile_interval_secs (60), max_concurrent_jobs (4, at most 1024), spool_path; deny_unknown_fields, every key optional, validated at load. init writes the commented block; extenddb.sample.toml and the admin guide carry it. ServerComponentsOptions.max_concurrent_jobs lets the PostgreSQL factory raise pool_size and catalog_pool_size to 2 * max_concurrent_jobs + 10 with a warning naming both numbers.

Engine (crates/engine/src/backup.rs, backup/runtime.rs, backup/reconciler.rs). BackupRuntime holds the job slots (JobPermit::acquire_within: a request that finds every slot held waits up to PERMIT_WAIT, 30 s, for one, the wait main's PostgreSQL and MongoDB restores had, and is then LimitExceededException; try_acquire stays for callers that must not wait; the service accepted five restores and six creates sent together) and the configured store, which is None on this head, so every backup_index row is a row of another store: Delete and Restore of one return the ValidationException that names no root. Describe, List and Delete read the catalog index first and fall back to the reader; List merges list_account_backup_rows (the time range as bounds, is_listed statuses) with the reader's list under the existing paging, filtering and cursor exceptions. CreateBackup runs the design's step 1 order: the handler's ACTIVE probe, check_backup_arn_fits (the region alone, refusing with a message naming server.region when it is outside [a-z0-9-]; a name over 255 bytes or made only of dots is backed up as on main, in-catalog, where an earlier head of this PR refused it: nothing the in-catalog create writes needs a store prefix, and the service backs such a table up, so the store-prefix name rules are the store-backed writer's, PR 8), the vector refusal, the permit, the deprecation warning, then LegacyBackupReader::create(permit); restore resolves the backup, then checks the target name through describe_table (a table of the name TableAlreadyExistsException, a CREATING restore target TableInUseException naming its source backup), before the overrides are checked against the source, the order the service uses (a restore of a missing backup into an existing table, a table CreateTable is still creating or a CREATING restore target is BackupNotFoundException, measured on DynamoDB twice each; an earlier head checked the name first), then takes the permit last and passes it in; ListBackups and CreateBackup refuse a TableName ARN whose region is not server.region (compared as written) with the service's ValidationException "1 validation error detected: Invalid AWS region in ''" (an earlier head resolved it to the server region's table); the backend's fenced name check under the registration stays. Both spellings of a backup ARN (as written and re-spelled in this region) parse into BackupArn where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing; DeleteStart::Status(AVAILABLE) is an internal error. The reconciler runs a cycle at startup and every reconcile_interval_secs, spawned by serve beside the backend workers and drained with them: the maintenance-claim check, each dead legacy restore failed (fail_restore_job) and its target removed (remove_restore_target; a target the removal finds no longer CREATING is left to its owner and reported as such), the target of a FAILED record whose removal did not land removed once the record is stale, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), a dead manifest restore logged once per record and left to PR 9a (the set of logged records lives in serve, outside the worker set, so a worker restart after the serving registration is taken again does not log the same record again), a record whose row this build cannot read (a kind or status no release writes) logged and skipped by every backend's listing rather than failing the cycle, each abandoned in-catalog CREATING backup deleted through the reader; it records the worker metrics (WorkerLastSuccess, WorkerCycleLatency, WorkerErrorCount under BackupReconciler). It replaces the four backends' sweeps and their slot semaphores.

PostgreSQL. Migration 6 (migrations/0006_backup_catalog_index.sql): backup_index (primary key (account_id, backup_id), backup_id unique, backup_arn COLLATE "C" so the listings' (created_at, backup_arn) order is byte order whatever the database's collation, account/table and store/status indexes), twelve columns on table_restores (status, kind, store_kind, store_root, store_id, owner_generation, heartbeat_at, spool_host, file_index, items_applied, failure_reason, completed_at) with the backfill of the registrations that exist (kind = legacy; DONE with completed_at = creation_date_time, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING, then NOT NULL on the four claim columns), backup_maintenance, and settings.catalog_id written once. The claim is pg_try_advisory_lock(hashtextextended('backup:' || backup_id, RESTORE_LOCK_SEED)) on a dedicated, non-pooled connection (ClaimSession) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation), every write WHERE owner_generation = $gen; the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends; fail_restore_job writes WHERE status = 'RUNNING' AND owner_generation = $listed AND heartbeat_at = $listed. The legacy restore registers RUNNING, kind = legacy, owner_generation, heartbeat_at = now() in the transaction that creates the target, marks DONE in the transaction that flips it ACTIVE, and on failure marks FAILED (conditioned on RUNNING and the generation) before abort_restore, removing the target only when that matched. The per-account and per-server restore slots, sweep_abandoned_restores and the control-plane pass's call are removed; RESTORE_LOCK_SEED and abort_restore are pub and #[doc(hidden)] so the tests bind them. docs/schema/postgres.txt is regenerated from a fresh init.

SQLite. Migration 6 as the code body m0006_backup_catalog_index.rs (fingerprint 0006_backup_catalog_index/sqlite/1, golden fixture blessed): the same tables and indexes, column-checked ADD COLUMNs (nullable; ADD COLUMN on a populated table cannot add NOT NULL without a default, and the application writes every one), the backfill, the catalog id. One process serves a file (the serve lock), so the claim is an in-process slot (JobSlots, a set behind a mutex with a drop guard) and a restart is proof of death: expire_restore_job_heartbeats ages every RUNNING record to the epoch at startup so the reconciler's first cycle removes an interrupted restore, and abandoned_creating_backups removes, in one write under the engine write lock (BEGIN IMMEDIATE), every CREATING backups row whose slot no task holds, items and definition, with no DELETED tombstone (a backup that never finished was never visible as one), ignoring stale_after (there is no heartbeat column on backups), and reports what it removed; an earlier head read the rows and filtered them after the read, so a create that committed AVAILABLE and freed its slot between the read and the filter lost its backup through the reader's delete, which the window test now holds open. A legacy restore holds a JobSlots slot (restore_slot_key(table_id)) for its life, and the dead listing skips a RUNNING record whose slot a task of this process holds, whatever its heartbeat says: the heartbeat waits for the write lock behind every writer (a GSI backfill), so under the slot rule of design 4(e) the slot is the liveness test and the heartbeat decides only once the slot is free (a restart frees every slot). The restore's registration carries the job fields in the same write transaction as the CREATING target; the copy marks DONE with the ACTIVE flip, FAILED before removal, and heartbeats every JOB_HEARTBEAT_INTERVAL (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock (SqliteJobClaim::heartbeat's doc). WriteOutcome::Unknown is never produced. The two startup sweeps are removed; backup_server_slot_count stays (PR 8 keeps it as the backend's admission bound and raises the pool for it): its slots are shared by backups and table snapshots (#395), and the legacy create tries a slot without waiting, as a snapshot open does, answering the one LimitExceededException text (limit_exceeded_message), naming the [storage.sqlite] pool_size snapshot-slot bound, for both. docs/schema/sqlite.txt regenerated.

MongoDB. The backup_index collection (top-level account_id/backup_id under a unique compound index, backup_id unique, account/table/created and store/status indexes), backup_maintenance, and a restore_job subdocument (status, kind, owner_generation, heartbeat_at, failure_reason, completed_at) on a restore target's tables document beside the provenance fields DescribeTable reads; migration 6, a code body on the numbered runner (src/migrations/m0006_backup_catalog_index.rs, fingerprint 0006_backup_catalog_index/mongodb/1), creates them, backfills the registrations (DONE or RUNNING with the epoch heartbeat as on PostgreSQL, restore_operation_id and restore_heartbeat_at removed) and writes catalog_id; every request it sends is safe to send again (a collection that exists accepted by its NamespaceExists code, the backfill limited to registrations without a job record, the id an upsert with $setOnInsert), so a migrator stopped part way runs it again. The registration writes its first heartbeat as $$NOW inside the registration's transaction (an earlier head wrote the process's clock, so a host behind the database could be listed dead by another server within the window). The claim is the row: an update_one upsert filtered on owner_generation: {$exists: false} with heartbeat_at: "$$NOW", so a second claim trips the unique index. The lock for DeleteBackup and a restore's registration is a session transaction that $incs a lock_seq field on the backup's document while it is AVAILABLE, so the two conflict and order wholly; this replaces the restore_in_progress/delete_in_progress claim markers and the sweep that cleared stale ones. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at local, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after, and describe/list hide DELETING as they hide DELETED; a retry of DeleteBackup on a DELETING backup finishes the drop. The restore's hand-over to the GSI backfill worker and the record's DONE are one update_one conditioned on the generation. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves (CREATING, holding the backup in use and the name until the copy ends, found through DescribeTable) is stated in the differences document's restore row. WriteOutcome::Unknown is the driver's Io and ConnectionPoolCleared. The factory's order is main's (#681): it registers and reads the catalog's status (register_and_check) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test on scratch databases plants a document stored without its key under a catalog at migration 5, shows register_and_check refusing it as behind (migration 6 pending, no registration left) with the document untouched, and shows a repair started once the migrated catalog is accepted rewriting it. The restore slots and sweep_stale_backup_claims, at startup and in the GSI backfill worker, are removed. backup_protocol_model.rs is rewritten for the new steps (registration, copy, hand-over, decide, drop, mark, stop, resume, the reconciler's removal).

Cassandra. Migration 6, a code body on the numbered runner (src/migrations/m0006_backup_catalog_index.rs, fingerprint 0006_backup_catalog_index/cassandra/1), which executes migrations/catalog/0006_backup_catalog_index.cql as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement) and then backfills; a run that fails part way leaves migration 6 unrecorded, serve keeps refusing the catalog, and the next migrate --yes runs the body again. The file: backups_by_arn gains backup_id, store_kind, store_root, store_id, owner_generation, heartbeat_at, completed_at, failure_reason, source_table_details_json; backup_index_by_status (PRIMARY KEY ((store_root, backup_status), backup_arn), with account_id, store_kind, store_id so a listing can re-read the row and scope to the store); tables gains the twelve restore_* job columns; backup_maintenance (every statement an ALTER TABLE ... ADD IF NOT EXISTS or a CREATE TABLE IF NOT EXISTS, which the runner qualifies with the keyspace); the body's backfill (an LWT IF table_id = ? AND restore_status = null per registration, so a second run changes nothing) writes the job fields, and the catalog id is inserted IF NOT EXISTS. Code bodies get the lease check and the file execution on CassandraStores (check, progress, run_file, which moves there from the runner). An index row is a backups_by_arn row with store_kind set; the legacy reader treats such a row as not found and the catalog ignores rows without it. Every claim is a lightweight transaction IF owner_generation = ?; backups_by_arn is written first and the list and status entries follow, every reader re-reading the authoritative row. The legacy restore's heartbeat is IF table_id = ? AND table_status = 'CREATING' AND restore_status = 'RUNNING', so a FAILED write, the reconciler's or the job's own, stops the copy at its next heartbeat as on the other three backends (an earlier head's heartbeat did not require RUNNING, and a copy went on after a FAILED write). The legacy create holds a claim on its CREATING row, heartbeats it and polls the claim once per 64-item bucket, so a copy that lost its row writes at most one bucket after the loss; abandoned_creating_backups fences a stale CREATING row to DELETING under a fresh owner_generation before reporting it, and reports stale DELETING rows (scanned first, so a row it just fenced is reported once). A job stops after three heartbeat errors in a row (a beat that hangs is not counted; the per-bucket poll of the claim bounds the copy then), as after a heartbeat that does not apply, and touches neither its row nor its data again; one error does not stop it. WriteOutcome::Unknown is the driver's Io and Timeout and the server's WriteTimeout and WriteFailure. An orphan backup_index_by_status entry left by a crash between the row's LWT and the batch is skipped by every reader and costs a wasted read until the row is removed under its current status (remove_backup_row deletes the entry under the status it read); the sweep that removes orphans is backup sync's (PR 10). poll_abandoned_restores, sweep_abandoned_restores and the worker are removed. The crate gains a test-hooks feature (as MongoDB's), compiled only for the direct_integration binary, which CI runs with --features test-hooks: a per-account heartbeat period and heartbeat-error injection, and a pause between a delete's in-use read and its claim.

Migration. One change, one number on every backend: migration 6, backup_catalog_index, recorded in each store's ledger by the runner (migration 5 is #335's drop_continuous_backups). The design's PR 6 section and section 4 state the number (revision log items 26 and 27).

Docs. The upgrade manual (the SQLite startup sentence: the server ages every RUNNING restore record at startup and the reconciler's first cycle fails an in-catalog restore and removes its target, the cycle running beside the listener; migration 6: its objects on each backend, the stop-first rule, the first reconciler pass, and on Cassandra and MongoDB the rerun of a body that failed part way; its rollback: the pre-upgrade backup restored, the previous binary refusing the migrated catalog with exit 4, and the [backup] section removed from the configuration, which the previous binary refuses as an unknown field; the 0.0.4 section's 15 s sweep line now attributed to the 0.1.x server), the admin guide (the [backup] table; the job bound in place of the per-account and half-pool restore slots; the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain + max_concurrent_jobs), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document (the ListBackups paging row, the DeleteBackup-during-restore row, the interrupted-restore row, the two-servers row, the RestoreTableFromBackup row's MongoDB registration bound, the concurrent-restores row's read-concern window and its restore-during-delete answer (BackupNotFoundException on every backend), the restore-order row (the name checked before the job permit), the ListBackups cursor row under an invalid region, the BackupArn and backup creation rows: CreateBackup refused under a server.region outside [a-z0-9-] and for a name no backup ARN can carry), the admin guide's region row, the limits document's BackupName/TableName row, the upgrade manual's server.region paragraph, CONTRIBUTING.md's migration example renumbered to 7, the differences document's restore-order, ListBackups, CreateBackup and MongoDB two-servers rows and the limits document's TableName row for this round's changes (the name after the backup; the region refusal; the 30 s slot wait; the dot-only name backed up; $$NOW), the admin guide's max_concurrent_jobs row and the deployment guide's bound sentence, design revision item 28, the sample config, docs/schema/postgres.txt regenerated for the collation and sqlite.txt, cassandra.txt and mongodb.txt for the new objects, the backup/restore design's migration numbers, its PR 6 interface block (now with list_dead_restore_jobs, fail_restore_job and LegacyBackupReader::abandoned_creating_backups) and revision log items 26 and 27.

Why

#396 section 4 and the PR 6 section. The engine-owned backup path (PR 8) needs a row per backup it can list without the store, a claim that excludes a second owner and lets a successor detect a dead one, and one place every backend's rules meet; the legacy in-catalog paths of #384 stay reachable so backups taken before the series restore, but as a reader whose only unfrozen part is the restore's registration through the new job record. Replacing the four backends' sweeps and slot semaphores with one reconciler and one max_concurrent_jobs bound is what makes the next PRs backend-agnostic. The migration lands first, in its own PR, because every later PR writes the rows this one creates.

Decisions taken while integrating the per-backend work, each stated in the module doc of the backend it concerns: JobPermit travels as a trait-method argument (keeps RestoreRequest Clone); the migration is one id on every backend, so one number names one change; SQLite's abandoned_creating_backups ignores stale_after and its dead-create delete leaves no tombstone; MongoDB's lock replaces the claim markers and DELETING precedes the drop; the WriteOutcome::Unknown classes differ per driver; the 15 s heartbeat is one constant, extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL, that every backend's heartbeat loop reads and the config floor is computed from, until PR 8 wires heartbeat_interval_secs; a Cassandra job stops after three heartbeat errors in a row (a beat that hangs is not counted; the per-bucket poll of the claim bounds the copy then), as after a heartbeat that does not apply, and touches neither its row nor its data again, one error not stopping it (design PR 6 section); the in-catalog create on Cassandra polls its claim once per 64-item bucket; Cassandra's orphan status entries wait for PR 10's sweep; the [backup] doc says why the stale floor is a floor on SQLite.

Testing done

Third review round (one inline Minor; design 4(g) from the review body): a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_its_in_use_rule now runs its row-first case over a CREATING row as well as a DELETED one: a CREATING backup restored into an existing table answers BackupInUseException ("Backup is being created: ") before the name is read, so moving the CREATING check after the name check fails the cell (it passed before). Design 4(g) now says what the code does for a row of another store: Describe answers it by its lifecycle state as it answers any row, with no warning, and Delete and Restore check the row's own state first (CREATING BackupInUseException, DELETED or FAILED BackupNotFoundException, and for Restore a DELETING row too), then refuse what is left, an AVAILABLE row and for Delete a DELETING one, with the other-store ValidationException; the code is unchanged. Engine backup:: 106 passed on this head, clippy 1.97 clean.

Second review round (two inline items): on the catalog-row path of a restore (the rows #700 writes), the target name is now checked once the row is known to exist and AVAILABLE (check_catalog_row_status), before the backup's in-use rule and the store checks (check_catalog_row_for_restore), as on the legacy path: an in-use backup restored into an existing table answers TableAlreadyExistsException, as the service does. a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_its_in_use_rule covers the existing name, the CREATING restore target, the free name reaching the in-use rule and then the store refusal, and a DELETED row answering before the name is read. create_backup_table_name checks the ARN's account before its region, so another account's ARN in another region is AccessDeniedException on CreateBackup and ListBackups (another_accounts_arn_in_another_region_is_access_denied_before_the_region_check). Engine backup:: 106 passed on this head (259 engine tests listed), clippy 1.97 clean. The MongoDB live_ count below is corrected to 48, what --list gives on this head.

Counts are on this tree; added and removed count test functions against main. After the move onto e8ca7df the counts are re-measured: the workspace run (PostgreSQL and SQLite) on this tree, the MongoDB and Cassandra counts listed on this tree (--list) with their suites run on the top of the stack (#740); the other bullets are from 03baa32.

• extenddb-storage: 215 (13 added; 216 beside a crate that enables its test-hooks feature, as cargo test --workspace does: the LimitExceededException texts naming their bound, BackupStatus, BackupKey::from_arn, TimeBounds, the description and summary derived from the manifest fixture crates/core/testdata/extenddb-manifest/v1.json, a summary needing only the source table id and ARN, pool_size_for_jobs).
• extenddb-config: 14 (6 added: defaults, the stale floor as three times the period the jobs heartbeat at, the data-file floor, the max_concurrent_jobs upper bound, unknown keys refused).
• extenddb-engine: 255 (25 added: the target name checked before the backup and the permit, the region refusal naming server.region, a removal that finds the target moved left to its owner, Describe, List and Delete over the catalog with the legacy fallback, Delete and Restore of a row of another store, Delete of a configured store's row, the in-use check before the other-store check, an unreadable source table left out of the listing and stopping a delete before the write, the permit taken last for restore and create, the ARN-fits rule, a name the store prefix refuses resolved through the legacy reader, TableName as an ARN, the time range reaching the catalog as bounds, the reconciler's cycle, FAILED-target duty, once-per-record manifest log, worker metrics and worker with the Fake implementing both traits).
• extenddb-storage-postgres (PostgreSQL 16 at 127.0.0.1:55432, EXTENDDB_TEST_PG_REQUIRED=1): lib 75; tests/backup_restore 37 (27 kept with renames, abandoned_restore_is_removed_only_when_its_heartbeat_is_stale rewritten over the reconciler's methods, 2 slot tests removed, 9 added: a copy whose record the reconciler failed stopping at its next heartbeat; a claim excludes a second owner, fences its writes and holds the advisory lock while it lives; start_backup_delete's status, in-use and tombstone outcomes; start_backup_delete waiting for the row lock and then seeing the registration; listings scoped and ordered; migration 6's backfill plus the first reconciler pass; a legacy restore registers RUNNING and finishes DONE; a FAILED record whose target removal did not land listed once stale and removed; a target removed only while CREATING and only under its full key); doc_version_literals 3, migrations_fresh 6, migrations_released 12, migrations_golden 7, migrations_crash 9, migrations_guards 7, migrations_runs 7, migrations_owners 3, migrations_serving 5, migrations_tag_blobs 1, vector_control_plane 42, gsi_online_build 12, gsi_stale_writer 1, key_collation 3, migrate_data_db_name 3, put_create_race 9, stream_records_page_order 1, stream_shard_ownership 6, stream_shard_retention 1, twi_conflict 8, table_snapshot 81 (2 ignored).
• extenddb-storage-sqlite: 248 (3 rewritten from the startup sweeps to the reconciler's methods and expire_restore_job_heartbeats, 1 slot test now refusing without the wait; 13 added, migration 6's golden test among them: a target removed only while CREATING and only under its full key (the right table id under another account or name removes nothing), the PostgreSQL catalog-index module ported, a dead CREATING backup reported and removed through delete, a restart aging an interrupted restore, a legacy restore's record RUNNING mid-copy then DONE, a failed restore marking FAILED before removal, a create refused at once when no snapshot slot is free, a FAILED record whose target removal did not land listed once stale, slots and generations; 1 ignored benchmark).
• extenddb-storage-mongodb (replica set at mongodb://127.0.0.1:27117/?directConnection=true, EXTENDDB_TEST_MONGODB_REQUIRED=1): lib 165 with feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395's a_standalone_server_is_unsupported skipped (it needs the standalone mongod that CI's workflow names in EXTENDDB_TEST_MONGODB_STANDALONE_URI); -- --ignored --test-threads=1 live_ 48 (backup, stream and fix(mongodb): store the key when UpdateItem creates an item #473's key repair live tests, one of the key repair's new here: a catalog at migration 5 refused by register_and_check as behind with no repair started, and a repair started after the migrated catalog is accepted rewriting the planted document); 29 added and 27 removed with the slots, the sweep and the claim markers. The live tests cover a heartbeat lost while $out runs leaving no orphan collection, the PostgreSQL catalog-index module ported, migration 6's backfill and the reconciler's first pass, a legacy restore's record, a delete and three restores ordered both ways over six rounds (accounted by the catalog), a FAILED record whose target removal did not land listed once stale, DELETING left by a failed or dropped delete and finished by a retry, a restore of another instance not dead while it heartbeats, and a slow-clock process not dead; migrations_fresh 8, migrations_guards 7, migrations_holders 10, migrations_locks 10, migrations_released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20. The 7 protocol-model tests run by default (1.5 billion interleavings memoised, under a second).
• extenddb-storage-cassandra (Cassandra 4.1 at 127.0.0.1:9042, EXTENDDB_TEST_CASSANDRA=required, one binary at a time): lib 128 (4 added, 1 removed: the unknown-outcome classes from a synthetic error body, generations, listing order, the consecutive heartbeat-error rule); direct_integration 205 with --features test-hooks (13 added, 1 rewritten, 1 removed; the added: a FAILED record whose target the removal's claim left DELETING listed once stale and its removal resumed from the drop, claim, start_backup_delete, scoped listings, migration 6's backfill and the first reconciler pass on an isolated catalog, a legacy restore's record, an abandoned in-catalog create reported once stale and finished (counting the report and showing the dead generation fenced out), a FAILED record whose target removal did not land listed once stale and removed, a migration 6 whose backfill fails staying unrecorded and resuming, serve's registration refusing a catalog without migration 6 and accepting it once migrated, the create's heartbeat observed mid-copy, three heartbeat errors stopping a create within a bucket and two not, a restore registering between a delete's in-use read and its claim refusing the delete on both delete paths); migrations 21; migrations_stopped 8 (migration 6's golden test added); ttl_integration 26 passed, 1 ignored, and main's 1 failure (below).
• tests/test_resource_tag_migration.py's released-catalog fixture strips migration 6's objects (the twelve table_restores columns and their index, backup_index, backup_maintenance, catalog_id) before it clears the ledger, since the check after migration 1 refuses a table_restores carrying the job columns; both cells pass on the SQLite build.
• Through the binary, release builds of this tree against catalogs that release builds of main created (SQLite, PostgreSQL, Cassandra, MongoDB): migrate --list shows migrations 1 to 6; serve refuses the catalog at migration 5 with exit 3 naming migration 6 and the migrate --yes command; migrate --yes applies migration 6 (the ledger records it applied, settings.catalog_id is written); serve then starts, and main's binary refuses the migrated catalog with exit 4. A fresh init's schema equals docs/schema/<backend>.txt on all four.
• integration-mongodb.yml, the CI way, on a release build of this tree with mongodb-test-hooks: mongodb-pytest (run-mongodb-tests -- --pytest --comprehensive --parallel: 1636 passed, 50 skipped, 9 xfailed, 3 xpassed; comprehensive 502), mongodb-rust (--rust-live -- --rust-integration: tests/rust 514, records live 6, the migration binaries, live Rust 18; the --rust leg as the workspace cargo test without service env), mongodb-snapshot (11), mongodb-catalog (the schema, catalog-lock, auth, released and 0.1.14 older-binary checks), mongodb-production-build.
• integration-cassandra.yml, the CI way: cassandra-storage (the five Cassandra commands below), cassandra-rust (tests/rust 514), cassandra-pytest's comprehensive leg (502), cassandra-catalog-checks (schema, catalog lock, and the v0.1.14 tag's binary against migrated catalogs), cassandra-two-nodes (migrations 21 on a two-node cluster), cassandra-authorizer, cassandra-production-build. cassandra-pytest's --pytest --parallel leg on the review host ends with 30 s request timeouts on account creation (1 failed, 291 errors, no other failure); main's binary and tests meet the same timeouts there (2 failed, 233 errors), and every file with a timeout passes run without --parallel (326 passed). tests/test_backup_restore_fidelity.py (36 passed, 1 skipped) and tests/test_table_operations.py (26 passed) ran inside both backends' pytest jobs. The PostgreSQL and SQLite fidelity runs (64 and 63 passed) are from the tree before its rebases.

Review round (Lee's review of 2026-11-04), on this head (cargo fmt --check, cargo check --workspace --tests, cargo +1.97.0 clippy --all-targets --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks -- -D warnings clean):

• extenddb-engine backup:: 104, extenddb-storage 228, extenddb-config 14, extenddb-storage-sqlite backup 50, PostgreSQL backup_restore 40 (3 added: the FAILED write matching only the listed generation and heartbeat, a record failed after the last batch leaving the CREATING target to the reconciler, an unreadable record skipped by the dead listing), MongoDB backup_engine::live_tests:: 24 (2 added: the registration's first heartbeat on the server clock, the FAILED write's conditions), Cassandra direct_integration --features test-hooks backup_engine:: 31 (5 added: a FAILED record and a target claimed to DELETING each stopping the copy at its next heartbeat, a create that heartbeats between the scan and the fence keeping its row, a stale CREATING index row skipped by the abandoned-create scan, the FAILED write's conditions). Engine: the target name checked after the backup and before the permit, a request over the bound waiting a bounded time, a name the store prefix refuses accepted, the reconciler leaving a record whose owner heartbeated after the listing. SQLite: the abandoned-create removal under the write lock and a create committing AVAILABLE during the scan keeping its backup, a running restore holding its slot and not listed dead by its heartbeat, a copy whose record was failed stopping at its next batch, an unreadable record skipped, the heartbeat requiring RUNNING. Storage: 10,000 generations each non-zero and below 2^63, a permit waited for within the bound, a source without features reporting {}.
• Mutants, each applied, the one test run, the file restored: SQLite heartbeat without AND status = 'RUNNING' (the heartbeat test fails); SQLite per-batch check without AND r.status = 'RUNNING' (the batch test fails); the generation's >> 1 removed (the generation test fails); PostgreSQL done == 1 replaced by true (the last-batch test fails); Cassandra's fence without AND heartbeat_at = ? and with >= (the scan-and-fence test fails both ways); Cassandra's stale CREATING skip disabled (the skip test fails).
• devtools/run-mongodb-tests --rust-live gains the backup_engine::live_tests:: step with EXTENDDB_TEST_MONGODB_REQUIRED=1, so the mongodb-rust job runs the 24 live backup tests that ran in no CI job; .github/workflows/clippy.yml lints with the Cassandra and MongoDB crates' test-hooks features. Neither was run in CI from here.
• Not re-run on this head: the full workspace suite, the pytest suites and the CI workflows (CI runs them on push); the counts below from the previous head are carried where not named above.

cargo fmt --all -- --check
cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo doc --no-deps --workspace                                          # no warning in a file this PR adds or at a line it changes
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_STANDALONE_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb --lib -- --ignored live_
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --test migrations
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --test migrations_stopped
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --features test-hooks --test direct_integration
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --test ttl_integration   # 26 passed, 1 failed (main's), 1 ignored

Not changed here

• The backup store: [backup.store], the store identity file (extenddb-store.json), backup purge (PR 7). BackupRuntime.configured_store is None, so on this head Delete and Restore of a backup_index row return the other-store ValidationException, and claim_backup, set_backup_deleted and remove_backup_row have no production caller yet; each backend's tests drive them.
• The store-backed CreateBackup writer and worker, and the reconciler duties on backup rows (resolving a dead CREATING row, finishing DELETING, deleting prefixes, tombstone retention) (PR 8). heartbeat_interval_secs reaches no job yet: every in-catalog job heartbeats at JOB_HEARTBEAT_INTERVAL (15 s), which the stale floor is computed from.
• The manifest restore job (PR 9a): a dead kind = manifest record is logged and left; remove_restore_target deletes restore_progress rows only where that table exists.
• The maintenance claim's holder and expiry (PR 10): check_maintenance_claim reports any backup_maintenance row. Cassandra's orphan backup_index_by_status entries wait for backup sync.
• The Cassandra ttl_integration::test_ttl_enable_rejects_asynchronous_gsi failure is main's: the test expects "asynchronously propagated GSIs" in the refusal and metadata_engine::async_gsi_ttl_refusal says "asynchronously propagated global secondary index"; this PR leaves both files as on main.
• The design's PR 6 acceptance cell running the two DeleteBackup/RestoreTableFromBackup orderings over a backup_index row is deferred to PR 9a, which adds the first restore of such a row; this PR observes the row lock with a two-transaction test of start_backup_delete on PostgreSQL and a paused-delete test on Cassandra.
• The upgrade manual's migration 6 section states the rollback: the pre-upgrade dump restored (the previous binary refuses the migrated catalog, exit 4) and the [backup] section removed from the configuration, which the previous binary refuses as an unknown field; no store exists on this head, so the dump loses nothing.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), main's Cassandra TTL failure above excepted
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 3 and 4 and the PR 6 section; ADR-0023 (catalog migrations; migration 6 is one more numbered migration on every backend). Storage trait: BackupEngine is replaced by BackupCatalog and LegacyBackupReader on the StorageEngine supertrait bound; StorageError gains AlreadyExists.

Breaking changes

• Catalog migration 6 on every backend, breaking: stop every server of the previous release before extenddb migrate --yes; a server with pending migrations refuses to start (exit 3), and the previous release refuses the migrated catalog (exit 4). A restore in flight at the stop is failed and its target removed on the new server's first reconciler pass.
• The per-backend restore slot semaphores are removed (PostgreSQL and MongoDB: 4 per account and half the pool per server with a 30 s wait); the bound is [backup] max_concurrent_jobs (default 4, at most 1024) per instance for backups and restores together, waited for up to 30 s and then refused with LimitExceededException whose text names the bound met beside its number ([backup] max_concurrent_jobs, the [storage.sqlite] pool_size snapshot-slot bound, or the PostgreSQL server's connection slots when the job's claim connection is refused). SQLite's backup_server_slot_count stays (PR 8 keeps it as the backend's admission bound and raises the pool for it), its slots shared by backups and table snapshots (feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395); a CreateBackup that finds no slot free is refused at once with the same text where it waited up to 30 s, so snapshots holding every slot refuse every CreateBackup with no backup running. On PostgreSQL pool_size and catalog_pool_size are raised to at least 2 * max_concurrent_jobs + 10 (18 at the defaults), logged.
• A new [backup] section, all keys optional with defaults; a config file with an unknown key under it is refused at load, and stale_job_after_secs under three times the heartbeat period (45) or max_concurrent_jobs over 1024 is refused at load.
• Under a server.region outside [a-z0-9-] CreateBackup is refused with a ValidationException naming server.region (no backup ARN can be formed), where the previous release created a backup that DescribeBackup, DeleteBackup and RestoreTableFromBackup then refused as not valid; backups taken before the upgrade under such a region answer as before. The startup warning says so.
• Out-of-tree storage backends must implement BackupCatalog and LegacyBackupReader in place of BackupEngine, and handle StorageError::AlreadyExists in an exhaustive match.
• ListBackups accepts a table ARN in TableName (it was a ValidationException); an ARN whose region is not server.region is ValidationException "1 validation error detected: Invalid AWS region in ''" on ListBackups and CreateBackup (the service's answer; main answered the pattern-constraint text).
• RestoreTableFromBackup checks the target name after the backup: a restore of a missing backup into an existing table or a CREATING target is BackupNotFoundException, as on main and the service.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I built the server at this head for SQLite, PostgreSQL and Cassandra. I ran the PR's backup tests against PostgreSQL 16, a MongoDB 7 replica set and Cassandra 4.1. I sent the requests this PR changes to Amazon DynamoDB in us-east-1, at least twice per shape. I sent the same requests to this head on SQLite and PostgreSQL, and to main on SQLite. I ran two PostgreSQL servers on one catalog under SIGSTOP and SIGKILL, ended claim and registration sessions with pg_terminate_backend, and killed a SQLite server mid-copy. I also ran 67 mutants of this PR's code against its tests.

The per-crate test counts in the description match the CI logs for this head, apart from MongoDB's lib (166 in CI, 165 in the description).

A SQLite restart ages an interrupted restore, and the first cycle removes its target. Ending the PostgreSQL claim session stops the copy at the next heartbeat and removes the target. In 620 races of DeleteBackup against RestoreTableFromBackup on PostgreSQL and SQLite, no DeleteBackup succeeded while a target of the backup was CREATING. Every ACTIVE target contained all the backup's items. Two servers racing one restore claim made exactly one CREATING target in each of 40 runs.

This head describes, lists and refuses catalog index rows in each status as the description says. With max_concurrent_jobs = 1, every check that needs no job slot answered before the bound. The new PostgreSQL and Cassandra tests run against a server in CI.

The table ARN in the caller's region, another account's ARN, backup ARN ids, the ListBackups cursor checks and the restore response members match the service. Two restore answers now match it where main did not. An existing target with an override the backup refuses, and an in-use backup into an existing table, both answer TableAlreadyExistsException. 51 of the 67 mutants fail a test that CI runs. Clippy 1.97.0 and the 1.88.0 check with test-hooks are clean, and the PR adds no third-party crate.

Four Blocking items and seventeen more are inline. Once they are fixed, I will approve.

[LeeroyHannigan]

Blocking: this checks the target name before reading the backup, so a restore of a missing backup into an existing table answers TableAlreadyExistsException Table already exists: <name>. Into a table that CreateTable is still creating, it answers the same, and into a CREATING restore target it answers TableInUseException Table: <name> is already being restored from backup: <arn>. Amazon DynamoDB (twice each) and main answer BackupNotFoundException Backup not found: <arn> in all three cases. Please move this check after resolve_backup and before validate_override_for_source, the order the service uses. Please also change restore_checks_the_target_name_before_the_backup_and_the_permit and the differences row to match.

[LeeroyHannigan]

Blocking: ListBackups now parses TableName with create_backup_table_name. So arn:aws:dynamodb:us-west-2:<account>:table/<name> or an ARN with the region US-EAST-1 answers 200 with the server-region table's backups on SQLite and PostgreSQL. Amazon DynamoDB answers ValidationException 1 validation error detected: Invalid AWS region in '<arn>' (twice each), and main answers ValidationException 1 validation error detected: Value '<arn>' at 'tableName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+. Please refuse an ARN whose region is not server.region in that function, with the service's text. The refusal also covers the same gap CreateBackup has on main. The CreateBackup and ListBackups rows of the differences document need the same change.

[LeeroyHannigan]

Blocking: with the default max_concurrent_jobs = 4, the fifth of five restores sent together answers LimitExceededException Too many backup and restore jobs are running on this server (at most 4 at once, the [backup] max_concurrent_jobs bound); retry later on SQLite and PostgreSQL. A CreateBackup while four jobs run answers the same. Amazon DynamoDB accepted all five restores, and it accepted six CreateBackup calls sent together in each of two runs. Main accepted both requests on SQLite, and its PostgreSQL and MongoDB restores wait up to 30 s for a slot. Please wait a bounded time for a slot, or raise the default. In both cases, please name the bound in the differences rows for CreateBackup and RestoreTableFromBackup.

[LeeroyHannigan]

Nit: this refuses CreateBackup of a table named ... (and, from the code, of a name over 255 bytes) with ValidationException TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name "..." is only dots. Main answered 200 AVAILABLE. Amazon DynamoDB answered ContinuousBackupsUnavailableException Backups are being enabled for the table: .... Please retry later 13.2 s after CreateTable, and at 33.7 s it answered 200 with BackupStatus CREATING and the ARN arn:aws:dynamodb:us-east-1:<account>:table/.../backup/<id>. Nothing the in-catalog create writes here needs a store prefix. Please keep it as on main and apply these rules in #700's store-backed writer. The CreateBackup rows of the differences and limits documents then need the same change.

[LeeroyHannigan]

Blocking: this filter runs after the read of the CREATING rows. The reconciler removes each listed row through delete, which also removes an AVAILABLE backup. A create that commits AVAILABLE and frees its slot between the read and this filter loses its backup, and no test covers that window. With a 25 s sleep added before this line, CreateBackup answered AVAILABLE, and 26 s later the reconciler removed the backup and its 5,000 items. Please remove abandoned creates in a write under the engine write lock. Limit the write to CREATING rows with a free slot, and test that window.

[LeeroyHannigan]

This is true per worker start: run creates ReconcilerState, and serve restarts the workers each time it takes the serving registration again. On PostgreSQL the reconciler logged the same dead manifest record again 17 s later, after I ended the serving-registration session and the server registered again. Please keep the logged set in serve, or say "once per worker start".

[LeeroyHannigan]

On MongoDB the registration writes the first heartbeat with the process's own clock (local_now). $$NOW replaces that heartbeat only once the target's collections and indexes exist. If a host's clock is behind the database's by more than stale_job_after_secs, another server can list that host's new restore dead in that window. I read this from the code and did not run it. Please write $$NOW in the registration, or state the exception in this row.

[LeeroyHannigan]

Amazon DynamoDB's BackupInUseException text for a restore of an in-use backup is Backup is being used to restore another table: <arn>, with no table name (nine samples). Main's MongoDB answered BackupInUseException with that text when another restore held the claim, and this head answers Backup is being used to restore table <name>: <arn> through this function. I read the MongoDB side from the code at main and at this head, as I ran no MongoDB server. Please return the service's text for the restore refusals, and keep the table name in a log line.

[LeeroyHannigan]

This test reads its expected count from HEARTBEAT_ERRORS_TO_STOP, so it still passes with the constant at 4. Please use the literals 3 and 2, as the direct test does.

[LeeroyHannigan]

CI runs clippy and the 1.88.0 build without this feature. So CI never lints src/test_hooks.rs and the hook call sites, or builds them at the MSRV. Clippy 1.97.0 and the 1.88.0 check with the feature are clean at this head. Please add --features extenddb-storage-cassandra/test-hooks, and the MongoDB crate's test-hooks, to the clippy step so clippy lints the next change here.

[robinnsc]

Thanks for the runs and the mutants. Every item is addressed on the new head; the tests you asked for were each run against the mutation you named and fail there.

• crates/engine/src/backup.rs:1367 (Blocking, the target name before the backup): restore_table_from_backup resolves the backup first, then checks the name through describe_table, then validates the overrides against the source, so a missing backup is BackupNotFoundException into an existing table, a table CreateTable is still creating and a CREATING restore target alike. The test is restore_checks_the_target_name_after_the_backup_and_before_the_permit; the "Restoring into a name that exists" row says the order and names the earlier head's answer.

• crates/engine/src/backup.rs:735 (Blocking, a TableName ARN in another region): create_backup_table_name refuses an ARN whose region is not server.region, compared as written, with ValidationException "1 validation error detected: Invalid AWS region in ''", for ListBackups and CreateBackup alike. Both differences rows carry it.

• crates/engine/src/backup/runtime.rs:66 (Blocking, a refusal with no wait): the permit is waited for. JobPermit::acquire_within(slots, wait) waits up to PERMIT_WAIT (30 s, the wait main's PostgreSQL and MongoDB restores had) and take_permit is then LimitExceededException; try_acquire stays for the callers that must not wait. Test a_request_over_the_job_bound_waits_a_bounded_time_for_a_slot (engine) and a_permit_is_waited_for_within_the_bound (storage). The CreateBackup and RestoreTableFromBackup rows, the limits document's concurrent-restores row and the admin guide's max_concurrent_jobs row name the 30 s.

• crates/engine/src/backup.rs:443 (Nit, a dot-only name refused): kept as on main. check_backup_arn_fits keeps the region rule alone; a name over 255 bytes or made only of dots is backed up in-catalog, and the store-prefix rules move to the store-backed writer of feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700. Test create_backup_accepts_a_name_the_store_prefix_refuses; the differences and limits rows say so, with your DynamoDB measurement.

• crates/storage-sqlite/src/backup.rs:1448 (Blocking, the filter after the read): abandoned_creating_backups now removes, under the engine write lock (BEGIN IMMEDIATE), only the CREATING rows whose slot no task holds, and reports what it removed; the reader's delete no longer runs on them. a_create_that_commits_available_during_the_abandoned_scan_keeps_its_backup holds the write lock across the scan while a create commits AVAILABLE and frees its slot, and the backup and its items survive; an_abandoned_creating_backup_is_removed_by_the_call_that_reports_it covers the removal.

• crates/engine/src/backup/reconciler.rs:234 (the FAILED write over a late heartbeat): fail_restore_job(key, owner_generation, heartbeat_at, reason) is conditioned on RUNNING, the listed generation and the listed heartbeat on all four backends, and writes a fresh generation. Engine test the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listing; backend tests the_failed_write_matches_only_the_listed_generation_and_heartbeat on PostgreSQL, SQLite, MongoDB (live) and Cassandra. Design 4(e) and the PR 6 interface block carry the signature.

• crates/storage-sqlite/src/backup_catalog.rs:140 (the restore judged by a heartbeat behind the write lock): a legacy restore holds a JobSlots slot (restore_slot_key(table_id)) for its life, and list_dead_restore_jobs skips a RUNNING record whose slot a task of this process holds, whatever its heartbeat. Test a_running_restore_holds_its_slot_and_is_not_listed_dead_by_its_heartbeat, with the test holding the write lock as your UpdateTable did.

• crates/storage/src/backup_catalog.rs:412 (SourceTableFeatureDetails omitted): always present, {} when the source recorded nothing, the inner members still omitted when empty. Test a_source_without_features_reports_empty_feature_details.

• crates/storage-mongodb/src/backup_live_tests.rs:1330 (the ignored tests in no CI job): devtools/run-mongodb-tests --rust-live gains a step running cargo test -p extenddb-storage-mongodb --lib -- --ignored --test-threads=1 backup_engine::live_tests:: with EXTENDDB_TEST_MONGODB_REQUIRED=1, which the mongodb-rust job runs. 24 live tests pass here against the replica set.

• crates/storage-sqlite/src/backup_catalog.rs:197 (AND status = 'RUNNING' untested): the_restore_heartbeat_requires_a_running_record fails with it removed from the heartbeat, and a_copy_whose_record_was_failed_stops_at_its_next_batch fails with AND r.status = 'RUNNING' removed from the per-batch check.

• crates/storage-postgres/src/backup_engine.rs:557 (done == 1 untested): a_record_failed_after_the_last_batch_leaves_the_target_to_the_reconciler fails the record after the last batch, asserts the target stays CREATING and that the reconciler's listing and removal take it; with true in place of done == 1 the target flips ACTIVE and the test fails.

• crates/storage/src/backup_catalog.rs:54 (the >> 1 caught by chance): generations_are_non_zero_and_fit_a_signed_integer_column draws 10,000 and requires each non-zero and below 2^63; it fails with the shift removed.

• crates/storage-cassandra/src/backup_engine.rs:1229 (AND heartbeat_at = ? untested): a test-hooks pause after the scan lets test_a_create_that_heartbeats_between_the_scan_and_the_fence_keeps_its_row land the owner's beat before the fence; the row keeps its status and generation. It fails with the condition removed and with >= in its place.

• crates/storage-cassandra/src/backup_engine.rs:917 (the heartbeat without RUNNING): the heartbeat's LWT is IF table_id = ? AND table_status = 'CREATING' AND restore_status = 'RUNNING'. test_a_failed_record_stops_the_restore_copy_at_its_next_heartbeat and test_a_target_claimed_to_deleting_stops_the_restore_copy_at_its_next_heartbeat each stop a copy mid-way and assert the count settles short of the total and the record is left as written.

• crates/storage-cassandra/src/backup_engine.rs:2026 (the skip untested): test_a_stale_creating_index_row_is_skipped_by_the_abandoned_create_scan plants a stale CREATING index row and asserts the scan neither fences nor reports it; it fails with the skip disabled.

• crates/engine/src/backup/reconciler.rs:155 (one unreadable row failing the listing): every backend's list_dead_restore_jobs logs and skips a record whose kind or status it cannot read, as ListBackups skips a row it cannot read. Tests an_unreadable_record_is_skipped_by_the_dead_listing on PostgreSQL and SQLite (a RUNNING row of kind manifest-v2 beside a dead legacy one, which is listed).

• crates/engine/src/backup/reconciler.rs:19 (once per worker start): the logged set now lives in serve, outside the worker set, and is handed to the reconciler each time the workers start, so a worker restart after the serving registration is taken again logs nothing twice. The doc says where it lives.

• docs/differences-from-dynamodb.md:125 (the registration's first heartbeat on the process clock): the MongoDB registration writes $$NOW inside its transaction. Live test live_a_registration_writes_its_first_heartbeat_on_the_server_clock; the two-servers row says every heartbeat, the first included, is the database's clock.

• crates/storage/src/backup_catalog.rs:654 (the in-use text): backup_in_use_message(backup_arn) is "Backup is being used to restore another table: "; the target's name goes to a log line where the backends knew it.

• crates/storage-cassandra/src/backup_engine.rs:2202 (the count from the constant): the test asserts HEARTBEAT_ERRORS_TO_STOP == 3 and uses the literals 3 and 2.

• crates/storage-cassandra/Cargo.toml:38 (the feature unlinted): .github/workflows/clippy.yml runs clippy with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks.

On the CodeQL note at tests/backup_restore.rs:3362: other_account is the suite's second 12-digit test account id, a constant, in an assertion message; not changed.

From #672's review, the DELETING row of a store other than the configured one: DescribeBackup reports it DELETED because BackupStatus reports DELETING as DELETED on purpose (the delete has committed and no reader will see the backup again), while DeleteBackup of such a row answers that it is in another store because this server cannot run the deletion; the two answer different questions and are left as they are.

[LeeroyHannigan]

I checked every item from my last review at 03aa0bd. I ran the round-one reproductions again on the backends each item named. I applied the round-one mutants and ran the tests your reply names. I also sent the ARN shapes the new region check covers to Amazon DynamoDB in us-east-1, twice each.

All 21 inline items and the item in the review body are fixed. Each reproduction now gives the right answer, and each named test fails on its mutant and passes after the revert.

The change that moved the target name check brought one new Major item and one Minor item, both inline. The description also gives -- --ignored --test-threads=1 live_ as 46 tests, and this head lists 48 ignored tests whose names contain live_.

Once you fix the Major item, I will approve.

[LeeroyHannigan]

Major: for a catalog index row, which #700 writes, this call runs the in-use, CREATING and store checks before the target name check. On SQLite, a restore of such a row, written by hand and in use, into an existing ACTIVE table answered BackupInUseException twice. Amazon DynamoDB answers TableAlreadyExistsException for an in-use backup into an existing table, and the round-one head did too. Please check the name first, as for a legacy backup, and add a test of a catalog row restored into an existing name.

[robinnsc]

Fixed. For a catalog row the request now checks the row's own state first (check_catalog_row_status: CREATING is BackupInUseException, DELETING, DELETED or FAILED BackupNotFoundException), then the target name (check_restore_target_name), then the in-use rule and the store checks (check_catalog_row_for_restore), so the order matches the legacy path and the service: an in-use backup into an existing ACTIVE table is TableAlreadyExistsException, and a backup into its own CREATING target TableInUseException. a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_its_in_use_rule restores an in-use row into an existing table (TableAlreadyExistsException), into a CREATING restore target (TableInUseException naming the backup), into a free name (the in-use rule, then, with the backup free, the store refusal), and a DELETED row into an existing table (BackupNotFoundException with no table read). Design 5.3 step 1 states the order; #700 and #740 carry the same order on their store-backed paths.

[LeeroyHannigan]

Minor: this region check runs before the account check at line 385. In a dev-mode build, an ARN with another account and another region answers ValidationException "1 validation error detected: Invalid AWS region in ''". Amazon DynamoDB answers AccessDeniedException "Access is denied" for that ARN on CreateBackup and ListBackups, twice each. Please check the account first.

[robinnsc]

Fixed. create_backup_table_name checks the ARN's account before its region, so another account's ARN in another region is AccessDeniedException "Access is denied" on CreateBackup and ListBackups; another_accounts_arn_in_another_region_is_access_denied_before_the_region_check covers both, and the differences document's backup-creation row says the account is checked first.

[robinnsc]

Both inline items are fixed as above, and the description's MongoDB live_ count is corrected to 48, what --list gives on this head. Engine backup:: is 106 tests here (259 in the crate), clippy 1.97 clean.

[LeeroyHannigan]

I checked both items from my last review and the description at bfa5732. You fixed all three.

On SQLite, a catalog row that is in use, restored into an existing ACTIVE table, now answers TableAlreadyExistsException, as the service does. The new test fails when I put back either earlier order of the checks, and it passes on this head. Another account's ARN in another region now answers "Access is denied" on CreateBackup and ListBackups. Its test fails with the region checked first. The description gives 48 live_ tests, and --list gives 48.

The new test brings one Minor item, inline. I also tried to measure the service's answer for a restore of a CREATING backup into an existing table. Every DescribeBackup, the first sent 4 ms after CreateBackup answered, read AVAILABLE. So no restore met a CREATING backup, and that answer is still unmeasured.

From #672, the answers for a DELETING row of another store still disagree. At #672's head, DescribeBackup answers DELETED, RestoreTableFromBackup answers BackupNotFoundException, and DeleteBackup answers that the backup is in another store. The server logs no warning for the row. Design 4(g) says that a restore of such a row returns the ValidationException. Please make the design and the code agree, here or in a follow-up. It is not a condition for this approval.

[LeeroyHannigan]

Minor: this checks a DELETED row only. With the CREATING check moved after the name check, all 106 backup:: tests still pass. Please add a CREATING row restored into an existing table.

[robinnsc]

Fixed. The cell now runs its row-first case over a CREATING row as well as a DELETED one: a CREATING backup restored into an existing table answers BackupInUseException ("Backup is being created: ") before the name is read. With the CREATING check moved after the name check the cell fails on that case (it passed before, as you found); back in place, backup:: is 106 passed.

[robinnsc]

The Minor is fixed as above (head 6ff37487). Design 4(g) now says what the code does for a row of another store, so the two agree without a code change: Describe answers the row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning, since which store a row belongs to is the operator's concern (backup list prints every row's store); Delete and Restore check the row's own state first, as for a row of the configured store (CREATING BackupInUseException, DELETED or FAILED BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for Delete a DELETING one, with the other-store ValidationException. That is the order you measured at #672's head (DescribeBackup DELETED, Restore BackupNotFoundException, Delete the other-store text). On the CREATING-backup restore the service's answer stays unmeasured, as you say; the code's BackupInUseException with the service's "Backup is being created" text is the row-state refusal the design gives, and if a measurement ever shows otherwise it is one message in check_catalog_row_status.

[LeeroyHannigan]

I checked the test item and the design note at 6ff3748.

You fixed the test item. With the CREATING check moved after the name check, the cell fails, and it passes on this head.

The new 4(g) text gives the answers I measured for a DELETING row of another store. Two of its statements still differ from the code, inline. The code's answers match section 4(a), so the fix is to the text. As I said last time, it is not a condition for this approval.

On this head, the cassandra-storage job fails in test_ttl_enable_rejects_asynchronous_gsi and a_backfill_the_recovery_worker_started_ends_before_the_worker_returns. Both also fail on main, and this PR does not touch them.

[LeeroyHannigan]

Minor: on SQLite at this head, DescribeBackup of a DELETED row answers BackupNotFoundException, as section 4(a) says, where this text says DELETED. The in-use rule and a restore's name check run before the other-store ValidationException, so an AVAILABLE row restored into an existing table answers TableAlreadyExistsException. Please make these two statements match the code.

[LeeroyHannigan]

LGTM 🚀