[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · #740 the restore job, the cross-backend matrix and backup remove-target · #825 backup sync, --prune-missing, --adopt-store · #855 the CI filters, the open measurement cells and the backup and restore manual · this PR: backup sync --move-store, --forget-store, --forget-backup and the report of the rows of other stores · next: #821 migrate-legacy, #748 export and import through the store.
Closes #820 (part of #624).

Stacked on #855 (feat/backup-docs, head 88d445da), itself on #825 (feat/backup-sync, 358ff8bd) and #740 (9803ab7f); opened as a draft against feat/backup-docs. One commit. No migration, no schema change, no new dependency; Cargo.lock is unchanged. The design's PR 10 ships as three PRs (revision item 34): #825 is the catalog-loss recovery, this PR the options that rewrite or delete rows of other stores, #821 migrate-legacy. Every binary below is a debug build of this tree (88d445da plus this diff).

What

Storage trait (crates/storage). Three BackupCatalog methods, with no default body. list_backup_rows_by_root(kind, root, status): every row of the kind and root whatever its store id, of one status or all, in list_backup_rows' order, the root matched byte for byte (an S3 root's ?endpoint= suffix included). rewrite_store(from_kind, from_root, keys, to) -> u64: each row of keys that is still of the old kind and root and not CREATING gets to's kind, root and id; the count of rows rewritten. The design's signature had no "from" arguments; with them a row another writer moved, or the CREATING row of a server still configured with the old root, is left rather than moved without its bytes. A partial rewrite (between MongoDB chunks, between Cassandra keys) leaves rows under both roots, which the trait doc states: every step is idempotent and the command names what is left. summarize_store_roots() -> Vec<StoreRootCount>: the rows per store (kind, root, id) and status, in one order on every backend (StoreRootCount::sort), for sync's report; the design had no method that lists the roots. BackupStatus::ALL. The docs of forget_available_backup_row, remove_backup_row, set_backup_deleted and remove_dead_creating_backup_row name their new callers.

PostgreSQL, SQLite, MongoDB, Cassandra. PostgreSQL: the by-root listing is list_backup_rows' statement without the id (the (store_kind, store_root) prefix of idx_backup_index_store_status); rewrite_store one UPDATE ... WHERE (account_id, backup_id) IN (SELECT * FROM unnest($4, $5)) AND store_kind = $6 AND store_root = $7 AND status <> 'CREATING', whose row lock orders it against a restore's FOR UPDATE and DeleteBackup's transaction; the summary one GROUP BY. SQLite: the same statements, rewrite_store one BEGIN IMMEDIATE transaction under the write lock, all rows or none. MongoDB: find on kind, root and status; update_many in chunks of REWRITE_CHUNK_KEYS = 1,000 keys, filtered on the old kind and root and status: {$ne: "CREATING"}, that also $inc lock_seq, so a concurrent claim_restore_tx or start_backup_delete_tx on a moved row conflicts, re-runs and reads the new store; a $group for the summary. Cassandra: the by-root listing reads the five (root, status) partitions of backup_index_by_status, filters entries on the kind, and keeps a row only when its re-read backups_by_arn row has that status, kind and root (root_index_arns, refactored from status_index_arns). rewrite_store, per key: a LOCAL_SERIAL read, skipped unless of the old kind and root and not CREATING; the new root's status entry written; the lightweight transaction UPDATE backups_by_arn SET store_kind, store_root, store_id ... IF store_kind = ? AND store_root = ? AND backup_status = <read>; the old root's entry deleted (a failure logged). An unknown outcome is settled by a serial read (moved when the row now carries to's store); NotMatched removes the new entry and reads again, up to three attempts, after which the key is left uncounted. The module doc gives the one exception this makes to its row-first rule: a stop after the new entry leaves only an entry every reader skips, while the row first would leave a moved row that no listing of the configured store returns. The summary counts the status entries, unverified, as the trait doc says. Test hooks fail_rewrite_writes and stop_rewrite (RewriteStop::{AfterNewEntry, AfterRow}).

Engine (crates/engine/src/backup/). New store_maintenance.rs, written against BackupCatalog and the configured store:

• locate_root(storage, root, configured_root): the kind of the rows under a root (both kinds are tried through read_store_ids; an S3 label starts with s3:// and a filesystem label is an absolute path) and the store ids the catalog records for it; RootRefusal::ConfiguredRoot, and NoRows with the summary, so the command can print the stores the catalog knows.
• move_store: reads the old root's rows (each row's source_table_details_json dropped as it is read), then checks each against the configured store and names every row that stops the move in one pass. A CREATING row goes through reconciler::resolve_dead_creating_row, split out of resolve_dead_creating_rows so both callers share one body (CreatingOutcome), with the configured store's calls; a row not handed over has a live owner. A DELETING row with a verified manifest under its copied prefix stops the move; otherwise finish_backup_delete deletes what the copy brought behind the fence and sets DELETED. A DELETED or FAILED row with a verified manifest there stops the move, naming --forget-backup. An AVAILABLE row is probed with probe_manifest (now pub(super)) and its manifest's size and item total must be the row's. A row that would give an account a second row of one backup id (against the configured root's rows, and among the moved rows, which only Cassandra can hold) stops the move. With none stopping, StoreFence::check_catalog re-reads the identity file once, rewrite_store runs, and the old root is listed again: a row still there is reported as left (a CreateBackup of a server still on the old root, a row rewrite_store did not count); when that re-list fails after a rewrite that counted every key, every key is taken as moved. The resolutions and finished deletions are kept when the move stops. Audit lines (extenddb::audit, backup sync --move-store) for each row moved, resolved or finished.
• forget_store and forget_backups: rows removed outright with the removals that exist (forget_row): AVAILABLE forget_available_backup_row, DELETED and FAILED remove_backup_row, CREATING remove_dead_creating_backup_row on the catalog's clock, DELETING set_backup_deleted then remove_backup_row (the DELETED state between them is safe: the DELETED-prefix and tombstone duties do not run under the claim). A removal that meets another status re-reads once and applies that status's removal. A FAILED row whose heartbeat is within the owner window is left as having a live owner (FailedOwners, one list_backup_rows_older_than per store). --forget-backup reads the ARN as given, then under its other spelling (BackupArnSpellings), then through LegacyBackupReader::describe, and stops at the first ARN it does not remove. After --forget-store the root's store ids are read again and those with no row behind them reported.
• other_stores and group_counts: the summary grouped per store other than the configured one, and each CREATING and DELETING row named through list_backup_rows(Some(status), <that store>).

maintenance::wait_for_quiet and busy_jobs take &[StoreIdentity]. reconciler.rs: Duty::fenced re-reads the row after the fence passes and skips one that is gone or whose status or store changed (logged), for the FAILED-prefix, DELETED-prefix and tombstone duties; the module doc says so and words the OTHER_STORE line for the move. restore.rs: OTHER_STORE reads "...; configure that store in [backup.store], or move the backup's rows to the configured store with extenddb backup sync --move-store <root>, and restore again", as design 5.3 step 7 says. sync.rs: the module doc says that sync reads and changes no row of another store and that the command reports them.

App (crates/app/src/cmd_backup.rs). backup sync --move-store <old root>, --forget-store <root>, --forget-backup <arn> (repeatable), each with clap help, conflicting with each other, with --adopt-store and with --prune-missing (exit 2). --move-store needs Identity::Known, locates the root before the claim, takes the claim on every backend (on SQLite too, beside the serve lock, as the other options do), waits over the configured identity and the old root's ids, runs the move, prints its sections (Resolved, Finished, Stopping the move, Moved, Left under the old root) and the summary line, exits 1 unless every row moved, and only then syncs the configured root. The forget options keep the catalog open when no store is configured (open returns Opening::CatalogOnly for them), print the store's state in the header, locate the root (--forget-store), take the claim on every backend, wait (over the configured identity when known, and the root's ids), remove, and print each row's outcome and the summary line; after --forget-backup, when a store is configured, check_configured_store runs once and record_check stores its outcome, printed as "The configured store's check passes (store id ...)" or the problem, which for ManifestMissing names the next ARN and the command for it. Plain sync's ManifestMissing refusal names --forget-backup. Every sync that listed the root to its end prints "Rows of other stores": per store the counts per status, the CREATING and DELETING rows, and the line naming --move-store and --forget-store; a failed read of them is printed and exits 1. The module doc gains the three options. take_claim, adopt_wait_identity and root_refusal are factored out; the line and exit decisions are pure functions with unit tests.

Harness. tests/test_backup_job.py: five new cells and three extended (below); helpers _set_root, _set_store_disabled, _copy_root, _section, _age_row. tests/test_backup_store_config.py: the serve-lock cell runs the three options beside a SQLite server.

Docs and texts. The backup manual: the restored-catalog paragraph now gives the --forget-backup <arn> procedure in place of "This release has no command that clears it"; the claim paragraph names the three options among those that wait; the configuration section points a root that has to change at the move; a new section, "Moving a store to a new root" (reconfigure every server first, copy whole with the modes, finish the copy, the identity file, the root label byte for byte, the checks per status, the stop and its way out, the summary line, the re-list, what is left untouched, OTHER_STORE, memory, and --forget-store to drop a root's rows). The admin guide: the identity paragraph (the move), the synopsis, the exit-1 list, the sample output with the other stores' section, the newest-manifest sentence (--forget-backup), the full report in place of "reported in part", and two paragraphs on the three options with the measured memory and a sample output. The deployment guide's shared-root bullet says how to change the root later; the differences doc's "Where backups live" row says a moved root keeps its ARNs. Design: revision item 36; the PR 10 section's split line, interfaces block (rewrite_store's arguments, summarize_store_roots) and sync contract, and section 7's memory paragraph, edited with it.

Why

Design 4(c), 4(g) and the PR 10 section. #825 left three gaps that this PR closes. A store root could not change without losing its rows: a tree copied to a new root has no rows there, and sync skips every prefix whose backup id a row of the old root holds, so the backups were reachable only by adopting them again under the new root's store, and a failed restore of another store had no remedy but configuring that store. Rows of a root that is gone could not be dropped without a tombstone, and a tombstone is what a deletion duty acts on. And a catalog restored from a database backup whose newest AVAILABLE backup was deleted after the catalog's backup stays unhealthy ("manifest probe failed"), which --prune-missing refuses, with the prefix put back as the only way out. Plain sync also reported the rows of other roots only in part.

Choices to weigh

• A third trait method, summarize_store_roots. The design's interfaces gave list_backup_rows_by_root, which takes a root; nothing listed the roots the report has to name. The alternative was one list_account_backup_rows per account, which reads every row of the catalog whole (about 22.5 KiB per row of a five-GSI table) to count them. The method is one grouping query on three backends; on Cassandra it counts status entries without re-reading the rows, so a hint left by a write that stopped part-way is counted, which the report and the trait doc say. The CREATING and DELETING rows it names are read through list_backup_rows of each store, which re-reads every row.
• rewrite_store carries the old kind and root and excludes CREATING. Between the move's checks and its rewrite a server still on the old root can write (its CreateBackup after the wait, its DeleteBackup, which the claim does not exclude). Moving a CREATING row whose job writes to the old root would give an AVAILABLE row of the new root with no bytes there. A row another writer moved to DELETING may still move; the configured store's reconciler then finishes that deletion under the new root, which is what the DeleteBackup asked for. The rows not counted are found by the re-list and reported.
• Cassandra writes the new root's status entry before the row. The module doc's rule is "authoritative row first". With the row first, a stop before the new entry would leave a moved row invisible to every listing of the configured store: to the reconciler, to sync's row read (sync would then insert, answer false and report a row it cannot list) and to the manifest probe. With the entry first, a stop leaves an entry whose row still names the old root, which every reader skips. read_store_ids of the new root then counts that entry; its id is the configured id, so no identity mismatch follows. One race stays at the hint level the module doc already names: a DeleteBackup that read a row before the rewrite and writes DELETING after it moves the status entry under the old root.
• Duty::fenced re-reads the row: one catalog read per deletion. A reconciler cycle reads the maintenance claim once, at its start, and its FAILED-prefix, DELETED-prefix and tombstone duties act on rows listed when each began. A cycle that started just before a forget's claim could delete the prefix of a FAILED row the operator forgot precisely to keep it, and with no running jobs the wait returns at once, so the window is real. The re-read also covers a rewrite_store landing mid-cycle. finish_deleting_rows is unchanged: a DELETING row's deletion is the one an operator asked for. a_duty_whose_row_is_forgotten_or_moved_after_its_listing_acts_on_nothing pins it with a Fake hook between the listing and the deletion.
• The forget options run on any store state and with no store. --forget-backup is the way out of ManifestMissing, so it cannot require a healthy store, and a deployment that turned kind = "disabled" must still be able to drop an old root's rows; neither option deletes an object or reads the store. --prune-missing keeps refusing on ManifestMissing, which is also the host-local-root detector (4(c)): pruning while it holds would remove rows whose bytes are on another host. The store's check after --forget-backup is printed and does not change the exit: the rows asked for were removed, and the check names the next ARN to forget when another newest manifest is gone.
• No fourth removal method. DELETING is removed as set_backup_deleted then remove_backup_row. The DELETED state between them is never acted on: the DELETED-prefix and tombstone duties do not run while the claim is held. On a DELETING row of the configured store finish_deleting_rows, which does run under the claim, may finish it first; the forget then finds DELETED and removes it.
• The FAILED live-owner rule applies to both forget options. A FAILED row whose heartbeat is within the owner window may still have its job writing under the prefix (the job has not yet seen its claim lost); the wait already holds on such a row, and the removal leaves it rather than removing a row whose prefix is still changing. The rule applies to --forget-store and to --forget-backup alike.
• The move claims and waits on SQLite too, and a stopped move runs no sync. On SQLite the serve lock already excludes every other writer; the claim is taken as the other waiting options take it, so the stop closure and the claim-lost path are one code path on every backend. A move that stops exits 1 before the sync, so its report is the move's; the resolutions it made are kept and the next run sees them.
• Memory is the backends' read of the old root's rows. The move keeps each row without its parsed description, plus the configured root's (account, backup id) pairs for the two-rows check, and reads one manifest and one prefix listing at a time; the backends still collect each listing whole first, as for sync and backup list. Measured at 20,000 rows (below), within 12 % of backup list over the same rows. Streaming the listings out of the backends is not done here; the texts state the figure.

Testing done

Measured on this tree, debug builds, a 16-core Xeon Platinum 8124M host (30 GiB, Linux 5.10), the services local.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, and with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks, with --features extenddb-engine/backup-test-hooks, and on -p extenddb-storage-cassandra --all-targets --features test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean, no warning.
• Engine --lib: 479 passed, 13 of them new (move_store_resolves_checks_and_moves_each_status_of_the_old_root, move_store_names_every_row_that_stops_it_and_moves_none, rows_the_rewrite_does_not_move_are_listed_as_left, move_store_on_a_root_another_catalog_adopted_moves_nothing, move_store_refuses_a_second_row_of_one_backup_id, the_wait_covers_every_store_it_is_given, locate_root_finds_the_kind_and_ids_and_refuses_the_rest, forget_store_removes_the_roots_rows_outright_and_reports_what_it_left, forget_backup_removes_each_row_in_order_and_stops_at_the_first_refusal, a_stop_before_the_first_row_forgets_nothing, forget_backup_of_the_newest_rows_deleted_backup_heals_the_store, a_duty_whose_row_is_forgotten_or_moved_after_its_listing_acts_on_nothing, the_report_of_other_stores_counts_and_names_their_rows_and_nothing_acts_on_them); every existing cell unchanged by the reconciler split and the Duty::fenced re-read. App --lib 55 passed (6 new: the conflicts and the repeat, the other-stores lines, and the exit and line functions of the move, the forgets and the store check). Server --lib backup 19 passed (StubCatalog). Storage --lib 233 passed (the_root_tally_counts_each_store_and_status_apart_in_one_order).
• Backends, against the live services: SQLite --lib backup 67 passed, PostgreSQL --test backup_restore 61 passed (--test-threads=2), each with list_backup_rows_by_root_returns_every_id_and_status_of_the_root_and_nothing_else, rewrite_store_moves_only_the_old_roots_rows_that_are_not_creating and summarize_store_roots_counts_the_rows_of_each_store_and_status; MongoDB --lib -- --ignored --test-threads=1 live_ 71 passed, the same three as live_ cells plus live_rewrite_store_moves_every_chunk_of_keys (1,001 keys); Cassandra, on a single 4.1 node of its own, the three plus test_a_stopped_rewrite_leaves_the_row_listed_under_one_root (each RewriteStop) and test_an_unknown_rewrite_outcome_is_settled_by_a_serial_read, with test_read_store_ids_lists_the_distinct_ids_of_a_root: 6 passed (--features test-hooks --test direct_integration -- --exact).
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 39 passed, 9 skipped (the PostgreSQL-only cells; -n 4), on the PostgreSQL build 32 passed, 16 skipped (the SQLite-only cells; -n 2); the four new cells that run on MongoDB passed on the MongoDB build. No PostgreSQL database left (the list equal to the one taken before).
• Binary-level checks before the wire cells, on the SQLite and PostgreSQL builds against a filesystem root: each option beside a SQLite server (exit 6), a clap conflict (exit 2), a copy with a data file missing, --forget-backup and the move finishing, a server on the new root describing and restoring a moved backup, --forget-store and the root's return, and the forget options with kind = "disabled".
• Mutants, each a targeted one-occurrence edit, the named tests run, the original bytes written back and checked by hash: 26 in the four backends and the storage crate (the old-root condition dropped, CREATING not excluded, the by-root listing matching a root prefix or dropping the kind, MongoDB's chunks cut to one, a status or an id folded in the summary or its tally, the tally's sort reversed; on Cassandra also the old entry not deleted, the new entry written after the lightweight transaction, and the by-root listing trusting the entry's root); 14 in the engine (the tombstone refusal, for DELETING too, the data-file check, the fence before the rewrite, the live-owner stop, the duty's re-read, the two-rows check, the CREATING and DELETING rows not named in the report, the wait reading only the first store, a live FAILED row removed, --forget-backup going past a refusal, the left rows not reported, locate_root accepting the configured root, the configured store in the report); 4 in the CLI's pure functions; 5 in the CLI's wiring against the wire cells, each with the binary rebuilt (the other stores' section always empty, no check after --forget-backup, a stopped move's exit ignored, --forget-store not knowing the configured root, and on PostgreSQL the move's wait over the configured store only). 49 of 49 killed, each on its test's assertion.
• Memory and time of --move-store (design section 7 and the admin guide state it): a root of 20,000 AVAILABLE rows of a five-GSI table, each prefix complete under the copy (one CreateBackup and 19,999 copies with their manifests rewritten to their own ARN, adopted by backup sync; the copy made with hard links): 569 MiB peak RSS and 37.1 s on the SQLite build, 500 MiB and 36.9 s on the PostgreSQL build (51 and 49 MiB at 50 rows); backup list over the same rows 508 and 472 MiB, and the plain sync of the new root after the move 554 and 495 MiB. Filesystem roots on a local NVMe disk, PostgreSQL on localhost: an S3 store pays a request per manifest, sidecar and prefix listing, and a network database the read's round trips, neither measured.

Acceptance lines of #820 (the design's PR 10 acceptance list) and where each is tested:

Line	Tests
rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes	engine the_report_of_other_stores_...; wire test_another_roots_rows_are_left_alone_until_forget_store_drops_them (a FAILED and a DELETED row dated 2000 and a DELETING row of another root keep their rows and bytes through three cycles of a server on the configured root, and the report counts and names them) and the move cell's report
after a store tree is copied to a new root, --move-store re-points its rows and Describe and Restore work against the new root	engine move_store_resolves_...; wire test_move_store_moves_a_copied_tree_and_stops_on_an_incomplete_copy (SQLite, PostgreSQL, MongoDB)
--move-store resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest	engine move_store_resolves_... and move_store_names_every_row_... (each status and each incomplete shape); wire, the move cell (a FAILED row over a complete copy, a missing data file)
--forget-backup refuses a CREATING row with a live owner and a backup from which a restore target is CREATING, and says so and does nothing for a legacy ARN	engine forget_backup_removes_each_row_...; the backends' existing forget_available_backup_row in-use cells (PostgreSQL, SQLite, MongoDB, Cassandra), which the option calls; wire test_forget_backup_leaves_a_live_creating_row_and_an_in_catalog_backup (the live CREATING row on PostgreSQL; on SQLite the serve lock makes every owner dead)
a second maintenance command while one holds the claim fails at once	wire test_a_second_maintenance_command_fails_at_once with the three options (PostgreSQL)
on SQLite every writing command refuses to run beside a server	wire, the sync cell and test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock with the three options (exit 6)
with the volume unmounted the store is unhealthy, sync without the option that resolves it refuses, and no row changes	wire test_prune_missing_and_the_unhealthy_store_refusals with --move-store on each shape; the forget options run on an unhealthy store by design (the clears cell)
after --forget-store and the store's return, sync adopts its prefixes	engine forget_store_removes_...; wire, the another-roots cell
a CreateBackup started on another instance during --move-store either finishes before the rewrite or is refused	engine rows_the_rewrite_does_not_move_are_listed_as_left (a CREATING row landing after the checks is left and reported) and the_wait_covers_every_store_it_is_given; wire test_a_create_backup_during_the_moves_wait_is_refused (PostgreSQL)
after --move-store A stops on a row with one data file missing and --forget-backup lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied	wire, the move cell
a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler has finished the one and failed the other	not a new cell: the engine's the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead_owners covers the cycle under the claim, and the_wait_covers_every_store_it_is_given the wait the options share; the Fake lists RUNNING records from a table its FAILED write does not update, so a cell joining both needs a Fake change
4(g): --forget-store deletes that root's rows outright, leaving no DELETED tombstone	backends' by-root listing after the removals; engine forget_store_removes_... (a cycle afterwards deletes nothing); wire, the another-roots cell (backup list empty, the tree byte-identical)
--move-store and --forget-store match store_root exactly as stored, the ?endpoint= suffix included	backends' by-root listing (a trailing /, a prefix); engine locate_root_... (the S3 label shape); wire, the move cell (a trailing /)
the reconciler cleans the leftovers of the moved tombstones	engine move_store_resolves_... (the next cycle deletes a moved FAILED row's prefix and a DELETED row's objects and removes the tombstones)
the restored catalog whose newest AVAILABLE backup was deleted: --forget-backup is the way out	engine forget_backup_of_the_newest_rows_deleted_backup_heals_the_store; wire test_forget_backup_clears_a_store_whose_newest_manifests_are_gone (SQLite, PostgreSQL, MongoDB; beside servers the running server's /health turns healthy)

Cassandra has no wire leg (its CreateBackup stays in-catalog, item 19); its share is the direct tests above.

Not in this PR

• Backup/restore: backup migrate-legacy #821: extenddb backup migrate-legacy <arn>|--all, with its acceptance cells.
• A sweep of Cassandra's orphan backup_index_by_status entries: --forget-store reports the store ids they keep for a root, and nothing removes them yet (item 34 says the same of the status-entry orphans).
• Detecting a server left configured with the old root after a move: its later rows appear in the next sync's report of other stores, and the manual's procedure reconfigures every instance first.

Breaking changes

None. backup sync's report gains a section, and a sync that cannot read the rows of other stores now exits 1. OTHER_STORE's text names the move; tests assert the constant by identity. Each FAILED-prefix, DELETED-prefix and tombstone deletion of the reconciler reads its row once more.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com