[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · #740 the restore job, the cross-backend matrix and backup remove-target · #825 backup sync, --prune-missing, --adopt-store · #855 the CI filters, the open measurement cells and the backup and restore manual · #880 backup sync --move-store, --forget-store, --forget-backup · this PR: backup migrate-legacy · next: #748 export and import through the store, #749 the in-catalog reader's removal.
Closes #821 (part of #624).

Stacked on #880 (feat/backup-store-maintenance, head 6793fae2), itself on #855 (feat/backup-docs, 88d445da) and #825; opened as a draft against feat/backup-store-maintenance. One commit. No migration, no schema change, no new dependency; Cargo.lock is unchanged. This is the third of the three PRs the design's PR 10 ships as (revision item 34). Every binary below is a debug build of this tree (6793fae2 plus this diff).

What

Format (crates/core). SnapshotDescriptor::KIND_LEGACY_IN_CATALOG = legacy_in_catalog, whose doc says the marker is empty and that source_backend and source_extenddb_version name the binary that wrote the copy (the in-catalog creator's version was never recorded). No format version change and no fixture byte change; a unit test parses and round-trips a manifest of that kind. is_valid_backup_name is exported, so the command refuses a name the manifest would refuse before it writes anything.

The definition (crates/storage/src/backup_definition.rs). source_table_from_legacy(source, vector_state, region, account), beside restore_input: the table the legacy restore creates, by one rule on every backend. It runs ensure_restorable, refuses LegacyVectorState::Unknown (a SQLite backup with no definition), and otherwise takes restore_input with no overrides (a backup with no definition keeps its keys, its key attribute definitions and the 5/5 fallback) plus the definition's table class, TTL and stream and SseDescription::from_specification of its SSE specification. The vector state is [] on PostgreSQL and SQLite with no names (PostgreSQL with no definition and a NULL or zero snapshot too) and null on MongoDB and Cassandra. Tests: an exhaustive property over every combination of key schema, billing, recorded throughput, GSI shapes, LSI, four SSE specifications, on-demand throughput, class, TTL and stream and no definition, times two vector states and four override sets, that restore_input_from_manifest(source_table_from_legacy(s)) restores what s.restore_input restores; the one difference, a KMSMasterKeyId not carried, has a test of its own.

Storage trait (crates/storage/src/backup_catalog.rs). LegacyBackupReader gains items(account, arn) -> LegacyBackupItems (the name, the recorded source, the vector state and a 'static item stream), list_all() -> Vec<LegacyBackupKey> (the in-catalog backups of every account that are not DELETED, by creation time then ARN) and replace_with_index_row(row) -> Replaced { Replaced, AlreadyReplaced, NotAvailable(String), InUse(String) }; an index row of the id under another ARN or store is AlreadyExists with nothing written. The trait doc names the replacement as its second write beside the restore's registration.

PostgreSQL, SQLite, MongoDB, Cassandra. Each stream is a frozen copy of the backend's restore read: PostgreSQL a REPEATABLE READ READ ONLY catalog transaction whose first statement re-checks AVAILABLE, owned by a spawned task sending through a one-slot channel; SQLite the restore's keyset batches (500 rows or 4 MiB), each in its own read transaction re-checking AVAILABLE; MongoDB a cursor on _backup_<backup_id> with batch_size(DEFAULT_FETCH_BATCH) and legacy_item_from_document, a copy of document_to_item, and Unsupported for a document with no backup_id; Cassandra the buckets in order, each read in pages of LEGACY_ITEMS_PAGE = 8 rows by item_index. The replacement: PostgreSQL one transaction (the backups row FOR UPDATE, the CREATING-target check, the insert statement insert_backup_row used, now insert_unowned_row, the items and definition deleted, DELETED); SQLite the same under the write lock and BEGIN IMMEDIATE; MongoDB start_replacement, one session transaction ($inc lock_seq while AVAILABLE, the CREATING-target check, the upsert, DELETING with migrated_to_store: true), then finish_legacy_delete's drop and DELETED, a DELETING document with an index row of the same ARN and store being finished as AlreadyReplaced; Cassandra replace_legacy_row, the row converted in place by one lightweight transaction IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null after creating_restore_target, then write_index_entries (sent once more on a failure) and cleanup_backup_payload, an unknown outcome settled by read_index_row_serial, NotMatched read again up to three attempts, and a converted row without its entries finished by the next call. Three frozen deletes change, each to order a DeleteBackup wholly before or after a replacement: PostgreSQL's reads the status under its FOR UPDATE lock and answers not found for anything but AVAILABLE; MongoDB's reads a DELETING document marked migrated_to_store as not found, in delete and in the lock's miss; Cassandra's DELETING claim carries AND store_kind = null. SQLite's is unchanged: the command holds the serve lock. Module docs name the stream and the replacement. Fake implements the three with fault knobs; Cassandra's test-hooks gain fail_convert_writes and stop_convert_after_row.

Engine (crates/engine/src/backup/migrate_legacy.rs). migrate(storage, &Migration, Selection::{Arn, All}, on_backup) -> MigrateReport, written against BackupCatalog + LegacyBackupReader and the configured store. Per backup, in this order: the id (no instant: refused before anything is read, the text naming 0.1.x and the way out); the table name by the store-prefix rules and the partition; a catalog row under the ARN (the configured store's: finish, the replacement with that row; another store's: refused), before the status, so MongoDB's DELETING migrated_to_store document is finished, not skipped; the status (CREATING and DELETING skipped, named); the open (Unsupported refused); the name; ensure_restorable ("the legacy restore refuses it too"); the unknown vector state. Then the copy: a listing of the prefix, whose objects with no row are an earlier run's and are deleted through the fence; the fence before the first put; writer::write_data_files with the item validator under the instance's limits; the count check; the three documents and the manifest (unordered, legacy_in_catalog, no marker); verify_prefix and the second count check; replace_with_index_row with the row built as sync builds it (the manifest's name, size, item total and source_table, the id's instant as created_at, unowned). A refusal or failure after the first put deletes the copy through the fence. Outcome::{Migrated, Finished, InStore, Refused, Skipped, Failed}; MigrateReport::stopped for a lost claim, a signal, an unhealthy or fenced store, a failed listing, a catalog call and a replacement whose call fails. The writer (job/writer.rs) runs under a ClaimWatch (confirms_the_claim, the loss token, the reason): Pipeline<'a, W: ?Sized = HeartbeatHandle>, so the job's Pipeline<'_> is unchanged, and impl ClaimWatch for HeldClaim, whose timer now cancels a lost_signal token when it records the loss. is_item_refusal shares refusal_reason's words through one constant. put_document is generic over the watch. Under backup-test-hooks the writer's after_data_file and the command's before_manifest and after_manifest kill points pause it as they pause a job.

Sync and ListBackups. sync.rs: SkipReason::InCatalog { backup_arn, status } ("an in-catalog backup (STATUS) holds this ARN; run extenddb backup migrate-legacy "), not transient, from LegacyBackupReader::describe under both spellings before the row read under them; sync, finish_group, adopt_prefix and startup_sync now need LegacyBackupReader, which every caller's storage has. backup.rs: ListBackups drops an in-catalog summary whose ARN a listed catalog row has.

App (crates/app/src/cmd_backup_migrate.rs, a child of cmd_backup). BackupCommand::MigrateLegacy { arn, all } with a required ArgGroup (usage migrate-legacy <ARN|--all>, exit 2 for neither or both), Mode::Write, the registration name extenddb backup migrate-legacy. The command needs Identity::Known, takes the claim on every backend, starts HeldClaim, runs wait_for_jobs over the configured identity (on SQLite after resolve_dead_owners, with a zero window), runs the migration, releases the claim, prints one line per backup (with its notes) and the summary line, logs extenddb::audit per migration and per finish, and exits 0 only when every backup reached is in the store. lib.rs's help and the module doc name it.

Harness. tests/test_backup_job.py: two new cells, one extended (below); helpers _catalog_rows, _in_catalog_rows, _copy_in_catalog_backup, _typed_canonical, _migrate. tests/test_backup_store_config.py: the serve-lock cell runs migrate-legacy beside a SQLite server.

Docs and texts. The backup manual's Legacy backups section: "Migrating legacy backups into the store" in place of "This release has no command that converts a legacy backup" (the two forms; what is kept; the check and the one-step replacement per backend; the cost: the claim, every backup and restore job answering LimitExceededException, the wait, SQLite stopped, PostgreSQL's held REPEATABLE READ transaction and the vacuum it holds back; <arn> to migrate in pieces; sample output; the outcomes and exit; the refusals, every 0.1.x backup first; the restore-and-back-up way out, kept from the earlier text; what a migrated backup reports; a stopped run and sync's skip; Cassandra's interrupted conversion and its recurring --all; memory); the vector-index sentence corrected; the Strategy section's Cassandra paragraph and the catalog-lost paragraph point to it. The upgrade manual: the store paragraph, the 0.1.x paragraph (refused on their ids) and the rollback paragraph (a migrated backup is in-catalog again in a restored pre-upgrade catalog). The admin guide: the command's paragraph and sample output, the registration names. The differences doc: "Where backups live" and the maintenance-claim row. Design: revision item 37, with sections 3, 4(c), 4(f), 4(g) and 7, the PR 10 paragraph and transitions, the PR 14 lines and section 10's upgrade-manual row edited with it.

Why

Design section 3, 4(f), 4(g) and the PR 10 section's migrate-legacy paragraph, and PR 14's precondition: before the in-catalog reader is removed, operators need a way to move the backups it holds into the store without losing their ARNs. Until now the only way was a restore and a new backup, which gives a new ARN, a new creation time and, for a table that still exists, an interim table. Cassandra keeps writing in-catalog backups until PR 14 settles its backup path, so there the command is a recurring step.

Choices to weigh

• One replacement step per backend, not "insert the row, then the reader's delete". DeleteBackup does not read the maintenance claim. In the design's order a DeleteBackup that landed between the AVAILABLE check and the insert was answered DELETED for a backup that then came back as the migrated row, and a restore target in CREATING made the delete refuse after the insert, leaving both rows and two ListBackups entries. On Cassandra the order cannot work at all: the legacy row and the index row share the backups_by_arn key, and the frozen delete removes the list rows the migrated row uses and leaves a DELETED row that every later insert answers false for. The cost is one write method on the frozen trait (as the restore's registration already is) and three small changes to frozen deletes, each pinned by a test and a mutant; PR 14 removes all of it with the trait.
• Every 0.1.x backup is refused. The released fixtures carry 13-digit ids (v0.1.14 is the newest tag, and the shared 14-digit id came after it), and the store format's ids carry their creation instant, which the manifest parser checks. So the command migrates only in-catalog backups taken by this release line: without a store, with kind = "disabled", and on Cassandra. The texts say so plainly, and the refusal names the way out. The pre-0.0.4 SQLite refusal is in practice a subset of this one; it stays as a defensive check, reachable through a hand-seeded row, with its own SQLite cell.
• The source table is what the legacy restore creates. One rule on every backend, restore_input with no overrides plus the features DescribeBackup reports, so a migrated backup restores the table the in-catalog backup restored. The design refused only the pre-0.0.4 SQLite case; a backup with no definition on PostgreSQL, MongoDB and Cassandra migrates with its keys and the 5/5 fallback, as its legacy restore creates it. What the legacy restore refuses (vector indexes, a multi-part key) is refused here too, so the migration lifts no refusal. The SSE specification's KMSMasterKeyId does not survive the manifest's SSE description; the manual names it.
• Extra refusals the design does not list: a table name the store-prefix rules refuse, a backup name the manifest refuses (the name survives with the ARN or not at all), a MongoDB backup with no item collection, and an ARN a row of another store holds. Each keeps the in-catalog backup and writes nothing.
• The claim on every backend, and the deletion fence. The design's invariants say every writing command takes the maintenance claim, and 4(f)'s list did not name this one. On SQLite it costs nothing under the serve lock and gives one code path. The fence runs before the first put, as CreateBackup's does, and before each deletion of a copy. The cost, stated in the texts: every backup and restore job answers LimitExceededException for the run, and on SQLite the server is stopped. <arn> migrates in pieces.
• Always rewrite after a crash. Objects under the prefix with no row are deleted through the fence and the copy is written again. Reusing a complete copy would save one copy after a crash and needs a second rule; one rule is simpler to state and to test. Meanwhile backup sync skips such a prefix, naming the command, so a sync between the crash and the rerun cannot give the backup two rows. Cassandra already skipped it through the insert's false.
• The count is checked twice, against the recorded count. A DeleteBackup during the read ends the SQLite, MongoDB and Cassandra streams short; PostgreSQL reads a pinned snapshot. The manifest's table_size_bytes is the sum of the copied items' sizes, as the job records it, so BackupSizeBytes and TableSizeBytes can differ from what the in-catalog backup reported; the name, ARN, id, creation times and item count do not.
• --all goes on past a refused, skipped or failed backup and exits 1 at the end with the counts; a global condition (claim lost, signal, store unhealthy or fenced, listing or catalog failure, a replacement whose answer did not come) stops the run at once.
• A Cassandra conversion that stopped after its row is an index row, which list_all does not list, so --all does not finish it; migrate-legacy <arn> does (its "already in the store" path runs the replacement once more). Until then the backup is served by Describe, List and Restore, and backup list, which reads status entries, does not show it. The manual says so.
• ListBackups dedupes by ARN. With sync's skip and the one-step replacement no build of this PR leaves both rows; an older build's crash between its two writes could, so the filter keeps the catalog row's summary for an ARN both list.
• The stream reads are frozen copies, not calls into the data path, so the data path stays free to change before PR 14. Cassandra's bucket read is paged (8 rows of a 64-row bucket), where the restore's copy reads a whole bucket, about 1.7 GiB at the line cap; the restore's copy stays as it was.

Testing done

Measured on this tree, debug builds, a 16-core Xeon Platinum 8124M host (30 GiB, Linux 5.10), the services local.

• cargo fmt --all -- --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean, and with --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks and with --features extenddb-engine/backup-test-hooks; cargo +1.88.0 check --locked --workspace --all-targets clean.
• Core: a_manifest_of_the_legacy_in_catalog_kind_parses_and_round_trips; --lib backup_format 181 passed, 1 ignored. Storage: --lib 237 passed, with the four source_table_from_legacy cells (the exhaustive property; the described features and the legacy identity; the KMS key id; the refusals and the unknown vector state).
• Backends, against the live services: SQLite legacy_migration 6 cells (the stream across both batch bounds with 1,201 small and three 3 MiB items; a stream ending at a DeleteBackup after its batch; list_all across accounts; the replacement's insert and removal in one step, an adopted copy finished, another store refused, in-use and unavailable refused with nothing written); PostgreSQL legacy_migration 7 cells (the stream from its pinned snapshot while another transaction deletes every item row; the same replacement cells; a legacy delete queued behind a replacement answers not found); MongoDB 6 live cells (2,500 items past a cursor batch; a drop ending a stream; the replacement; a replacement stopped before its drop finished by the next call, DeleteBackup leaving it; the in-use and unavailable refusals); Cassandra 5 direct cells on a node of their own (200 items over four buckets in pages of 8; list_all; the conversion in place, the legacy describe, list and delete no longer seeing it; in-use and unavailable; a lost answer either way, a stop after the row finished by the next call, and a DeleteBackup paused between its read and its claim answering not found), run with the 4 existing cells that reach the changed delete, restore_source and insert.
• Engine --lib: 511 passed, 16 of them new (backup::tests::migrate_legacy): the copy under the ARN and its row; each refusal before the copy writing nothing; a refused item deleting the copy and --all going on; a short or failing read; the fence on a root another catalog adopted; a copy that does not read back; a copy an earlier run left; a backup whose row is in the store; a replacement that fails; deleted or in use at the replacement; CREATING and DELETING skipped; a signal and a lost claim; the claim's loss token; the ARN's spellings; sync's skip under both spellings; the ListBackups dedupe. The job, writer, sync and purge cells pass unchanged. App --lib: 58 passed.
• Wire, Python 3.10, moto variables: tests/test_backup_store_config.py tests/test_backup_job.py on the SQLite build 42 passed, 12 skipped (-n 4), on the PostgreSQL build 38 passed, 16 skipped (-n 2); the two new cells passed on the MongoDB build. New: test_migrate_legacy_carries_in_catalog_backups_into_the_store_under_their_arns (303 items with a GSI and NS, B, nested M and L, BOOL and NULL values; two in-catalog backups; <arn> spelt in another region, --all, reruns; the manifests; the in-catalog rows DELETED with no item or definition row; ListBackups once each, DescribeBackup as before, the restore's items and GSI, DeleteBackup; beside the server on PostgreSQL and MongoDB) and test_migrate_legacy_refuses_a_backup_with_no_instant_or_an_item_618_let_in_and_keeps_it (--all past a BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 backup and, on SQLite and PostgreSQL, a 13-digit copy of a backup's rows; the three refused items logged; both kept and AVAILABLE; the 13-digit one restored and backed up again into the store). Extended: test_a_second_maintenance_command_fails_at_once (PostgreSQL: migrate-legacy --all refused while --prune-missing holds the claim, then holding it while it waits for a live job, backup purge --confirm refused beside it, a SIGTERM stopping it before any change) and the serve-lock cell (exit 6 beside a SQLite server).
• Binary-level checks before the wire cells (SQLite, PostgreSQL; this diff over feat(backup): backup sync --move-store, --forget-store, --forget-backup [backup/restore 10b] #880's earlier head 11544ecd): init with kind = "disabled", two CreateBackups, the store configured, the command run (refused with exit 6 beside a SQLite server, nothing written), boto3 afterwards: 33 and 30 checks, none failed.
• SIGKILL at each kill point, on SQLite and PostgreSQL builds with backup-test-hooks, a 40,000-item table in two data files: after the first data file, before the manifest and after it, the in-catalog backup was whole with no row; after the manifest a plain backup sync skipped the prefix naming the command and wrote no row; each rerun deleted the copy, migrated 40,000 items and left the in-catalog row DELETED; a restore of the last held every item. On PostgreSQL the killed command's advisory lock ended with its session, so the rerun took the claim at once. Cassandra's stop after the conversion is the direct cell above (no binary kill point and no Cassandra wire leg).
• Mutants, each a targeted one-occurrence edit, the named tests run, the original bytes written back and checked by hash: 32 in core, the storage crate and the four backends (the kind's text; the definition untrimmed, the unknown state accepted, ensure_restorable ignored, null written as [], the class or the SSE description dropped; a batch or bucket or page boundary skipped, a stream truncated or read outside its snapshot, the per-batch AVAILABLE check dropped; the in-use check skipped, deletion before the insert, Cassandra's store_kind left null or the buckets deleted before the conversion, a lost answer read as applied; each frozen delete's new check dropped; DELETED rows or index rows listed), 10 in the engine (the id refusal, each count check and both, the fence before the first put, --all stopping at a refusal, the leftover kept, the re-read skipped, the sync skip, the dedupe, the loss token), 3 on the wire (a run with refusals exiting 0, the <ARN>/--all group optional, the wait skipped): 45, all killed.
• Memory and time of the command (design section 7, the manual and the admin guide state it): an in-catalog backup of 1,000 and of 100,000 items of the job's measurement shape (a composite key, two GSIs, a 200-byte pad; on Cassandra without GSIs), ru_maxrss of the command:

Backend	1,000 items	100,000 items	100,000 items, duration
SQLite	54.8 MiB	121.9 MiB	7.6 s
PostgreSQL	55.2 MiB	55.2 MiB	8.6 s
MongoDB	56.1 MiB	56.1 MiB	12.4 s
Cassandra	55.2 MiB	55.2 MiB	10.6 s

SQLite's excess is the command's connections' page cache (cache_size, 32 MB each) over a 172 MB database file. With the two GSIs, the Cassandra node's asynchronous GSI queue over the 100,000 writes left enough tombstones in gsi_pending that it timed out before the measurement, so that run has no GSI; the read path does not read them.

Acceptance lines of #821 (the design's PR 10 acceptance list) and where each is tested:

Line	Tests
migrate-legacy on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone	engine an_in_catalog_backup_is_copied_under_its_arn_and_replaced_by_its_row; the backends' replacement cells (no non-DELETED row, no items, definition, collection or buckets); wire test_migrate_legacy_carries_in_catalog_backups_... (SQLite, PostgreSQL, MongoDB; the in-catalog rows read on SQLite and PostgreSQL)
migrate-legacy refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason	engine each_refusal_before_the_copy_... and a_refused_item_deletes_the_copy_...; storage a_copy_refuses_what_the_legacy_restore_refuses_and_an_unknown_vector_state and SQLite's stream cell (the definition row removed: Unknown); wire test_migrate_legacy_refuses_... (the 13-digit id on SQLite and PostgreSQL, #618 on all three)
a second maintenance command while one holds the claim fails at once	wire test_a_second_maintenance_command_fails_at_once (PostgreSQL, both ways)
on SQLite every writing command refuses to run beside a server	wire, the serve-lock cell (exit 6)
PR 10 paragraph: the re-read and count check before the row	engine a_read_short_..., a_copy_that_does_not_read_back_...
4(c): the fence before the first put	engine migrate_legacy_on_a_root_another_catalog_adopted_writes_nothing
transitions: legacy rows replaced by an AVAILABLE row	the backends' replacement cells (the row unowned, completed_at set)
4(g): sync skips a prefix an in-catalog backup holds	engine sync_skips_a_prefix_...; SIGKILL after the manifest, then backup sync (SQLite, PostgreSQL)

Cassandra has no wire leg (no pytest fixture runs it); its share is the direct cells above and the memory run.

Not in this PR

• Backup/restore: legacy reader removal #749: removing the in-catalog reader (PR 14), whose refusal counts the rows that are not DELETED, as revision item 37 records.
• Carrying a backup with a 13-digit id (every 0.1.x backup) into the store format: refused by design, since the format's ids carry their creation instant; the way out is a restore and a new backup.
• A Cassandra wire leg (direct tests only).

Breaking changes

None. backup sync skips a prefix whose ARN an in-catalog backup holds, where it could adopt it before; no release wrote such a prefix. ListBackups lists an ARN both readers hold once. On PostgreSQL, MongoDB and Cassandra an in-catalog DeleteBackup that read the backup before a replacement committed answers BackupNotFoundException, where it could have answered DELETED for a backup that lives on in the store.

DCO

Signed-off-by: Scott Robinson robinnsc@amazon.com