From f702c8d23dfe76f40dba0f2954fb594803dc6f9f Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Fri, 6 Nov 2026 20:34:14 +0000 Subject: [PATCH] feat(backup): backup migrate-legacy `extenddb backup migrate-legacy ` or `--all` (exactly one) copies in-catalog backups into the configured store, each under its own ARN, backup id, name and creation time, and replaces each by its catalog row, AVAILABLE and unowned as a row `backup sync` inserts. `` is matched as written and rebuilt in this server's region; `--all` takes every in-catalog backup of every account that is not DELETED, oldest first, and goes on past one it does not migrate. The command needs the configured store healthy, opens the catalog as `serve` does (on SQLite under the serve lock, so the server is stopped for it), takes the maintenance claim on every backend for its whole run and waits as `purge` does, so every backup and restore job of the catalog answers LimitExceededException until it ends. Per backup, in this order: an id with no creation instant is refused before anything is read (that is every backup a 0.1.x release took: a store prefix carries only the 14-digit shape); then a table name the store-prefix rules refuse; a catalog row under the ARN (the configured store's: an earlier run's, finished; another store's: refused); CREATING and DELETING are skipped; a backup the backend cannot read (MongoDB's with no item collection), a name the manifest refuses, what the legacy restore refuses (a vector index, a multi-part key) and a SQLite backup taken before catalog 0.0.4 are refused. Then the copy: objects under the prefix with no row are an earlier run's and are deleted through the deletion fence first; the fence runs again before the first put; the items stream through the backup job's writer, each checked with the item validator under this instance's limits; the manifest is written last with `snapshot.kind` `legacy_in_catalog`, no marker, `unordered`, and `source_backend` and `source_extenddb_version` naming this binary. The prefix is read back through `verify_prefix`, sync's check, and the item count is checked against the in-catalog backup's twice. A refused item, a short read or a copy that does not read back deletes the copy through the fence and keeps the in-catalog backup. A lost claim, a signal, an unhealthy store, a failed listing or catalog call, and a replacement whose answer did not come stop the run at once with exit 1; a signal is acted on between backups and between data files. Exit 0 only when every backup reached is in the store. Each migration is logged under `extenddb::audit`. The manifest's source table is the table the legacy restore creates (`source_table_from_legacy`, pinned by an exhaustive property against `restore_input` and `restore_input_from_manifest`), with the recorded table class, TTL and stream and the SSE description of the recorded specification, so a `KMSMasterKeyId` is not carried over. The vector state is `[]` on PostgreSQL and SQLite and `null` on MongoDB and Cassandra. The replacement is one step per backend, `LegacyBackupReader::replace_with_index_row`, in place of an insert followed by the reader's delete, which let a DeleteBackup between the two answer DELETED for a backup that then came back, and which cannot work on Cassandra, where both rows share one key. PostgreSQL: one transaction (the row `FOR UPDATE`, the CREATING-target check, the insert, the items and definition deleted, DELETED). SQLite: the same under the write lock and `BEGIN IMMEDIATE`. MongoDB: one session transaction that inserts the row and marks the document DELETING with `migrated_to_store`, then the collection's drop and DELETED; a second call finishes one that stopped. Cassandra: the row converted in place by a lightweight transaction `IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null`, then its list and status entries and the item buckets' deletion; an unknown outcome is settled by a serial read, and a second call finishes a conversion that stopped after the row. The answers are `Replaced`, `AlreadyReplaced`, `NotAvailable` and `InUse`. Three frozen deletes change so a DeleteBackup orders wholly before or after a replacement: PostgreSQL's reads the status under its row lock, MongoDB's reads a `migrated_to_store` document as not found, Cassandra's claim carries `store_kind = null`. `LegacyBackupReader` also gains `items(account, arn)`, a frozen copy of each restore's read as a stream (PostgreSQL a REPEATABLE READ catalog transaction held for the copy, SQLite keyset batches each re-checking AVAILABLE, MongoDB a cursor with `batch_size` and a frozen decode, Cassandra each bucket paged 8 rows at a time), and `list_all()`. The core format gains `SnapshotDescriptor::KIND_LEGACY_IN_CATALOG`; no format version or fixture byte changes. The writer runs under a `ClaimWatch`, the job's heartbeat or the maintenance claim, whose loss now cancels a token; the job's tests run unchanged. `backup sync` skips a prefix whose ARN an in-catalog backup holds, under either spelling, naming `migrate-legacy `, and ListBackups lists an ARN that both a catalog row and the in-catalog reader hold once, with the row's summary. Tests: per-backend cells for the stream (batch and bucket boundaries, a pinned snapshot, a delete ending it), `list_all` across accounts and the replacement's four answers, a racing DeleteBackup and Cassandra's lost answer and stop; 16 engine cells on `Fake` and `MemStore` (each refusal, the fence, the counts, a copy that does not read back, every rerun rule, a stop, the sync skip, the dedupe); wire cells on SQLite, PostgreSQL and MongoDB (two in-catalog backups migrated, described, listed, restored and deleted under their ARNs with the in-catalog rows gone; `--all` past a #618 backup and a 13-digit one, both kept; the claim against a second command, and the serve lock); SIGKILL at the three kill points on SQLite and PostgreSQL, each rerun finishing. 45 mutants of the new tests, all killed. Peak RSS 55 to 56 MiB for 1,000 and for 100,000 items on PostgreSQL, MongoDB and Cassandra debug builds, and 122 MiB on SQLite, its page cache. Docs: the backup manual's legacy section (the command, its cost, what it refuses and the way out, what a migrated backup reports, a stopped run, Cassandra's recurring step, memory); the upgrade manual's store, 0.1.x and rollback paragraphs; the admin guide's paragraph and sample output; the differences doc's two rows. Design revision item 37, with sections 3, 4(c), 4(f), 4(g) and 7, the PR 10 paragraph and transitions, the PR 14 lines (rows that are not DELETED) and section 10's row edited with it. Not here: removing the in-catalog reader (#749); a 13-digit backup carried into the store format; a Cassandra wire leg (direct tests only). Co-authored-by: Lee Signed-off-by: Scott Robinson --- crates/app/src/cmd_backup.rs | 43 +- crates/app/src/cmd_backup_migrate.rs | 224 +++++ crates/app/src/lib.rs | 4 +- crates/core/src/backup_format/manifest.rs | 37 +- crates/core/src/backup_format/mod.rs | 2 +- crates/engine/src/backup.rs | 11 +- crates/engine/src/backup/job.rs | 6 +- crates/engine/src/backup/job/writer.rs | 58 +- crates/engine/src/backup/maintenance.rs | 19 +- crates/engine/src/backup/migrate_legacy.rs | 827 +++++++++++++++++ crates/engine/src/backup/sync.rs | 59 +- crates/engine/src/backup/tests.rs | 182 +++- .../engine/src/backup/tests/migrate_legacy.rs | 852 ++++++++++++++++++ crates/engine/src/lib.rs | 4 +- .../storage-cassandra/src/backup_catalog.rs | 197 +++- crates/storage-cassandra/src/backup_engine.rs | 280 +++++- crates/storage-cassandra/src/test_hooks.rs | 59 ++ .../tests/direct/backup_engine.rs | 525 +++++++++++ crates/storage-mongodb/src/backup_catalog.rs | 36 +- crates/storage-mongodb/src/backup_engine.rs | 325 ++++++- .../storage-mongodb/src/backup_live_tests.rs | 452 ++++++++++ crates/storage-postgres/src/backup_catalog.rs | 112 ++- crates/storage-postgres/src/backup_engine.rs | 229 ++++- .../storage-postgres/tests/backup_restore.rs | 473 ++++++++++ crates/storage-sqlite/src/backup.rs | 640 ++++++++++++- crates/storage-sqlite/src/backup_catalog.rs | 112 ++- crates/storage/src/backup_catalog.rs | 146 ++- crates/storage/src/backup_definition.rs | 432 ++++++++- docs/design/15-backup-restore.md | 22 +- docs/differences-from-dynamodb.md | 4 +- docs/manuals/05-admin-guide.md | 15 +- docs/manuals/07-upgrade-manual.md | 6 +- docs/manuals/14-backup-and-restore.md | 109 ++- tests/test_backup_job.py | 395 +++++++- tests/test_backup_store_config.py | 7 +- 35 files changed, 6692 insertions(+), 212 deletions(-) create mode 100644 crates/app/src/cmd_backup_migrate.rs create mode 100644 crates/engine/src/backup/migrate_legacy.rs create mode 100644 crates/engine/src/backup/tests/migrate_legacy.rs diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs index ed35a7aaf..17ef76c01 100644 --- a/crates/app/src/cmd_backup.rs +++ b/crates/app/src/cmd_backup.rs @@ -87,9 +87,15 @@ //! backends purge runs beside live servers; the maintenance claim is the //! exclusion. //! +//! `backup migrate-legacy |--all` copies in-catalog backups into the +//! configured store and replaces each by its catalog row, under the +//! maintenance claim on every backend (on SQLite under the serve lock as +//! well); its module (`cmd_backup_migrate.rs`) states its output and exit. +//! //! Every command opens the catalog through the backend's factory, under its own //! name (`extenddb backup list`, `extenddb backup purge`, `extenddb backup -//! remove-target`, `extenddb backup sync`), and none runs the startup recovery +//! remove-target`, `extenddb backup sync`, `extenddb backup migrate-legacy`), +//! and none runs the startup recovery //! a server runs (`ServerComponentsOptions::startup_recovery` off): that work //! belongs to the servers the command runs beside, and a command that ran it //! could exit with it half done. On PostgreSQL, MongoDB and Cassandra the @@ -220,6 +226,9 @@ use extenddb_storage::error::StorageError; use extenddb_storage::hooks::ServingRegistrar; use futures::TryStreamExt; +#[path = "cmd_backup_migrate.rs"] +mod migrate_legacy; + #[derive(Args)] #[allow(clippy::doc_markdown)] // Clap help text, not rustdoc pub struct BackupArgs { @@ -312,6 +321,21 @@ enum BackupCommand { )] forget_backup: Vec, }, + /// Copy in-catalog backups into the configured backup store, each under its own ARN, and + /// replace each by its catalog row. Takes the maintenance claim first (every backup and + /// restore job answers LimitExceededException until it ends) and waits for running jobs; on + /// SQLite it needs the server stopped. Refuses a backup a 0.1.x release took (its id carries + /// no instant) and one with an item or a definition the store format cannot hold, keeping its + /// in-catalog backup; restore such a backup and take a new one. + #[command(group(clap::ArgGroup::new("backups").required(true).args(["arn", "all"])))] + MigrateLegacy { + /// The backup ARN of the in-catalog backup to migrate + #[arg(value_name = "ARN")] + arn: Option, + /// Migrate every in-catalog backup of every account, going on past a backup it refuses + #[arg(long)] + all: bool, + }, } /// How often the waiting purge re-reads the rows. @@ -325,6 +349,7 @@ const LIST: &str = "extenddb backup list"; const PURGE: &str = "extenddb backup purge"; const REMOVE_TARGET: &str = "extenddb backup remove-target"; const SYNC: &str = "extenddb backup sync"; +const MIGRATE_LEGACY: &str = "extenddb backup migrate-legacy"; pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> { if !std::path::Path::new(&args.config).exists() { @@ -391,6 +416,7 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> BackupCommand::Purge { confirm: true } => (PURGE, Mode::Write), BackupCommand::Sync { .. } if forgetting => (SYNC, Mode::Forget), BackupCommand::Sync { .. } => (SYNC, Mode::Sync), + BackupCommand::MigrateLegacy { .. } => (MIGRATE_LEGACY, Mode::Write), BackupCommand::RemoveTarget { .. } => unreachable!("handled above"), }; let opening = open( @@ -500,6 +526,21 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> done } (BackupCommand::Sync { .. }, Opening::NoStore(reason)) => anyhow::bail!("{reason}"), + (BackupCommand::MigrateLegacy { arn, .. }, Opening::Store(opened)) => { + let done = migrate_legacy::migrate_legacy( + &opened, + &app_config, + release, + arn.as_deref(), + interrupt, + ) + .await; + opened.close().await; + done + } + (BackupCommand::MigrateLegacy { .. }, Opening::NoStore(reason)) => { + anyhow::bail!("{reason}; backup migrate-legacy copies into the configured store") + } (_, Opening::CatalogOnly(catalog, reason)) => { catalog.close().await; anyhow::bail!("{reason}") diff --git a/crates/app/src/cmd_backup_migrate.rs b/crates/app/src/cmd_backup_migrate.rs new file mode 100644 index 000000000..b7e11559c --- /dev/null +++ b/crates/app/src/cmd_backup_migrate.rs @@ -0,0 +1,224 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! `extenddb backup migrate-legacy |--all`: the in-catalog backups +//! copied into the configured store and replaced there by their catalog +//! rows, under their own ARNs (`extenddb_engine::migrate_legacy` states +//! each step and every refusal). +//! +//! The command opens the catalog as `serve` does (on SQLite under the serve +//! lock, so the server is stopped for it) and needs the configured store +//! healthy, its identity known. It takes the maintenance claim on every +//! backend and heartbeats it for its whole run, so every backup and restore +//! job of the catalog answers `LimitExceededException` until it ends (a +//! second maintenance command fails at once), and it waits, as `purge` +//! does, until no CREATING or FAILED backup row of the store has a live +//! owner and no restore job is RUNNING (a legacy restore of a backup about +//! to be replaced among them). On SQLite it first runs the reconciler's +//! dead-owner resolutions itself and waits with a zero window, every owner +//! being dead under the serve lock. +//! +//! `` names one backup, as written or rebuilt in this server's region +//! (the spellings DescribeBackup resolves); `--all` takes every in-catalog +//! backup that is not DELETED, of every account, in creation order, and +//! goes on past a backup it refuses. Each backup gets one line: `migrated`, +//! `finished` (an earlier run had written its row), `refused` (with the +//! reason and the way out; its in-catalog backup is kept), `skipped` (not +//! AVAILABLE, or deleted meanwhile) or `failed` (a store or catalog call; +//! a later run tries again), then a summary. Each migration is logged under +//! `extenddb::audit` with the account and the ARN. A first `SIGINT` or +//! `SIGTERM` stops the run before the next backup or the next data file +//! (the copy in progress is deleted), and a lost claim, a store the writer +//! finds unhealthy or fenced, or a catalog call that fails stops it at +//! once; the claim is then released and the command exits 1. It exits 0 +//! when every backup it reached is in the store, and 1 otherwise, after the +//! others ran. + +use std::time::Duration; + +use extenddb_core::validation::RestorableLimits; +use extenddb_engine::HeldClaim; +use extenddb_engine::maintenance::Interrupt; +use extenddb_engine::migrate_legacy::{BackupOutcome, Migration, Outcome, Selection, migrate}; +use extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL; +use extenddb_storage::error::StorageError; + +use super::{Identity, Opened, config, release, wait_for_jobs}; + +/// `backup migrate-legacy` (module doc). `arn` is `None` for `--all`; +/// `version` is this binary's, which the manifests name as their writer. +pub(super) async fn migrate_legacy( + opened: &Opened, + app_config: &config::AppConfig, + version: &str, + arn: Option<&str>, + mut interrupt: Interrupt, +) -> anyhow::Result<()> { + let storage = &*opened.catalog.storage; + let configured = opened.store(); + println!("=== extenddb backup migrate-legacy ==="); + let identity = match &opened.identity { + Identity::Known(identity) => { + println!( + "Store: {} {} (store id {})", + identity.store_kind, identity.store_root, identity.store_id + ); + identity.clone() + } + Identity::Empty => anyhow::bail!( + "the {} backup store {} has no {}: its check did not run", + configured.kind, + configured.root, + extenddb_engine::STORE_IDENTITY_KEY + ), + Identity::Problem(problem) => anyhow::bail!( + "the {} backup store {} is unhealthy: {problem}; nothing is migrated into a store \ + whose check fails", + configured.kind, + configured.root + ), + }; + + let backup = &app_config.backup; + let stale_after = Duration::from_secs(backup.stale_job_after_secs); + let holder = format!( + "extenddb backup migrate-legacy on {} (pid {})", + extenddb_server::backup_store::hostname(), + std::process::id() + ); + let taken = storage + .maintenance_claim(&holder, stale_after) + .await + .map_err(|e| match e { + StorageError::AlreadyExists(m) => anyhow::anyhow!( + "{m}; one maintenance command runs at a time, and a claim whose holder died \ + expires after stale_job_after_secs ({} s)", + backup.stale_job_after_secs + ), + other => anyhow::anyhow!("cannot take the maintenance claim: {other}"), + })?; + println!( + "Took the maintenance claim as {holder:?}: until the command ends every backup and \ + restore job of this catalog answers LimitExceededException." + ); + let claim = HeldClaim::start(taken, JOB_HEARTBEAT_INTERVAL); + if let Err(refusal) = wait_for_jobs( + &opened.catalog, + std::slice::from_ref(&identity), + &claim, + &mut interrupt, + stale_after, + false, + ) + .await + { + release(claim).await; + return Err(refusal); + } + + let migration = Migration { + configured, + identity: &identity, + claim: &claim, + interrupt: &interrupt, + limits: RestorableLimits::from(&app_config.limits), + data_file_target_bytes: backup.data_file_target_bytes, + region: &app_config.server.region, + source_backend: &opened.catalog.backend, + source_extenddb_version: version, + metrics: None, + }; + let selection = arn.map_or(Selection::All, Selection::Arn); + let report = migrate(storage, &migration, selection, print_outcome).await; + release(claim).await; + + let migrated = report.count(|o| matches!(o, Outcome::Migrated { .. })); + let finished = report.count(|o| matches!(o, Outcome::Finished | Outcome::InStore)); + let refused = report.count(|o| matches!(o, Outcome::Refused(_))); + let skipped = report.count(|o| matches!(o, Outcome::Skipped(_))); + let failed = report.count(|o| matches!(o, Outcome::Failed(_))); + println!(); + if report.backups.is_empty() && report.stopped.is_none() { + println!("No in-catalog backup to migrate."); + } + println!( + "=== extenddb backup migrate-legacy {}: {migrated} migrated, {finished} finished, \ + {refused} refused, {skipped} skipped, {failed} failed ===", + if report.stopped.is_some() { + "stopped" + } else { + "complete" + } + ); + if let Some(reason) = &report.stopped { + anyhow::bail!( + "stopped before the remaining backups: {reason}. The maintenance claim is released; \ + run the command again for the rest" + ); + } + if !report.complete() { + anyhow::bail!( + "{} backup(s) were not migrated; see above", + refused + skipped + failed + ); + } + Ok(()) +} + +/// One backup's lines, and the audit line of a migration. +fn print_outcome(done: &BackupOutcome) { + let BackupOutcome { + account_id, + backup_arn, + outcome, + notes, + } = done; + match outcome { + Outcome::Migrated { + item_count, + size_bytes, + } => { + tracing::info!( + target: "extenddb::audit", + action = "backup migrate-legacy", + account_id = %account_id, + backup_arn = %backup_arn, + "copied an in-catalog backup into the store and replaced it by its catalog row" + ); + println!( + "migrated {backup_arn} (account {account_id}): {item_count} items, {size_bytes} \ + bytes" + ); + } + Outcome::Finished => { + tracing::info!( + target: "extenddb::audit", + action = "backup migrate-legacy", + account_id = %account_id, + backup_arn = %backup_arn, + "removed what was left of an in-catalog backup whose catalog row was written" + ); + println!( + "finished {backup_arn} (account {account_id}): an earlier run had written its \ + row; what was left of the in-catalog backup is removed" + ); + } + Outcome::InStore => println!( + "{backup_arn} (account {account_id}) is already in the store: nothing to migrate" + ), + Outcome::Refused(reason) => println!( + "refused {backup_arn} (account {account_id}): {reason}. Its in-catalog backup is \ + kept" + ), + Outcome::Skipped(reason) => { + println!("skipped {backup_arn} (account {account_id}): {reason}") + } + Outcome::Failed(reason) => println!( + "failed {backup_arn} (account {account_id}): {reason}. Its in-catalog backup is \ + kept; a later run tries again" + ), + } + for note in notes { + println!(" {note}"); + } +} diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index 6dfcc54c0..c7667c423 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -69,8 +69,8 @@ enum Command { Init(cmd_init::InitArgs), /// Tear down a extenddb deployment (the databases; the backup store is left untouched) Destroy(cmd_destroy::DestroyArgs), - /// List, purge or sync the backups of the configured backup store, or remove a restore - /// target + /// List, purge or sync the backups of the configured backup store, migrate in-catalog + /// backups into it, or remove a restore target Backup(cmd_backup::BackupArgs), /// Validate a extenddb deployment Verify(cmd_verify::VerifyArgs), diff --git a/crates/core/src/backup_format/manifest.rs b/crates/core/src/backup_format/manifest.rs index 5afa2afd9..158402ea6 100644 --- a/crates/core/src/backup_format/manifest.rs +++ b/crates/core/src/backup_format/manifest.rs @@ -130,6 +130,14 @@ impl SnapshotDescriptor { /// MongoDB: a session with snapshot read concern; the marker is the /// cluster time `:`. pub const KIND_MONGODB_SNAPSHOT_READ_CONCERN: &'static str = "mongodb_snapshot_read_concern"; + /// A copy of an in-catalog backup, written by `extenddb backup + /// migrate-legacy`; the marker is empty. The items are those the + /// in-catalog backup stored when it was taken, so the snapshot is the + /// one its legacy create took. For this kind `source_backend` and + /// `source_extenddb_version` name the backend and the release that + /// wrote the copy, not the release that took the in-catalog backup, + /// whose version was never recorded. + pub const KIND_LEGACY_IN_CATALOG: &'static str = "legacy_in_catalog"; } /// Order of items within and across a backup's data files. @@ -1456,8 +1464,10 @@ pub fn check_format_version(found: u32) -> Result<(), FormatError> { /// only copy of the alphabet: the engine's `validate_name_member` /// (`BackupName` on `CreateBackup`) and `validate_index_name` spell it again, /// and the engine counts the bounds in characters where this counts bytes -/// (the same for this ASCII alphabet). -fn is_valid_backup_name(name: &str) -> bool { +/// (the same for this ASCII alphabet). `extenddb backup migrate-legacy` +/// checks an in-catalog backup's name with it before it writes the copy. +#[must_use] +pub fn is_valid_backup_name(name: &str) -> bool { (3..=255).contains(&name.len()) && crate::validation::validate_table_name_chars(name).is_ok() } @@ -1853,6 +1863,29 @@ mod tests { assert_eq!(parsed, manifest); } + #[test] + fn a_manifest_of_the_legacy_in_catalog_kind_parses_and_round_trips() { + assert_eq!( + SnapshotDescriptor::KIND_LEGACY_IN_CATALOG, + "legacy_in_catalog" + ); + let mut manifest = sample_manifest(); + manifest.snapshot = SnapshotDescriptor { + kind: SnapshotDescriptor::KIND_LEGACY_IN_CATALOG.to_owned(), + marker: String::new(), + }; + let bytes = manifest.to_json_bytes().unwrap(); + let text = std::str::from_utf8(&bytes).unwrap(); + assert!( + text.contains(r#""snapshot":{"kind":"legacy_in_catalog","marker":""}"#), + "the kind is written as named: {text}" + ); + let parsed = ExtenddbManifest::from_json_bytes(&bytes).unwrap(); + assert_eq!(parsed, manifest); + assert_eq!(parsed.snapshot.kind, "legacy_in_catalog"); + assert_eq!(parsed.snapshot.marker, ""); + } + #[test] fn snapshot_descriptor_serde_round_trip() { let snapshot = SnapshotDescriptor { diff --git a/crates/core/src/backup_format/mod.rs b/crates/core/src/backup_format/mod.rs index ed2e02e5b..d391468bd 100644 --- a/crates/core/src/backup_format/mod.rs +++ b/crates/core/src/backup_format/mod.rs @@ -94,5 +94,5 @@ pub use layout::{ pub use manifest::{ CURRENT_FORMAT_VERSION, DataFileEntry, DataFileReader, ExtenddbManifest, ItemOrder, KeyRange, KeyRangeCheck, KeyRangeTracker, SnapshotDescriptor, SourceTableSchema, check_format_version, - verify_manifest_checksum, + is_valid_backup_name, verify_manifest_checksum, }; diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs index 41e820b75..9686b9750 100755 --- a/crates/engine/src/backup.rs +++ b/crates/engine/src/backup.rs @@ -936,11 +936,19 @@ async fn list_backups( .try_collect() .await .map_err(storage_err_to_dynamo)?; + // A backup both readers list (an in-catalog backup whose copy a `backup + // sync` adopted before `backup migrate-legacy` replaced it, or a + // replacement an earlier build left half done) is listed once, with the + // catalog row's summary. + let indexed: std::collections::HashSet = + summaries.iter().map(|s| s.backup_arn.clone()).collect(); summaries.extend( storage .list(account_id, table_name) .await - .map_err(storage_err_to_dynamo)?, + .map_err(storage_err_to_dynamo)? + .into_iter() + .filter(|s| !indexed.contains(&s.backup_arn)), ); let start = start.map(|cursor| cursor.resolve(&summaries)).transpose()?; let summaries = summaries.into_iter().filter(|s| filter.keeps(s)).collect(); @@ -1992,6 +2000,7 @@ fn storage_err_to_dynamo(e: StorageError) -> DynamoDbError { mod job; pub mod maintenance; +pub mod migrate_legacy; mod overrides; mod pitr; pub mod reconciler; diff --git a/crates/engine/src/backup/job.rs b/crates/engine/src/backup/job.rs index f0ee2bda0..35b8e5e30 100644 --- a/crates/engine/src/backup/job.rs +++ b/crates/engine/src/backup/job.rs @@ -644,7 +644,11 @@ async fn write_documents( /// One document's put under the backup prefix, after the pre-put check; /// returns the bytes the store reported. -pub(crate) async fn put_document(p: &Pipeline<'_>, name: &str, bytes: &[u8]) -> Result { +pub(crate) async fn put_document( + p: &Pipeline<'_, W>, + name: &str, + bytes: &[u8], +) -> Result { if p.shutdown.is_cancelled() { return Err(Stop::Cancelled); } diff --git a/crates/engine/src/backup/job/writer.rs b/crates/engine/src/backup/job/writer.rs index ba479a168..0556042c0 100644 --- a/crates/engine/src/backup/job/writer.rs +++ b/crates/engine/src/backup/job/writer.rs @@ -40,9 +40,11 @@ //! number rule over every member); an item within the default is validated //! on the feeder's own task. //! -//! Before every `put` the uploader checks the heartbeat -//! ([`HeartbeatHandle::confirms_the_claim`]) and starts none otherwise; a -//! lost claim or the shutdown token ends the pipeline at its next await. +//! Before every `put` the uploader checks the claim the writer runs under +//! ([`ClaimWatch::confirms_the_claim`]: the job's heartbeat, +//! [`HeartbeatHandle`], or for `backup migrate-legacy` the maintenance +//! claim's) and starts none otherwise; a lost claim or the shutdown token +//! ends the pipeline at its next await. //! When the uploader fails on its own (a put refused, an encoder error) it //! cancels a token the feeder selects on, so the feeder stops reading at //! once rather than at the stream's next item and the FAILED write does not @@ -128,8 +130,34 @@ pub(crate) struct DataFiles { pub bytes_uploaded: u64, } +/// The claim the pipeline writes under, as the uploader and the feeder read +/// it: the backup job's heartbeat ([`HeartbeatHandle`]), or the maintenance +/// claim `backup migrate-legacy` holds. +pub(crate) trait ClaimWatch: Send + Sync { + /// The pre-put check: `true` while the claim is confirmed held. + fn confirms_the_claim(&self) -> bool; + /// Cancelled once the claim is known to be lost. + fn lost(&self) -> &CancellationToken; + /// Why the claim is lost, once it is. + fn reason(&self) -> String; +} + +impl ClaimWatch for HeartbeatHandle { + fn confirms_the_claim(&self) -> bool { + HeartbeatHandle::confirms_the_claim(self) + } + + fn lost(&self) -> &CancellationToken { + HeartbeatHandle::lost(self) + } + + fn reason(&self) -> String { + HeartbeatHandle::reason(self) + } +} + /// What the uploader needs from the job. -pub(crate) struct Pipeline<'a> { +pub(crate) struct Pipeline<'a, W: ?Sized = HeartbeatHandle> { pub store: &'a dyn BackupStore, /// The store as the runtime holds it: its health and identity, read /// before every put. @@ -140,7 +168,8 @@ pub(crate) struct Pipeline<'a> { pub prefix: &'a str, /// The ARN, for the log lines. pub arn: &'a str, - pub heartbeat: &'a HeartbeatHandle, + /// The claim every put is checked against. + pub heartbeat: &'a W, pub shutdown: &'a CancellationToken, pub metrics: Option<&'a MetricsCollector>, /// `data_file_target_bytes`. @@ -151,7 +180,7 @@ pub(crate) struct Pipeline<'a> { pub bucket: &'a str, } -impl Pipeline<'_> { +impl Pipeline<'_, W> { /// The check before every put: the store is healthy as the runtime holds /// it (the reconciler's check sets it) and its identity is the one the /// row records. A store marked unhealthy mid-job, or whose identity file @@ -189,6 +218,9 @@ impl Pipeline<'_> { } } +/// The words between the count and the first item in [`refusal_reason`]. +const REFUSED_ITEMS: &str = " item(s) cannot be restored; first: "; + /// The reason the FAILED row carries for refused items (module doc). fn refusal_reason(count: u64, first: &RestorableItemError) -> String { let issue = if matches!(first.reason, RestorableItemReason::InvalidNumber(_)) { @@ -196,7 +228,15 @@ fn refusal_reason(count: u64, first: &RestorableItemError) -> String { } else { "" }; - format!("{count} item(s) cannot be restored; first: {first}{issue}") + format!("{count}{REFUSED_ITEMS}{first}{issue}") +} + +/// Whether a [`Stop::Fail`] reason is the validator's refusal of items +/// ([`refusal_reason`]) rather than a store or encoder failure. +pub(crate) fn is_item_refusal(reason: &str) -> bool { + reason + .split_once(REFUSED_ITEMS) + .is_some_and(|(count, _)| !count.is_empty() && count.bytes().all(|b| b.is_ascii_digit())) } /// What the feeder found by the end of the stream. @@ -482,8 +522,8 @@ fn close( } /// The pipeline (module doc). `tracker` is `Some` for a key-ordered backup. -pub(crate) async fn write_data_files( - p: &Pipeline<'_>, +pub(crate) async fn write_data_files( + p: &Pipeline<'_, W>, items: BoxStream<'static, Result>, key: RestorableKeySchema, limits: RestorableLimits, diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs index 3889348bf..0b8bfbc03 100644 --- a/crates/engine/src/backup/maintenance.rs +++ b/crates/engine/src/backup/maintenance.rs @@ -54,7 +54,7 @@ use extenddb_storage::backup_catalog::{ StaleClock, StoreIdentity, is_backup_not_found, }; use extenddb_storage::error::StorageError; -use extenddb_storage::{BackupCatalog, MaintenanceClaim}; +use extenddb_storage::{BackupCatalog, CancellationToken, MaintenanceClaim}; use futures::TryStreamExt; use tokio::sync::watch; use tokio::time::Instant; @@ -65,6 +65,8 @@ use super::StoreFence; pub struct HeldClaim { claim: Arc, lost: Arc>>, + /// Cancelled when `lost` is set, for a writer that selects on the loss. + lost_signal: CancellationToken, timer: tokio::task::JoinHandle<()>, } @@ -77,9 +79,11 @@ impl HeldClaim { pub fn start(claim: Box, interval: Duration) -> Self { let claim: Arc = Arc::from(claim); let lost = Arc::new(Mutex::new(None)); + let lost_signal = CancellationToken::new(); let timer = tokio::spawn({ let claim = Arc::clone(&claim); let lost = Arc::clone(&lost); + let lost_signal = lost_signal.clone(); async move { let mut ticks = tokio::time::interval(interval); ticks.tick().await; @@ -94,11 +98,22 @@ impl HeldClaim { Err(e) => format!("the maintenance claim's heartbeat failed: {e}"), }; *lost.lock().unwrap_or_else(PoisonError::into_inner) = Some(reason); + lost_signal.cancel(); return; } } }); - Self { claim, lost, timer } + Self { + claim, + lost, + lost_signal, + timer, + } + } + + /// Cancelled once [`Self::lost`] reports the claim lost. + pub(crate) fn lost_signal(&self) -> &CancellationToken { + &self.lost_signal } /// The claim's fencing token. diff --git a/crates/engine/src/backup/migrate_legacy.rs b/crates/engine/src/backup/migrate_legacy.rs new file mode 100644 index 000000000..cd6478777 --- /dev/null +++ b/crates/engine/src/backup/migrate_legacy.rs @@ -0,0 +1,827 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! `extenddb backup migrate-legacy |--all`: in-catalog backups copied +//! into the configured store, each under its own ARN, and replaced there by +//! a catalog index row. +//! +//! [`migrate`] runs under the maintenance claim the command holds +//! ([`HeldClaim`]) and takes the backups from +//! [`LegacyBackupReader::list_all`]: every one with `--all`, in creation +//! order, or the one whose ARN is `` as written or rebuilt in this +//! server's region, the spellings DescribeBackup resolves. Each backup goes +//! through these steps in this order, and every refusal before the first +//! put reads nothing more and writes nothing: +//! +//! 1. The id: one that carries no instant (the 13-digit and bare shapes +//! every 0.1.x release issued) is refused, since a store prefix carries +//! only the 14-digit shape. The way out is a restore of the backup and a +//! new backup of the restored table, which has a new ARN. +//! 2. The table name: one the store-prefix rules refuse (`BackupArn`: over +//! 255 bytes, or dots only) is refused. +//! 3. The status: a CREATING or DELETING in-catalog backup is skipped and +//! named, unless a catalog row of the configured store already holds its +//! ARN (an earlier run's replacement that stopped before its end on +//! MongoDB), which is finished. +//! 4. A catalog row under the ARN: one of the configured store is an earlier +//! run's (or a `backup sync` that adopted an earlier run's copy), and +//! [`LegacyBackupReader::replace_with_index_row`] with that row removes +//! what is left of the in-catalog backup ("finished"); one of another +//! store refuses the backup. +//! 5. The open ([`LegacyBackupReader::items`]): the backup's name, its +//! recorded source and its vector state. A backend that cannot read the +//! backup (MongoDB's without an item collection) refuses it. +//! 6. The name: one the manifest refuses ([`is_valid_backup_name`]) is +//! refused; the name survives with the ARN or not at all. +//! 7. The definition: [`source_table_from_legacy`], the table the legacy +//! restore creates; it refuses what the legacy restore refuses (vector +//! indexes, a multi-part base key) and a SQLite backup taken before +//! catalog 0.0.4, whose vector state is unknown. +//! +//! Then the copy. Objects under the backup's prefix with no catalog row +//! for it are what an earlier run left (it stopped before its +//! replacement): they are deleted through the deletion fence first, so +//! every copy is written whole by one run. The fence runs before the first +//! put as CreateBackup's does. The items stream through the backup job's +//! writer ([`writer::write_data_files`]), each checked with the item +//! validator under this instance's limits, into data files under the +//! backup's prefix; then the three documents, the claim check, and +//! `extenddb-manifest.json`, whose `snapshot.kind` is +//! [`SnapshotDescriptor::KIND_LEGACY_IN_CATALOG`] with no marker, whose +//! item order is `unordered`, and whose `source_backend` and +//! `source_extenddb_version` name this binary. The prefix is then read back +//! through [`verify_prefix`] (the check `backup sync` and the dead-owner +//! duty apply), and the manifest's item total must equal both the items +//! written and the count the in-catalog backup records. Only then is the +//! row written, by [`LegacyBackupReader::replace_with_index_row`]: AVAILABLE, +//! unowned (`owner_generation` 0, as `backup sync` writes one), in the +//! configured store's identity, with the manifest's name, size, item total +//! and `source_table`, and the id's instant as `created_at`. A refusal or +//! failure after the first put deletes the prefix through the fence and +//! keeps the in-catalog backup. +//! +//! A refusal is a property of one backup, or a race it lost (a DeleteBackup +//! or a restore that came first): it is reported and the next backup runs. +//! A lost claim, a signal, a store the writer finds unhealthy or fenced, a +//! failed listing and a catalog call that fails stop the run at once +//! ([`MigrateReport::stopped`]). A signal is acted on between backups and +//! inside the writer between data files; a copy stopped by one is deleted. +//! The replacement is not interrupted once the manifest is written. A run +//! that stops with a copy under a prefix and no row for it (a lost claim, +//! a crash, a replacement whose answer did not come) leaves the in-catalog +//! backup in place, and the next run deletes the copy and writes it again; +//! `backup sync` skips such a prefix while the in-catalog backup holds the +//! ARN. + +use extenddb_core::backup_format::{ + BackupArn, BackupDocuments, ItemOrder, ManifestHeader, SnapshotDescriptor, + backup_creation_date_time, is_valid_backup_name, +}; +use extenddb_core::metrics::MetricsCollector; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits}; +use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, LegacyBackupKey, Replaced, StoreIdentity, UNOWNED_GENERATION, + is_backup_not_found, +}; +use extenddb_storage::backup_definition::source_table_from_legacy; +use extenddb_storage::error::StorageError; +use extenddb_storage::util::ARN_PARTITION; +use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader}; +use futures::TryStreamExt; + +use super::BackupArnSpellings; +use super::job::writer::{self, ClaimWatch, Pipeline, Stop}; +use super::job::{KillPoint, kill_point, put_document}; +use super::maintenance::{HeldClaim, Interrupt}; +use super::reconciler::{StoreCalls, Verified, manifest_item_count, verify_prefix}; +use super::runtime::ConfiguredStore; + +/// The maintenance claim as the writer reads it: confirmed while no +/// heartbeat has found it lost. +impl ClaimWatch for HeldClaim { + fn confirms_the_claim(&self) -> bool { + self.lost().is_none() + } + + fn lost(&self) -> &CancellationToken { + self.lost_signal() + } + + fn reason(&self) -> String { + HeldClaim::lost(self).unwrap_or_else(|| "the maintenance claim is lost".to_owned()) + } +} + +/// What a migration runs with. +pub struct Migration<'a> { + /// The configured store, healthy. + pub configured: &'a ConfiguredStore, + /// The configured store's identity, which the rows are written under. + pub identity: &'a StoreIdentity, + /// The maintenance claim the command holds. + pub claim: &'a HeldClaim, + /// The command's signal. + pub interrupt: &'a Interrupt, + /// The limits the item validator holds every item to. + pub limits: RestorableLimits, + /// `data_file_target_bytes`. + pub data_file_target_bytes: u64, + /// The server's region: `` is also looked up rebuilt in it, and a + /// backup of another region is noted. + pub region: &'a str, + /// The backend and the release the manifests name as their writer. + pub source_backend: &'a str, + pub source_extenddb_version: &'a str, + /// The collector failed store calls are recorded in; `None` in a + /// command. + pub metrics: Option<&'a MetricsCollector>, +} + +/// Which backups to migrate. +#[derive(Debug, Clone, Copy)] +pub enum Selection<'a> { + /// The in-catalog backup this ARN names (as written, or rebuilt in the + /// server's region). + Arn(&'a str), + /// Every in-catalog backup that is not DELETED, of every account. + All, +} + +/// What happened to one backup. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Outcome { + /// Copied into the store and replaced by its row. + Migrated { item_count: u64, size_bytes: i64 }, + /// A catalog row of the configured store already held the ARN (an + /// earlier run's); what was left of the in-catalog backup is removed. + Finished, + /// `` names no in-catalog backup and a catalog row of the + /// configured store holds it: already in the store. + InStore, + /// Not migrated, for a reason of its own; the in-catalog backup is + /// kept. The text gives the reason and the way out. + Refused(String), + /// Not AVAILABLE, or no longer AVAILABLE when its row was written; the + /// text says which. + Skipped(String), + /// A store or catalog call for this backup failed; the in-catalog + /// backup is kept and a later run tries again. + Failed(String), +} + +/// One backup's outcome. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct BackupOutcome { + pub account_id: String, + /// The ARN as the in-catalog backup stores it. + pub backup_arn: String, + pub outcome: Outcome, + /// What else the operator is told about it: a copy an earlier run left + /// and this one deleted, a copy that could not be deleted, a region + /// other than the server's, a creation time that differs from the id's + /// instant. + pub notes: Vec, +} + +/// What a run did. +#[derive(Debug, Default)] +pub struct MigrateReport { + /// Each backup the run reached, in order. + pub backups: Vec, + /// Why the run stopped before the end, when it did. + pub stopped: Option, +} + +impl MigrateReport { + /// How many backups ended with an outcome `pick` keeps. + #[must_use] + pub fn count(&self, pick: impl Fn(&Outcome) -> bool) -> usize { + self.backups.iter().filter(|b| pick(&b.outcome)).count() + } + + /// Whether every backup the run reached is in the store (migrated, + /// finished or already there) and the run reached its end. + #[must_use] + pub fn complete(&self) -> bool { + self.stopped.is_none() + && self.backups.iter().all(|b| { + matches!( + b.outcome, + Outcome::Migrated { .. } | Outcome::Finished | Outcome::InStore + ) + }) + } +} + +/// The way out named for a backup this release cannot copy but can +/// restore. +const CARRY_FORWARD: &str = "restore it (RestoreTableFromBackup) and take a new backup of the \ + restored table, which has a new ARN"; + +/// One backup's end: an outcome, or a stop of the whole run. +enum Step { + Done(Outcome), + Stop(String), +} + +/// What the steps of one backup share. +struct Run<'m, 'a> { + m: &'m Migration<'a>, + /// Cancelled on the command's signal: the writer's shutdown token. + shutdown: CancellationToken, +} + +/// Migrate the backups `selection` names (module doc). `on_backup` is +/// called with each backup's outcome as it ends. +pub async fn migrate( + storage: &S, + m: &Migration<'_>, + selection: Selection<'_>, + mut on_backup: impl FnMut(&BackupOutcome), +) -> MigrateReport { + let shutdown = CancellationToken::new(); + let watcher = tokio::spawn({ + let mut interrupt = m.interrupt.clone(); + let shutdown = shutdown.clone(); + async move { + interrupt.wait().await; + shutdown.cancel(); + } + }); + let run = Run { m, shutdown }; + let mut report = MigrateReport::default(); + let keys = match storage.list_all().await { + Ok(keys) => keys, + Err(e) => { + watcher.abort(); + report.stopped = Some(format!("the in-catalog backups could not be listed: {e}")); + return report; + } + }; + let keys = match selection { + Selection::All => keys, + Selection::Arn(arn) => match named(storage, &run, arn, keys).await { + Ok(Named::Key(key)) => vec![key], + Ok(Named::Done(done)) => { + on_backup(&done); + report.backups.push(done); + watcher.abort(); + return report; + } + Err(stop) => { + watcher.abort(); + report.stopped = Some(stop); + return report; + } + }, + }; + for key in keys { + if let Some(reason) = m.claim.lost() { + report.stopped = Some(reason); + break; + } + if let Some(signal) = m.interrupt.signalled() { + report.stopped = Some(format!("{signal} received")); + break; + } + let mut notes = Vec::new(); + match migrate_one(storage, &run, &key, &mut notes).await { + Step::Done(outcome) => { + let done = BackupOutcome { + account_id: key.account_id, + backup_arn: key.backup_arn, + outcome, + notes, + }; + on_backup(&done); + report.backups.push(done); + } + Step::Stop(reason) => { + // What the backup's steps noted before the stop (a copy + // left) is the stop's to report. + if !notes.is_empty() { + let done = BackupOutcome { + account_id: key.account_id, + backup_arn: key.backup_arn, + outcome: Outcome::Failed(reason.clone()), + notes, + }; + on_backup(&done); + report.backups.push(done); + } + report.stopped = Some(reason); + break; + } + } + } + watcher.abort(); + report +} + +/// What `` resolved to. +enum Named { + /// The in-catalog backup it names. + Key(LegacyBackupKey), + /// Nothing to copy: the outcome. + Done(BackupOutcome), +} + +/// The in-catalog backup `arn` names among `keys` (the ARN as written, or +/// rebuilt in the server's region, of the ARN's account), or, when none +/// does, what holds the ARN in the catalog index. +async fn named( + storage: &S, + run: &Run<'_, '_>, + arn: &str, + keys: Vec, +) -> Result { + let parts = extenddb_storage::util::check_backup_arn(arn.trim()) + .map_err(|e| format!("{arn} is not a backup ARN: {e}"))?; + let spellings = BackupArnSpellings::new(&parts, run.m.region); + let account_id = parts.account_id.to_owned(); + for spelling in std::iter::once(spellings.canonical()).chain(spellings.as_written.as_deref()) { + if let Some(key) = keys + .iter() + .find(|k| k.account_id == account_id && k.backup_arn == spelling) + { + return Ok(Named::Key(key.clone())); + } + } + let mut done = BackupOutcome { + account_id, + backup_arn: spellings.canonical().to_owned(), + outcome: Outcome::Refused( + "no in-catalog backup and no catalog row has this ARN".to_owned(), + ), + notes: Vec::new(), + }; + let held = super::read_catalog_row(storage, &spellings) + .await + .map_err(|e| format!("the catalog could not be read: {e}"))?; + if let Some((spelling, row)) = held { + done.backup_arn = spelling.to_arn_string(); + done.outcome = if row.store == *run.m.identity { + // On Cassandra a replacement that stopped after its conversion + // left the item buckets; the call removes them. Elsewhere it + // finds nothing left. + match finish(storage, &row).await { + Step::Done(Outcome::Finished) => Outcome::InStore, + Step::Done(other) => other, + Step::Stop(stop) => return Err(stop), + } + } else { + Outcome::Refused(format!( + "it is a backup of another store ({} {}, store id {}), not an in-catalog backup", + row.store.store_kind, row.store.store_root, row.store.store_id + )) + }; + } + Ok(Named::Done(done)) +} + +/// The replacement with a row that is already in the catalog index: what +/// is left of the in-catalog backup is removed. +async fn finish(storage: &S, row: &BackupRow) -> Step { + match storage.replace_with_index_row(row).await { + Ok(Replaced::Replaced | Replaced::AlreadyReplaced) => Step::Done(Outcome::Finished), + Ok(Replaced::NotAvailable(what)) => Step::Done(Outcome::Skipped(what)), + Ok(Replaced::InUse(table)) => Step::Done(Outcome::Refused(in_use(&table))), + Err(StorageError::AlreadyExists(what)) => Step::Done(Outcome::Refused(what)), + Err(e) => Step::Stop(format!( + "the catalog could not finish the replacement of {}: {e}", + row.backup_arn + )), + } +} + +fn in_use(table: &str) -> String { + format!( + "a restore into {table} is reading it (the table is still CREATING); once that restore \ + ends, or its target is removed with extenddb backup remove-target, run the command again" + ) +} + +/// One backup (module doc). `notes` collects what the operator is told +/// beside the outcome. +#[allow(clippy::too_many_lines)] +async fn migrate_one( + storage: &S, + run: &Run<'_, '_>, + key: &LegacyBackupKey, + notes: &mut Vec, +) -> Step { + let m = run.m; + let arn_text = key.backup_arn.as_str(); + // 1. The id, before anything is read. + let Ok(parts) = extenddb_storage::util::check_backup_arn(arn_text) else { + return Step::Done(Outcome::Refused( + "its ARN does not parse as a backup ARN".to_owned(), + )); + }; + let Some(created_at) = backup_creation_date_time(parts.backup_id) else { + return Step::Done(Outcome::Refused(format!( + "its backup id {} carries no instant (an id a 0.1.x release issued), and a store \ + prefix carries only the 14-digit shape; {CARRY_FORWARD}", + parts.backup_id + ))); + }; + // 2. The table name, by the store-prefix rules. + let arn = match BackupArn::parse(arn_text) { + Ok(arn) if arn.partition() == ARN_PARTITION => arn, + Ok(arn) => { + return Step::Done(Outcome::Refused(format!( + "its ARN is in partition {}, which no request to this server can address", + arn.partition() + ))); + } + Err(e) => { + return Step::Done(Outcome::Refused(format!( + "its table name cannot name a store prefix ({e}); restore it into a table with \ + another name and take a new backup of that table, which has a new ARN" + ))); + } + }; + // 3. The status, and 4. a row under the ARN. + let held = match storage.read_backup_row(&arn).await { + Ok(held) => held, + Err(e) => return Step::Stop(format!("the catalog could not be read: {e}")), + }; + if let Some(row) = &held { + if row.store == *m.identity { + return finish(storage, row).await; + } + return Step::Done(Outcome::Refused(format!( + "a catalog row of another store ({} {}, store id {}) holds its ARN", + row.store.store_kind, row.store.store_root, row.store.store_id + ))); + } + match key.status { + BackupStatus::Available => {} + BackupStatus::Creating => { + return Step::Done(Outcome::Skipped( + "it is CREATING (a create in flight, or one that stopped, which the reconciler \ + removes)" + .to_owned(), + )); + } + BackupStatus::Deleting => { + return Step::Done(Outcome::Skipped( + "it is DELETING (a DeleteBackup that stopped; DeleteBackup again finishes it)" + .to_owned(), + )); + } + other => { + return Step::Done(Outcome::Skipped(format!("it is {other}"))); + } + } + // 5. The open. + let opened = match storage.items(&key.account_id, arn_text).await { + Ok(opened) => opened, + Err(e) if is_backup_not_found(&e) => { + return Step::Done(Outcome::Skipped( + "it is no longer AVAILABLE (deleted since it was listed)".to_owned(), + )); + } + Err(StorageError::Unsupported(why)) => { + return Step::Done(Outcome::Refused(format!( + "{why}; the legacy restore cannot read it either" + ))); + } + Err(e) => { + return Step::Stop(format!("the in-catalog backup could not be opened: {e}")); + } + }; + // 6. The name. + if !is_valid_backup_name(&opened.backup_name) { + return Step::Done(Outcome::Refused(format!( + "its name {:?} is not a backup name the store format accepts (3 to 255 characters of \ + a-z, A-Z, 0-9, _, - and .); {CARRY_FORWARD}, under a valid name", + opened.backup_name + ))); + } + // 7. The definition: what the legacy restore refuses (it has no way + // out in this release), then what only the copy cannot state. + if let Err(e) = opened.source.ensure_restorable() { + return Step::Done(Outcome::Refused(format!( + "{}; the legacy restore refuses it too", + storage_text(&e) + ))); + } + let mut source_table = match source_table_from_legacy( + &opened.source, + opened.vector_state, + arn.region(), + &key.account_id, + ) { + Ok(source_table) => source_table, + Err(e) => { + return Step::Done(Outcome::Refused(format!( + "{}; {CARRY_FORWARD}", + storage_text(&e) + ))); + } + }; + let key_schema = match RestorableKeySchema::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) { + Ok(key_schema) => key_schema, + Err(e) => { + return Step::Done(Outcome::Refused(format!( + "its key schema cannot be checked: {e}" + ))); + } + }; + if (opened.source.backup_creation_date_time - created_at).abs() >= 0.001 { + notes.push(format!( + "its in-catalog creation time {} is not its id's instant {created_at}; the copy \ + records the id's", + opened.source.backup_creation_date_time + )); + } + if arn.region() != m.region { + notes.push(format!( + "its region {} is not this server's; requests find it by its ARN as written", + arn.region() + )); + } + + let configured = m.configured; + let prefix = arn.store_prefix(); + let calls = StoreCalls { + store: &*configured.store, + metrics: m.metrics, + }; + // A copy an earlier run left (no row names the prefix: the read above + // found none). + match calls.store.list(&prefix).try_next().await { + Ok(None) => {} + Ok(Some(_)) => { + if let Err(problem) = configured.fence().check_catalog(storage).await { + return Step::Stop(format!( + "the deletion fence refused the deletion of the copy an earlier run left \ + under {prefix}: {problem}" + )); + } + if let Err(e) = calls.store.delete_prefix(&prefix).await { + return Step::Done(Outcome::Failed(format!( + "the copy an earlier run left under {prefix} could not be deleted: {}", + calls.failed("delete_prefix", e) + ))); + } + notes.push(format!( + "deleted the copy an earlier run left under {prefix}" + )); + } + Err(e) => { + return Step::Done(Outcome::Failed(format!( + "the store could not list {prefix}: {}", + calls.failed("list", e) + ))); + } + } + // The fence before the first put. + if let Err(problem) = configured.fence().check_catalog(storage).await { + return Step::Stop(format!( + "the deletion fence refused the first write: {problem}" + )); + } + + let pipeline = Pipeline { + store: &*configured.store, + configured, + identity: m.identity, + prefix: &prefix, + arn: arn_text, + heartbeat: m.claim, + shutdown: &run.shutdown, + metrics: m.metrics, + target_bytes: m.data_file_target_bytes, + key_prefix: &configured.key_prefix, + bucket: &configured.bucket, + }; + let recorded = opened.source.item_count; + let files = + match writer::write_data_files(&pipeline, opened.items, key_schema, m.limits, None).await { + Ok(files) => files, + Err(stop) => return stopped(storage, run, &prefix, stop, notes).await, + }; + // The count check: a DeleteBackup that landed while the stream ran + // (SQLite, MongoDB, Cassandra) leaves it short. + if i64::try_from(files.item_count).ok() != Some(recorded) { + let n = files.item_count; + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Refused(format!( + "the in-catalog backup records {recorded} items and its read gave {n} (a \ + DeleteBackup during the read ends it short)" + ))); + } + source_table.item_count = recorded; + source_table.table_size_bytes = i64::try_from(files.item_bytes).unwrap_or(i64::MAX); + let item_count = files.item_count; + let header = ManifestHeader { + backup_arn: arn.clone(), + backup_name: opened.backup_name.clone(), + source_backend: m.source_backend.to_owned(), + source_extenddb_version: m.source_extenddb_version.to_owned(), + snapshot: SnapshotDescriptor { + kind: SnapshotDescriptor::KIND_LEGACY_IN_CATALOG.to_owned(), + marker: String::new(), + }, + item_order: ItemOrder::Unordered, + source_table, + }; + let docs = match BackupDocuments::build( + header, + files.entries, + &configured.key_prefix, + &configured.bucket, + ) { + Ok(docs) => docs, + Err(e) => { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Failed(format!("building the manifest: {e}"))); + } + }; + for (name, bytes) in docs.documents_before_manifest() { + if let Err(stop) = put_document(&pipeline, name, bytes).await { + return stopped(storage, run, &prefix, stop, notes).await; + } + } + kill_point(KillPoint::BeforeManifest, &run.shutdown).await; + if !m.claim.confirms_the_claim() { + return Step::Stop(ClaimWatch::reason(m.claim)); + } + let (name, bytes) = docs.manifest_document(); + if let Err(stop) = put_document(&pipeline, name, bytes).await { + return stopped(storage, run, &prefix, stop, notes).await; + } + kill_point(KillPoint::AfterManifest, &run.shutdown).await; + + // The re-read: what a restore, sync and the dead-owner duty will read. + let manifest = match verify_prefix(calls, &prefix, None).await { + Ok(Verified::Complete(manifest)) => manifest, + Ok(Verified::Incomplete(why)) => { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Failed(format!( + "the copy does not read back as a complete backup: {why}" + ))); + } + Err(e) => { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Failed(format!( + "the copy could not be read back: {e}" + ))); + } + }; + if manifest.backup_arn().ok().as_ref() != Some(&arn) { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Failed( + "the copy's manifest names another backup".to_owned(), + )); + } + let total = manifest_item_count(&manifest); + if total != recorded { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Refused(format!( + "the copy's manifest counts {total} items and the in-catalog backup records \ + {recorded}" + ))); + } + let source_table_details_json = match serde_json::to_value(&manifest.source_table) { + Ok(value) => value, + Err(e) => { + delete_copy(storage, run, &prefix, notes).await; + return Step::Done(Outcome::Failed(format!( + "the source table does not serialize: {e}" + ))); + } + }; + let row = BackupRow { + backup_id: arn.backup_id().to_owned(), + account_id: key.account_id.clone(), + table_name: arn.table_name().to_owned(), + backup_arn: arn_text.to_owned(), + backup_name: manifest.backup_name.clone(), + status: BackupStatus::Available, + store: m.identity.clone(), + size_bytes: manifest.backup_size_bytes, + item_count: total, + created_at, + completed_at: None, + failure_reason: None, + source_table_details_json, + owner_generation: UNOWNED_GENERATION, + heartbeat_at: 0.0, + }; + // The claim before the replacement: a run that lost it leaves the copy, + // which the next run rewrites. + if !m.claim.confirms_the_claim() { + return Step::Stop(format!( + "{}; the copy under {prefix} is left for the next run, which writes it again", + ClaimWatch::reason(m.claim) + )); + } + match storage.replace_with_index_row(&row).await { + Ok(Replaced::Replaced) => Step::Done(Outcome::Migrated { + item_count, + size_bytes: manifest.backup_size_bytes, + }), + Ok(Replaced::AlreadyReplaced) => Step::Done(Outcome::Finished), + Ok(Replaced::NotAvailable(what)) => { + delete_copy(storage, run, &prefix, notes).await; + Step::Done(Outcome::Skipped(format!( + "it was no longer AVAILABLE when its row was to be written ({what})" + ))) + } + Ok(Replaced::InUse(table)) => { + delete_copy(storage, run, &prefix, notes).await; + Step::Done(Outcome::Refused(in_use(&table))) + } + Err(StorageError::AlreadyExists(what)) => { + delete_copy(storage, run, &prefix, notes).await; + Step::Done(Outcome::Refused(what)) + } + // The outcome is unknown: the row may be in. The copy stays, and the + // next run finishes (a row there) or rewrites it (none). + Err(e) => Step::Stop(format!( + "the replacement failed: {e}; the copy under {prefix} is left for the next run, which \ + finishes the replacement or writes the copy again" + )), + } +} + +/// A [`StorageError`]'s text without the variant's wrapper. +fn storage_text(e: &StorageError) -> String { + match e { + StorageError::Validation(text) + | StorageError::Unsupported(text) + | StorageError::Internal(text) => text.clone(), + other => other.to_string(), + } +} + +/// The writer's stop (`write_data_files` or a document's put), as the +/// backup's end: a failure or a refused item ends this backup with its copy +/// deleted; a signal stops the run with the copy deleted; a lost claim and +/// a store found unhealthy or fenced stop the run with no further store +/// call. +async fn stopped( + storage: &S, + run: &Run<'_, '_>, + prefix: &str, + stop: Stop, + notes: &mut Vec, +) -> Step { + match stop { + Stop::Fail(reason) => { + delete_copy(storage, run, prefix, notes).await; + if writer::is_item_refusal(&reason) { + Step::Done(Outcome::Refused(format!( + "{reason}; restore it (RestoreTableFromBackup), correct the items the log \ + names, and take a new backup of the restored table, which has a new ARN" + ))) + } else { + Step::Done(Outcome::Failed(reason)) + } + } + Stop::Cancelled => { + delete_copy(storage, run, prefix, notes).await; + Step::Stop(match run.m.interrupt.signalled() { + Some(signal) => format!("{signal} received during the copy"), + None => "the copy was stopped".to_owned(), + }) + } + Stop::ClaimLost(why) => Step::Stop(format!( + "{why}; the copy under {prefix} is left for the next run, which writes it again" + )), + Stop::StoreUnhealthy(why) => Step::Stop(why), + } +} + +/// Delete the backup's copy through the deletion fence, noting what was +/// left when it could not be deleted. +async fn delete_copy( + storage: &S, + run: &Run<'_, '_>, + prefix: &str, + notes: &mut Vec, +) { + let configured = run.m.configured; + if let Err(problem) = configured.fence().check_catalog(storage).await { + notes.push(format!( + "its copy under {prefix} was not deleted ({problem}); the next run deletes it" + )); + return; + } + let calls = StoreCalls { + store: &*configured.store, + metrics: run.m.metrics, + }; + if let Err(e) = calls.store.delete_prefix(prefix).await { + notes.push(format!( + "its copy under {prefix} could not be deleted ({}); the next run deletes it", + calls.failed("delete_prefix", e) + )); + } +} diff --git a/crates/engine/src/backup/sync.rs b/crates/engine/src/backup/sync.rs index 855809249..79729522d 100644 --- a/crates/engine/src/backup/sync.rs +++ b/crates/engine/src/backup/sync.rs @@ -46,7 +46,11 @@ //! another root, or one whose list entries are missing) is not seen by it; //! within one run, the row check keeps a second prefix of an id this run //! adopted out. A row under the manifest's own spelling is left to the -//! insert. A +//! insert. A prefix whose ARN, under either spelling, an in-catalog backup +//! of the legacy reader holds (`LegacyBackupReader::describe` finds it) is +//! skipped, naming `extenddb backup migrate-legacy `: that backup's +//! copy is the migration's to finish, and a row inserted here would leave +//! the backup in the catalog twice. A //! verified prefix gets an AVAILABLE row through //! [`BackupCatalog::insert_backup_row`], unowned, in the configured store's //! identity, with the manifest's name, ARN, size, item total and @@ -126,13 +130,13 @@ use extenddb_core::backup_format::{ backup_creation_date_time, classify_store_key, }; use extenddb_core::metrics::MetricsCollector; -use extenddb_storage::BackupCatalog; use extenddb_storage::backup_catalog::{ BackupKey, BackupRow, BackupStatus, ForgetRow, StoreIdentity, UNOWNED_GENERATION, is_backup_not_found, }; use extenddb_storage::error::StorageError; use extenddb_storage::util::ARN_PARTITION; +use extenddb_storage::{BackupCatalog, LegacyBackupReader}; use futures::TryStreamExt; use super::BackupRuntime; @@ -242,6 +246,9 @@ pub enum SkipReason { /// The manifest's ARN is in another partition: no request to this /// server can address it. OtherPartition(String), + /// An in-catalog backup holds the ARN: its ARN as stored and its + /// status. + InCatalog { backup_arn: String, status: String }, /// A read of the prefix's manifest or sidecar failed; the store's /// error. Store(String), @@ -287,6 +294,11 @@ impl std::fmt::Display for SkipReason { Self::RowExists(what) => f.write_str(what), Self::Incomplete(why) => write!(f, "not a complete backup: {why}"), Self::ArnMismatch(why) | Self::OtherPartition(why) => f.write_str(why), + Self::InCatalog { backup_arn, status } => write!( + f, + "an in-catalog backup ({status}) holds this ARN; run extenddb backup \ + migrate-legacy {backup_arn}" + ), Self::Store(e) => write!(f, "the store could not be read: {e}"), Self::Catalog(e) => write!(f, "the catalog could not be read or written: {e}"), } @@ -437,7 +449,7 @@ type Known = HashMap<(String, String), (BackupStatus, String, String)>; /// /// [`SyncError`]: the catalog's read failed (nothing written), or the /// listing failed (the rows inserted before it stay). -pub async fn sync( +pub async fn sync( storage: &S, target: SyncTarget<'_>, ) -> Result { @@ -512,7 +524,7 @@ pub async fn sync( /// One prefix the listing has passed: the stop, then the row check, then /// the verification and the insert. Returns why the run stops, when the /// stop asked it to (the prefix is then left unread). -async fn finish_group( +async fn finish_group( storage: &S, calls: StoreCalls<'_>, target: SyncTarget<'_>, @@ -586,7 +598,7 @@ async fn finish_group( } /// Verify one prefix with no row and insert its row. -async fn adopt_prefix( +async fn adopt_prefix( storage: &S, calls: StoreCalls<'_>, target: SyncTarget<'_>, @@ -632,6 +644,13 @@ async fn adopt_prefix( let created_at = backup_creation_date_time(backup_id).ok_or_else(|| { SkipReason::NotABackup(format!("backup id {backup_id:?} carries no instant")) })?; + match in_catalog_holder(storage, &arn, target.region).await { + Ok(Some((backup_arn, status))) => { + return Err(SkipReason::InCatalog { backup_arn, status }); + } + Ok(None) => {} + Err(e) => return Err(SkipReason::Catalog(e.to_string())), + } // A row under the other spelling of the ARN: on Cassandra the insert // decides per spelling only (the trait's doc). A row under the // manifest's own spelling is left to the insert, which answers `false` @@ -709,6 +728,34 @@ async fn row_under_a_spelling( .map(|row| (arn.clone(), row))) } +/// The in-catalog backup that holds `arn` under the spelling every request +/// tries first (the ARN rebuilt in `region`) or as the manifest writes it, +/// as DescribeBackup resolves one: its ARN as stored and its status. +async fn in_catalog_holder( + storage: &S, + arn: &BackupArn, + region: Option<&str>, +) -> Result, StorageError> { + let written = arn.to_arn_string(); + let found = match (region, extenddb_storage::util::check_backup_arn(&written)) { + (Some(region), Ok(parts)) => { + let spellings = super::BackupArnSpellings::new(&parts, region); + super::resolve_backup(&spellings, by!(storage, arn.account_id(), describe)) + .await + .map(|(_, found)| found) + } + _ => storage.describe(arn.account_id(), &written).await, + }; + match found { + Ok(description) => Ok(Some(( + description.backup_details.backup_arn, + description.backup_details.backup_status, + ))), + Err(e) if is_backup_not_found(&e) => Ok(None), + Err(e) => Err(e), + } +} + /// The sentence for an ARN in another region than the server's, or `None`. fn foreign_note(arn: &BackupArn, region: Option<&str>) -> Option { region @@ -914,7 +961,7 @@ impl Drop for Rearm<'_> { /// store, whose check ran then) and every reconciler cycle calls it after /// the store's check (the S3 store, whose first check runs there, and a sync /// left pending). -pub async fn startup_sync( +pub async fn startup_sync( storage: &S, runtime: &BackupRuntime, ) -> Option> { diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index 3d1efad13..ab198c5b6 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -22,9 +22,9 @@ use extenddb_core::types::{ }; use extenddb_storage::backup_catalog::{ AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, - JobClaim, JobRow, JobStatus, RestoreJobRow, RestoreKind, RestoreRegistrationRow, - RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, StoreRootCount, - TimeBounds, UNOWNED_GENERATION, WriteOutcome, new_generation, + JobClaim, JobRow, JobStatus, LegacyBackupItems, LegacyBackupKey, Replaced, RestoreJobRow, + RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, + StoreKind, StoreRootCount, TimeBounds, UNOWNED_GENERATION, WriteOutcome, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; @@ -39,6 +39,8 @@ use serde_json::{Value, json}; use super::*; +mod migrate_legacy; + const ACCOUNT: &str = "123456789012"; const REGION: &str = "us-east-1"; const CREATED: f64 = 1_791_976_431.438; @@ -379,6 +381,38 @@ struct Fake { kept_entries: Mutex>, /// When set, `list_backup_rows_by_root` of this root fails. by_root_error_for: Mutex>, + /// The in-catalog backups `items` opens, by ARN: their name, vector + /// state and items. Their recorded source is `sources`'; a backup with a + /// source and no entry here opens with no items. + legacy_items: Mutex>, + /// What `list_all` answers. + legacy_keys: Mutex>, + /// When set, `list_all` fails with it (every call). + list_all_error: Mutex>, + /// When set, the next `items` fails with it before opening anything. + items_error: Mutex>, + /// When set, the next stream `items` opens yields this many items and + /// then fails with the error: a read that ends part way (a MongoDB + /// cursor whose collection was dropped, a lost connection). + items_fail_after: Mutex>, + /// Every `items` call, by ARN. + items_opened: Mutex>, + /// When set, the next `replace_with_index_row` answers this instead of + /// deciding; `Replaced` and `AlreadyReplaced` also make the change. + replace_answer: Mutex>, + /// When set, the next `replace_with_index_row` fails with it, writing + /// nothing. + replace_error: Mutex>, + /// Every row `replace_with_index_row` received, in order. + replaces_seen: Mutex>, +} + +/// One in-catalog backup the `Fake`'s `items` opens. +#[derive(Debug, Clone)] +struct FakeLegacyItems { + backup_name: String, + vector_state: extenddb_storage::backup_definition::LegacyVectorState, + items: Vec, } /// One `rewrite_store` call the `Fake` received: the old kind and root, the @@ -3438,6 +3472,148 @@ impl LegacyBackupReader for Fake { let found = self.abandoned_backups.lock().unwrap().clone(); Box::pin(async move { Ok(found) }) } + + fn list_all(&self) -> BoxedFuture<'_, Result, StorageError>> { + let error = self.list_all_error.lock().unwrap().clone(); + let keys = self.legacy_keys.lock().unwrap().clone(); + Box::pin(async move { + match error { + Some(e) => Err(StorageError::Internal(e)), + None => Ok(keys), + } + }) + } + + /// The source `restore_source` would answer, with the backup's entry in + /// `legacy_items`; a backup with no source is `Backup not found`. + fn items( + &self, + _account_id: &str, + backup_arn: &str, + ) -> BoxedFuture<'_, Result> { + self.items_opened + .lock() + .unwrap() + .push(backup_arn.to_owned()); + let error = self.items_error.lock().unwrap().take(); + let source = self.sources.lock().unwrap().get(backup_arn).cloned(); + let entry = self.legacy_items.lock().unwrap().get(backup_arn).cloned(); + let fail_after = self.items_fail_after.lock().unwrap().take(); + let backup_arn = backup_arn.to_owned(); + Box::pin(async move { + if let Some(e) = error { + return Err(e); + } + let source = source.ok_or_else(|| { + StorageError::Validation(format!("Backup not found: {backup_arn}")) + })?; + let entry = entry.unwrap_or(FakeLegacyItems { + backup_name: "nightly".to_owned(), + vector_state: extenddb_storage::backup_definition::LegacyVectorState::NoneRecorded, + items: Vec::new(), + }); + let mut items: Vec> = + entry.items.into_iter().map(Ok).collect(); + if let Some((after, e)) = fail_after { + items.truncate(after); + items.push(Err(e)); + } + Ok(LegacyBackupItems { + backup_name: entry.backup_name, + source, + vector_state: entry.vector_state, + items: futures::stream::iter(items).boxed(), + }) + }) + } + + /// The backends' replacement, in one step: `InUse` while `in_use` names + /// a target of the backup; `AlreadyReplaced` when a row under the ARN + /// with the account and store is there (whatever is left of the + /// in-catalog backup removed); an `AlreadyExists` error when another row + /// holds the backup id (or, with `unique_by_arn`, the ARN under another + /// store); `NotAvailable` when the in-catalog backup has no source (not + /// AVAILABLE); else the row inserted unowned and the in-catalog backup + /// removed (its source, description, items and `list_all` entry). + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxedFuture<'a, Result> { + self.replaces_seen.lock().unwrap().push(row.clone()); + let error = self.replace_error.lock().unwrap().take(); + let preset = self.replace_answer.lock().unwrap().take(); + let arn = row.backup_arn.clone(); + let remove_legacy = || { + self.sources.lock().unwrap().remove(&arn); + self.descriptions.lock().unwrap().remove(&arn); + self.legacy_items.lock().unwrap().remove(&arn); + self.legacy_keys + .lock() + .unwrap() + .retain(|k| k.backup_arn != arn); + }; + let insert = || { + let now = seconds_since_epoch(); + let mut inserted = row.clone(); + inserted.owner_generation = UNOWNED_GENERATION; + inserted.completed_at = Some(now); + inserted.heartbeat_at = now; + self.rows + .lock() + .unwrap() + .insert(row.backup_arn.clone(), inserted); + }; + let answer = if let Some(e) = error { + Err(StorageError::Internal(e)) + } else if let Some(preset) = preset { + match preset { + Replaced::Replaced => { + insert(); + remove_legacy(); + } + Replaced::AlreadyReplaced => remove_legacy(), + Replaced::NotAvailable(_) | Replaced::InUse(_) => {} + } + Ok(preset) + } else { + let available = self.sources.lock().unwrap().contains_key(&arn); + let in_use = self.in_use.lock().unwrap().get(&arn).cloned(); + let stored = self.rows.lock().unwrap().get(&arn).cloned(); + let by_arn = self + .unique_by_arn + .load(std::sync::atomic::Ordering::Acquire); + let other_holds_id = + self.rows.lock().unwrap().values().any(|r| { + r.backup_id == row.backup_id && r.backup_arn != row.backup_arn && !by_arn + }); + match (in_use, stored) { + (Some(target), _) if available => Ok(Replaced::InUse(target)), + (_, Some(stored)) + if stored.account_id == row.account_id && stored.store == row.store => + { + remove_legacy(); + Ok(Replaced::AlreadyReplaced) + } + (_, Some(stored)) => Err(StorageError::AlreadyExists(format!( + "backup {} has a row in store {}", + stored.backup_arn, stored.store.store_root + ))), + (_, None) if other_holds_id => Err(StorageError::AlreadyExists(format!( + "backup id {} has a row under another ARN", + row.backup_id + ))), + (_, None) if !available => { + Ok(Replaced::NotAvailable(format!("Backup not found: {arn}"))) + } + (_, None) => { + insert(); + remove_legacy(); + Ok(Replaced::Replaced) + } + } + }; + Box::pin(async move { answer }) + } } /// What a backend reports for a CREATING target: deliberately the shape the diff --git a/crates/engine/src/backup/tests/migrate_legacy.rs b/crates/engine/src/backup/tests/migrate_legacy.rs new file mode 100644 index 000000000..acea8761d --- /dev/null +++ b/crates/engine/src/backup/tests/migrate_legacy.rs @@ -0,0 +1,852 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! `backup migrate-legacy` (`super::super::migrate_legacy`) against the +//! `Fake`'s legacy reader and a `MemStore`, the `backup sync` skip of a +//! prefix an in-catalog backup holds, and ListBackups over a backup both +//! readers list. + +use extenddb_core::types::AttributeValue; +use extenddb_core::validation::RestorableLimits; +use extenddb_storage::backup_definition::LegacyVectorState; + +use super::super::migrate_legacy::{ + BackupOutcome, MigrateReport, Migration, Outcome, Selection, migrate, +}; +use super::*; + +/// The in-catalog backups of these tests: 14-digit ids, one per `n`, of +/// the `Music` table of the tests' account and region. +fn legacy_arn(n: u32) -> String { + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/0179197643{n:04}-1111{n:04}") +} + +/// An item of the `source` fixture's key schema (`pk`, `sk`). +fn legacy_item(i: usize) -> Item { + Item::from([ + ("pk".to_owned(), AttributeValue::S(format!("p{i:04}"))), + ("sk".to_owned(), AttributeValue::S("s".to_owned())), + ("n".to_owned(), AttributeValue::N(i.to_string())), + ]) +} + +fn legacy_items(n: usize) -> Vec { + (0..n).map(legacy_item).collect() +} + +/// An AVAILABLE in-catalog backup of `arn` holding `items`, its count +/// recorded as theirs, with a pay-per-request definition (a GSI and an +/// LSI), its name `nightly`, listed by `list_all`. +fn seed_legacy(fake: &Fake, arn: &str, items: Vec) { + let mut source = source(Some(definition("PAY_PER_REQUEST"))); + source.backup_arn = arn.to_owned(); + source.item_count = i64::try_from(items.len()).unwrap(); + // An id with no instant (a 0.1.x backup's) keeps the fixture's time. + source.backup_creation_date_time = + extenddb_storage::util::backup_creation_date_time(arn).unwrap_or(CREATED); + fake.sources.lock().unwrap().insert(arn.to_owned(), source); + fake.legacy_items.lock().unwrap().insert( + arn.to_owned(), + FakeLegacyItems { + backup_name: "nightly".to_owned(), + vector_state: LegacyVectorState::NoneRecorded, + items, + }, + ); + fake.legacy_keys.lock().unwrap().push(LegacyBackupKey { + account_id: ACCOUNT.to_owned(), + backup_arn: arn.to_owned(), + status: BackupStatus::Available, + }); +} + +/// The in-catalog backup of `arn` is still there: its source and its +/// `list_all` entry. +fn legacy_kept(fake: &Fake, arn: &str) -> bool { + fake.sources.lock().unwrap().contains_key(arn) + && fake + .legacy_keys + .lock() + .unwrap() + .iter() + .any(|k| k.backup_arn == arn) +} + +/// The objects under `arn`'s prefix. +fn copy_of(mem: &MemStore, arn: &str) -> Vec { + objects_under(mem, arn) +} + +/// A healthy runtime on `mem`, and the claim and signal of a command. +struct Command { + runtime: BackupRuntime, + holder: Arc, + claim: maintenance::HeldClaim, + signal: tokio::sync::watch::Sender>, + interrupt: maintenance::Interrupt, +} + +fn new_command(mem: &Arc) -> Command { + let holder = Arc::new(FakeHolder::default()); + let claim = held(&holder); + let (signal, interrupt) = maintenance::Interrupt::new(); + Command { + runtime: runtime_on(mem, store()), + holder, + claim, + signal, + interrupt, + } +} + +/// Run the migration of `selection` as the command would, collecting each +/// backup's outcome as it is reported. +async fn run( + fake: &Fake, + command: &Command, + selection: Selection<'_>, +) -> (MigrateReport, Vec) { + let identity = store(); + let migration = Migration { + configured: command.runtime.configured_store().unwrap(), + identity: &identity, + claim: &command.claim, + interrupt: &command.interrupt, + limits: RestorableLimits::from(&LimitsConfig::default()), + data_file_target_bytes: 64 * 1024 * 1024, + region: REGION, + source_backend: "sqlite", + source_extenddb_version: "9.9.9", + metrics: None, + }; + let mut seen = Vec::new(); + let report = migrate(fake, &migration, selection, |b| seen.push(b.clone())).await; + (report, seen) +} + +fn outcome_of<'a>(report: &'a MigrateReport, arn: &str) -> &'a Outcome { + &report + .backups + .iter() + .find(|b| b.backup_arn == arn) + .unwrap_or_else(|| panic!("no outcome for {arn}: {report:?}")) + .outcome +} + +fn refusal(outcome: &Outcome) -> &str { + match outcome { + Outcome::Refused(why) => why, + other => panic!("expected a refusal, got {other:?}"), + } +} + +/// The items of every data file of `arn`'s copy, decoded and verified +/// against the manifest's entries. +fn copied_items(mem: &MemStore, arn: &str) -> Vec { + let manifest = stored_manifest(mem, arn); + let prefix = prefix_of(arn); + let mut items = Vec::new(); + for file in &manifest.data_files { + let bytes = mem.objects.lock().unwrap()[&format!("{prefix}{}", file.key)].clone(); + let mut decoder = file.decoder(&bytes[..]).unwrap(); + while let Some(item) = decoder.next_item().unwrap() { + items.push(item); + } + } + items +} + +/// An AVAILABLE in-catalog backup is copied under its own ARN: the +/// manifest says it is a copy of an in-catalog backup (no marker, +/// unordered), names this binary as its writer, keeps the backup's name and +/// creation instant and records the table the legacy restore creates; its +/// data files hold every item; the prefix reads back as `backup sync` +/// reads one. The replacement then gets the row `backup sync` would write +/// (AVAILABLE, unowned, the configured store, the manifest's size, count +/// and `source_table`, the id's instant), and the in-catalog backup is +/// gone. The migrated backup is described from its row, and a second run +/// finds nothing to do. +#[tokio::test] +async fn an_in_catalog_backup_is_copied_under_its_arn_and_replaced_by_its_row() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = legacy_arn(1); + seed_legacy(&fake, &arn, legacy_items(5)); + let command = new_command(&mem); + let (report, seen) = run(&fake, &command, Selection::All).await; + assert!(report.complete(), "{report:?}"); + assert_eq!(seen, report.backups); + let size = match outcome_of(&report, &arn) { + Outcome::Migrated { + item_count, + size_bytes, + } => { + assert_eq!(*item_count, 5); + *size_bytes + } + other => panic!("{other:?}"), + }; + + let manifest = stored_manifest(&mem, &arn); + assert_eq!( + manifest.snapshot.kind, + extenddb_core::backup_format::SnapshotDescriptor::KIND_LEGACY_IN_CATALOG + ); + assert_eq!(manifest.snapshot.marker, ""); + assert_eq!( + manifest.item_order, + extenddb_core::backup_format::ItemOrder::Unordered + ); + assert_eq!(manifest.source_backend, "sqlite"); + assert_eq!(manifest.source_extenddb_version, "9.9.9"); + assert_eq!(manifest.backup_name, "nightly"); + assert_eq!(manifest.backup_arn().unwrap().to_arn_string(), arn); + let instant = extenddb_storage::util::backup_creation_date_time(&arn).unwrap(); + assert!((manifest.backup_creation_date_time - instant).abs() < 1e-9); + let source = source(Some(definition("PAY_PER_REQUEST"))); + let expected = extenddb_storage::backup_definition::source_table_from_legacy( + &BackupRestoreSource { + item_count: 5, + ..source + }, + LegacyVectorState::NoneRecorded, + REGION, + ACCOUNT, + ) + .unwrap(); + assert_eq!(manifest.source_table.key_schema, expected.key_schema); + assert_eq!( + manifest.source_table.global_secondary_indexes, + expected.global_secondary_indexes + ); + assert_eq!(manifest.source_table.vector_indexes, Some(Vec::new())); + assert_eq!(manifest.source_table.item_count, 5); + let mut items = copied_items(&mem, &arn); + items.sort_by_key(|item| format!("{item:?}")); + let mut expected_items = legacy_items(5); + expected_items.sort_by_key(|item| format!("{item:?}")); + assert_eq!(items, expected_items); + + let replaced = fake.replaces_seen.lock().unwrap().clone(); + assert_eq!(replaced.len(), 1); + let row = &replaced[0]; + assert_eq!(row.backup_arn, arn); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!(row.owner_generation, UNOWNED_GENERATION); + assert_eq!(row.store, store()); + assert_eq!(row.item_count, 5); + assert_eq!(row.size_bytes, size); + assert_eq!(row.size_bytes, manifest.backup_size_bytes); + assert_eq!(row.backup_name, "nightly"); + assert!((row.created_at - instant).abs() < 1e-9); + assert_eq!( + row.source_table_details_json, + serde_json::to_value(&manifest.source_table).unwrap() + ); + assert!(!legacy_kept(&fake, &arn)); + assert!(fake.rows.lock().unwrap().contains_key(&arn)); + + let description = describe_backup(&fake, ACCOUNT, REGION, &own(&arn)) + .await + .unwrap(); + assert_eq!(description.backup_details.backup_status, "AVAILABLE"); + assert_eq!(description.backup_details.backup_name, "nightly"); + + let (again, _) = run(&fake, &command, Selection::All).await; + assert!(again.backups.is_empty() && again.complete(), "{again:?}"); +} + +/// What a refusal case changes in the seeded backup. +type Prepare = Box; + +/// Every refusal that needs no copy keeps the in-catalog backup and writes +/// nothing to the store: an id with no instant (13 digits with a suffix, +/// and bare), a table name the store-prefix rules refuse, a name the +/// manifest refuses, a definition whose vector state is unknown (a SQLite +/// backup from before catalog 0.0.4), vector indexes (the legacy restore's +/// own refusal), a backend that cannot read the backup (MongoDB's without +/// an item collection), and a row of another store under the ARN. The id +/// and table-name refusals come before the backup is opened. +#[tokio::test] +async fn each_refusal_before_the_copy_keeps_the_in_catalog_backup_and_writes_nothing() { + let thirteen = + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/1793254932797-4116998c"); + let bare = format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/1793254932797"); + let dots = + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/.../backup/01791976431001-11110001"); + let cases: Vec<(String, Prepare, &str, bool)> = vec![ + (thirteen, Box::new(|_, _| {}), "carries no instant", false), + (bare, Box::new(|_, _| {}), "carries no instant", false), + ( + dots, + Box::new(|_, _| {}), + "cannot name a store prefix", + false, + ), + ( + legacy_arn(2), + Box::new(|fake, arn| { + fake.legacy_items + .lock() + .unwrap() + .get_mut(arn) + .unwrap() + .backup_name = "x".to_owned(); + }), + "is not a backup name", + true, + ), + ( + legacy_arn(3), + Box::new(|fake, arn| { + fake.legacy_items + .lock() + .unwrap() + .get_mut(arn) + .unwrap() + .vector_state = LegacyVectorState::Unknown; + }), + "no recorded table definition", + true, + ), + ( + legacy_arn(4), + Box::new(|fake, arn| { + fake.sources + .lock() + .unwrap() + .get_mut(arn) + .unwrap() + .definition + .as_mut() + .unwrap() + .vector_index_names = vec!["v".to_owned()]; + }), + "the legacy restore refuses it too", + true, + ), + ( + legacy_arn(5), + Box::new(|fake, _| { + *fake.items_error.lock().unwrap() = Some(StorageError::Unsupported( + "the backup has no item collection to read".to_owned(), + )); + }), + "no item collection", + true, + ), + ( + legacy_arn(6), + Box::new(|fake, arn| { + let mut other = row(arn, BackupStatus::Available); + other.store.store_root = "/srv/elsewhere".to_owned(); + fake.rows.lock().unwrap().insert(arn.to_owned(), other); + }), + "another store", + false, + ), + ]; + for (arn, prepare, says, opened) in cases { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + seed_legacy(&fake, &arn, legacy_items(3)); + prepare(&fake, &arn); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + let why = refusal(outcome_of(&report, &arn)); + assert!(why.contains(says), "{arn}: {why}"); + assert!(report.stopped.is_none() && !report.complete(), "{report:?}"); + assert!(legacy_kept(&fake, &arn), "{arn}"); + assert!( + mem.puts.lock().unwrap().is_empty(), + "{arn}: nothing written" + ); + assert!(fake.replaces_seen.lock().unwrap().is_empty(), "{arn}"); + assert_eq!( + !fake.items_opened.lock().unwrap().is_empty(), + opened, + "{arn}: opened" + ); + } +} + +/// An item the validator refuses (#618's number) fails the copy: the +/// prefix written so far is deleted, the in-catalog backup kept, and the +/// reason names the item and the way out. `--all` goes on: the next +/// backup is migrated, and the run is incomplete. +#[tokio::test] +async fn a_refused_item_deletes_the_copy_and_all_goes_on_to_the_next_backup() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let bad = legacy_arn(1); + let good = legacy_arn(2); + let mut items = legacy_items(3); + items[1].insert("n".to_owned(), AttributeValue::N("1E126".to_owned())); + seed_legacy(&fake, &bad, items); + seed_legacy(&fake, &good, legacy_items(2)); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + let why = refusal(outcome_of(&report, &bad)); + assert!(why.contains("1 item(s) cannot be restored"), "{why}"); + assert!( + why.contains("(#618)") && why.contains("correct the items"), + "{why}" + ); + assert!(copy_of(&mem, &bad).is_empty(), "{:?}", copy_of(&mem, &bad)); + assert!(legacy_kept(&fake, &bad)); + assert!(matches!( + outcome_of(&report, &good), + Outcome::Migrated { item_count: 2, .. } + )); + assert!(!report.complete() && report.stopped.is_none()); +} + +/// The count check: a read that gives fewer items than the in-catalog +/// backup records (a DeleteBackup landing during the read) refuses the +/// backup and deletes its copy; a read that fails part way fails it the +/// same way. The in-catalog backup is kept either way. +#[tokio::test] +async fn a_read_short_of_the_recorded_count_or_failing_deletes_the_copy() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let short = legacy_arn(1); + seed_legacy(&fake, &short, legacy_items(3)); + fake.sources + .lock() + .unwrap() + .get_mut(&short) + .unwrap() + .item_count = 4; + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + let why = refusal(outcome_of(&report, &short)); + assert!(why.contains("records 4 items and its read gave 3"), "{why}"); + assert!(copy_of(&mem, &short).is_empty()); + assert!( + !mem.puts.lock().unwrap().is_empty(), + "a data file was written first" + ); + assert!(legacy_kept(&fake, &short)); + assert!(fake.replaces_seen.lock().unwrap().is_empty()); + + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let failing = legacy_arn(2); + seed_legacy(&fake, &failing, legacy_items(3)); + *fake.items_fail_after.lock().unwrap() = Some(( + 2, + StorageError::Internal("the cursor's collection was dropped".to_owned()), + )); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + match outcome_of(&report, &failing) { + Outcome::Failed(why) => assert!(why.contains("collection was dropped"), "{why}"), + other => panic!("{other:?}"), + } + assert!(copy_of(&mem, &failing).is_empty()); + assert!(legacy_kept(&fake, &failing)); +} + +/// The deletion fence before the first put: a root another catalog +/// adopted since the store's check gets nothing from this catalog. The run +/// stops (a store this catalog no longer owns is no place for the next +/// backup either), the store is marked unhealthy, and the in-catalog +/// backups are kept. +#[tokio::test] +async fn migrate_legacy_on_a_root_another_catalog_adopted_writes_nothing() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let first = legacy_arn(1); + seed_legacy(&fake, &first, legacy_items(3)); + seed_legacy(&fake, &legacy_arn(2), legacy_items(3)); + let command = new_command(&mem); + mem.insert(STORE_IDENTITY_KEY, &adopted_file()); + let (report, _) = run(&fake, &command, Selection::All).await; + let stopped = report.stopped.clone().expect("the run stopped"); + assert!( + stopped.contains("deletion fence refused the first write"), + "{stopped}" + ); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(legacy_kept(&fake, &first) && legacy_kept(&fake, &legacy_arn(2))); + assert!(fake.replaces_seen.lock().unwrap().is_empty()); + assert_fenced(&command.runtime); +} + +/// The re-read: a manifest that does not read back as written (its bytes +/// changed after the put, so its sidecar no longer matches) fails the +/// backup before any row is written; the copy is deleted and the in-catalog +/// backup kept. +#[tokio::test] +async fn a_copy_that_does_not_read_back_is_deleted_before_any_row() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = legacy_arn(1); + seed_legacy(&fake, &arn, legacy_items(3)); + let weak = Arc::downgrade(&mem); + *mem.on_put.lock().unwrap() = Some(Box::new(move |key: &str| { + if key.ends_with("extenddb-manifest.json") + && let Some(mem) = weak.upgrade() + { + mem.insert(key, b"{}"); + } + })); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + match outcome_of(&report, &arn) { + Outcome::Failed(why) => { + assert!( + why.contains("does not read back") && why.contains("sidecar"), + "{why}" + ); + } + other => panic!("{other:?}"), + } + assert!(copy_of(&mem, &arn).is_empty(), "{:?}", copy_of(&mem, &arn)); + assert!(legacy_kept(&fake, &arn)); + assert!(fake.replaces_seen.lock().unwrap().is_empty()); +} + +/// A copy an earlier run left (it stopped after some puts and before the +/// replacement; no row names the prefix) is deleted through the fence and +/// the backup is copied again whole: the stale object is gone. +#[tokio::test] +async fn a_copy_an_earlier_run_left_is_deleted_and_written_again() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = legacy_arn(1); + seed_legacy(&fake, &arn, legacy_items(3)); + let stale = format!("{}data/00009.json.gz", prefix_of(&arn)); + mem.insert(&stale, b"left by a run that stopped"); + let command = new_command(&mem); + let (report, seen) = run(&fake, &command, Selection::All).await; + assert!(report.complete(), "{report:?}"); + assert!(!mem.keys().contains(&stale)); + assert!( + seen[0] + .notes + .iter() + .any(|n| n.starts_with("deleted the copy an earlier run left")), + "{seen:?}" + ); + assert_eq!(copied_items(&mem, &arn).len(), 3); +} + +/// A catalog row of the configured store under the ARN (a `backup sync` +/// that adopted an earlier run's copy before its replacement, or a +/// MongoDB replacement that stopped before its drop, whose in-catalog +/// backup is DELETING) is finished: the replacement with that row removes +/// what is left of the in-catalog backup, and nothing is copied. +#[tokio::test] +async fn a_backup_whose_row_is_in_the_store_is_finished_without_a_copy() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let adopted = legacy_arn(1); + let deleting = legacy_arn(2); + seed_legacy(&fake, &adopted, legacy_items(3)); + seed_legacy(&fake, &deleting, legacy_items(3)); + fake.legacy_keys.lock().unwrap()[1].status = BackupStatus::Deleting; + for arn in [&adopted, &deleting] { + let mut existing = row(arn, BackupStatus::Available); + existing.owner_generation = UNOWNED_GENERATION; + fake.rows.lock().unwrap().insert(arn.clone(), existing); + } + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + assert!(report.complete(), "{report:?}"); + assert_eq!(outcome_of(&report, &adopted), &Outcome::Finished); + assert_eq!(outcome_of(&report, &deleting), &Outcome::Finished); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(fake.items_opened.lock().unwrap().is_empty()); + assert!(!legacy_kept(&fake, &adopted)); + let finished: Vec = fake + .replaces_seen + .lock() + .unwrap() + .iter() + .map(|r| r.backup_arn.clone()) + .collect(); + assert_eq!(finished, [adopted, deleting]); +} + +/// A replacement whose call fails (its outcome unknown) stops the run with +/// the copy left; the next run, finding no row, deletes the copy, writes it +/// again and replaces the backup. +#[tokio::test] +async fn a_replacement_that_fails_leaves_the_copy_for_the_next_run() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = legacy_arn(1); + seed_legacy(&fake, &arn, legacy_items(3)); + seed_legacy(&fake, &legacy_arn(2), legacy_items(1)); + *fake.replace_error.lock().unwrap() = Some("the connection was reset".to_owned()); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + let stopped = report.stopped.clone().expect("stopped"); + assert!( + stopped.contains("the connection was reset") && stopped.contains("left for the next run"), + "{stopped}" + ); + assert!(report.backups.is_empty(), "{report:?}"); + assert!(mem.keys().contains(&manifest_key(&arn))); + assert!(legacy_kept(&fake, &arn)); + assert!( + fake.items_opened.lock().unwrap().len() == 1, + "the second backup was not reached" + ); + + let (again, seen) = run(&fake, &command, Selection::All).await; + assert!(again.complete(), "{again:?}"); + assert!(matches!(outcome_of(&again, &arn), Outcome::Migrated { .. })); + assert!(seen[0].notes[0].starts_with("deleted the copy an earlier run left")); + assert!(!legacy_kept(&fake, &arn)); +} + +/// A race lost at the replacement: a DeleteBackup that came first +/// (`NotAvailable`) skips the backup, a restore whose target is still +/// CREATING (`InUse`) refuses it naming `remove-target`; either way its copy +/// is deleted and the next backup runs. The in-use backup keeps its +/// in-catalog rows. +#[tokio::test] +async fn a_backup_deleted_or_in_use_at_the_replacement_has_its_copy_deleted() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let gone = legacy_arn(1); + let in_use = legacy_arn(2); + let next = legacy_arn(3); + seed_legacy(&fake, &gone, legacy_items(1)); + seed_legacy(&fake, &in_use, legacy_items(1)); + seed_legacy(&fake, &next, legacy_items(1)); + *fake.replace_answer.lock().unwrap() = + Some(Replaced::NotAvailable(format!("Backup not found: {gone}"))); + fake.in_use + .lock() + .unwrap() + .insert(in_use.clone(), "MusicRestored".to_owned()); + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + assert!( + matches!(outcome_of(&report, &gone), Outcome::Skipped(why) if why.contains("no longer AVAILABLE")), + "{report:?}" + ); + let why = refusal(outcome_of(&report, &in_use)); + assert!( + why.contains("MusicRestored") && why.contains("remove-target"), + "{why}" + ); + assert!(copy_of(&mem, &gone).is_empty() && copy_of(&mem, &in_use).is_empty()); + assert!(legacy_kept(&fake, &in_use)); + assert!(matches!( + outcome_of(&report, &next), + Outcome::Migrated { .. } + )); +} + +/// A CREATING or DELETING in-catalog backup with no row is skipped and +/// named, unopened; the AVAILABLE one after them is migrated. +#[tokio::test] +async fn creating_and_deleting_in_catalog_backups_are_skipped() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + for n in 1..=3 { + seed_legacy(&fake, &legacy_arn(n), legacy_items(1)); + } + fake.legacy_keys.lock().unwrap()[0].status = BackupStatus::Creating; + fake.legacy_keys.lock().unwrap()[1].status = BackupStatus::Deleting; + let command = new_command(&mem); + let (report, _) = run(&fake, &command, Selection::All).await; + assert!( + matches!(outcome_of(&report, &legacy_arn(1)), Outcome::Skipped(why) if why.contains("CREATING")) + ); + assert!( + matches!(outcome_of(&report, &legacy_arn(2)), Outcome::Skipped(why) if why.contains("DeleteBackup again")) + ); + assert!(matches!( + outcome_of(&report, &legacy_arn(3)), + Outcome::Migrated { .. } + )); + assert_eq!(*fake.items_opened.lock().unwrap(), [legacy_arn(3)]); + assert!(!report.complete()); +} + +/// A signal received before a backup stops the run there, and a claim a +/// heartbeat found lost stops it too; neither touches the store or the +/// in-catalog backups. +#[tokio::test] +async fn a_signal_or_a_lost_claim_stops_the_run_before_the_next_backup() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + seed_legacy(&fake, &legacy_arn(1), legacy_items(1)); + let command = new_command(&mem); + command.signal.send(Some("SIGINT")).unwrap(); + let (report, _) = run(&fake, &command, Selection::All).await; + assert_eq!(report.stopped.as_deref(), Some("SIGINT received")); + assert!(report.backups.is_empty()); + + let command = new_command(&mem); + let lost = maintenance::HeldClaim::start( + Box::new(Forward(Arc::clone(&command.holder))), + Duration::from_millis(10), + ); + *command.holder.lost.lock().unwrap() = true; + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lost.lost().is_none() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + let command = Command { + claim: lost, + ..command + }; + let (report, _) = run(&fake, &command, Selection::All).await; + assert!( + report + .stopped + .as_deref() + .is_some_and(|s| s.contains("lost")), + "{report:?}" + ); + assert!(report.backups.is_empty()); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(legacy_kept(&fake, &legacy_arn(1))); +} + +/// The maintenance claim as the writer reads it: once a heartbeat finds it +/// lost, the pre-put check fails, the loss token the writer selects on is +/// cancelled (so a copy stops at its next await, not at the next backup), +/// and the reason is the heartbeat's. +#[tokio::test] +async fn a_lost_maintenance_claim_cancels_the_writers_token() { + use super::super::job::writer::ClaimWatch; + let holder = Arc::new(FakeHolder::default()); + let claim = maintenance::HeldClaim::start( + Box::new(Forward(Arc::clone(&holder))), + Duration::from_millis(10), + ); + assert!(ClaimWatch::confirms_the_claim(&claim)); + assert!(!ClaimWatch::lost(&claim).is_cancelled()); + *holder.lost.lock().unwrap() = true; + tokio::time::timeout( + Duration::from_secs(10), + ClaimWatch::lost(&claim).cancelled(), + ) + .await + .expect("the loss token was cancelled"); + assert!(!ClaimWatch::confirms_the_claim(&claim)); + assert!( + ClaimWatch::reason(&claim).contains("another holder took it over"), + "{}", + ClaimWatch::reason(&claim) + ); +} + +/// `` names the backup as DescribeBackup resolves it: another region +/// written by the operator finds the backup this server issued. An ARN no +/// in-catalog backup has is refused; one a row of the configured store +/// holds is already in the store. +#[tokio::test] +async fn an_arn_names_the_backup_describe_backup_resolves() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = legacy_arn(1); + seed_legacy(&fake, &arn, legacy_items(2)); + seed_legacy(&fake, &legacy_arn(2), legacy_items(2)); + let command = new_command(&mem); + let written = arn.replace(REGION, "eu-west-1"); + let (report, _) = run(&fake, &command, Selection::Arn(&written)).await; + assert!(report.complete(), "{report:?}"); + assert_eq!(report.backups.len(), 1); + assert!(matches!( + outcome_of(&report, &arn), + Outcome::Migrated { .. } + )); + assert!(legacy_kept(&fake, &legacy_arn(2)), "only the named backup"); + + let (again, _) = run(&fake, &command, Selection::Arn(&arn)).await; + assert_eq!(outcome_of(&again, &arn), &Outcome::InStore); + assert!(again.complete()); + + let unknown = legacy_arn(9); + let (none, _) = run(&fake, &command, Selection::Arn(&unknown)).await; + assert!(refusal(outcome_of(&none, &unknown)).contains("no in-catalog backup")); + assert!(!none.complete()); +} + +/// `backup sync` skips a prefix whose ARN an in-catalog backup holds, under +/// the manifest's spelling and under the one this server issues, naming +/// `migrate-legacy`; no row is inserted, so the backup is not in the +/// catalog twice. +#[tokio::test] +async fn sync_skips_a_prefix_whose_arn_an_in_catalog_backup_holds() { + use super::super::sync::SkipReason; + let identity = store(); + for (described, written) in [ + (sync_arn(1), sync_arn(1)), + (sync_arn(2), sync_arn(2).replace(REGION, "eu-west-1")), + ] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut description = backup_description("AVAILABLE", "PAY_PER_REQUEST"); + description.backup_details.backup_arn = described.clone(); + fake.descriptions + .lock() + .unwrap() + .insert(described.clone(), description); + put_backup(&mem, &written); + let report = super::super::sync::sync(&*fake, sync_target(&mem, &identity)) + .await + .unwrap(); + assert!(report.adopted.is_empty(), "{report:?}"); + let reason = &skipped_by_prefix(&report)[&prefix_of(&written)]; + assert_eq!( + reason, + &SkipReason::InCatalog { + backup_arn: described.clone(), + status: "AVAILABLE".to_owned(), + } + ); + assert!( + reason + .to_string() + .contains(&format!("extenddb backup migrate-legacy {described}")), + "{reason}" + ); + assert!(!reason.is_transient()); + assert!(fake.insert_attempts.lock().unwrap().is_empty()); + } +} + +/// ListBackups lists a backup both readers hold (a copy `backup sync` +/// adopted before the replacement) once, with the catalog row's summary; +/// the other in-catalog backups are listed as before. +#[tokio::test] +async fn list_backups_lists_a_backup_both_readers_hold_once_with_its_row() { + let both = summary(1, 1_791_976_431_001); + let legacy_only = summary(2, 1_791_976_431_002); + let fake = fake_with_summaries(vec![ + BackupSummary { + backup_name: "legacy-name".to_owned(), + ..both.clone() + }, + legacy_only.clone(), + ]); + let mut indexed = row(&both.backup_arn, BackupStatus::Available); + indexed.backup_name = "indexed-name".to_owned(); + fake.rows + .lock() + .unwrap() + .insert(both.backup_arn.clone(), indexed); + let page = list(&fake, json!({})).await.unwrap(); + let mut listed = arns(&page); + listed.sort(); + assert_eq!(listed, [both.backup_arn.clone(), legacy_only.backup_arn]); + let names: Vec<&str> = page["BackupSummaries"] + .as_array() + .unwrap() + .iter() + .filter(|s| s["BackupArn"] == both.backup_arn.as_str()) + .map(|s| s["BackupName"].as_str().unwrap()) + .collect(); + assert_eq!(names, ["indexed-name"]); +} diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index fd7a2faa4..f56cb6080 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -12,8 +12,8 @@ pub use backup::{ BackupRuntime, ConfiguredStore, HeldClaim, JobSettings, KillPoint, LOCAL_CHECK_TIMEOUT, OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, StoreFence, StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, TARGET_NOT_EMPTY, - check_configured_store, check_configured_store_read_only, maintenance, read_identity_bounded, - reconciler, store_identity, store_maintenance, sync, + check_configured_store, check_configured_store_read_only, maintenance, migrate_legacy, + read_identity_bounded, reconciler, store_identity, store_maintenance, sync, }; mod batch_get_item; mod batch_write_item; diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs index e1ca1caac..2f07b9090 100644 --- a/crates/storage-cassandra/src/backup_catalog.rs +++ b/crates/storage-cassandra/src/backup_catalog.rs @@ -8,7 +8,9 @@ //! //! An index row is a `backups_by_arn` row with `store_kind` set; a row without //! it is an in-catalog backup of the legacy reader (`backup_engine.rs`), and -//! each path skips the other's rows. `backups_by_arn`, partitioned by +//! each path skips the other's rows. `extenddb backup migrate-legacy` turns +//! an in-catalog row into its index row in place +//! (`CassandraEngine::replace_legacy_row`), so an ARN never has both. `backups_by_arn`, partitioned by //! `(account_id, backup_arn)`, is authoritative and is written first by every //! writer; the list rows and the `backup_index_by_status` row follow, and a //! reader of those re-reads the authoritative row and skips an entry it no @@ -118,7 +120,7 @@ use extenddb_core::backup_format::BackupArn; use extenddb_core::types::CreateTableInput; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, ForgetRow, Heartbeat, JobClaim, - JobRow, JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + JobRow, JobStatus, MaintenanceClaim, Replaced, RestoreJobRow, RestoreKind, RestoreRegistered, RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, StoreRootCount, TimeBounds, UNOWNED_GENERATION, WriteOutcome, billing_mode_name, new_generation, @@ -3091,6 +3093,197 @@ impl MaintenanceClaim for CassandraMaintenanceClaim { } } +/// The conversion `replace_with_index_row` makes: the in-catalog row turned +/// into `row`'s index row in place, conditioned on the row being an +/// AVAILABLE in-catalog row with no registered restore (the condition +/// DeleteBackup's claim and a restore's registration are ordered against on +/// the same partition). +fn convert_statement(catalog: &str) -> String { + format!( + "UPDATE {catalog}.backups_by_arn SET store_kind = ?, store_root = ?, store_id = ?, \ + backup_id = ?, backup_name = ?, table_id = ?, table_name = ?, table_arn = ?, \ + backup_size_bytes = ?, item_count = ?, key_schema = ?, attribute_definitions = ?, \ + billing_mode = ?, table_created_at = ?, failure_reason = null, \ + source_table_details_json = ?, owner_generation = ?, \ + completed_at = toTimestamp(now()), heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND backup_arn = ? \ + IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null" + ) +} + +impl CassandraEngine { + /// `replace_with_index_row` on Cassandra, where the in-catalog row and + /// the index row of one ARN are the same `backups_by_arn` row: one + /// lightweight transaction sets the row's store columns, sizes, source + /// table and an unowned generation (the in-catalog reader skips a row + /// with `store_kind` set from then on, and the index reader reads it), + /// then the list and status entries are written and the item buckets + /// deleted. The list rows keep their keys: both kinds key them by the + /// id's millisecond. The `definition` column stays, unread. + /// + /// Before the transaction a registered restore whose target is still + /// CREATING answers `InUse`, and stale registrations are cleared so the + /// condition can apply. `NotMatched` reads the row again and decides + /// anew, up to three times. A lost answer (`Unknown`) is settled by a + /// serial read: the row converted with this store is a conversion, the + /// row still in-catalog means the proposal did not commit and the write + /// is sent again. A row already converted with this account and store + /// (an earlier call that stopped after the transaction) is + /// `AlreadyReplaced`, and its entries and buckets are written and + /// deleted again; an index row under another store is `AlreadyExists`. + pub(crate) async fn replace_legacy_row( + &self, + row: &BackupRow, + ) -> Result { + /// Reads and transactions before the call gives up on a row whose + /// condition keeps failing (a registration or a delete racing it). + const ATTEMPTS: usize = 3; + + Self::validate_account_id(&row.account_id)?; + let source = row.source_table()?; + let key_schema = serde_json::to_string(&source.key_schema) + .map_err(|e| StorageError::Internal(format!("Serialize key schema: {e}")))?; + let attribute_definitions = serde_json::to_string(&source.attribute_definitions) + .map_err(|e| StorageError::Internal(format!("Serialize attributes: {e}")))?; + let source_table_details_json = row.source_table_details_json.to_string(); + let convert = convert_statement(&self.catalog_keyspace()); + let values = || { + cdrs_tokio::query_values!( + row.store.store_kind.as_str(), + row.store.store_root.as_str(), + row.store.store_id.as_str(), + row.backup_id.as_str(), + row.backup_name.as_str(), + source.table_id.as_str(), + row.table_name.as_str(), + source.table_arn.as_str(), + row.size_bytes, + row.item_count, + key_schema.as_str(), + attribute_definitions.as_str(), + billing_mode_name(source.billing_mode), + seconds_to_millis(source.table_creation_date_time), + source_table_details_json.as_str(), + generation_column(UNOWNED_GENERATION), + row.account_id.as_str(), + row.backup_arn.as_str() + ) + }; + let is_this_row = |read: &BackupRow| { + read.account_id == row.account_id + && read.backup_arn == row.backup_arn + && read.store == row.store + }; + let mut sent = false; + for _ in 0..ATTEMPTS { + let Some(stored) = self + .load_stored_backup(&row.account_id, &row.backup_arn) + .await? + else { + return Ok(Replaced::NotAvailable(format!( + "Backup not found: {}", + row.backup_arn + ))); + }; + if !stored.is_in_catalog() { + let read = self + .read_index_row_serial(&row.account_id, &row.backup_arn) + .await?; + return match read { + Some(read) if is_this_row(&read) => { + self.finish_conversion(row, &source, stored.item_count) + .await?; + Ok(if sent { + Replaced::Replaced + } else { + Replaced::AlreadyReplaced + }) + } + _ => Err(StorageError::AlreadyExists(format!( + "backup {} has an index row in the {} store at {:?} ({})", + row.backup_arn, + stored.store_kind.as_deref().unwrap_or("?"), + stored.store_root.as_deref().unwrap_or("?"), + stored.store_id.as_deref().unwrap_or("?"), + ))), + }; + } + if stored.backup_status != "AVAILABLE" { + return Ok(Replaced::NotAvailable(format!( + "the in-catalog backup is {}", + stored.backup_status + ))); + } + if let Some(target) = self.creating_restore_target(&stored).await? { + return Ok(Replaced::InUse(target)); + } + let fault = crate::test_hooks::convert_write_fault(&row.backup_arn); + let outcome = if fault == Some(crate::test_hooks::ProgressWriteFault::Lost) { + WriteOutcome::Unknown + } else { + let outcome = conditional_write(&self.session, &convert, values()).await?; + if fault.is_some() { + WriteOutcome::Unknown + } else { + outcome + } + }; + match outcome { + WriteOutcome::Matched => { + crate::test_hooks::convert_stop(&row.backup_arn)?; + self.finish_conversion(row, &source, stored.item_count) + .await?; + return Ok(Replaced::Replaced); + } + // Read again: the next pass decides from what is there. + WriteOutcome::NotMatched => {} + WriteOutcome::Unknown => { + sent = true; + if self + .read_index_row_serial(&row.account_id, &row.backup_arn) + .await? + .is_some_and(|read| is_this_row(&read)) + { + crate::test_hooks::convert_stop(&row.backup_arn)?; + self.finish_conversion(row, &source, stored.item_count) + .await?; + return Ok(Replaced::Replaced); + } + } + } + } + Err(StorageError::Internal(format!( + "the conversion of backup {} did not apply after {ATTEMPTS} attempts", + row.backup_arn + ))) + } + + /// The writes after a conversion: the list and status entries (sent + /// once more on a failure, as the insert does), then the in-catalog item + /// buckets, of which `item_count` the row accounted for (and any after + /// them, as the frozen delete finds them). + async fn finish_conversion( + &self, + row: &BackupRow, + source: &extenddb_core::backup_format::SourceTableSchema, + item_count: i64, + ) -> Result<(), StorageError> { + if let Err(first) = self + .write_index_entries(row, &source.table_id, &source.table_arn, row.status) + .await + { + tracing::warn!( + "backup {}: index entries not written, sending them once more: {first}", + row.backup_arn + ); + self.write_index_entries(row, &source.table_id, &source.table_arn, row.status) + .await?; + } + self.cleanup_backup_payload(&row.account_id, &row.backup_arn, item_count) + .await + } +} + #[cfg(test)] mod tests { use std::net::{IpAddr, Ipv4Addr, SocketAddr}; diff --git a/crates/storage-cassandra/src/backup_engine.rs b/crates/storage-cassandra/src/backup_engine.rs index ecb73e318..7c0a2bc7a 100755 --- a/crates/storage-cassandra/src/backup_engine.rs +++ b/crates/storage-cassandra/src/backup_engine.rs @@ -69,6 +69,19 @@ //! copies; the request future waits only for the registration outcome and //! returns the CREATING description, so a dropped request does not stop the //! restore. +//! +//! # `extenddb backup migrate-legacy` +//! +//! `items` reads a backup's buckets in order, each in pages by `item_index` +//! (a copy of the restore's read, which fetches a bucket whole). +//! `replace_with_index_row` converts the `backups_by_arn` row into its +//! catalog index row in place with one lightweight transaction conditioned +//! as DeleteBackup's claim is plus `store_kind = null` +//! (`backup_catalog.rs`), then writes its list and status entries and +//! deletes the buckets; the row is never both kinds, and no DELETED row is +//! left. DeleteBackup's claim carries `store_kind = null` too, so a delete +//! that read the in-catalog row before the conversion does not claim the +//! index row after it. use std::collections::HashMap; use std::sync::Arc; @@ -81,14 +94,15 @@ use cdrs_tokio::types::map::Map; use cdrs_tokio::types::{AsRust, IntoRustByName, rows::Row}; use extenddb_core::types::{ AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, CreateTableInput, - DeleteTableInput, GsiInput, KeySchemaElement, LsiInput, ProvisionedThroughput, + DeleteTableInput, GsiInput, Item, KeySchemaElement, LsiInput, ProvisionedThroughput, SourceTableDetails, TableDescription, TableKeyInfo, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, RestoreRegistrationRow, new_generation, + AbandonedLegacyBackup, BackupRow, BackupStatus, LegacyBackupItems, LegacyBackupKey, Replaced, + RestoreRegistrationRow, new_generation, }; use extenddb_storage::backup_definition::{ - BACKUP_DEFINITION_VERSION, BackupTableDefinition, throughput_from_catalog, + BACKUP_DEFINITION_VERSION, BackupTableDefinition, LegacyVectorState, throughput_from_catalog, }; use extenddb_storage::error::StorageError; use extenddb_storage::{ @@ -182,7 +196,7 @@ pub(crate) struct StoredBackup { pub(crate) backup_status: String, backup_type: String, backup_size_bytes: i64, - item_count: i64, + pub(crate) item_count: i64, key_schema: String, attribute_definitions: String, billing_mode: String, @@ -561,7 +575,7 @@ impl CassandraEngine { /// The create writes buckets in order and publishes `item_count` last, so /// a create that died mid-copy leaves a row whose count is 0 and whose /// buckets are contiguous from 0; the probe finds them all. - async fn cleanup_backup_payload( + pub(crate) async fn cleanup_backup_payload( &self, account_id: &str, backup_arn: &str, @@ -2141,7 +2155,7 @@ impl LegacyBackupReader for CassandraEngine { "UPDATE {}.backups_by_arn SET backup_status = 'DELETING', \ heartbeat_at = toTimestamp(now()) \ WHERE account_id = ? AND backup_arn = ? \ - IF backup_status = 'AVAILABLE' AND active_restores = null", + IF backup_status = 'AVAILABLE' AND active_restores = null AND store_kind = null", self.catalog_keyspace() ); let mut attempt = 0; @@ -2217,44 +2231,7 @@ impl LegacyBackupReader for CassandraEngine { Box::pin(async move { Self::validate_account_id(&account_id)?; let backup = self.load_available_backup(&account_id, &backup_arn).await?; - let definition = backup.parsed_definition()?; - let provisioned_throughput = match &definition { - Some(definition) => definition.provisioned_throughput.clone(), - // A backup row written before V005: the throughput column it - // has always carried. That column holds {0, 0} for an - // on-demand source; treat a non-positive value as unrecorded, - // as the restore of catalog 0.0.3 did (it fell back to 5/5). - None => { - let value = backup - .provisioned_throughput - .as_deref() - .map(serde_json::from_str::) - .transpose() - .map_err(|e| StorageError::Internal(format!("Parse throughput: {e}")))?; - throughput_from_catalog(value.as_ref()) - .filter(|pt| pt.read_capacity_units > 0 && pt.write_capacity_units > 0) - } - }; - Ok(BackupRestoreSource { - backup_arn: backup.backup_arn.clone(), - backup_creation_date_time: crate::cassandra_util::millis_to_seconds_f64( - backup.created_at, - ), - source_table_name: backup.table_name.clone(), - source_table_arn: backup.table_arn.clone(), - source_table_id: backup.table_id.clone(), - key_schema: backup.parsed_key_schema()?, - attribute_definitions: backup.parsed_attribute_definitions()?, - billing_mode: backup.billing_mode.clone(), - provisioned_throughput, - item_count: backup.item_count, - table_size_bytes: backup.backup_size_bytes, - table_creation_date_time: crate::cassandra_util::millis_to_seconds_f64( - backup.table_created_at, - ), - definition, - legacy_vector_index_count: 0, - }) + restore_source_of(&backup) }) } @@ -2372,6 +2349,221 @@ impl LegacyBackupReader for CassandraEngine { Ok(abandoned) }) } + + /// A range read of `backups_by_arn` at quorum (no index serves every + /// account; the restore-registration sweep reads the table the same + /// way), keeping the in-catalog rows (`store_kind` null) that are not + /// DELETED. + fn list_all(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async move { + let scan = format!( + "SELECT account_id, backup_arn, backup_status, store_kind, created_at \ + FROM {}.backups_by_arn", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &scan, + cdrs_tokio::query_values!(), + "in-catalog backups", + ) + .await?; + let mut keys = Vec::new(); + for row in rows { + let store_kind: Option = row.get_by_name("store_kind").ok().flatten(); + let status: String = + crate::cassandra_util::get_column(&row, "backup_status", "in-catalog backups")?; + if store_kind.is_some() || status == "DELETED" { + continue; + } + let created_at: i64 = row.get_by_name("created_at").ok().flatten().unwrap_or(0); + keys.push(( + created_at, + LegacyBackupKey { + account_id: crate::cassandra_util::get_column( + &row, + "account_id", + "in-catalog backups", + )?, + backup_arn: crate::cassandra_util::get_column( + &row, + "backup_arn", + "in-catalog backups", + )?, + status: BackupStatus::parse(&status)?, + }, + )); + } + keys.sort_by(|(a_at, a), (b_at, b)| { + a_at.cmp(b_at).then_with(|| a.backup_arn.cmp(&b.backup_arn)) + }); + Ok(keys.into_iter().map(|(_, key)| key).collect()) + }) + } + + /// The AVAILABLE in-catalog row (its source as `restore_source` reads + /// it), then its item buckets in order, each read in pages of + /// [`LEGACY_ITEMS_PAGE`] rows by `item_index` rather than whole as the + /// restore's copy reads it, so one response holds at most a page of + /// items at the line cap. + fn items( + &self, + account_id: &str, + backup_arn: &str, + ) -> BoxFuture<'_, Result> { + let account_id = account_id.to_owned(); + let backup_arn = backup_arn.to_owned(); + Box::pin(async move { + Self::validate_account_id(&account_id)?; + let backup = self.load_available_backup(&account_id, &backup_arn).await?; + let source = restore_source_of(&backup)?; + let cursor = LegacyItemsCursor { + engine: self.clone(), + query: format!( + "SELECT item_index, item_data FROM {}.backup_items \ + WHERE backup_arn = ? AND bucket = ? AND item_index > ? LIMIT ?", + self.account_keyspace(&account_id) + ), + backup_arn, + buckets: bucket_count(backup.item_count), + bucket: 0, + after: -1, + page: std::collections::VecDeque::new(), + done: false, + }; + Ok(LegacyBackupItems { + backup_name: backup.backup_name.clone(), + source, + vector_state: LegacyVectorState::Unsupported, + items: Box::pin(futures::stream::unfold(cursor, LegacyItemsCursor::next)), + }) + }) + } + + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(self.replace_legacy_row(row)) + } +} + +/// Rows of one item bucket `items` reads per query. +const LEGACY_ITEMS_PAGE: i32 = 8; + +/// Where `items`' stream is: the bucket, the last `item_index` read in it, +/// and the page not yet handed out. +struct LegacyItemsCursor { + engine: CassandraEngine, + query: String, + backup_arn: String, + buckets: i64, + bucket: i64, + after: i64, + page: std::collections::VecDeque, + done: bool, +} + +impl LegacyItemsCursor { + async fn next(mut self) -> Option<(Result, Self)> { + while self.page.is_empty() { + if self.done || self.bucket >= self.buckets { + return None; + } + if let Err(e) = self.read_page().await { + self.done = true; + return Some((Err(e), self)); + } + } + let item_data = self.page.pop_front()?; + let item = serde_json::from_str::(&item_data) + .map_err(|e| StorageError::Internal(format!("Deserialize backup item: {e}"))); + if item.is_err() { + self.done = true; + } + Some((item, self)) + } + + /// The next page of the current bucket; a short page ends the bucket. + async fn read_page(&mut self) -> Result<(), StorageError> { + #[allow(clippy::cast_possible_truncation)] + let bucket = self.bucket as i32; + let rows = self + .engine + .session + .query_with_values( + &self.query, + cdrs_tokio::query_values!( + self.backup_arn.as_str(), + bucket, + self.after, + LEGACY_ITEMS_PAGE + ), + ) + .await + .map_err(|e| StorageError::Internal(format!("Read backup payload: {e}")))? + .response_body() + .map_err(|e| StorageError::Internal(format!("Parse backup payload: {e}")))? + .into_rows() + .unwrap_or_default(); + let full = rows.len() >= usize::try_from(LEGACY_ITEMS_PAGE).unwrap_or(usize::MAX); + for row in rows { + let item_index: i64 = row + .get_r_by_name("item_index") + .map_err(|e| StorageError::Internal(format!("Parse backup item: {e}")))?; + let item_data: String = row + .get_r_by_name("item_data") + .map_err(|e| StorageError::Internal(format!("Parse backup item: {e}")))?; + self.after = item_index; + self.page.push_back(item_data); + } + if !full { + self.bucket += 1; + self.after = -1; + } + Ok(()) + } +} + +/// The recorded source of an AVAILABLE in-catalog `backup`, as +/// `restore_source` answers it. +fn restore_source_of(backup: &StoredBackup) -> Result { + let definition = backup.parsed_definition()?; + let provisioned_throughput = match &definition { + Some(definition) => definition.provisioned_throughput.clone(), + // A backup row written before V005: the throughput column it + // has always carried. That column holds {0, 0} for an + // on-demand source; treat a non-positive value as unrecorded, + // as the restore of catalog 0.0.3 did (it fell back to 5/5). + None => { + let value = backup + .provisioned_throughput + .as_deref() + .map(serde_json::from_str::) + .transpose() + .map_err(|e| StorageError::Internal(format!("Parse throughput: {e}")))?; + throughput_from_catalog(value.as_ref()) + .filter(|pt| pt.read_capacity_units > 0 && pt.write_capacity_units > 0) + } + }; + Ok(BackupRestoreSource { + backup_arn: backup.backup_arn.clone(), + backup_creation_date_time: crate::cassandra_util::millis_to_seconds_f64(backup.created_at), + source_table_name: backup.table_name.clone(), + source_table_arn: backup.table_arn.clone(), + source_table_id: backup.table_id.clone(), + key_schema: backup.parsed_key_schema()?, + attribute_definitions: backup.parsed_attribute_definitions()?, + billing_mode: backup.billing_mode.clone(), + provisioned_throughput, + item_count: backup.item_count, + table_size_bytes: backup.backup_size_bytes, + table_creation_date_time: crate::cassandra_util::millis_to_seconds_f64( + backup.table_created_at, + ), + definition, + legacy_vector_index_count: 0, + }) } #[cfg(test)] diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs index 67e698b85..b1e937634 100644 --- a/crates/storage-cassandra/src/test_hooks.rs +++ b/crates/storage-cassandra/src/test_hooks.rs @@ -156,6 +156,8 @@ mod armed { pub(super) after_delete_claim: HashMap>, pub(super) after_deleted_write: HashMap>, pub(super) before_tombstone_removal: HashMap>, + pub(super) convert_write_faults: HashMap>, + pub(super) convert_stops: std::collections::HashSet, } pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default); @@ -464,6 +466,63 @@ pub fn stop_rewrite(backup_arn: &str, at: RewriteStop) { }); } +/// Make the next lightweight transactions that convert the in-catalog row +/// of `backup_arn` into its index row (`replace_with_index_row`) answer +/// `WriteOutcome::Unknown`, one fault per write in the order given, as +/// [`fail_insert_writes`] does for the insert. +#[cfg(feature = "test-hooks")] +pub fn fail_convert_writes(backup_arn: &str, faults: &[ProgressWriteFault]) { + armed::with_hooks(|hooks| { + hooks + .convert_write_faults + .insert(backup_arn.to_owned(), faults.iter().copied().collect()); + }); +} + +/// Stop the next `replace_with_index_row` of `backup_arn` with an error +/// right after its row's conversion, before the list and status entries and +/// the item buckets, as a process that dies there would. Taken once. +#[cfg(feature = "test-hooks")] +pub fn stop_convert_after_row(backup_arn: &str) { + armed::with_hooks(|hooks| { + hooks.convert_stops.insert(backup_arn.to_owned()); + }); +} + +/// The fault the next conversion lightweight transaction of `backup_arn` +/// takes, if a test armed one. +pub(crate) fn convert_write_fault(backup_arn: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + armed::with_hooks(|hooks| { + hooks + .convert_write_faults + .get_mut(backup_arn) + .and_then(std::collections::VecDeque::pop_front) + }) + } + #[cfg(not(feature = "test-hooks"))] + { + let _ = backup_arn; + None + } +} + +/// The error a conversion of `backup_arn` stops with after its row, if a +/// test armed [`stop_convert_after_row`]. +pub(crate) fn convert_stop(backup_arn: &str) -> Result<(), StorageError> { + #[cfg(feature = "test-hooks")] + { + if armed::with_hooks(|hooks| hooks.convert_stops.remove(backup_arn)) { + return Err(StorageError::Internal(format!( + "test hook: the conversion of {backup_arn} stopped after its row" + ))); + } + } + let _ = backup_arn; + Ok(()) +} + /// The fault the next rewrite lightweight transaction of `backup_arn` takes, /// if a test armed one. pub(crate) fn rewrite_write_fault(backup_arn: &str) -> Option { diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index f6181255f..38cd965bf 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -10439,3 +10439,528 @@ async fn test_the_entries_deleters_leave_behind_a_rewrite_are_swept_from_the_old } remove_index_rows(&engine, &all.iter().collect::>()).await; } + +// --- `extenddb backup migrate-legacy`: the item stream, `list_all`, the +// conversion of an in-catalog row into its index row. + +/// The index row of the store copy of the in-catalog backup `backup_arn`, as +/// the migration builds it. +#[cfg(feature = "test-hooks")] +async fn migrated_row( + engine: &CassandraEngine, + account_id: &str, + backup_arn: &str, + store: &StoreIdentity, +) -> BackupRow { + let source = engine + .restore_source(account_id, backup_arn) + .await + .expect("source"); + let table = extenddb_storage::backup_definition::source_table_from_legacy( + &source, + extenddb_storage::backup_definition::LegacyVectorState::Unsupported, + "us-east-1", + account_id, + ) + .expect("source table"); + let description = engine + .describe(account_id, backup_arn) + .await + .expect("describe"); + BackupRow { + backup_id: BackupArn::parse(backup_arn) + .expect("arn") + .backup_id() + .to_owned(), + account_id: account_id.to_owned(), + table_name: source.source_table_name.clone(), + backup_arn: backup_arn.to_owned(), + backup_name: description.backup_details.backup_name, + status: BackupStatus::Available, + store: store.clone(), + size_bytes: 4321, + item_count: source.item_count, + created_at: source.backup_creation_date_time, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::to_value(&table).expect("json"), + owner_generation: 77, + heartbeat_at: 0.0, + } +} + +/// `item_data` of `backup_arn`'s payload rows in (bucket, `item_index`) +/// order, the order the create wrote them. +#[cfg(feature = "test-hooks")] +async fn payload_in_order( + engine: &CassandraEngine, + account_id: &str, + backup_arn: &str, + buckets: i32, +) -> Vec { + use cdrs_tokio::types::IntoRustByName; + let mut items = Vec::new(); + for bucket in 0..buckets { + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT item_data FROM {}.backup_items WHERE backup_arn = ? AND bucket = ?", + engine.account_keyspace(account_id) + ), + cdrs_tokio::query_values!(backup_arn, bucket), + "payload", + ) + .await + .expect("read a bucket"); + for row in rows { + let data: String = row.get_r_by_name("item_data").expect("item_data"); + items.push(data); + } + } + items +} + +/// The stream yields every item of every bucket, in bucket and +/// `item_index` order, past several pages of each bucket and a last bucket +/// that ends on a page bound (200 items: three full buckets of 64 and one +/// of 8, read in pages of 8), with the backup's name, source and the vector +/// state of a backend without vector support; a missing backup is refused. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_legacy_items_stream_every_bucket_in_pages() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let (account, key_info, backup_arn) = backed_up_plain_table(&engine, "LmSource", 200).await; + let account_id = account.account_id.clone(); + let opened = engine.items(&account_id, &backup_arn).await.expect("items"); + assert_eq!(opened.backup_name, "copy-timing"); + assert_eq!( + opened.source, + engine + .restore_source(&account_id, &backup_arn) + .await + .expect("source") + ); + assert_eq!( + opened.vector_state, + extenddb_storage::backup_definition::LegacyVectorState::Unsupported + ); + let streamed: Vec = opened + .items + .map(|item| serde_json::to_string(&item.expect("an item")).expect("text")) + .collect() + .await; + let stored: Vec = payload_in_order(&engine, &account_id, &backup_arn, 5) + .await + .iter() + .map(|text| { + let item: Item = serde_json::from_str(text).expect("item"); + serde_json::to_string(&item).expect("text") + }) + .collect(); + assert_eq!(stored.len(), 200); + assert_eq!(streamed.len(), 200); + assert_eq!( + streamed, stored, + "every item once, in bucket and index order" + ); + assert_eq!(scan_count(&engine, &key_info).await, 200); + + let err = engine + .items(&account_id, &flip_last_hex(&backup_arn)) + .await + .err(); + assert!( + matches!(&err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{err:?}" + ); +} + +/// `list_all` reads every account's in-catalog rows and leaves out the +/// index rows and the DELETED rows. The catalog keyspace is shared with +/// other tests, so the listing is checked for this test's two accounts. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_list_all_lists_the_in_catalog_rows_of_every_account() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let (account, _, backup_arn) = backed_up_plain_table(&engine, "LmList", 3).await; + let account_id = account.account_id.clone(); + let other = unique_test_account(); + let catalog = engine.catalog_keyspace(); + let raw = |account: String, arn: String, status: &'static str, created_at: i64| { + let engine = &engine; + let catalog = catalog.clone(); + async move { + engine + .session() + .query_with_values( + &format!( + "INSERT INTO {catalog}.backups_by_arn (account_id, backup_arn, \ + backup_name, table_name, backup_status, created_at) \ + VALUES (?, ?, 'raw', 'LmList', ?, ?)" + ), + cdrs_tokio::query_values!(account.as_str(), arn.as_str(), status, created_at), + ) + .await + .expect("an in-catalog row"); + } + }; + let other_arn = extenddb_storage::util::backup_arn( + "us-east-1", + &other, + "LmList", + &extenddb_storage::util::new_backup_id(), + ); + let deleted_arn = extenddb_storage::util::backup_arn( + "us-east-1", + &account_id, + "LmList", + &extenddb_storage::util::new_backup_id(), + ); + raw(other.clone(), other_arn.clone(), "DELETING", i64::MAX / 2).await; + raw(account_id.clone(), deleted_arn.clone(), "DELETED", 1).await; + let mut index = index_row(&account_id, "LmList", "index", &unique_store()); + index.status = BackupStatus::Available; + assert!(engine.insert_backup_row(&index).await.expect("insert")); + + let listed: Vec<_> = engine + .list_all() + .await + .expect("list") + .into_iter() + .filter(|k| k.account_id == account_id || k.account_id == other) + .collect(); + remove_index_rows(&engine, &[&index]).await; + engine + .session() + .query_with_values( + &format!( + "DELETE FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(other.as_str(), other_arn.as_str()), + ) + .await + .expect("remove the raw row"); + assert_eq!( + listed, + vec![ + extenddb_storage::backup_catalog::LegacyBackupKey { + account_id: account_id.clone(), + backup_arn: backup_arn.clone(), + status: BackupStatus::Available, + }, + extenddb_storage::backup_catalog::LegacyBackupKey { + account_id: other.clone(), + backup_arn: other_arn.clone(), + status: BackupStatus::Deleting, + }, + ] + ); +} + +/// The replacement converts the in-catalog row into its index row in place: +/// the index reader reads it unowned with the store and the source table, +/// its list and status entries are written, its buckets deleted, and the +/// in-catalog reader no longer sees it (describe, list, delete). A second +/// call is `AlreadyReplaced`; one with another store is `AlreadyExists`. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_replacement_converts_the_in_catalog_row_in_place() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let (account, _, backup_arn) = backed_up_plain_table(&engine, "LmConvert", 100).await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = migrated_row(&engine, &account_id, &backup_arn, &store).await; + assert_eq!( + payload_item_count(&engine, &account_id, &backup_arn).await, + 100 + ); + let answer = engine.replace_with_index_row(&row).await; + let read = engine.read_backup_row(&arn_of(&row)).await.expect("read"); + let listed = listed_in(&engine, &row).await; + let payload = payload_item_count(&engine, &account_id, &backup_arn).await; + let legacy_describe = engine.describe(&account_id, &backup_arn).await.err(); + let legacy_list = engine.list(&account_id, None).await.expect("list"); + let legacy_delete = engine.delete(&account_id, &backup_arn).await.err(); + let mut moved = row.clone(); + moved.store = unique_store(); + let again = engine.replace_with_index_row(&row).await; + let elsewhere = engine.replace_with_index_row(&moved).await.err(); + remove_index_rows(&engine, &[&row]).await; + + assert_eq!( + answer.expect("replace"), + extenddb_storage::backup_catalog::Replaced::Replaced + ); + let read = read.expect("the index row"); + assert_eq!(read.owner_generation, UNOWNED_GENERATION); + assert_eq!(read.status, BackupStatus::Available); + assert_eq!(read.store, store); + assert_eq!(read.backup_id, row.backup_id); + assert_eq!(read.backup_name, "copy-timing"); + assert_eq!(read.size_bytes, 4321); + assert_eq!(read.item_count, 100); + assert!(read.completed_at.is_some()); + assert!((read.created_at - row.created_at).abs() < 1e-3); + assert_eq!( + read.source_table().expect("source"), + row.source_table().expect("source") + ); + assert_eq!(listed, [true; 3], "status, account and table entries"); + assert_eq!(payload, 0, "the buckets are deleted"); + assert!( + matches!(&legacy_describe, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{legacy_describe:?}" + ); + assert!( + legacy_list.iter().all(|s| s.backup_arn != backup_arn), + "{legacy_list:?}" + ); + assert!( + matches!(&legacy_delete, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{legacy_delete:?}" + ); + assert_eq!( + again.expect("replace again"), + extenddb_storage::backup_catalog::Replaced::AlreadyReplaced + ); + assert!( + matches!(&elsewhere, Some(StorageError::AlreadyExists(_))), + "{elsewhere:?}" + ); +} + +/// The replacement refuses, changing nothing, while a restore target of the +/// backup is CREATING (its registration is in `active_restores`), and when +/// the backup is DELETED or missing. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_replacement_refuses_an_in_use_or_unavailable_backup_and_changes_nothing() { + use extenddb_storage::backup_catalog::Replaced; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let (account, _, backup_arn) = backed_up_plain_table(&engine, "LmRefuse", 10).await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = migrated_row(&engine, &account_id, &backup_arn, &store).await; + restore( + &engine, + &account_id, + &backup_arn, + "LmTarget", + &RestoreTableOverrides::default(), + ) + .await + .expect("restore"); + let target = wait_active(&engine, &account_id, "LmTarget", Duration::from_secs(120)).await; + // The target back in CREATING with its registration, as while its copy + // runs. + let catalog = engine.catalog_keyspace(); + for (cql, values) in [ + ( + format!( + "UPDATE {catalog}.tables SET table_status = 'CREATING' \ + WHERE account_id = ? AND table_name = ?" + ), + cdrs_tokio::query_values!(account_id.as_str(), "LmTarget"), + ), + ( + format!( + "UPDATE {catalog}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!( + target.table_id.as_str(), + "LmTarget", + account_id.as_str(), + backup_arn.as_str() + ), + ), + ] { + engine + .session() + .query_with_values(&cql, values) + .await + .expect("hold the target"); + } + let in_use = engine.replace_with_index_row(&row).await; + let read = engine.read_backup_row(&arn_of(&row)).await.expect("read"); + let payload = payload_item_count(&engine, &account_id, &backup_arn).await; + let status = backup_status_of(&engine, &account_id, &backup_arn).await; + engine + .session() + .query_with_values( + &format!( + "UPDATE {catalog}.tables SET table_status = 'ACTIVE' \ + WHERE account_id = ? AND table_name = ?" + ), + cdrs_tokio::query_values!(account_id.as_str(), "LmTarget"), + ) + .await + .expect("release the target"); + assert_eq!( + in_use.expect("replace"), + Replaced::InUse("LmTarget".to_owned()) + ); + assert!(read.is_none(), "still an in-catalog row"); + assert_eq!(payload, 10, "the buckets are kept"); + assert_eq!(status.as_deref(), Some("AVAILABLE")); + + // DELETED, then missing. + let second = engine + .create(&account_id, "LmRefuse", "second", JobPermit::detached()) + .await + .expect("second backup") + .backup_arn; + let second_row = migrated_row(&engine, &account_id, &second, &store).await; + engine.delete(&account_id, &second).await.expect("delete"); + assert_eq!( + engine + .replace_with_index_row(&second_row) + .await + .expect("replace"), + Replaced::NotAvailable("the in-catalog backup is DELETED".to_owned()) + ); + let mut missing = second_row.clone(); + missing.backup_arn = flip_last_hex(&second); + assert!(matches!( + engine.replace_with_index_row(&missing).await.expect("replace"), + Replaced::NotAvailable(m) if m.contains("Backup not found") + )); +} + +/// A lost answer of the conversion is settled by a serial read (the +/// transaction ran: `Replaced`; it never ran: sent again, `Replaced`); a +/// stop after the conversion leaves the index row with no entries and its +/// buckets, which the next call finishes as `AlreadyReplaced`; and a +/// DeleteBackup that read the in-catalog row before the conversion claims +/// nothing after it. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_conversion_whose_answer_is_lost_or_that_stopped_is_settled() { + use extenddb_storage::backup_catalog::Replaced; + use extenddb_storage_cassandra::test_hooks::{ + ProgressWriteFault, fail_convert_writes, pause_before_delete_claim, stop_convert_after_row, + }; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let (account, _, first) = backed_up_plain_table(&engine, "LmSettle", 70).await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let mut backups = vec![first]; + for name in ["second", "third", "fourth"] { + backups.push( + engine + .create(&account_id, "LmSettle", name, JobPermit::detached()) + .await + .expect("backup") + .backup_arn, + ); + } + let mut rows = Vec::new(); + for arn in &backups { + rows.push(migrated_row(&engine, &account_id, arn, &store).await); + } + + // The transaction ran; its answer was lost. + fail_convert_writes(&rows[0].backup_arn, &[ProgressWriteFault::Landed]); + let landed = engine.replace_with_index_row(&rows[0]).await; + let landed_read = engine + .read_backup_row(&arn_of(&rows[0])) + .await + .expect("read"); + let landed_listed = listed_in(&engine, &rows[0]).await; + let landed_payload = payload_item_count(&engine, &account_id, &backups[0]).await; + + // The transaction never ran: sent again. + fail_convert_writes(&rows[1].backup_arn, &[ProgressWriteFault::Lost]); + let lost = engine.replace_with_index_row(&rows[1]).await; + let lost_read = engine + .read_backup_row(&arn_of(&rows[1])) + .await + .expect("read"); + let lost_payload = payload_item_count(&engine, &account_id, &backups[1]).await; + + // A stop after the conversion, then the next call. + stop_convert_after_row(&rows[2].backup_arn); + let stopped = engine.replace_with_index_row(&rows[2]).await.err(); + let stopped_read = engine + .read_backup_row(&arn_of(&rows[2])) + .await + .expect("read"); + let stopped_listed = listed_in(&engine, &rows[2]).await; + let stopped_payload = payload_item_count(&engine, &account_id, &backups[2]).await; + let stopped_describe = engine.describe(&account_id, &backups[2]).await.err(); + let finished = engine.replace_with_index_row(&rows[2]).await; + let finished_listed = listed_in(&engine, &rows[2]).await; + let finished_payload = payload_item_count(&engine, &account_id, &backups[2]).await; + + // A DeleteBackup paused between its read and its claim while the + // conversion lands. + let pause = pause_before_delete_claim(&backups[3]); + let deleting = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let arn = backups[3].clone(); + tokio::spawn(async move { engine.delete(&account_id, &arn).await }) + }; + pause.reached().await; + let converted = engine.replace_with_index_row(&rows[3]).await; + pause.release(); + let deleted = deleting.await.expect("delete task"); + let after_delete = engine + .read_backup_row(&arn_of(&rows[3])) + .await + .expect("read"); + + remove_index_rows(&engine, &rows.iter().collect::>()).await; + + assert_eq!(landed.expect("landed"), Replaced::Replaced); + assert_eq!(landed_read.map(|r| r.store), Some(store.clone())); + assert_eq!(landed_listed, [true; 3]); + assert_eq!(landed_payload, 0); + + assert_eq!(lost.expect("lost"), Replaced::Replaced); + assert_eq!(lost_read.map(|r| r.store), Some(store.clone())); + assert_eq!(lost_payload, 0); + + assert!( + matches!(&stopped, Some(StorageError::Internal(m)) if m.contains("stopped after its row")), + "{stopped:?}" + ); + assert_eq!(stopped_read.map(|r| r.store), Some(store.clone())); + assert_eq!( + stopped_listed, + [false, true, true], + "no status entry yet; the list rows are the in-catalog backup's, keyed alike" + ); + assert_eq!(stopped_payload, 70, "the buckets are still there"); + assert!( + matches!(&stopped_describe, Some(StorageError::Validation(_))), + "the in-catalog reader no longer reads it: {stopped_describe:?}" + ); + assert_eq!(finished.expect("finished"), Replaced::AlreadyReplaced); + assert_eq!(finished_listed, [true; 3]); + assert_eq!(finished_payload, 0); + + assert_eq!(converted.expect("converted"), Replaced::Replaced); + assert!( + matches!(&deleted, Err(StorageError::Validation(m)) if m.contains("Backup not found")), + "{deleted:?}" + ); + let after_delete = after_delete.expect("the index row"); + assert_eq!(after_delete.status, BackupStatus::Available); + assert_eq!(after_delete.store, store); +} diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs index 553b2cb07..51108f92e 100644 --- a/crates/storage-mongodb/src/backup_catalog.rs +++ b/crates/storage-mongodb/src/backup_catalog.rs @@ -227,6 +227,25 @@ fn key_filter(key: &BackupKey) -> Document { doc! { "account_id": &key.account_id, "backup_id": &key.backup_id } } +/// `insert_backup_row`'s upsert of `row`, unowned, as a filter and a +/// pipeline update: an existing row never matches the filter (every row +/// carries a generation, this one 0), so the upsert's insert hits a unique +/// index, (`account_id`, `backup_id`) or `backup_id`, and nothing is +/// written. The legacy reader's `replace_with_index_row` sends it in its +/// session transaction. +pub(crate) fn unowned_insert(row: &BackupRow) -> Result<(Document, Vec), StorageError> { + let mut filter = key_filter(&row.key()); + filter.insert("owner_generation", doc! { "$exists": false }); + let mut set = row_fields(row, row.status)?; + if let Some(reason) = &row.failure_reason { + set.insert("failure_reason", reason); + } + set.insert("owner_generation", generation_field(UNOWNED_GENERATION)); + set.insert("heartbeat_at", "$$NOW"); + set.insert("completed_at", "$$NOW"); + Ok((filter, vec![doc! { "$set": literal_set(set) }])) +} + /// `set` as the `$set` stage of a pipeline update: `$$NOW` stays the server's /// clock and every other value is wrapped in `$literal`, so a string, or a /// key inside the source table object, is never read as a field path. @@ -902,23 +921,10 @@ impl BackupCatalog for MongoEngine { row: &'a BackupRow, ) -> BoxFuture<'a, Result> { Box::pin(async move { - // The claim's upsert, unowned: an existing row never matches the - // filter (every row carries a generation, this one 0), so the - // upsert's insert hits a unique index, (`account_id`, - // `backup_id`) or `backup_id`, and nothing is written. - let mut filter = key_filter(&row.key()); - filter.insert("owner_generation", doc! { "$exists": false }); - let mut set = row_fields(row, row.status)?; - if let Some(reason) = &row.failure_reason { - set.insert("failure_reason", reason); - } - set.insert("owner_generation", generation_field(UNOWNED_GENERATION)); - set.insert("heartbeat_at", "$$NOW"); - set.insert("completed_at", "$$NOW"); - let set = literal_set(set); + let (filter, update) = unowned_insert(row)?; match self .backup_index() - .update_one(filter, vec![doc! { "$set": set }]) + .update_one(filter, update) .upsert(true) .await { diff --git a/crates/storage-mongodb/src/backup_engine.rs b/crates/storage-mongodb/src/backup_engine.rs index 100223ce5..51c2a1652 100644 --- a/crates/storage-mongodb/src/backup_engine.rs +++ b/crates/storage-mongodb/src/backup_engine.rs @@ -15,6 +15,14 @@ //! from that id so the `backup_arn` (which contains slashes and colons) never //! appears in a collection name. //! +//! `extenddb backup migrate-legacy` reads a backup through `items`, a cursor +//! over that collection with a frozen copy of the decode, and ends with +//! `replace_with_index_row`: one session transaction that takes the lock +//! below, inserts the catalog index document and writes DELETING marked +//! `migrated_to_store`, then the delete's drop and DELETED. DeleteBackup +//! reads a marked DELETING document as not found instead of finishing it, +//! since its backup lives on in the store. +//! //! # The restore job record, DeleteBackup and the claim //! //! A restore registers through the job record on its target's `tables` @@ -90,26 +98,32 @@ use serde::Serialize; use serde::de::DeserializeOwned; use extenddb_core::types::{ - AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, GsiInput, + AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, GsiInput, Item, KeySchemaElement, LsiInput, Projection, ProvisionedThroughput, SourceTableDetails, StreamSpecification, TableDescription, TimeToLiveDescription, TimeToLiveStatus, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, BackupCatalog, JOB_HEARTBEAT_INTERVAL, JobStatus, RestoreKind, - RestoreTargetKey, StoreIdentity, new_generation, + AbandonedLegacyBackup, BackupCatalog, BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, + JobStatus, LegacyBackupItems, LegacyBackupKey, Replaced, RestoreKind, RestoreTargetKey, + StoreIdentity, new_generation, +}; +use extenddb_storage::backup_definition::{ + BACKUP_DEFINITION_VERSION, BackupTableDefinition, LegacyVectorState, }; -use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; +use extenddb_storage::snapshot::paging; use extenddb_storage::{BackupRestoreSource, JobPermit, LegacyBackupReader, RestoreRequest, util}; use crate::MongoEngine; use crate::backup_catalog::{ LOCK_SEQ, RESTORE_JOB, RESTORE_JOB_COMPLETED_AT, RESTORE_JOB_FAILURE_REASON, RESTORE_JOB_GENERATION, RESTORE_JOB_HEARTBEAT, RESTORE_JOB_STATUS, backup_in_use, - generation_field, + generation_field, unowned_insert, }; use crate::data::data_collection_name; -use crate::data_engine::{CommitOutcome, TRANSIENT_RETRY_ATTEMPTS, TxErr, backoff_sleep}; +use crate::data_engine::{ + CommitOutcome, TRANSIENT_RETRY_ATTEMPTS, TxErr, backoff_sleep, is_duplicate_key, +}; /// The definition version stored on a backup document written by this build. /// Backup documents written before it carry the same fields without the @@ -461,11 +475,18 @@ enum LockMiss { fn lock_miss(doc: Option<&Document>) -> LockMiss { match doc.and_then(|d| d.get_str("backup_status").ok()) { - Some("DELETING") => LockMiss::Deleting, + Some("DELETING") if !is_migrated(doc) => LockMiss::Deleting, _ => LockMiss::NotFound, } } +/// Whether `doc` is a backup document `replace_with_index_row` moved to +/// DELETING ([`MIGRATED_TO_STORE`]). +fn is_migrated(doc: Option<&Document>) -> bool { + doc.and_then(|d| d.get_bool(MIGRATED_TO_STORE).ok()) + .unwrap_or(false) +} + impl MongoEngine { /// Drop a backup's data collection (missing is success). In test builds /// [`BackupLiveness::after_backup_drop`] can make it fail or hang after the @@ -1681,6 +1702,14 @@ impl LegacyBackupReader for MongoEngine { "Backup not found: {backup_arn}" ))); } + // A replacement by its store copy (`replace_with_index_row`) + // that stopped before its drop: the backup lives on in the + // store, and that call, not this one, finishes the drop. + Some(Ok("DELETING")) if is_migrated(current.as_ref()) => { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } // A DeleteBackup that stopped after writing DELETING: resume // its drop and its DELETED write. Some(Ok("DELETING")) => { @@ -1841,6 +1870,288 @@ impl LegacyBackupReader for MongoEngine { ) -> BoxFuture<'_, Result, StorageError>> { Box::pin(async { Ok(Vec::new()) }) } + + fn list_all(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async move { + let mut cursor = self + .catalog_db + .collection::("backups") + .find(doc! { "backup_status": { "$ne": "DELETED" } }) + .sort(doc! { "created_at": 1, "_id": 1 }) + .await + .map_err(internal)?; + let mut keys = Vec::new(); + while let Some(doc) = cursor.try_next().await.map_err(internal)? { + keys.push(LegacyBackupKey { + account_id: doc.get_str("account_id").unwrap_or_default().to_owned(), + backup_arn: doc.get_str("_id").unwrap_or_default().to_owned(), + status: BackupStatus::parse( + doc.get_str("backup_status").unwrap_or("AVAILABLE"), + )?, + }); + } + Ok(keys) + }) + } + + /// The AVAILABLE backup document (its source as `restore_source` reads + /// it), then a cursor over its `_backup_` collection in + /// batches of [`paging::DEFAULT_FETCH_BATCH`] documents, each decoded as + /// the data path decoded `item_data` when the backup was taken + /// ([`legacy_item_from_document`]). The collection is immutable while the + /// backup is AVAILABLE, so no snapshot is held; a DeleteBackup's drop + /// ends the cursor with an error. + fn items( + &self, + account_id: &str, + backup_arn: &str, + ) -> BoxFuture<'_, Result> { + let account_id = account_id.to_owned(); + let backup_arn = backup_arn.to_owned(); + Box::pin(async move { + let doc = self + .catalog_db + .collection::("backups") + .find_one(doc! { + "_id": &backup_arn, + "account_id": &account_id, + "backup_status": "AVAILABLE", + }) + .await + .map_err(internal)? + .ok_or_else(|| { + StorageError::Validation(format!("Backup not found: {backup_arn}")) + })?; + let source = restore_source_from_doc(&doc, &self.region, &account_id, &backup_arn)?; + let Ok(collection_id) = doc.get_str("backup_id") else { + return Err(StorageError::Unsupported(format!( + "backup {backup_arn} was taken before backups copied their items into a \ + collection of their own, so it has no item collection to read" + ))); + }; + let cursor = self + .data_db + .collection::(&backup_collection_name(collection_id)) + .find(doc! {}) + .batch_size(u32::try_from(paging::DEFAULT_FETCH_BATCH).unwrap_or(u32::MAX)) + .await + .map_err(internal)?; + let items = futures::stream::unfold(Some(cursor), |cursor| async move { + let mut cursor = cursor?; + match cursor.try_next().await { + Ok(Some(doc)) => { + let item = legacy_item_from_document(&doc); + let next = item.is_ok().then_some(cursor); + Some((item, next)) + } + Ok(None) => None, + Err(e) => Some((Err(internal(e)), None)), + } + }); + Ok(LegacyBackupItems { + backup_name: doc.get_str("backup_name").unwrap_or_default().to_owned(), + source, + vector_state: LegacyVectorState::Unsupported, + items: Box::pin(items), + }) + }) + } + + /// One session transaction on the catalog database: the backup + /// document's lock (`$inc lock_seq` while AVAILABLE, the lock DeleteBackup + /// and a restore's registration take), the in-use rule, the index + /// document (`insert_backup_row`'s upsert) and DELETING with + /// [`MIGRATED_TO_STORE`] set. After the commit, the frozen delete's + /// second half drops the item collection and writes DELETED + /// (`finish_legacy_delete`). A stop between the two leaves DELETING with + /// the index document committed, which `list_all` lists and a later call + /// finishes as `AlreadyReplaced`; DeleteBackup of the legacy ARN reads + /// such a document as not found rather than finish it. + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let (outcome, finish) = self.start_replacement(row).await?; + if let Some(backup_doc) = finish { + self.finish_legacy_delete(&row.account_id, &row.backup_arn, &backup_doc) + .await?; + } + Ok(outcome) + }) + } +} + +/// Set (`true`) on a legacy backup document that `replace_with_index_row` +/// moved to DELETING: its backup lives on in the store under the same ARN, so +/// the frozen DeleteBackup must not finish the deletion and answer DELETED. +const MIGRATED_TO_STORE: &str = "migrated_to_store"; + +/// A frozen copy of the data path's decode of a stored document +/// (`crate::data::document_to_item`) as it was when the in-catalog `$out` +/// copies were taken, so the data path can change without changing what a +/// legacy backup reads as. +fn legacy_item_from_document(doc: &Document) -> Result { + let item_data = doc + .get("item_data") + .ok_or_else(|| StorageError::Internal("Document missing item_data field".to_string()))?; + let json_value: serde_json::Value = bson::from_bson(item_data.clone()) + .map_err(|e| StorageError::Internal(format!("BSON to JSON conversion error: {e}")))?; + serde_json::from_value(json_value) + .map_err(|e| StorageError::Internal(format!("JSON to Item conversion error: {e}"))) +} + +impl MongoEngine { + /// The transaction of `replace_with_index_row`: what it decided, and the + /// backup document (as it was) whose collection is still to drop. + async fn start_replacement( + &self, + row: &BackupRow, + ) -> Result<(Replaced, Option), StorageError> { + let backups = self.catalog_db.collection::("backups"); + let index = self.backup_index(); + let mut session = self.client.start_session().await.map_err(internal)?; + let tx_options = self.write_transaction_options(); + for attempt in 0..TRANSIENT_RETRY_ATTEMPTS { + session + .start_transaction() + .with_options(tx_options.clone()) + .await + .map_err(internal)?; + let body: Result), StorageError>, TxErr> = async { + let held = index + .find_one(doc! { "backup_id": &row.backup_id }) + .session(&mut session) + .await + .map_err(TxErr::from)?; + let same = held.as_ref().is_some_and(|held| { + held.get_str("account_id").ok() == Some(row.account_id.as_str()) + && held.get_str("backup_arn").ok() == Some(row.backup_arn.as_str()) + && held.get_str("store_kind").ok() == Some(row.store.store_kind.as_str()) + && held.get_str("store_root").ok() == Some(row.store.store_root.as_str()) + && held.get_str("store_id").ok() == Some(row.store.store_id.as_str()) + }); + let locked = backups + .find_one_and_update( + doc! { + "_id": &row.backup_arn, + "account_id": &row.account_id, + "backup_status": "AVAILABLE", + }, + doc! { "$inc": { LOCK_SEQ: 1_i64 } }, + ) + .session(&mut session) + .await + .map_err(TxErr::from)?; + let Some(backup_doc) = locked else { + let current = backups + .find_one(doc! { "_id": &row.backup_arn, "account_id": &row.account_id }) + .session(&mut session) + .await + .map_err(TxErr::from)?; + let status = current + .as_ref() + .and_then(|d| d.get_str("backup_status").ok().map(str::to_owned)); + return Ok(Ok(match (same, status.as_deref()) { + // An earlier call committed and stopped before the + // drop: finish it. + (true, Some("DELETING")) => (Replaced::AlreadyReplaced, current), + (true, _) => (Replaced::AlreadyReplaced, None), + (false, Some(status)) => ( + Replaced::NotAvailable(format!("the in-catalog backup is {status}")), + None, + ), + (false, None) => ( + Replaced::NotAvailable(format!("Backup not found: {}", row.backup_arn)), + None, + ), + })); + }; + if let Some(target) = self + .creating_restore_target(&row.account_id, &row.backup_arn, Some(&mut session)) + .await + .map_err(TxErr::from)? + { + return Ok(Ok((Replaced::InUse(target), None))); + } + let outcome = match held { + None => { + let (filter, update) = unowned_insert(row)?; + index + .update_one(filter, update) + .upsert(true) + .session(&mut session) + .await + .map_err(|e| { + // A row inserted since the read above: run the + // body again, which reads it. + if is_duplicate_key(&e) { + TxErr::Transient + } else { + TxErr::from(e) + } + })?; + Replaced::Replaced + } + Some(_) if same => Replaced::AlreadyReplaced, + Some(held) => { + return Ok(Err(StorageError::AlreadyExists(format!( + "backup id {} has a row under {} in the {} store at {:?} ({})", + row.backup_id, + held.get_str("backup_arn").unwrap_or("?"), + held.get_str("store_kind").unwrap_or("?"), + held.get_str("store_root").unwrap_or("?"), + held.get_str("store_id").unwrap_or("?"), + )))); + } + }; + backups + .update_one( + doc! { + "_id": &row.backup_arn, + "account_id": &row.account_id, + "backup_status": "AVAILABLE", + }, + doc! { "$set": { "backup_status": "DELETING", MIGRATED_TO_STORE: true } }, + ) + .session(&mut session) + .await + .map_err(TxErr::from)?; + Ok(Ok((outcome, Some(backup_doc)))) + } + .await; + match body { + Ok(Ok(decided)) => { + if decided.1.is_none() { + // Nothing was written: no commit to wait for. + let _ = session.abort_transaction().await; + return Ok(decided); + } + match self.commit_write_transaction(&mut session, None).await? { + CommitOutcome::Committed => return Ok(decided), + CommitOutcome::RerunBody => backoff_sleep(attempt).await, + } + } + Ok(Err(refusal)) => { + let _ = session.abort_transaction().await; + return Err(refusal); + } + Err(TxErr::Transient) => { + let _ = session.abort_transaction().await; + backoff_sleep(attempt).await; + } + Err(TxErr::Fatal(e)) => { + let _ = session.abort_transaction().await; + return Err(e); + } + } + } + Err(StorageError::Internal(format!( + "the replacement of {} did not commit after {TRANSIENT_RETRY_ATTEMPTS} conflicting \ + attempts", + row.backup_arn + ))) + } } #[cfg(test)] diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index 141bf0cd4..cc3b58cdf 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -5492,3 +5492,455 @@ async fn live_the_newest_available_row_and_the_running_restores_are_read_by_stat ); fx.finish().await; } + +// --- `extenddb backup migrate-legacy`: the item stream, `list_all`, the +// replacement by an index row. + +/// The index row of the store copy of the legacy backup `backup_arn`, as +/// the migration builds it. +async fn migrated_row(engine: &MongoEngine, backup_arn: &str) -> BackupRow { + let source = engine + .restore_source(ACCOUNT, backup_arn) + .await + .expect("source"); + let table = extenddb_storage::backup_definition::source_table_from_legacy( + &source, + extenddb_storage::backup_definition::LegacyVectorState::Unsupported, + REGION, + ACCOUNT, + ) + .expect("source table"); + BackupRow { + backup_id: BackupArn::parse(backup_arn) + .expect("arn") + .backup_id() + .to_owned(), + account_id: ACCOUNT.to_owned(), + table_name: source.source_table_name.clone(), + backup_arn: backup_arn.to_owned(), + backup_name: "b".to_owned(), + status: BackupStatus::Available, + store: store(), + size_bytes: 4321, + item_count: source.item_count, + created_at: source.backup_creation_date_time, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::to_value(&table).expect("json"), + owner_generation: 77, + heartbeat_at: 0.0, + } +} + +/// A legacy backup `b` of a new table `name` with `items` items, and the +/// name of its item collection. +async fn legacy_backup(engine: &MongoEngine, name: &str, items: usize) -> (String, String) { + source_table_with(engine, name, items).await; + let arn = engine + .create(ACCOUNT, name, "b", JobPermit::detached()) + .await + .expect("backup") + .backup_arn; + let collection = super::backup_collection_name( + backup_meta(engine, &arn) + .await + .get_str("backup_id") + .expect("backup_id"), + ); + (arn, collection) +} + +/// The items of `collection`, by their JSON text, sorted. +async fn collection_items(engine: &MongoEngine, collection: &str) -> Vec { + let docs: Vec = engine + .data_db + .collection::(collection) + .find(doc! {}) + .await + .expect("find") + .try_collect() + .await + .expect("documents"); + let mut items: Vec = docs + .iter() + .map(|d| serde_json::to_string(&document_to_item(d).expect("item")).expect("text")) + .collect(); + items.sort(); + items +} + +/// The stream reads every document of the backup's collection, past one +/// fetch batch (2,500 items, batches of 1,000), decoded as the data path +/// decodes them; it carries the backup's name and source, and the vector +/// state of a backend without vector support. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_legacy_items_stream_every_document_of_the_backup_collection() { + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let (arn, collection) = legacy_backup(&fx.engine, "src", 2_500).await; + let opened = fx.engine.items(ACCOUNT, &arn).await.expect("items"); + assert_eq!(opened.backup_name, "b"); + assert_eq!( + opened.source, + fx.engine + .restore_source(ACCOUNT, &arn) + .await + .expect("source") + ); + assert_eq!( + opened.vector_state, + extenddb_storage::backup_definition::LegacyVectorState::Unsupported + ); + let mut streamed: Vec = opened + .items + .map(|item| serde_json::to_string(&item.expect("an item")).expect("text")) + .collect() + .await; + streamed.sort(); + let expected = collection_items(&fx.engine, &collection).await; + assert_eq!(expected.len(), 2_500); + assert_eq!(streamed, expected); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} + +/// `items` refuses another account's backup, a missing one and a backup +/// with no `backup_id` (no item collection), and a DeleteBackup's drop +/// ends a running stream with an error. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_legacy_items_refuse_what_cannot_be_read_and_end_at_a_drop() { + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let (arn, _) = legacy_backup(&fx.engine, "src", 2_500).await; + for (account, backup_arn) in [(OTHER_ACCOUNT, arn.clone()), (ACCOUNT, flip_last_hex(&arn))] + { + let err = fx.engine.items(account, &backup_arn).await.err(); + assert!( + matches!(&err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{err:?}" + ); + } + let mut items = fx.engine.items(ACCOUNT, &arn).await.expect("items").items; + assert!(items.next().await.expect("one").is_ok()); + fx.engine.delete(ACCOUNT, &arn).await.expect("delete"); + let mut read = 1; + let mut ended_with = None; + while let Some(item) = items.next().await { + match item { + Ok(_) => read += 1, + Err(e) => { + ended_with = Some(e); + break; + } + } + } + assert!(ended_with.is_some(), "the drop ends the stream"); + assert!(read < 2_500, "{read} items read"); + assert!(items.next().await.is_none(), "nothing follows the error"); + + let (old, _) = legacy_backup(&fx.engine, "old", 3).await; + fx.engine + .catalog_db + .collection::("backups") + .update_one(doc! { "_id": &old }, doc! { "$unset": { "backup_id": "" } }) + .await + .expect("a backup from before $out"); + let err = fx.engine.items(ACCOUNT, &old).await.err(); + assert!( + matches!(&err, Some(StorageError::Unsupported(m)) if m.contains("no item collection")), + "{err:?}" + ); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_list_all_lists_every_account_and_leaves_deleted_backups_out() { + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let mut arns = Vec::new(); + for name in ["a", "b", "c", "d"] { + arns.push(legacy_backup(&fx.engine, name, 2).await.0); + } + let backups = fx.engine.catalog_db.collection::("backups"); + backups + .update_one( + doc! { "_id": &arns[1] }, + doc! { "$set": { "account_id": OTHER_ACCOUNT } }, + ) + .await + .expect("move"); + backups + .update_one( + doc! { "_id": &arns[2] }, + doc! { "$set": { "backup_status": "DELETING" } }, + ) + .await + .expect("deleting"); + fx.engine.delete(ACCOUNT, &arns[3]).await.expect("delete"); + let key = + |account: &str, arn: &str, status| extenddb_storage::backup_catalog::LegacyBackupKey { + account_id: account.to_owned(), + backup_arn: arn.to_owned(), + status, + }; + assert_eq!( + fx.engine.list_all().await.expect("list"), + vec![ + key(ACCOUNT, &arns[0], BackupStatus::Available), + key(OTHER_ACCOUNT, &arns[1], BackupStatus::Available), + key(ACCOUNT, &arns[2], BackupStatus::Deleting), + ] + ); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} + +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_a_replacement_inserts_the_unowned_row_and_removes_the_backup() { + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let (arn, collection) = legacy_backup(&fx.engine, "src", 30).await; + let row = migrated_row(&fx.engine, &arn).await; + assert!(collection_exists(&fx.engine, &collection).await); + assert_eq!( + fx.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + extenddb_storage::backup_catalog::Replaced::Replaced + ); + let read = fx + .engine + .read_backup_row(&arn_of(&row)) + .await + .expect("read") + .expect("the index row"); + assert_eq!(read.owner_generation, UNOWNED_GENERATION); + assert!(read.completed_at.is_some()); + assert_eq!(read.status, BackupStatus::Available); + assert_eq!(read.store, row.store); + assert_eq!( + read.source_table().expect("source"), + row.source_table().expect("source") + ); + let doc = backup_meta(&fx.engine, &arn).await; + assert_eq!(doc.get_str("backup_status"), Ok("DELETED")); + assert!(!collection_exists(&fx.engine, &collection).await); + assert!(fx.engine.describe(ACCOUNT, &arn).await.is_err()); + assert!( + fx.engine + .list(ACCOUNT, None) + .await + .expect("list") + .is_empty() + ); + assert!(fx.engine.list_all().await.expect("list").is_empty()); + assert_eq!( + fx.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + extenddb_storage::backup_catalog::Replaced::AlreadyReplaced + ); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} + +/// A replacement whose drop fails after the commit leaves the document +/// DELETING with the index row in: DeleteBackup of the legacy ARN reads it +/// as not found and leaves it, `list_all` lists it, and the next +/// replacement finishes it as `AlreadyReplaced`. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_a_replacement_that_stopped_before_its_drop_is_finished_by_the_next() { + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let (arn, collection) = legacy_backup(&fx.engine, "src", 30).await; + let row = migrated_row(&fx.engine, &arn).await; + let faulty = with_drop_fault(&fx.engine, AfterBackupDrop::Fail); + assert!(faulty.replace_with_index_row(&row).await.is_err()); + let doc = backup_meta(&fx.engine, &arn).await; + assert_eq!(doc.get_str("backup_status"), Ok("DELETING")); + assert_eq!(doc.get_bool(super::MIGRATED_TO_STORE), Ok(true)); + assert!( + fx.engine + .read_backup_row(&arn_of(&row)) + .await + .expect("read") + .is_some() + ); + let err = fx.engine.delete(ACCOUNT, &arn).await.err(); + assert!( + matches!(&err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{err:?}" + ); + assert_eq!( + backup_meta(&fx.engine, &arn).await.get_str("backup_status"), + Ok("DELETING"), + "DeleteBackup left it" + ); + assert_eq!( + fx.engine.list_all().await.expect("list"), + vec![extenddb_storage::backup_catalog::LegacyBackupKey { + account_id: ACCOUNT.to_owned(), + backup_arn: arn.clone(), + status: BackupStatus::Deleting, + }] + ); + // A collection the failed drop left (the fault fires after the drop + // ran) is dropped again; then DELETED. + assert_eq!( + fx.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + extenddb_storage::backup_catalog::Replaced::AlreadyReplaced + ); + assert_eq!( + backup_meta(&fx.engine, &arn).await.get_str("backup_status"), + Ok("DELETED") + ); + assert!(!collection_exists(&fx.engine, &collection).await); + assert!(fx.engine.list_all().await.expect("list").is_empty()); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} + +/// The replacement refuses, writing nothing, while a restore target of the +/// backup is CREATING, when the backup is not AVAILABLE or missing, and +/// when another store's row holds its id; a row `backup sync` adopted is +/// finished. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_a_replacement_refuses_an_in_use_or_unavailable_backup_and_writes_nothing() { + use extenddb_storage::backup_catalog::Replaced; + let fixture = Fixture::new().await; + let outcome = std::panic::AssertUnwindSafe(async { + let fx = &fixture; + let (arn, collection) = legacy_backup(&fx.engine, "src", 30).await; + let row = migrated_row(&fx.engine, &arn).await; + let (gated, gate) = gated(&fx.engine); + let request = restore_request(&fx.engine, &arn, "dst").await; + gated + .restore(ACCOUNT, request, JobPermit::detached()) + .await + .expect("registers"); + assert_eq!( + fx.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::InUse("dst".to_owned()) + ); + assert_eq!( + backup_meta(&fx.engine, &arn).await.get_str("backup_status"), + Ok("AVAILABLE") + ); + assert!(collection_exists(&fx.engine, &collection).await); + assert!( + fx.engine + .read_backup_row(&arn_of(&row)) + .await + .expect("read") + .is_none() + ); + gate.add_permits(1); + assert_fully_restored_with(&fx.engine, "dst", 30).await; + + // Another store's row holds the id. + let mut elsewhere = row.clone(); + elsewhere.store.store_root = "/srv/elsewhere".to_owned(); + assert!( + fx.engine + .insert_backup_row(&elsewhere) + .await + .expect("insert") + ); + let err = fx.engine.replace_with_index_row(&row).await.err(); + assert!( + matches!(&err, Some(StorageError::AlreadyExists(m)) if m.contains("/srv/elsewhere")), + "{err:?}" + ); + assert_eq!( + backup_meta(&fx.engine, &arn).await.get_str("backup_status"), + Ok("AVAILABLE") + ); + assert!(collection_exists(&fx.engine, &collection).await); + + // A copy `backup sync` adopted: finished. + let (adopted, adopted_collection) = legacy_backup(&fx.engine, "adopted", 3).await; + let adopted_row = migrated_row(&fx.engine, &adopted).await; + assert!( + fx.engine + .insert_backup_row(&adopted_row) + .await + .expect("insert") + ); + assert_eq!( + fx.engine + .replace_with_index_row(&adopted_row) + .await + .expect("replace"), + Replaced::AlreadyReplaced + ); + assert_eq!( + backup_meta(&fx.engine, &adopted) + .await + .get_str("backup_status"), + Ok("DELETED") + ); + assert!(!collection_exists(&fx.engine, &adopted_collection).await); + + // Not AVAILABLE, then missing. + let (gone, _) = legacy_backup(&fx.engine, "gone", 3).await; + let gone_row = migrated_row(&fx.engine, &gone).await; + fx.engine.delete(ACCOUNT, &gone).await.expect("delete"); + assert_eq!( + fx.engine + .replace_with_index_row(&gone_row) + .await + .expect("replace"), + Replaced::NotAvailable("the in-catalog backup is DELETED".to_owned()) + ); + let mut missing = gone_row.clone(); + missing.backup_arn = flip_last_hex(&gone); + missing.backup_id = BackupArn::parse(&missing.backup_arn) + .unwrap() + .backup_id() + .to_owned(); + assert!(matches!( + fx.engine.replace_with_index_row(&missing).await.expect("replace"), + Replaced::NotAvailable(m) if m.contains("Backup not found") + )); + assert!( + fx.engine + .read_backup_row(&arn_of(&gone_row)) + .await + .expect("read") + .is_none() + ); + }) + .catch_unwind() + .await; + fixture.finish_after(outcome).await; +} diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs index d5d8fb455..8d230b843 100644 --- a/crates/storage-postgres/src/backup_catalog.rs +++ b/crates/storage-postgres/src/backup_catalog.rs @@ -86,6 +86,82 @@ pub(crate) fn backup_in_use(table: &str, backup_arn: &str) -> StorageError { extenddb_storage::backup_catalog::backup_in_use(table, backup_arn) } +/// `insert_backup_row`'s statement, on `executor`: `row` unowned, if no row +/// with its key or its backup id exists. `Ok(false)`, writing nothing, +/// otherwise. The legacy reader's `replace_with_index_row` runs it inside +/// its transaction. +pub(crate) async fn insert_unowned_row<'e, E: PgExecutor<'e>>( + executor: E, + row: &BackupRow, +) -> Result { + // No conflict target: the primary key (`account_id`, `backup_id`) and + // the unique `backup_id` both make it a no-op. No advisory lock: a job + // that holds the id's lock and has not inserted its row yet finds this + // one and is refused, as it is by any row with the key. + let inserted = sqlx::query( + "INSERT INTO backup_index (backup_id, account_id, table_name, backup_arn, \ + backup_name, status, store_kind, store_root, store_id, size_bytes, item_count, \ + created_at, completed_at, failure_reason, source_table_details_json, \ + owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, to_timestamp($12), now(), \ + $13, $14, $15, now()) ON CONFLICT DO NOTHING", + ) + .bind(&row.backup_id) + .bind(&row.account_id) + .bind(&row.table_name) + .bind(&row.backup_arn) + .bind(&row.backup_name) + .bind(row.status.as_str()) + .bind(row.store.store_kind.as_str()) + .bind(&row.store.store_root) + .bind(&row.store.store_id) + .bind(row.size_bytes) + .bind(row.item_count) + .bind(row.created_at) + .bind(&row.failure_reason) + .bind(&row.source_table_details_json) + .bind(generation_column(UNOWNED_GENERATION)) + .execute(executor) + .await + .map_err(db_err)? + .rows_affected(); + Ok(inserted == 1) +} + +/// The account, ARN and store of the `backup_index` row holding +/// `backup_id`, if one does, on `executor`. +pub(crate) async fn index_row_holding<'e, E: PgExecutor<'e>>( + executor: E, + backup_id: &str, +) -> Result, StorageError> { + let found: Option<(String, String, String, String, String)> = sqlx::query_as( + "SELECT account_id, backup_arn, store_kind, store_root, store_id FROM backup_index \ + WHERE backup_id = $1", + ) + .bind(backup_id) + .fetch_optional(executor) + .await + .map_err(db_err)?; + found + .map(|(account_id, backup_arn, kind, store_root, store_id)| { + let store_kind = StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "backup {backup_arn} carries an unknown store kind {kind:?}" + )) + })?; + Ok(( + account_id, + backup_arn, + StoreIdentity { + store_kind, + store_root, + store_id, + }, + )) + }) + .transpose() +} + fn db_err(e: sqlx::Error) -> StorageError { StorageError::Internal(format!("Database error: {e}")) } @@ -546,41 +622,7 @@ impl BackupCatalog for PostgresEngine { &'a self, row: &'a BackupRow, ) -> BoxFuture<'a, Result> { - Box::pin(async move { - // No conflict target: the primary key (`account_id`, - // `backup_id`) and the unique `backup_id` both make it a no-op. - // No advisory lock: a job that holds the id's lock and has not - // inserted its row yet finds this one and is refused, as it is by - // any row with the key. - let inserted = sqlx::query( - "INSERT INTO backup_index (backup_id, account_id, table_name, backup_arn, \ - backup_name, status, store_kind, store_root, store_id, size_bytes, item_count, \ - created_at, completed_at, failure_reason, source_table_details_json, \ - owner_generation, heartbeat_at) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, to_timestamp($12), now(), \ - $13, $14, $15, now()) ON CONFLICT DO NOTHING", - ) - .bind(&row.backup_id) - .bind(&row.account_id) - .bind(&row.table_name) - .bind(&row.backup_arn) - .bind(&row.backup_name) - .bind(row.status.as_str()) - .bind(row.store.store_kind.as_str()) - .bind(&row.store.store_root) - .bind(&row.store.store_id) - .bind(row.size_bytes) - .bind(row.item_count) - .bind(row.created_at) - .bind(&row.failure_reason) - .bind(&row.source_table_details_json) - .bind(generation_column(UNOWNED_GENERATION)) - .execute(&self.pool) - .await - .map_err(db_err)? - .rows_affected(); - Ok(inserted == 1) - }) + Box::pin(insert_unowned_row(&self.pool, row)) } fn list_backup_rows<'a>( diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs index a37f1183e..0a37dcbd1 100755 --- a/crates/storage-postgres/src/backup_engine.rs +++ b/crates/storage-postgres/src/backup_engine.rs @@ -5,7 +5,11 @@ //! delete, restore-source and restore paths of [`LegacyBackupReader`], frozen //! apart from the restore's registration, which writes the restore job //! fields and holds the job claim ([`ClaimSession`]) the catalog index's jobs -//! hold too (`backup_catalog.rs`). +//! hold too (`backup_catalog.rs`). `extenddb backup migrate-legacy` reads a +//! backup through `items` (a copy of the restore's snapshot read) and ends +//! with `replace_with_index_row`, which inserts the catalog index row and +//! removes the in-catalog backup in one transaction under the row lock the +//! delete and the registration take. use std::sync::Arc; use std::time::Duration; @@ -16,11 +20,13 @@ use extenddb_core::types::{ TableDescription, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, JOB_HEARTBEAT_INTERVAL, JobBound, limit_exceeded_message, new_generation, + AbandonedLegacyBackup, BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, JobBound, + LegacyBackupItems, LegacyBackupKey, Replaced, StoreIdentity, limit_exceeded_message, + new_generation, }; use extenddb_storage::backup_definition::{ BACKUP_DEFINITION_VERSION, BackupTableDefinition, COPY_BATCH_BYTES, COPY_BATCH_ITEMS, - throughput_from_catalog, + LegacyVectorState, throughput_from_catalog, }; use extenddb_storage::error::StorageError; use extenddb_storage::util::{ @@ -1706,15 +1712,25 @@ impl LegacyBackupReader for PostgresEngine { // Refuse while a restore from this backup is still running, as the // service does. The backup row is locked first; a restore records // itself in `table_restores` while holding the same row FOR UPDATE, - // so one of the two always sees the other. - sqlx::query( - "SELECT 1 FROM backups WHERE backup_arn = $1 AND account_id = $2 FOR UPDATE", + // so one of the two always sees the other. The status is read + // under the lock too: a `replace_with_index_row` that committed + // after the lookup above has left the row DELETED, and this + // delete must not answer DELETED for a backup that now lives in + // the store. + let status: Option = sqlx::query_scalar( + "SELECT backup_status FROM backups WHERE backup_arn = $1 AND account_id = $2 \ + FOR UPDATE", ) .bind(&backup_arn) .bind(&account_id) - .execute(&mut *tx) + .fetch_optional(&mut *tx) .await .map_err(db_err)?; + if status.as_deref() != Some("AVAILABLE") { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } if let Some(table) = crate::backup_catalog::creating_restore_target(&mut *tx, &backup_arn).await? { @@ -1913,6 +1929,205 @@ impl LegacyBackupReader for PostgresEngine { ) -> BoxFuture<'_, Result, StorageError>> { Box::pin(async { Ok(Vec::new()) }) } + + fn list_all(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async move { + let rows: Vec<(String, String, String)> = sqlx::query_as( + "SELECT account_id, backup_arn, backup_status FROM backups \ + WHERE backup_status <> 'DELETED' ORDER BY created_at, backup_arn", + ) + .fetch_all(&self.pool) + .await + .map_err(read_err)?; + rows.into_iter() + .map(|(account_id, backup_arn, status)| { + Ok(LegacyBackupKey { + account_id, + backup_arn, + status: BackupStatus::parse(&status)?, + }) + }) + .collect() + }) + } + + /// The source as `restore_source` reads it, then one `REPEATABLE READ + /// READ ONLY` catalog transaction whose first statement re-checks + /// AVAILABLE and pins the snapshot the items are read in, as the + /// restore's copy reads them (`copy_backup_items`). `backup_items` has no + /// key to page by, so the transaction is held, on one pooled + /// connection, until the stream ends or is dropped; it holds the + /// catalog's xmin for that long. A task owns the transaction and hands + /// the items over one at a time. + fn items( + &self, + account_id: &str, + backup_arn: &str, + ) -> BoxFuture<'_, Result> { + let account_id = account_id.to_owned(); + let backup_arn = backup_arn.to_owned(); + Box::pin(async move { + let source = self.restore_source(&account_id, &backup_arn).await?; + let mut snapshot = self + .pool + .begin_with("BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY") + .await + .map_err(db_err)?; + let backup_name: Option = sqlx::query_scalar( + "SELECT backup_name FROM backups WHERE backup_arn = $1 AND account_id = $2 \ + AND backup_status = 'AVAILABLE'", + ) + .bind(&backup_arn) + .bind(&account_id) + .fetch_optional(&mut *snapshot) + .await + .map_err(read_err)?; + let Some(backup_name) = backup_name else { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + }; + let (sender, receiver) = tokio::sync::mpsc::channel(1); + tokio::spawn(send_legacy_items(snapshot, backup_arn, sender)); + Ok(LegacyBackupItems { + backup_name, + source, + // A definition's vector index names, or the snapshot's count + // for a backup without one, are what `ensure_restorable` + // refuses; a backup it accepts had none. + vector_state: LegacyVectorState::NoneRecorded, + items: Box::pin(futures::stream::unfold( + receiver, + |mut receiver| async move { receiver.recv().await.map(|item| (item, receiver)) }, + )), + }) + }) + } + + /// One catalog transaction: the `backups` row FOR UPDATE (the lock + /// DeleteBackup and a restore's registration take), the in-use rule, + /// `insert_backup_row`'s insert, then the frozen delete's statements + /// (items, definition, DELETED). + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let mut tx = self.pool.begin().await.map_err(db_err)?; + let status: Option = sqlx::query_scalar( + "SELECT backup_status FROM backups WHERE backup_arn = $1 AND account_id = $2 \ + FOR UPDATE", + ) + .bind(&row.backup_arn) + .bind(&row.account_id) + .fetch_optional(&mut *tx) + .await + .map_err(db_err)?; + let same = |held: &Option<(String, String, StoreIdentity)>| { + held.as_ref().is_some_and(|(account_id, arn, store)| { + *account_id == row.account_id && *arn == row.backup_arn && *store == row.store + }) + }; + if status.as_deref() != Some("AVAILABLE") { + let held = + crate::backup_catalog::index_row_holding(&mut *tx, &row.backup_id).await?; + tx.rollback().await.map_err(db_err)?; + return Ok(if same(&held) { + Replaced::AlreadyReplaced + } else { + Replaced::NotAvailable(match status { + Some(status) => format!("the in-catalog backup is {status}"), + None => format!("Backup not found: {}", row.backup_arn), + }) + }); + } + if let Some(table) = + crate::backup_catalog::creating_restore_target(&mut *tx, &row.backup_arn).await? + { + tx.rollback().await.map_err(db_err)?; + return Ok(Replaced::InUse(table)); + } + let outcome = if crate::backup_catalog::insert_unowned_row(&mut *tx, row).await? { + Replaced::Replaced + } else { + let held = + crate::backup_catalog::index_row_holding(&mut *tx, &row.backup_id).await?; + if !same(&held) { + tx.rollback().await.map_err(db_err)?; + return Err(match held { + Some((_, arn, store)) => StorageError::AlreadyExists(format!( + "backup id {} has a row under {arn} in the {} store at {:?} \ + ({})", + row.backup_id, store.store_kind, store.store_root, store.store_id + )), + None => StorageError::Internal(format!( + "the insert of backup {} met a conflict and no row", + row.backup_arn + )), + }); + } + Replaced::AlreadyReplaced + }; + for statement in [ + "DELETE FROM backup_items WHERE backup_arn = $1", + "DELETE FROM backup_definitions WHERE backup_arn = $1", + ] { + sqlx::query(statement) + .bind(&row.backup_arn) + .execute(&mut *tx) + .await + .map_err(db_err)?; + } + sqlx::query( + "UPDATE backups SET backup_status = 'DELETED' \ + WHERE backup_arn = $1 AND account_id = $2", + ) + .bind(&row.backup_arn) + .bind(&row.account_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + tx.commit().await.map_err(db_err)?; + Ok(outcome) + }) + } +} + +/// The task half of `items`: read the backup's items in `snapshot` and send +/// each, parsed, to `sender`; the first error is sent last. Ends, rolling +/// the transaction back, when the read ends or the stream is dropped. +async fn send_legacy_items( + mut snapshot: sqlx::Transaction<'static, sqlx::Postgres>, + backup_arn: String, + sender: tokio::sync::mpsc::Sender>, +) { + use futures::TryStreamExt; + + let failure = { + let mut rows = sqlx::query_scalar::<_, String>( + "SELECT item_data::text FROM backup_items WHERE backup_arn = $1", + ) + .bind(&backup_arn) + .fetch(&mut *snapshot); + loop { + match rows.try_next().await { + Ok(Some(item_json)) => { + let item = serde_json::from_str::(&item_json) + .map_err(|e| StorageError::Internal(format!("Parse backup item: {e}"))); + let parsed = item.is_ok(); + if sender.send(item).await.is_err() || !parsed { + break None; + } + } + Ok(None) => break None, + Err(e) => break Some(read_err(e)), + } + } + }; + if let Some(error) = failure { + let _ = sender.send(Err(error)).await; + } + let _ = snapshot.rollback().await; } #[cfg(test)] diff --git a/crates/storage-postgres/tests/backup_restore.rs b/crates/storage-postgres/tests/backup_restore.rs index bd0505e5b..acf9b87e9 100644 --- a/crates/storage-postgres/tests/backup_restore.rs +++ b/crates/storage-postgres/tests/backup_restore.rs @@ -6598,3 +6598,476 @@ mod reconciler_fencing { s.cleanup().await; } } + +/// The legacy reader's half of `extenddb backup migrate-legacy`: the item +/// stream, `list_all` and the replacement by an index row. +mod legacy_migration { + use extenddb_core::backup_format::BackupArn; + use extenddb_storage::BackupCatalog; + use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, LegacyBackupKey, Replaced, UNOWNED_GENERATION, + }; + use extenddb_storage::backup_definition::{LegacyVectorState, source_table_from_legacy}; + use futures::StreamExt; + + use super::*; + + /// A legacy backup named `nightly` of a new indexed table `name` with + /// `items` items, its definition recorded. + async fn legacy(s: &Scratch, name: &str, items: usize) -> String { + indexed_source_with(s, name, items).await; + s.engine + .create(ACCOUNT, name, "nightly", JobPermit::detached()) + .await + .expect("legacy backup") + .backup_arn + } + + /// The items `backup_arn` stores, by their JSON text, sorted. + async fn stored(s: &Scratch, backup_arn: &str) -> Vec { + let rows: Vec = + sqlx::query_scalar("SELECT item_data FROM backup_items WHERE backup_arn = $1") + .bind(backup_arn) + .fetch_all(&s.catalog) + .await + .expect("stored items"); + let mut rows: Vec = rows + .into_iter() + .map(|v| { + let item: Item = serde_json::from_value(v).expect("item"); + serde_json::to_string(&item).expect("text") + }) + .collect(); + rows.sort(); + rows + } + + /// Items, definitions and the `backups` status of `backup_arn`. + async fn counts(s: &Scratch, backup_arn: &str) -> (i64, i64, Option) { + let items: i64 = + sqlx::query_scalar("SELECT count(*) FROM backup_items WHERE backup_arn = $1") + .bind(backup_arn) + .fetch_one(&s.catalog) + .await + .expect("items"); + let definitions: i64 = + sqlx::query_scalar("SELECT count(*) FROM backup_definitions WHERE backup_arn = $1") + .bind(backup_arn) + .fetch_one(&s.catalog) + .await + .expect("definitions"); + let status: Option = + sqlx::query_scalar("SELECT backup_status FROM backups WHERE backup_arn = $1") + .bind(backup_arn) + .fetch_optional(&s.catalog) + .await + .expect("status"); + (items, definitions, status) + } + + /// The index row of the store copy of `backup_arn`, as the migration + /// builds it. + async fn migrated_row(s: &Scratch, backup_arn: &str) -> BackupRow { + let source = s + .engine + .restore_source(ACCOUNT, backup_arn) + .await + .expect("source"); + let table = + source_table_from_legacy(&source, LegacyVectorState::NoneRecorded, REGION, ACCOUNT) + .expect("source table"); + BackupRow { + backup_id: BackupArn::parse(backup_arn) + .expect("arn") + .backup_id() + .to_owned(), + account_id: ACCOUNT.to_owned(), + table_name: source.source_table_name.clone(), + backup_arn: backup_arn.to_owned(), + backup_name: "nightly".to_owned(), + status: BackupStatus::Available, + store: catalog_index::store(), + size_bytes: 4321, + item_count: source.item_count, + created_at: source.backup_creation_date_time, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::to_value(&table).expect("json"), + owner_generation: 77, + heartbeat_at: 0.0, + } + } + + /// Sessions of the scratch catalog waiting on a lock. + async fn lock_waiters(s: &Scratch) -> i64 { + sqlx::query_scalar( + "SELECT count(*) FROM pg_stat_activity \ + WHERE datname = current_database() AND wait_event_type = 'Lock'", + ) + .fetch_one(&s.catalog) + .await + .expect("lock waiters") + } + + async fn await_lock_waiters(s: &Scratch, n: i64) { + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(30); + while lock_waiters(s).await < n { + assert!( + tokio::time::Instant::now() < deadline, + "expected {n} sessions waiting on a lock" + ); + tokio::time::sleep(std::time::Duration::from_millis(10)).await; + } + } + + /// The stream reads every stored item from the snapshot its first + /// statement pinned: rows deleted after it, before the item read starts + /// (held behind a lock on `backup_items`), are still read. + #[tokio::test] + async fn items_stream_every_stored_item_from_the_pinned_snapshot() { + if base_conn().is_none() { + eprintln!( + "SKIP items_stream_every_stored_item_from_the_pinned_snapshot: no PostgreSQL" + ); + return; + } + let s = scratch().await; + let arn = legacy(&s, "lm_src", 30).await; + add_backup_rows(&s, &arn, &bulk_rows(2500)).await; + let expected = stored(&s, &arn).await; + assert_eq!(expected.len(), 2530); + + let mut gate = gate_copies(&s).await; + let opened = s.engine.items(ACCOUNT, &arn).await.expect("items"); + assert_eq!(opened.backup_name, "nightly"); + assert_eq!( + opened.source, + s.engine + .restore_source(ACCOUNT, &arn) + .await + .expect("source") + ); + assert_eq!(opened.vector_state, LegacyVectorState::NoneRecorded); + let reader = tokio::spawn(opened.items.collect::>()); + await_copies_waiting(&s, 1).await; + // Committed after the snapshot, before the stream's read of the + // items: not seen. + sqlx::query("DELETE FROM backup_items WHERE backup_arn = $1") + .bind(&arn) + .execute(&mut *gate) + .await + .expect("delete the rows"); + gate.commit().await.expect("release the gate"); + let streamed = reader.await.expect("reader"); + let mut streamed: Vec = streamed + .into_iter() + .map(|item| serde_json::to_string(&item.expect("an item")).expect("text")) + .collect(); + streamed.sort(); + assert_eq!(streamed.len(), 2530); + assert_eq!(streamed, expected); + s.cleanup().await; + } + + #[tokio::test] + async fn items_refuse_a_backup_that_is_not_available() { + if base_conn().is_none() { + eprintln!("SKIP items_refuse_a_backup_that_is_not_available: no PostgreSQL"); + return; + } + let s = scratch().await; + let arn = legacy(&s, "lm_src", 3).await; + for (account, backup_arn) in [(OTHER_ACCOUNT, arn.clone()), (ACCOUNT, flip_last_hex(&arn))] + { + let err = s.engine.items(account, &backup_arn).await.err(); + assert!( + matches!(&err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{err:?}" + ); + } + s.engine.delete(ACCOUNT, &arn).await.expect("delete"); + let err = s.engine.items(ACCOUNT, &arn).await.err(); + assert!(matches!(err, Some(StorageError::Validation(_))), "{err:?}"); + s.cleanup().await; + } + + #[tokio::test] + async fn list_all_lists_every_account_and_leaves_deleted_backups_out() { + if base_conn().is_none() { + eprintln!( + "SKIP list_all_lists_every_account_and_leaves_deleted_backups_out: no PostgreSQL" + ); + return; + } + let s = scratch().await; + let mut arns = Vec::new(); + for name in ["lm_a", "lm_b", "lm_c"] { + arns.push(legacy(&s, name, 2).await); + } + sqlx::query("UPDATE backups SET account_id = $1 WHERE backup_arn = $2") + .bind(OTHER_ACCOUNT) + .bind(&arns[1]) + .execute(&s.catalog) + .await + .expect("move"); + s.engine.delete(ACCOUNT, &arns[2]).await.expect("delete"); + assert_eq!( + s.engine.list_all().await.expect("list"), + vec![ + LegacyBackupKey { + account_id: ACCOUNT.to_owned(), + backup_arn: arns[0].clone(), + status: BackupStatus::Available, + }, + LegacyBackupKey { + account_id: OTHER_ACCOUNT.to_owned(), + backup_arn: arns[1].clone(), + status: BackupStatus::Available, + }, + ] + ); + s.cleanup().await; + } + + #[tokio::test] + async fn a_replacement_inserts_the_unowned_row_and_removes_the_backup_in_one_step() { + if base_conn().is_none() { + eprintln!( + "SKIP a_replacement_inserts_the_unowned_row_and_removes_the_backup_in_one_step: \ + no PostgreSQL" + ); + return; + } + let s = scratch().await; + let arn = legacy(&s, "lm_src", 7).await; + let row = migrated_row(&s, &arn).await; + assert_eq!(counts(&s, &arn).await, (7, 1, Some("AVAILABLE".to_owned()))); + assert_eq!( + s.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::Replaced + ); + let key = BackupArn::parse(&arn).unwrap(); + let read = s + .engine + .read_backup_row(&key) + .await + .expect("read") + .expect("the index row"); + assert_eq!(read.owner_generation, UNOWNED_GENERATION); + assert!(read.completed_at.is_some()); + assert!((read.created_at - row.created_at).abs() < 1e-3); + // The source table is compared parsed: a float read back from the + // catalog's JSON may differ from the one written in its last bit. + let mut read = read; + let read_table = read.source_table().expect("source table"); + let row_table = row.source_table().expect("source table"); + assert!( + (read_table.table_creation_date_time - row_table.table_creation_date_time).abs() < 1e-3 + ); + assert_eq!( + extenddb_core::backup_format::SourceTableSchema { + table_creation_date_time: row_table.table_creation_date_time, + ..read_table + }, + row_table + ); + read.source_table_details_json = row.source_table_details_json.clone(); + assert_eq!( + BackupRow { + owner_generation: row.owner_generation, + completed_at: None, + heartbeat_at: 0.0, + created_at: row.created_at, + ..read + }, + row + ); + assert_eq!(counts(&s, &arn).await, (0, 0, Some("DELETED".to_owned()))); + assert!(s.engine.describe(ACCOUNT, &arn).await.is_err()); + assert!(s.engine.list(ACCOUNT, None).await.expect("list").is_empty()); + assert!(s.engine.list_all().await.expect("list").is_empty()); + assert_eq!( + s.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::AlreadyReplaced + ); + s.cleanup().await; + } + + #[tokio::test] + async fn a_replacement_over_an_adopted_copy_finishes_and_one_of_another_store_refuses() { + if base_conn().is_none() { + eprintln!( + "SKIP a_replacement_over_an_adopted_copy_finishes_and_one_of_another_store_refuses: \ + no PostgreSQL" + ); + return; + } + let s = scratch().await; + let adopted = legacy(&s, "lm_src", 3).await; + let row = migrated_row(&s, &adopted).await; + assert!(s.engine.insert_backup_row(&row).await.expect("insert")); + assert_eq!( + s.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::AlreadyReplaced + ); + assert_eq!( + counts(&s, &adopted).await, + (0, 0, Some("DELETED".to_owned())) + ); + + let other = legacy(&s, "lm_other", 3).await; + let row = migrated_row(&s, &other).await; + let mut elsewhere = row.clone(); + elsewhere.store.store_root = "/srv/elsewhere".to_owned(); + assert!( + s.engine + .insert_backup_row(&elsewhere) + .await + .expect("insert") + ); + let err = s.engine.replace_with_index_row(&row).await.err(); + assert!( + matches!(&err, Some(StorageError::AlreadyExists(m)) if m.contains("/srv/elsewhere")), + "{err:?}" + ); + assert_eq!( + counts(&s, &other).await, + (3, 1, Some("AVAILABLE".to_owned())) + ); + s.cleanup().await; + } + + #[tokio::test] + async fn a_replacement_refuses_an_in_use_or_unavailable_backup_and_writes_nothing() { + if base_conn().is_none() { + eprintln!( + "SKIP a_replacement_refuses_an_in_use_or_unavailable_backup_and_writes_nothing: \ + no PostgreSQL" + ); + return; + } + let s = scratch().await; + let arn = legacy(&s, "lm_src", 4).await; + let row = migrated_row(&s, &arn).await; + let key = BackupArn::parse(&arn).unwrap(); + // A restore target of the backup, CREATING. + indexed_source(&s, "lm_target").await; + let target_id = table_id_of(&s, "lm_target").await.unwrap(); + sqlx::query( + "INSERT INTO table_restores (table_id, source_backup_arn, source_table_arn, \ + restore_date_time, status, kind, owner_generation, heartbeat_at) \ + VALUES ($1, $2, 'arn:aws:dynamodb:us-east-1:123456789012:table/lm_src', now(), \ + 'RUNNING', 'legacy', 7, now())", + ) + .bind(&target_id) + .bind(&arn) + .execute(&s.catalog) + .await + .expect("register a target"); + let creating = |status: &'static str| { + let catalog = s.catalog.clone(); + let target_id = target_id.clone(); + async move { + sqlx::query("UPDATE tables SET table_status = $1 WHERE table_id = $2") + .bind(status) + .bind(&target_id) + .execute(&catalog) + .await + .expect("target status"); + } + }; + creating("CREATING").await; + assert_eq!( + s.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::InUse("lm_target".to_owned()) + ); + assert_eq!(counts(&s, &arn).await, (4, 1, Some("AVAILABLE".to_owned()))); + assert!( + s.engine + .read_backup_row(&key) + .await + .expect("read") + .is_none() + ); + creating("ACTIVE").await; + + s.engine.delete(ACCOUNT, &arn).await.expect("delete"); + assert_eq!( + s.engine + .replace_with_index_row(&row) + .await + .expect("replace"), + Replaced::NotAvailable("the in-catalog backup is DELETED".to_owned()) + ); + let mut missing = row.clone(); + missing.backup_arn = flip_last_hex(&arn); + missing.backup_id = BackupArn::parse(&missing.backup_arn) + .unwrap() + .backup_id() + .to_owned(); + assert!(matches!( + s.engine.replace_with_index_row(&missing).await.expect("replace"), + Replaced::NotAvailable(m) if m.contains("Backup not found") + )); + assert!( + s.engine + .read_backup_row(&key) + .await + .expect("read") + .is_none() + ); + s.cleanup().await; + } + + /// A legacy DeleteBackup that looked the backup up before a replacement + /// committed, and waited on the row lock behind it, answers not found: + /// the backup lives on in the store and is not reported DELETED. + #[tokio::test] + async fn a_legacy_delete_queued_behind_a_replacement_answers_not_found() { + if base_conn().is_none() { + eprintln!( + "SKIP a_legacy_delete_queued_behind_a_replacement_answers_not_found: no PostgreSQL" + ); + return; + } + let s = scratch().await; + let arn = legacy(&s, "lm_src", 2).await; + // The replacement's side: the row lock, then its writes. + let mut replacement = s.catalog.begin().await.expect("begin"); + sqlx::query("SELECT 1 FROM backups WHERE backup_arn = $1 FOR UPDATE") + .bind(&arn) + .execute(&mut *replacement) + .await + .expect("lock the row"); + let engine = s.engine.clone(); + let deleting = { + let arn = arn.clone(); + tokio::spawn(async move { engine.delete(ACCOUNT, &arn).await }) + }; + await_lock_waiters(&s, 1).await; + sqlx::query("UPDATE backups SET backup_status = 'DELETED' WHERE backup_arn = $1") + .bind(&arn) + .execute(&mut *replacement) + .await + .expect("the replacement's write"); + replacement.commit().await.expect("commit"); + let answer = deleting.await.expect("delete task"); + assert!( + matches!(&answer, Err(StorageError::Validation(m)) if m.contains("Backup not found")), + "{answer:?}" + ); + s.cleanup().await; + } +} diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index e75a9b407..977a9493c 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -35,6 +35,15 @@ //! where main answered BackupNotFound; the service's BackupInUse for a //! CREATING backup is the catalog-index row's rule, not this reader's. //! +//! `extenddb backup migrate-legacy` reads a backup through `items`, keyset +//! batches on `backup_items.id` as the restore's copy reads them, and ends +//! with `replace_with_index_row`: one write transaction under the engine +//! write lock that inserts the catalog index row and removes the backup's +//! items and definition, leaving the row DELETED as the delete does. A +//! backup with no definition (taken before catalog 0.0.4) is opened with an +//! unknown vector state, since this backend recorded vector indexes only in +//! the definition. +//! //! A restore's heartbeat runs every //! [`JOB_HEARTBEAT_INTERVAL`](extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL), //! the one constant every backend's heartbeat loop reads and the stale floor @@ -60,12 +69,13 @@ use extenddb_core::types::{ TableDescription, TimeToLiveDescription, TimeToLiveStatus, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, JOB_HEARTBEAT_INTERVAL, JobBound, JobClaim, JobStatus, StatusFields, + AbandonedLegacyBackup, BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, JobBound, JobClaim, + JobStatus, LegacyBackupItems, LegacyBackupKey, Replaced, StatusFields, StoreIdentity, WriteOutcome, limit_exceeded_message, new_generation, }; use extenddb_storage::backup_definition::{ BACKUP_DEFINITION_VERSION, BackupTableDefinition, COPY_BATCH_BYTES, COPY_BATCH_ITEMS, - throughput_from_catalog, + LegacyVectorState, throughput_from_catalog, }; use extenddb_storage::error::StorageError; use extenddb_storage::util::{composite_pk_to_text, parse_sk, sk_column_n}; @@ -1523,6 +1533,233 @@ impl LegacyBackupReader for SqliteEngine { Ok(removed) }) } + + fn list_all(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async move { + let rows: Vec<(String, String, String)> = sqlx::query_as( + "SELECT account_id, backup_arn, backup_status FROM backups \ + WHERE backup_status <> 'DELETED' ORDER BY created_at, backup_arn", + ) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter() + .map(|(account_id, backup_arn, status)| { + Ok(LegacyBackupKey { + account_id, + backup_arn, + status: BackupStatus::parse(&status)?, + }) + }) + .collect() + }) + } + + /// The source as `restore_source` reads it, then the items in keyset + /// batches on `backup_items.id`, as the restore's copy reads them + /// (`copy_backup_items`): each batch in its own read transaction that + /// first re-checks AVAILABLE, at most [`COPY_BATCH_ITEMS`] rows or + /// [`COPY_BATCH_BYTES`] of item text (one row over the budget is still + /// read whole). No transaction is held between batches. + fn items( + &self, + account_id: &str, + backup_arn: &str, + ) -> BoxFuture<'_, Result> { + let account_id = account_id.to_owned(); + let backup_arn = backup_arn.to_owned(); + Box::pin(async move { + let source = self.restore_source(&account_id, &backup_arn).await?; + let backup_name: String = sqlx::query_scalar( + "SELECT backup_name FROM backups WHERE backup_arn = ? AND account_id = ? \ + AND backup_status = 'AVAILABLE'", + ) + .bind(&backup_arn) + .bind(&account_id) + .fetch_optional(&self.pool) + .await + .map_err(db_err)? + .ok_or_else(|| StorageError::Validation(format!("Backup not found: {backup_arn}")))?; + // A backup taken before catalog 0.0.4 has no definition, and this + // backend recorded vector indexes nowhere else. + let vector_state = if source.definition.is_some() { + LegacyVectorState::NoneRecorded + } else { + LegacyVectorState::Unknown + }; + let cursor = LegacyItemsCursor { + pool: self.pool.clone(), + backup_arn, + last_id: 0, + batch: std::collections::VecDeque::new(), + done: false, + }; + Ok(LegacyBackupItems { + backup_name, + source, + vector_state, + items: Box::pin(futures::stream::unfold(cursor, LegacyItemsCursor::next)), + }) + }) + } + + /// One `BEGIN IMMEDIATE` transaction under the engine write lock, the + /// lock DeleteBackup and a restore's registration take: the status, the + /// in-use rule, `insert_backup_row`'s insert, then the frozen delete's + /// statements (items, definition, DELETED). + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let status: Option = sqlx::query_scalar( + "SELECT backup_status FROM backups WHERE backup_arn = ? AND account_id = ?", + ) + .bind(&row.backup_arn) + .bind(&row.account_id) + .fetch_optional(&mut *tx) + .await + .map_err(db_err)?; + let same = |held: &Option<(String, String, StoreIdentity)>| { + held.as_ref().is_some_and(|(account_id, arn, store)| { + *account_id == row.account_id && *arn == row.backup_arn && *store == row.store + }) + }; + if status.as_deref() != Some("AVAILABLE") { + let held = + crate::backup_catalog::index_row_holding(&mut tx, &row.backup_id).await?; + tx.rollback().await.map_err(db_err)?; + return Ok(if same(&held) { + Replaced::AlreadyReplaced + } else { + Replaced::NotAvailable(match status { + Some(status) => format!("the in-catalog backup is {status}"), + None => format!("Backup not found: {}", row.backup_arn), + }) + }); + } + if let Some(table) = + crate::backup_catalog::creating_restore_target(&mut *tx, &row.backup_arn).await? + { + tx.rollback().await.map_err(db_err)?; + return Ok(Replaced::InUse(table)); + } + let outcome = if crate::backup_catalog::insert_unowned_row(&mut tx, row).await? { + Replaced::Replaced + } else { + let held = + crate::backup_catalog::index_row_holding(&mut tx, &row.backup_id).await?; + if !same(&held) { + tx.rollback().await.map_err(db_err)?; + return Err(match held { + Some((_, arn, store)) => StorageError::AlreadyExists(format!( + "backup id {} has a row under {arn} in the {} store at {:?} ({})", + row.backup_id, store.store_kind, store.store_root, store.store_id + )), + None => StorageError::Internal(format!( + "the insert of backup {} met a conflict and no row", + row.backup_arn + )), + }); + } + Replaced::AlreadyReplaced + }; + for table in ["backup_items", "backup_definitions"] { + sqlx::query(&format!("DELETE FROM {table} WHERE backup_arn = ?")) + .bind(&row.backup_arn) + .execute(&mut *tx) + .await + .map_err(db_err)?; + } + sqlx::query( + "UPDATE backups SET backup_status = 'DELETED' \ + WHERE backup_arn = ? AND account_id = ?", + ) + .bind(&row.backup_arn) + .bind(&row.account_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + tx.commit().await.map_err(db_err)?; + Ok(outcome) + }) + } +} + +/// Where `items`' stream is: the last `backup_items.id` read and the batch +/// not yet handed out. +struct LegacyItemsCursor { + pool: sqlx::SqlitePool, + backup_arn: String, + last_id: i64, + batch: std::collections::VecDeque, + done: bool, +} + +impl LegacyItemsCursor { + async fn next(mut self) -> Option<(Result, Self)> { + if self.done { + return None; + } + if self.batch.is_empty() { + match self.read_batch().await { + Ok(true) => {} + Ok(false) => return None, + Err(e) => { + self.done = true; + return Some((Err(e), self)); + } + } + } + let item_json = self.batch.pop_front()?; + let item = parse_json::(&item_json, "backup item"); + if item.is_err() { + self.done = true; + } + Some((item, self)) + } + + /// The next batch after `last_id`, in its own read transaction that + /// re-checks the backup is AVAILABLE; `Ok(false)` past the last row. + async fn read_batch(&mut self) -> Result { + let mut tx = self.pool.begin().await.map_err(db_err)?; + let available: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM backups WHERE backup_arn = ? \ + AND backup_status = 'AVAILABLE')", + ) + .bind(&self.backup_arn) + .fetch_one(&mut *tx) + .await + .map_err(db_err)?; + if !available { + return Err(StorageError::Validation(format!( + "Backup not found: {}", + self.backup_arn + ))); + } + let batch = next_batch( + &mut tx, + "backup_items", + "id", + Some(("backup_arn = ?", self.backup_arn.as_str())), + self.last_id, + ) + .await?; + tx.commit().await.map_err(db_err)?; + let Some(&(tail, _)) = batch.last() else { + self.done = true; + return Ok(false); + }; + self.last_id = tail; + self.batch = batch.into_iter().map(|(_, item_json)| item_json).collect(); + Ok(true) + } } #[cfg(test)] @@ -6514,6 +6751,405 @@ mod restore_tests { } } + mod legacy_migration { + //! The legacy reader's half of `extenddb backup migrate-legacy`: the + //! item stream, `list_all` and the replacement by an index row. + + use extenddb_core::backup_format::BackupArn; + use extenddb_storage::BackupCatalog; + use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, LegacyBackupKey, Replaced, StoreIdentity, StoreKind, + UNOWNED_GENERATION, + }; + use extenddb_storage::backup_definition::{LegacyVectorState, source_table_from_legacy}; + use futures::StreamExt; + + use super::*; + + const REGION: &str = "us-east-1"; + const OTHER_ACCOUNT: &str = "111111111111"; + + fn store() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/var/lib/extenddb/backups".to_owned(), + store_id: "store-1".to_owned(), + } + } + + /// A legacy backup named `bkp` of a new table `name` holding `items` + /// 1 KB items and then `large` items of 3 MiB, so the stream crosses + /// both the row and the byte bound of a batch. + async fn legacy_backup( + engine: &SqliteEngine, + name: &str, + items: i64, + large: i64, + ) -> String { + let key_info = plain_table(engine, name).await; + fill_table(engine, &key_info, items).await; + let table = crate::data::data_table_name(&key_info.table_id); + let payload = "y".repeat(3 * 1024 * 1024); + for i in 0..large { + let item = json!({"pk": {"S": format!("large{i}")}, "payload": {"S": payload}}); + sqlx::query(&format!( + "INSERT INTO {table} (pk, item_data) VALUES (?, ?)" + )) + .bind(format!("large{i}")) + .bind(item.to_string()) + .execute(&engine.pool) + .await + .expect("a large item"); + } + engine + .create(ACCOUNT, name, "bkp", JobPermit::detached()) + .await + .expect("backup") + .backup_arn + } + + /// The stored items of `backup_arn`, in `id` order, parsed. + async fn stored_items(engine: &SqliteEngine, backup_arn: &str) -> Vec { + let rows: Vec = sqlx::query_scalar( + "SELECT item_data FROM backup_items WHERE backup_arn = ? ORDER BY id", + ) + .bind(backup_arn) + .fetch_all(&engine.pool) + .await + .expect("stored items"); + rows.iter() + .map(|r| serde_json::from_str(r).expect("item")) + .collect() + } + + async fn counts(engine: &SqliteEngine, backup_arn: &str) -> (i64, i64, Option) { + let items: i64 = + sqlx::query_scalar("SELECT COUNT(*) FROM backup_items WHERE backup_arn = ?") + .bind(backup_arn) + .fetch_one(&engine.pool) + .await + .expect("items"); + let definitions: i64 = + sqlx::query_scalar("SELECT COUNT(*) FROM backup_definitions WHERE backup_arn = ?") + .bind(backup_arn) + .fetch_one(&engine.pool) + .await + .expect("definitions"); + let status: Option = + sqlx::query_scalar("SELECT backup_status FROM backups WHERE backup_arn = ?") + .bind(backup_arn) + .fetch_optional(&engine.pool) + .await + .expect("status"); + (items, definitions, status) + } + + /// The index row of the store copy of `backup_arn`, built as the + /// migration builds it. + async fn migrated_row(engine: &SqliteEngine, backup_arn: &str) -> BackupRow { + let source = engine + .restore_source(ACCOUNT, backup_arn) + .await + .expect("source"); + let table = + source_table_from_legacy(&source, LegacyVectorState::NoneRecorded, REGION, ACCOUNT) + .expect("source table"); + let arn = BackupArn::parse(backup_arn).expect("arn"); + BackupRow { + backup_id: arn.backup_id().to_owned(), + account_id: ACCOUNT.to_owned(), + table_name: source.source_table_name.clone(), + backup_arn: backup_arn.to_owned(), + backup_name: "bkp".to_owned(), + status: BackupStatus::Available, + store: store(), + size_bytes: 4321, + item_count: source.item_count, + created_at: source.backup_creation_date_time, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::to_value(&table).expect("json"), + owner_generation: 77, + heartbeat_at: 0.0, + } + } + + #[tokio::test] + async fn items_stream_every_stored_item_in_id_order_across_batch_bounds() { + let engine = engine().await; + let arn = legacy_backup(&engine, "src", 1201, 3).await; + let opened = engine.items(ACCOUNT, &arn).await.expect("items"); + assert_eq!(opened.backup_name, "bkp"); + assert_eq!( + opened.source, + engine.restore_source(ACCOUNT, &arn).await.expect("source") + ); + assert_eq!(opened.vector_state, LegacyVectorState::NoneRecorded); + let streamed: Vec = opened + .items + .map(|item| item.expect("an item")) + .collect() + .await; + let stored = stored_items(&engine, &arn).await; + assert_eq!(stored.len(), 1204, "the backup holds every item"); + assert_eq!(streamed.len(), 1204); + assert_eq!(streamed, stored, "every item, once, in id order"); + + // A backup taken before catalog 0.0.4 has no definition, and its + // vector state is not known. + sqlx::query("DELETE FROM backup_definitions WHERE backup_arn = ?") + .bind(&arn) + .execute(&engine.pool) + .await + .expect("drop the definition"); + let opened = engine.items(ACCOUNT, &arn).await.expect("items"); + assert!(opened.source.definition.is_none()); + assert_eq!(opened.vector_state, LegacyVectorState::Unknown); + } + + #[tokio::test] + async fn items_refuse_what_is_not_available_and_stop_at_a_delete() { + let engine = engine().await; + let arn = legacy_backup(&engine, "src", 1201, 0).await; + for (account, backup_arn) in + [(OTHER_ACCOUNT, arn.clone()), (ACCOUNT, flip_last_hex(&arn))] + { + let err = engine.items(account, &backup_arn).await.err(); + assert!( + matches!(&err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{err:?}" + ); + } + let mut items = engine.items(ACCOUNT, &arn).await.expect("items").items; + assert!(items.next().await.expect("one").is_ok()); + engine.delete(ACCOUNT, &arn).await.expect("delete"); + let mut read = 1; + let mut ended_with = None; + while let Some(item) = items.next().await { + match item { + Ok(_) => read += 1, + Err(e) => { + ended_with = Some(e); + break; + } + } + } + assert!( + matches!(&ended_with, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "{ended_with:?}" + ); + assert_eq!(read, 500, "the batch read before the delete, and no more"); + assert!(items.next().await.is_none(), "nothing follows the error"); + let err = engine.items(ACCOUNT, &arn).await.err(); + assert!(matches!(err, Some(StorageError::Validation(_))), "{err:?}"); + } + + #[tokio::test] + async fn list_all_lists_every_account_and_leaves_deleted_backups_out() { + let engine = engine().await; + let mut arns = Vec::new(); + for name in ["a", "b", "c", "d"] { + arns.push(legacy_backup(&engine, name, 2, 0).await); + } + // Another account's backup, a CREATING one and a deleted one. + sqlx::query("UPDATE backups SET account_id = ? WHERE backup_arn = ?") + .bind(OTHER_ACCOUNT) + .bind(&arns[1]) + .execute(&engine.pool) + .await + .expect("move"); + sqlx::query("UPDATE backups SET backup_status = 'CREATING' WHERE backup_arn = ?") + .bind(&arns[2]) + .execute(&engine.pool) + .await + .expect("creating"); + engine.delete(ACCOUNT, &arns[3]).await.expect("delete"); + assert_eq!( + engine.list_all().await.expect("list"), + vec![ + LegacyBackupKey { + account_id: ACCOUNT.to_owned(), + backup_arn: arns[0].clone(), + status: BackupStatus::Available, + }, + LegacyBackupKey { + account_id: OTHER_ACCOUNT.to_owned(), + backup_arn: arns[1].clone(), + status: BackupStatus::Available, + }, + LegacyBackupKey { + account_id: ACCOUNT.to_owned(), + backup_arn: arns[2].clone(), + status: BackupStatus::Creating, + }, + ] + ); + } + + #[tokio::test] + async fn a_replacement_inserts_the_unowned_row_and_removes_the_backup_in_one_step() { + let engine = engine().await; + let arn = legacy_backup(&engine, "src", 7, 0).await; + let row = migrated_row(&engine, &arn).await; + assert_eq!( + counts(&engine, &arn).await, + (7, 1, Some("AVAILABLE".to_owned())) + ); + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::Replaced + ); + let stored = engine + .read_backup_row(&BackupArn::parse(&arn).unwrap()) + .await + .expect("read") + .expect("the index row"); + assert_eq!(stored.owner_generation, UNOWNED_GENERATION); + assert!(stored.completed_at.is_some()); + // The source table is compared parsed: a float read back from the + // catalog's JSON may differ from the one written in its last bit. + let mut stored = stored; + let read_table = stored.source_table().expect("source table"); + let row_table = row.source_table().expect("source table"); + assert!( + (read_table.table_creation_date_time - row_table.table_creation_date_time).abs() + < 1e-3 + ); + assert_eq!( + extenddb_core::backup_format::SourceTableSchema { + table_creation_date_time: row_table.table_creation_date_time, + ..read_table + }, + row_table + ); + stored.source_table_details_json = row.source_table_details_json.clone(); + assert_eq!( + BackupRow { + owner_generation: row.owner_generation, + completed_at: None, + heartbeat_at: 0.0, + created_at: row.created_at, + ..stored + }, + row + ); + // No in-catalog backup is left but the DELETED row. + assert_eq!( + counts(&engine, &arn).await, + (0, 0, Some("DELETED".to_owned())) + ); + assert!(engine.describe(ACCOUNT, &arn).await.is_err()); + assert!(engine.list(ACCOUNT, None).await.expect("list").is_empty()); + assert!(engine.list_all().await.expect("list").is_empty()); + // Again: the row is there, nothing is left to remove. + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::AlreadyReplaced + ); + } + + #[tokio::test] + async fn a_replacement_over_an_adopted_copy_finishes_and_one_of_another_store_refuses() { + let engine = engine().await; + // `backup sync` adopted the copy before the replacement ran. + let adopted = legacy_backup(&engine, "src", 3, 0).await; + let row = migrated_row(&engine, &adopted).await; + assert!(engine.insert_backup_row(&row).await.expect("insert")); + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::AlreadyReplaced + ); + assert_eq!( + counts(&engine, &adopted).await, + (0, 0, Some("DELETED".to_owned())) + ); + + // A row of the id in another store: refused, nothing removed. + let other = legacy_backup(&engine, "other", 3, 0).await; + let row = migrated_row(&engine, &other).await; + let mut elsewhere = row.clone(); + elsewhere.store.store_root = "/srv/elsewhere".to_owned(); + assert!(engine.insert_backup_row(&elsewhere).await.expect("insert")); + let err = engine.replace_with_index_row(&row).await.err(); + assert!( + matches!(&err, Some(StorageError::AlreadyExists(m)) if m.contains("/srv/elsewhere")), + "{err:?}" + ); + assert_eq!( + counts(&engine, &other).await, + (3, 1, Some("AVAILABLE".to_owned())) + ); + assert_eq!( + engine + .read_backup_row(&BackupArn::parse(&other).unwrap()) + .await + .expect("read") + .expect("row") + .store, + elsewhere.store + ); + } + + #[tokio::test] + async fn a_replacement_refuses_an_in_use_or_unavailable_backup_and_writes_nothing() { + let engine = engine().await; + let arn = legacy_backup(&engine, "src", 4, 0).await; + let row = migrated_row(&engine, &arn).await; + // A restore target of the backup still CREATING. + restore(&engine, "dst", &arn).await.expect("restore"); + sqlx::query("UPDATE tables SET table_status = 'CREATING' WHERE table_name = 'dst'") + .execute(&engine.pool) + .await + .expect("creating"); + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::InUse("dst".to_owned()) + ); + assert_eq!( + counts(&engine, &arn).await, + (4, 1, Some("AVAILABLE".to_owned())) + ); + let key = BackupArn::parse(&arn).unwrap(); + assert!(engine.read_backup_row(&key).await.expect("read").is_none()); + sqlx::query("UPDATE tables SET table_status = 'ACTIVE' WHERE table_name = 'dst'") + .execute(&engine.pool) + .await + .expect("active"); + + // Not AVAILABLE: CREATING, then deleted, then missing. + sqlx::query("UPDATE backups SET backup_status = 'CREATING' WHERE backup_arn = ?") + .bind(&arn) + .execute(&engine.pool) + .await + .expect("creating"); + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::NotAvailable("the in-catalog backup is CREATING".to_owned()) + ); + sqlx::query("UPDATE backups SET backup_status = 'AVAILABLE' WHERE backup_arn = ?") + .bind(&arn) + .execute(&engine.pool) + .await + .expect("available"); + engine.delete(ACCOUNT, &arn).await.expect("delete"); + assert_eq!( + engine.replace_with_index_row(&row).await.expect("replace"), + Replaced::NotAvailable("the in-catalog backup is DELETED".to_owned()) + ); + let mut missing = row.clone(); + missing.backup_arn = flip_last_hex(&arn); + missing.backup_id = BackupArn::parse(&missing.backup_arn) + .unwrap() + .backup_id() + .to_owned(); + assert!(matches!( + engine.replace_with_index_row(&missing).await.expect("replace"), + Replaced::NotAvailable(m) if m.contains("Backup not found") + )); + assert!(engine.read_backup_row(&key).await.expect("read").is_none()); + } + } + /// The reconciler's fencing on SQLite (design 4(e)): the restore slot a /// running copy holds, the FAILED write conditioned on the listed claim, /// a copy stopping once its record is failed, and a record this build diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs index 31a77102e..8e3fa210f 100644 --- a/crates/storage-sqlite/src/backup_catalog.rs +++ b/crates/storage-sqlite/src/backup_catalog.rs @@ -444,6 +444,83 @@ pub(crate) fn backup_in_use(table: &str, backup_arn: &str) -> StorageError { extenddb_storage::backup_catalog::backup_in_use(table, backup_arn) } +/// `insert_backup_row`'s statement, in the caller's write transaction: +/// `row` unowned, if no row with its key or its backup id exists. +/// `Ok(false)`, writing nothing, otherwise. The legacy reader's +/// `replace_with_index_row` runs it in its own transaction. +pub(crate) async fn insert_unowned_row( + tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>, + row: &BackupRow, +) -> Result { + let source_table = serde_json::to_string(&row.source_table_details_json) + .map_err(|e| StorageError::Internal(format!("source table details: {e}")))?; + // `ON CONFLICT DO NOTHING` covers the primary key (`account_id`, + // `backup_id`) and the unique `backup_id`, and no other constraint: a + // NOT NULL or CHECK violation is an error. + let inserted = sqlx::query(&format!( + "INSERT INTO backup_index (backup_id, account_id, table_name, \ + backup_arn, backup_name, status, store_kind, store_root, store_id, size_bytes, \ + item_count, created_at, completed_at, failure_reason, \ + source_table_details_json, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, {NOW}, ?, ?, ?, {NOW}) \ + ON CONFLICT DO NOTHING" + )) + .bind(&row.backup_id) + .bind(&row.account_id) + .bind(&row.table_name) + .bind(&row.backup_arn) + .bind(&row.backup_name) + .bind(row.status.as_str()) + .bind(row.store.store_kind.as_str()) + .bind(&row.store.store_root) + .bind(&row.store.store_id) + .bind(row.size_bytes) + .bind(row.item_count) + .bind(format_epoch_seconds(row.created_at)) + .bind(&row.failure_reason) + .bind(&source_table) + .bind(generation_column(UNOWNED_GENERATION)) + .execute(&mut **tx) + .await + .map_err(db_err)? + .rows_affected(); + Ok(inserted == 1) +} + +/// The account, ARN and store of the `backup_index` row holding +/// `backup_id`, if one does, in the caller's transaction. +pub(crate) async fn index_row_holding( + tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>, + backup_id: &str, +) -> Result, StorageError> { + let found: Option<(String, String, String, String, String)> = sqlx::query_as( + "SELECT account_id, backup_arn, store_kind, store_root, store_id FROM backup_index \ + WHERE backup_id = ?", + ) + .bind(backup_id) + .fetch_optional(&mut **tx) + .await + .map_err(db_err)?; + found + .map(|(account_id, backup_arn, kind, store_root, store_id)| { + let store_kind = StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "backup {backup_arn} carries an unknown store kind {kind:?}" + )) + })?; + Ok(( + account_id, + backup_arn, + StoreIdentity { + store_kind, + store_root, + store_id, + }, + )) + }) + .transpose() +} + fn db_err(e: sqlx::Error) -> StorageError { StorageError::Internal(format!("Database error: {e}")) } @@ -834,46 +911,15 @@ impl BackupCatalog for SqliteEngine { // No slot: an unowned row has no job. A job that holds the id's // slot and has not inserted its row yet finds this one and is // refused, as it is by any row with the key. - let source_table = serde_json::to_string(&row.source_table_details_json) - .map_err(|e| StorageError::Internal(format!("source table details: {e}")))?; let _writer = self.write_lock.lock().await; let mut tx = self .pool .begin_with("BEGIN IMMEDIATE") .await .map_err(db_err)?; - // `ON CONFLICT DO NOTHING` covers the primary key (`account_id`, - // `backup_id`) and the unique `backup_id`, and no other - // constraint: a NOT NULL or CHECK violation is an error. - let inserted = sqlx::query(&format!( - "INSERT INTO backup_index (backup_id, account_id, table_name, \ - backup_arn, backup_name, status, store_kind, store_root, store_id, size_bytes, \ - item_count, created_at, completed_at, failure_reason, \ - source_table_details_json, owner_generation, heartbeat_at) \ - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, {NOW}, ?, ?, ?, {NOW}) \ - ON CONFLICT DO NOTHING" - )) - .bind(&row.backup_id) - .bind(&row.account_id) - .bind(&row.table_name) - .bind(&row.backup_arn) - .bind(&row.backup_name) - .bind(row.status.as_str()) - .bind(row.store.store_kind.as_str()) - .bind(&row.store.store_root) - .bind(&row.store.store_id) - .bind(row.size_bytes) - .bind(row.item_count) - .bind(format_epoch_seconds(row.created_at)) - .bind(&row.failure_reason) - .bind(&source_table) - .bind(generation_column(UNOWNED_GENERATION)) - .execute(&mut *tx) - .await - .map_err(db_err)? - .rows_affected(); + let inserted = insert_unowned_row(&mut tx, row).await?; tx.commit().await.map_err(db_err)?; - Ok(inserted == 1) + Ok(inserted) }) } diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs index 2cad85fdc..6cfce2c9d 100644 --- a/crates/storage/src/backup_catalog.rs +++ b/crates/storage/src/backup_catalog.rs @@ -29,8 +29,8 @@ use std::time::Duration; use extenddb_core::backup_format::{BackupArn, SourceTableSchema}; use extenddb_core::types::{ BackupDescription, BackupDetails, BackupSummary, BillingMode, CreateTableInput, - GlobalSecondaryIndexInfo, LocalSecondaryIndexInfo, ProvisionedThroughput, SourceTableDetails, - SourceTableFeatureDetails, TableDescription, + GlobalSecondaryIndexInfo, Item, LocalSecondaryIndexInfo, ProvisionedThroughput, + SourceTableDetails, SourceTableFeatureDetails, TableDescription, }; use futures::future::BoxFuture; use futures::stream::BoxStream; @@ -878,6 +878,63 @@ pub struct AbandonedLegacyBackup { pub removed: bool, } +/// One in-catalog backup that is not DELETED, as +/// [`LegacyBackupReader::list_all`] lists it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LegacyBackupKey { + pub account_id: String, + pub backup_arn: String, + /// CREATING, AVAILABLE or DELETING. + pub status: BackupStatus, +} + +/// An AVAILABLE in-catalog backup opened for copying, as +/// [`LegacyBackupReader::items`] answers it. +pub struct LegacyBackupItems { + /// The backup's name, which its copy keeps. + pub backup_name: String, + /// The backup's recorded source, as + /// [`LegacyBackupReader::restore_source`] answers it. + pub source: BackupRestoreSource, + /// What the backend can state about the source table's vector indexes. + pub vector_state: crate::backup_definition::LegacyVectorState, + /// Every item the backup stored, in the order the backend stored them + /// (not key order), each as the legacy restore reads it: parsed from the + /// stored item JSON, with no validation. The first `Err` ends the stream. + /// Drop it inside a Tokio runtime (the sqlx backends hold a pooled + /// connection until it ends). + pub items: BoxStream<'static, Result>, +} + +impl std::fmt::Debug for LegacyBackupItems { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("LegacyBackupItems") + .field("backup_name", &self.backup_name) + .field("source", &self.source) + .field("vector_state", &self.vector_state) + .finish_non_exhaustive() + } +} + +/// What [`LegacyBackupReader::replace_with_index_row`] did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Replaced { + /// The index row was written and the in-catalog backup removed, in one + /// step. + Replaced, + /// The index row was there already (an earlier run's, or a `backup sync` + /// that adopted the copy), so only what was left of the in-catalog + /// backup was removed. + AlreadyReplaced, + /// The in-catalog backup is not AVAILABLE (missing, another account's, + /// CREATING, DELETING or DELETED) and no index row of it exists; nothing + /// was written. The text names what was found. + NotAvailable(String), + /// A restore target of the backup is CREATING; nothing was written. The + /// text is the target's name. + InUse(String), +} + /// One of the `max_concurrent_jobs` slots of this instance, held by a backup /// or restore job for its life. The engine takes it last among a request's /// checks and hands it to the backend with the job; dropping it frees the @@ -1596,11 +1653,16 @@ pub trait JobClaim: Send + Sync { /// in-catalog format, which each backend keeps as a copy of its own code for /// reading and decoding that shape. The engine consults the catalog index /// first and falls back to this reader; `create` is the CreateBackup path -/// while no `[backup.store]` is configured. +/// while no `[backup.store]` is configured. `extenddb backup migrate-legacy` +/// reads a backup through [`Self::items`] (a frozen copy of each backend's +/// restore read), finds every backup through [`Self::list_all`], and ends +/// with [`Self::replace_with_index_row`]. /// -/// The one part that is not frozen is the restore's registration: a restore -/// registers through the restore job fields (`kind = legacy`, the claim), so -/// the reconciler finds a restore whose owner died as it finds any other. +/// Two parts are not frozen. A restore registers through the restore job +/// fields (`kind = legacy`, the claim), so the reconciler finds a restore +/// whose owner died as it finds any other. And +/// [`Self::replace_with_index_row`] writes a catalog index row in the step +/// that removes the in-catalog backup. pub trait LegacyBackupReader: Send + Sync { /// Create an in-catalog backup of a table, snapshotting all items. The /// permit is the job's slot, released when the create ends. @@ -1705,6 +1767,78 @@ pub trait LegacyBackupReader: Send + Sync { &self, stale_after: Duration, ) -> BoxFuture<'_, Result, StorageError>>; + + /// Every in-catalog backup that is not DELETED, of every account + /// (including an account since deleted), ordered by creation time and + /// then ARN. A frozen `delete` leaves a DELETED row, which is not a + /// backup and is not listed. + fn list_all(&self) -> BoxFuture<'_, Result, StorageError>>; + + /// Open the AVAILABLE in-catalog backup `backup_arn` owned by + /// `account_id` for copying: its name, its recorded source + /// ([`Self::restore_source`]'s answer), what the backend can state about + /// its vector indexes, and a stream of every item it stored. + /// + /// The stream is a copy of the backend's restore read, kept apart from + /// it so either can change alone: PostgreSQL reads the items in one + /// `REPEATABLE READ` catalog transaction, held until the stream ends or + /// is dropped, whose first statement re-checks AVAILABLE; SQLite reads + /// keyset batches, each in its own read transaction that re-checks + /// AVAILABLE; MongoDB reads the backup's collection with a cursor; and + /// Cassandra reads the item buckets in pages. On SQLite, MongoDB and + /// Cassandra a DeleteBackup that lands while the stream runs ends it + /// with an error or leaves it short; the caller compares the count with + /// `source.item_count`. + /// + /// # Errors + /// + /// `StorageError::Validation("Backup not found: ")` if the backup is + /// missing, another account's, or not AVAILABLE. + /// `StorageError::Unsupported` for a definition this build cannot read + /// and, on MongoDB, for a backup taken before the `$out` copy, which has + /// no item collection to read. + fn items( + &self, + account_id: &str, + backup_arn: &str, + ) -> BoxFuture<'_, Result>; + + /// Replace the in-catalog backup `row.backup_arn` of `row.account_id` by + /// `row`, the catalog index row of its copy in the store, in one step + /// that DeleteBackup and a restore's registration order wholly before or + /// after: the in-catalog backup is read AVAILABLE and locked, the in-use + /// rule runs (a CREATING restore target of the backup), `row` is + /// inserted as [`BackupCatalog::insert_backup_row`] inserts it (unowned, + /// `completed_at` and `heartbeat_at` on the database's clock), and the + /// backup's items and definition are removed with the row left DELETED, + /// as the frozen `delete` leaves it. PostgreSQL and SQLite do it in one + /// transaction (SQLite under the engine write lock). MongoDB writes the + /// index document and DELETING in one session transaction and then drops + /// the item collection and writes DELETED, as the frozen `delete` does; + /// a stop after the commit leaves DELETING, which describe and list + /// leave out and a later call or DeleteBackup finishes. Cassandra, whose + /// in-catalog and index rows share `backups_by_arn`, converts the row in + /// place with one lightweight transaction (`store_kind` set, so the + /// in-catalog reader no longer reads it), then writes its list and status + /// entries and deletes the item buckets; no DELETED row is left there. + /// + /// [`Replaced::AlreadyReplaced`] when an index row with `row`'s account, + /// ARN and store is there already: whatever is left of the in-catalog + /// backup is removed as above, the in-use rule first, and nothing else + /// is written. An index row with the backup's id under another ARN + /// spelling or another store is an `AlreadyExists` error, with nothing + /// written. + /// + /// # Errors + /// + /// `StorageError::AlreadyExists` as above; the backend's errors + /// otherwise, after which the in-catalog backup is as it was or (MongoDB + /// after the commit, Cassandra after the conversion) the call can be + /// repeated to finish it. + fn replace_with_index_row<'a>( + &'a self, + row: &'a BackupRow, + ) -> BoxFuture<'a, Result>; } #[cfg(test)] diff --git a/crates/storage/src/backup_definition.rs b/crates/storage/src/backup_definition.rs index db7f5247a..7802001fe 100644 --- a/crates/storage/src/backup_definition.rs +++ b/crates/storage/src/backup_definition.rs @@ -27,7 +27,7 @@ use extenddb_core::backup_format::SourceTableSchema; use extenddb_core::types::{ AttributeDefinition, BillingMode, CreateTableInput, GsiInput, KeySchemaElement, KeyType, - LsiInput, OnDemandThroughput, ProvisionedThroughput, StreamSpecification, + LsiInput, OnDemandThroughput, ProvisionedThroughput, SseDescription, StreamSpecification, TimeToLiveDescription, }; use serde::{Deserialize, Serialize}; @@ -438,6 +438,95 @@ impl BackupRestoreSource { } } +/// What a backend can state about the vector indexes of an in-catalog +/// backup's source table: the three states of a manifest's +/// `source_table.vector_indexes`, of which `Unknown` has no manifest form. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum LegacyVectorState { + /// The backend has no vector support (MongoDB, Cassandra): `null`. + Unsupported, + /// The backend supports vector indexes and its record states that the + /// table had none: a definition with no `VectorIndexNames`, or on + /// PostgreSQL a backup without a definition whose `vector_indexes` + /// snapshot is NULL or empty. `[]`. + NoneRecorded, + /// The backend supports vector indexes and recorded nothing about them: + /// a SQLite backup taken before catalog 0.0.4, which has no definition. + Unknown, +} + +/// The manifest `source_table` of a copy of an in-catalog backup: the table +/// the legacy restore of `source` creates, so a restore of the copy creates +/// the same table (`restore_input_from_manifest` of the result equals +/// `source.restore_input` for the same overrides, up to the SSE +/// specification, below). +/// +/// The keys, the attribute definitions (trimmed to the keys for a backup +/// with no definition, as the legacy restore trims them), the billing mode, +/// the throughput (5/5 for a provisioned backup that recorded none), the +/// secondary indexes (a provisioned GSI that recorded no throughput carries +/// the table's), the on-demand throughput and the SSE settings are those of +/// `source.restore_input` with no overrides. `table_class`, `time_to_live` +/// and `stream_specification` come from the definition, for DescribeBackup; +/// the SSE specification is recorded as the description DescribeBackup +/// reports for it ([`SseDescription::from_specification`]). A restore maps +/// an `ENABLED` description back to `{"Enabled": true, "SSEType": "KMS"}`, +/// so a `KMSMasterKeyId` the specification carried is not in the copy. +/// `vector_indexes` is `null` for [`LegacyVectorState::Unsupported`] and `[]` +/// for [`LegacyVectorState::NoneRecorded`]. +/// +/// # Errors +/// +/// [`StorageError::Unsupported`] for what +/// [`BackupRestoreSource::ensure_restorable`] refuses (a multi-part base key, +/// vector indexes), and for [`LegacyVectorState::Unknown`]: the copy would +/// have to guess the table's vector indexes. +pub fn source_table_from_legacy( + source: &BackupRestoreSource, + vector_state: LegacyVectorState, + region: &str, + account_id: &str, +) -> Result { + source.ensure_restorable()?; + let vector_indexes = match vector_state { + LegacyVectorState::Unsupported => None, + LegacyVectorState::NoneRecorded => Some(Vec::new()), + LegacyVectorState::Unknown => { + return Err(StorageError::Unsupported(format!( + "backup {} has no recorded table definition (it was taken before catalog \ + 0.0.4), so its vector indexes cannot be stated", + source.backup_arn + ))); + } + }; + let input = source.restore_input(&source.source_table_name, &RestoreTableOverrides::default()); + let definition = source.definition.as_ref(); + Ok(SourceTableSchema { + table_name: source.source_table_name.clone(), + table_id: source.source_table_id.clone(), + table_arn: source.source_table_arn.clone(), + table_creation_date_time: source.table_creation_date_time, + key_schema: input.key_schema, + attribute_definitions: input.attribute_definitions, + billing_mode: input.billing_mode.unwrap_or(BillingMode::PayPerRequest), + provisioned_throughput: input.provisioned_throughput, + on_demand_throughput: input.on_demand_throughput, + table_class: definition.and_then(|d| d.table_class.clone()), + sse_description: SseDescription::from_specification( + input.sse_specification.as_ref(), + region, + account_id, + ), + global_secondary_indexes: input.global_secondary_indexes.unwrap_or_default(), + local_secondary_indexes: input.local_secondary_indexes.unwrap_or_default(), + vector_indexes, + stream_specification: definition.and_then(|d| d.stream_specification.clone()), + time_to_live: definition.and_then(|d| d.time_to_live.clone()), + item_count: source.item_count, + table_size_bytes: source.table_size_bytes, + }) +} + /// What the engine answers a `RestoreTableFromBackup` with, beside the /// target's description: the three members of the `RestoreSummary`, from a /// legacy backup's [`BackupRestoreSource`] or a store-backed backup's @@ -1054,4 +1143,345 @@ mod tests { b.trim_start_matches("backup arn") ); } + + /// Every combination of the definition shapes a legacy backup can carry, + /// as a [`BackupRestoreSource`]. + #[allow(clippy::too_many_lines)] + fn legacy_sources() -> Vec { + let key = |n: &str, t| KeySchemaElement { + attribute_name: n.to_owned(), + key_type: t, + }; + let attr = |n: &str| AttributeDefinition { + attribute_name: n.to_owned(), + attribute_type: extenddb_core::types::ScalarAttributeType::S, + }; + let throughput = |r, w| ProvisionedThroughput { + read_capacity_units: r, + write_capacity_units: w, + }; + let gsi = |t: Option| GsiInput { + index_name: "g".to_owned(), + key_schema: vec![key("gpk", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::KeysOnly, + non_key_attributes: None, + }, + provisioned_throughput: t, + }; + let lsi = || LsiInput { + index_name: "l".to_owned(), + key_schema: vec![key("pk", KeyType::Hash), key("lsk", KeyType::Range)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + }; + let key_schemas = [ + vec![key("pk", KeyType::Hash)], + vec![key("pk", KeyType::Hash), key("sk", KeyType::Range)], + ]; + let gsis = [ + Vec::new(), + vec![gsi(Some(throughput(3, 4)))], + vec![gsi(None)], + vec![gsi(Some(throughput(0, 0)))], + ]; + let sses = [ + None, + Some(serde_json::json!({"Enabled": true, "SSEType": "KMS"})), + Some(serde_json::json!({"Enabled": true})), + Some(serde_json::json!({"Enabled": false})), + ]; + let ttl = TimeToLiveDescription { + time_to_live_status: extenddb_core::types::TimeToLiveStatus::Enabled, + attribute_name: Some("ttl".to_owned()), + }; + let stream = StreamSpecification { + stream_enabled: true, + stream_view_type: Some(extenddb_core::types::StreamViewType::KeysOnly), + }; + let mut sources = Vec::new(); + for key_schema in &key_schemas { + for billing in ["PROVISIONED", "PAY_PER_REQUEST"] { + for recorded in [None, Some(throughput(8, 9))] { + let mut attributes: Vec = + key_schema.iter().map(|k| attr(&k.attribute_name)).collect(); + // The index attributes, which a pre-0.0.4 backup recorded + // with its keys and no indexes to name them. + attributes.extend([attr("gpk"), attr("lsk")]); + let base = BackupRestoreSource { + backup_arn: "arn:aws:dynamodb:us-east-1:123456789012:table/src/backup/\ + 01757500000123-73d8d5bc" + .to_owned(), + backup_creation_date_time: 1_757_500_000.123, + source_table_name: "src".to_owned(), + source_table_arn: "arn:aws:dynamodb:us-east-1:123456789012:table/src" + .to_owned(), + source_table_id: "id".to_owned(), + key_schema: key_schema.clone(), + attribute_definitions: attributes, + billing_mode: billing.to_owned(), + provisioned_throughput: recorded.clone(), + item_count: 12, + table_size_bytes: 345, + table_creation_date_time: 1_757_400_000.0, + definition: None, + legacy_vector_index_count: 0, + }; + // A backup taken before catalog 0.0.4: keys, billing and + // the throughput column only. + sources.push(base.clone()); + for gsis in &gsis { + for lsis in [Vec::new(), vec![lsi()]] { + for sse in &sses { + for on_demand in [ + None, + Some(OnDemandThroughput { + max_read_request_units: Some(10), + max_write_request_units: None, + }), + ] { + for extras in [false, true] { + let mut source = base.clone(); + source.definition = Some(BackupTableDefinition { + version: BACKUP_DEFINITION_VERSION, + billing_mode: billing.to_owned(), + provisioned_throughput: recorded.clone(), + global_secondary_indexes: gsis.clone(), + local_secondary_indexes: lsis.clone(), + table_class: extras + .then(|| "STANDARD_INFREQUENT_ACCESS".to_owned()), + sse_specification: sse.clone(), + on_demand_throughput: on_demand.clone(), + vector_index_names: Vec::new(), + time_to_live: extras.then(|| ttl.clone()), + stream_specification: extras.then(|| stream.clone()), + }); + sources.push(source); + } + } + } + } + } + } + } + } + sources + } + + /// The SSE settings a target created from `input` reports, which is what + /// the two paths must agree on: the specification itself differs in + /// spelling (`{"Enabled": false}` against none, `{"Enabled": true}` + /// against the same with `"SSEType": "KMS"`). + fn reported_sse(input: &CreateTableInput) -> Option { + SseDescription::from_specification( + input.sse_specification.as_ref(), + "us-east-1", + "123456789012", + ) + } + + #[test] + fn a_copy_of_a_legacy_backup_restores_the_table_its_legacy_restore_creates() { + let overrides = [ + RestoreTableOverrides::default(), + RestoreTableOverrides { + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + }, + RestoreTableOverrides { + sse_specification: Some(serde_json::json!({"Enabled": true, "SSEType": "KMS"})), + ..Default::default() + }, + RestoreTableOverrides { + sse_specification: Some(serde_json::json!({"Enabled": false})), + ..Default::default() + }, + ]; + let sources = legacy_sources(); + assert!(sources.len() > 1000, "{} shapes", sources.len()); + let mut checked = 0; + for source in &sources { + for state in [ + LegacyVectorState::Unsupported, + LegacyVectorState::NoneRecorded, + ] { + let table = + source_table_from_legacy(source, state, "us-east-1", "123456789012").unwrap(); + // The manifest member survives its own serialization, as the + // manifest writer and parser carry it. + let json = serde_json::to_value(&table).unwrap(); + let back: SourceTableSchema = serde_json::from_value(json).unwrap(); + assert_eq!(back, table); + assert_eq!( + table.vector_indexes, + (state == LegacyVectorState::NoneRecorded).then(Vec::new) + ); + for o in &overrides { + let legacy = source.restore_input("dst", o); + let mut copy = restore_input_from_manifest(&table, "dst", o); + // Without an SSE override, the copy restores the one + // specification an ENABLED description maps back to. + assert!( + o.sse_specification.is_some() + || copy.sse_specification.is_none() + || copy.sse_specification + == Some(serde_json::json!({"Enabled": true, "SSEType": "KMS"})), + "{:?}", + copy.sse_specification + ); + assert_eq!( + reported_sse(©), + reported_sse(&legacy), + "{source:?} {o:?}" + ); + copy.sse_specification.clone_from(&legacy.sse_specification); + assert_eq!(copy, legacy, "{source:?} {o:?}"); + checked += 1; + } + } + } + assert_eq!(checked, sources.len() * 2 * overrides.len()); + } + + #[test] + fn a_copy_records_the_described_features_and_the_legacy_identity() { + let source = legacy_sources() + .into_iter() + .find(|s| { + s.definition.as_ref().is_some_and(|d| { + d.table_class.is_some() + && d.sse_specification.is_some() + && d.billing_mode == "PROVISIONED" + && d.provisioned_throughput.is_some() + && d.global_secondary_indexes + .first() + .is_some_and(|g| g.provisioned_throughput.is_none()) + }) + }) + .unwrap(); + let table = source_table_from_legacy( + &source, + LegacyVectorState::NoneRecorded, + "us-east-1", + "123456789012", + ) + .unwrap(); + let definition = source.definition.clone().unwrap(); + assert_eq!(table.table_name, "src"); + assert_eq!(table.table_id, "id"); + assert_eq!(table.table_arn, source.source_table_arn); + assert!((table.table_creation_date_time - 1_757_400_000.0).abs() < f64::EPSILON); + assert_eq!(table.item_count, 12); + assert_eq!(table.table_size_bytes, 345); + assert_eq!(table.table_class, definition.table_class); + assert_eq!(table.time_to_live, definition.time_to_live); + assert_eq!(table.stream_specification, definition.stream_specification); + assert_eq!( + table.sse_description.map(|d| d.status), + Some("ENABLED".to_owned()) + ); + // The provisioned GSI with no throughput carries the table's, as the + // legacy restore creates it. + let gsi_throughput = table.global_secondary_indexes[0] + .provisioned_throughput + .clone() + .map(|t| (t.read_capacity_units, t.write_capacity_units)); + assert_eq!(gsi_throughput, Some((8, 9))); + + // A backup with no definition records its keys and nothing more. + let mut bare = source.clone(); + bare.definition = None; + let table = source_table_from_legacy( + &bare, + LegacyVectorState::Unsupported, + "us-east-1", + "123456789012", + ) + .unwrap(); + assert_eq!( + table + .attribute_definitions + .iter() + .map(|a| a.attribute_name.as_str()) + .collect::>(), + table + .key_schema + .iter() + .map(|k| k.attribute_name.as_str()) + .collect::>() + ); + assert!(table.global_secondary_indexes.is_empty()); + assert!(table.table_class.is_none()); + assert!(table.time_to_live.is_none()); + assert!(table.stream_specification.is_none()); + assert!(table.vector_indexes.is_none()); + } + + #[test] + fn a_copy_drops_a_kms_key_id_and_nothing_else() { + let mut def = sample(); + def.sse_specification = Some(serde_json::json!({ + "Enabled": true, "SSEType": "KMS", "KMSMasterKeyId": "alias/k" + })); + let source = source(Some(def)); + let table = source_table_from_legacy( + &source, + LegacyVectorState::NoneRecorded, + "us-east-1", + "123456789012", + ) + .unwrap(); + let none = RestoreTableOverrides::default(); + let legacy = source.restore_input("dst", &none); + let mut copy = restore_input_from_manifest(&table, "dst", &none); + assert_eq!( + legacy.sse_specification.as_ref().unwrap()["KMSMasterKeyId"], + "alias/k" + ); + assert_eq!( + copy.sse_specification, + Some(serde_json::json!({"Enabled": true, "SSEType": "KMS"})) + ); + copy.sse_specification.clone_from(&legacy.sse_specification); + assert_eq!(copy, legacy); + } + + #[test] + fn a_copy_refuses_what_the_legacy_restore_refuses_and_an_unknown_vector_state() { + let state = LegacyVectorState::NoneRecorded; + let copy = |s: &BackupRestoreSource, state| { + source_table_from_legacy(s, state, "us-east-1", "123456789012") + }; + // No definition and nothing recorded about vector indexes. + assert!(matches!( + copy(&source(None), LegacyVectorState::Unknown), + Err(StorageError::Unsupported(m)) + if m.contains("no recorded table definition") && m.contains("vector indexes") + )); + // A definition makes the state known whatever the backend says. + assert!(copy(&source(Some(sample())), state).is_ok()); + let mut def = sample(); + def.vector_index_names.push("v".to_owned()); + assert!(matches!( + copy(&source(Some(def)), state), + Err(StorageError::Unsupported(m)) if m.contains("1 vector index") + )); + let mut counted = source(None); + counted.legacy_vector_index_count = 2; + assert!(matches!( + copy(&counted, state), + Err(StorageError::Unsupported(m)) if m.contains("2 vector index") + )); + let mut multi = source(None); + multi.key_schema.push(KeySchemaElement { + attribute_name: "pk2".to_owned(), + key_type: KeyType::Hash, + }); + assert!(matches!( + copy(&multi, state), + Err(StorageError::Unsupported(m)) if m.contains("multi-part key") + )); + } } diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 0f4d75a58..c3cd49700 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -84,7 +84,7 @@ Three store facts that rules of sections 4 and 5 rest on, and the one method a l The interfaces later PRs add to `extenddb-storage`, each with its contract; the signatures and the row keys are in the section of the PR that adds them (Phasing): - `RestoreEngine` (PR 9a), a trait in `StorageEngine`'s bound (`StorageEngine` has no methods of its own) that each backend implements: `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> Result` writes the items of one verified data file from item `start_at` on, and their index rows, into a CREATING restore target, records progress in the write that applied the items as 5.3 step 5 specifies for the backend (the job's `claim` carries the generation the record's writes are conditioned on) and returns how many items it applied; `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }` is the ACTIVE flip of 5.3 step 6; `count_restore_target(target)` is the count of section 1; `read_restore_progress` and `sweep_restore_progress(running)` read and sweep PostgreSQL's `restore_progress` rows (defaults elsewhere). There is one spelling of each signature, in PR 9a's section; 5.3 step 5 calls `bulk_load`. - The engine is backend-agnostic (main's `crates/engine` reaches every backend through the `extenddb_storage` traits), and nothing in sections 4 and 5 that touches a catalog row (the claim, the heartbeat, the conditional status writes, the store-scoped listings, the restore job record) can be written in it. `BackupCatalog`, a trait in `extenddb-storage` that PR 6 adds and each backend implements, is how the engine reaches the catalog index of section 4; PRs 8, 9a and 10 add methods to the same trait as their sections need them. Section 4's "Authority" line names this trait beside the engine module that binds it. -- `LegacyBackupReader` is a trait in `extenddb-storage` with `describe`, `list`, `restore` and `delete`, plus `create`, which every backend keeps until PR 14: the engine calls it for CreateBackup when no `[backup.store]` is configured (section 7), logging a deprecation warning that names `[backup.store]` on each call, and on Cassandra always, since it has no `SnapshotEngine` (the decision before PR 14 settles whether that stays), and a legacy create takes a `max_concurrent_jobs` permit like any job, returning `LimitExceededException` when none is free (5.1 step 1), since PR 6 removes the slot semaphores #384's creates wait on; each backend implements it with a frozen copy of its #384 code for reading and decoding the legacy shape (not calls through trait methods later PRs remove), and the engine holds `Option>`. PR 14 removes the trait. +- `LegacyBackupReader` is a trait in `extenddb-storage` with `describe`, `list`, `restore` and `delete`, plus `create`, which every backend keeps until PR 14: the engine calls it for CreateBackup when no `[backup.store]` is configured (section 7), logging a deprecation warning that names `[backup.store]` on each call, and on Cassandra always, since it has no `SnapshotEngine` (the decision before PR 14 settles whether that stays), and a legacy create takes a `max_concurrent_jobs` permit like any job, returning `LimitExceededException` when none is free (5.1 step 1), since PR 6 removes the slot semaphores #384's creates wait on; each backend implements it with a frozen copy of its #384 code for reading and decoding the legacy shape (not calls through trait methods later PRs remove), and the engine holds `Option>`. For `migrate-legacy` (PR 10) it gains `items(account, arn)` (the backup's name, its recorded source, its vector state and a stream of its items, each backend's a frozen copy of its restore's read), `list_all()` (the in-catalog backups of every account that are not DELETED) and `replace_with_index_row(row)`, which writes the backup's catalog row and removes the in-catalog backup in one step, the trait's second write beside the restore's registration (revision item 37). PR 14 removes the trait. ### 4. Catalog index @@ -94,15 +94,15 @@ The invariants and state machines every later PR honours, by lettered part; the (b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, its progress (`file_index` and `items_applied`: per batch on SQLite and Cassandra, `file_index` per file on MongoDB; PostgreSQL records progress in `restore_progress` rows of the data database instead, 5.3 step 5), `failure_reason` and `completed_at` (PR 6's schema; `RestoreJobRow` carries every member). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `RestoreJobs{failed}` (section 9). -(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10; a server on an in-memory database adopts, at startup, a root whose file another in-memory catalog wrote, revision item 34). The previous owner keeps acting on the root until a later check that can read its catalog finds the store unhealthy, at the earliest one reconciler interval after the cycle in progress ends, unless its deletion fence stops it first: each deletion from the store, DeleteBackup's and purge's row change before it, and each CreateBackup before its first put, re-reads the file and is refused, the store marked unhealthy at once, when the file names another catalog (revision item 34). A write already past the fence's read still lands, so every server of the previous catalog is stopped before an adoption. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL, PR 8's review; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. +(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10; a server on an in-memory database adopts, at startup, a root whose file another in-memory catalog wrote, revision item 34). The previous owner keeps acting on the root until a later check that can read its catalog finds the store unhealthy, at the earliest one reconciler interval after the cycle in progress ends, unless its deletion fence stops it first: each deletion from the store, DeleteBackup's and purge's row change before it, and each CreateBackup and each `migrate-legacy` copy before its first put, re-reads the file and is refused, the store marked unhealthy at once, when the file names another catalog (revision item 34). A write already past the fence's read still lands, so every server of the previous catalog is stopped before an adoption. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL, PR 8's review; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. (d) Identities. The ARN encodes partition, region, account, table, and backup id, and the key prefix is derived from account, table and id deterministically by `BackupArn::store_prefix`, under the configured store's prefix, and no row stores it, so restore, sync, purge and the reconciler derive it the same way and `describe_backup(arn)` works from the store alone after a sync. The engine, the writer and sync all parse and format ARNs through one implementation, `BackupArn` in `extenddb_core::backup_format` (#394); the engine keeps #384's region handling: it resolves a backup under the ARN as written and under the ARN re-spelled in its own region, and refuses a partition other than its own. `backup_creation_date_time` and `created_at` are the id's instant, so the ARN, the manifest, the API's reported time and the ListBackups cursor agree by construction. One row is addressed by `BackupKey` (`account_id`, `backup_id`, `backup_arn`): PostgreSQL, SQLite and MongoDB key `backup_index` by (`account_id`, `backup_id`), Cassandra keys `backups_by_arn` by (`account_id`, `backup_arn`); no method takes a bare backup id, and the engine treats a row whose `account_id`, `table_name` or `backup_arn` (after that re-spelling) differs from the request's as absent. One target is addressed by `RestoreTargetKey` (`account_id`, `table_name`, `table_id`), the table id guarding against a table re-created under the name. (e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No CREATING row exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); the AVAILABLE rows `backup sync` inserts are unowned (`owner_generation` 0) and have no job, and no duty reads an AVAILABLE row's generation (revision item 34); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING backup row and every RUNNING store-backed restore record at startup (the restart ages every RUNNING restore's heartbeat to the epoch), and at runtime a row or record whose in-process slot no task holds (on SQLite the store-backed restore's claim holds the `restore:/` slot from before the target exists to the task's end; the legacy restore holds no slot, and a legacy and a store-backed restore into one name exclude each other through the name's `tables` row under the write transaction (`BEGIN IMMEDIATE`: the CREATING-target check, then the unique name)); on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A dead RUNNING store-backed restore record is taken over through `claim_restore_dead(key, stale_after, spool_host)` under the same rules (PostgreSQL's name lock and SQLite's slot refuse a taker outright; MongoDB and Cassandra condition on the stale heartbeat, Cassandra's lightweight transaction also on the heartbeat it read and on `kind = manifest`), which writes the claimant's `spool_host` with the fresh generation and never takes over a legacy record; the successor resumes the restore from the recorded progress (5.3 step 7). A restore record's takeover differs from a backup row's in one write: `claim_restore_dead` sets `heartbeat_at` to the database's clock in the same conditional write as the generation and `spool_host` (PostgreSQL `now()`, SQLite `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, MongoDB `$$NOW`, Cassandra `toTimestamp(now())`), so a second instance's reconciler, whose cycle lands before the successor's first beat (one `heartbeat_interval_secs` after the takeover), finds the record live and is refused. The backup takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale window counts from the owner's last beat; nothing waits on a restore record's old heartbeat after a takeover (the FAILED-target duty ages on `COALESCE(completed_at, heartbeat_at)` and every FAILED write of a resume stamps `completed_at`; the maintenance claim waits on RUNNING, not on the beat). Per backend, the second taker is refused by: PostgreSQL the successor's advisory lock on the target name, then the fresh heartbeat; SQLite the successor's in-process slot (one process serves the file), and the record the startup step had aged is no longer listed as dead once the takeover refreshed its heartbeat; MongoDB the `$expr` staleness term of the takeover's own filter against the refreshed `heartbeat_at`; Cassandra the takeover's serial read, which sees the refreshed `restore_heartbeat_at` and stops before its lightweight transaction. So the two-taker argument of the backup takeover is, for a restore, "a second taker finds the record live"; the belt under it is that every progress write is fenced by the generation (a first taker's next write would match nothing and its job end with no table access) and that a resume's first file is `reapplying`, which covers a partial file on MongoDB and which Cassandra's upserts collapse, so a later change to either mechanism (a progress write without the generation, `reapplying` on a later file) is seen to matter here. The takeover on Cassandra also re-applies the target's `active_restores[table_id] = name` entry on the backup's row when it is absent (one serial read, then `UPDATE ... IF backup_status = 'AVAILABLE'`): the registration writes the target's row and its DDL before the entry, so an owner that died between the two left a resumable record whose backup nothing held in use, and `DeleteBackup` would have succeeded under the resumed load. A backup no longer AVAILABLE keeps no entry; the resumed load fails on the missing prefix and the target is removed. The engine's first confirmed beat of a fresh restore is the return of the one heartbeat the task sends right after `claim_restore` returns and before the reply (5.3 step 4), and of a resumed restore the instant `claim_restore_dead` returned, taken by the reconciler beside the takeover and handed to the job; a takeover's `heartbeat_at` was stamped inside that write, so the instant is at or after the beat it stands for by the write's own duration, and the catalog reads between a takeover and the resumed job's start count against the two-period window rather than widen it. The registration's own `heartbeat_at` stamp is the catalog's clock at some point of the registration, not its end: on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL commits a stale `heartbeat_at`; the job's first beat, sent before the reply, refreshes it. Two readers judge a RUNNING record's owner by the heartbeat alone and need that beat: the reconciler under a maintenance claim (a dead record is failed and its target removed, or failed `target_not_empty`) and `extenddb backup remove-target`; the lock fences only `claim_restore_dead`. A first beat that finds the claim lost, or fails, ends the task before the reply: the record is failed with reason "the claim was lost before the job started" (a write that matches nothing when the claim is another's) and the request is answered `InternalServerError`. What bounds a registration's duration per backend: PostgreSQL holds the `backup_index` row's `FOR UPDATE` lock and the target name's advisory lock through the registration, so `claim_restore_dead` is refused however long it takes; SQLite holds the in-process `restore:/
` slot from before the target exists; MongoDB runs the registration in a transaction bounded by `transactionLifetimeLimitSeconds` (default 60 s); Cassandra's registration writes are lightweight transactions bounded by the coordinator's `write_request_timeout_in_ms` and `cas_contention_timeout_in_ms` (Cassandra defaults 2 s and 1 s), and the manifest registration runs no heartbeat during the target's DDL; a registration whose DDL outlasts the stale window can lose its record to a takeover, and its refusal path then conditions the target's removal on its own generation and removes nothing (the registration answers an error naming the lost claim, and the record's owner finishes the restore). A registration that took two periods does not end the job at its first check, since the two-period rule counts from the first beat's return. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. PostgreSQL's lock rule applies to the resume's takeover alone: the two writes the reconciler makes to a dead RUNNING manifest record without resuming it, FAILED `OWNER_DEAD` (the target removed) or `target_not_empty` (the target left, 5.3 step 7) under a maintenance claim and FAILED `OTHER_STORE`, test the heartbeat alone on every backend, because the operator's command waits on the first and the second is a configuration this design forbids (4(c): instances of one catalog share one root), so neither waits for a stopped owner's session to end: both go through `fail_restore_job`, which conditions on RUNNING alone after `list_dead_restore_jobs` selected the record by heartbeat age, and on PostgreSQL they do not try the owner's advisory lock, so a stopped owner whose session lives keeps its record on the resume path and loses it on these two. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. While the server holds no serving registration (PostgreSQL, MongoDB and Cassandra) it does what the module doc of `crates/server/src/registration.rs` says: it refuses new requests, stops its background workers, the reconciler among them, and starts a new set once it registers again, and the work requests started goes on. Two points that doc leaves to this design: a reconciler cycle running at the stop spawns no resume once its own token has fired (the record is left for the next set's reconciler, which starts from a fresh state), and the backup and restore jobs, resumes included, run on the jobs' token, a child of the server's shutdown token and not of the workers', so a job running when the registration is lost goes on writing the catalog and the store until it ends or the server stops, beside a migration that may already have taken the catalog. Gating the running jobs on the registration (a job stopping while the server is unregistered) is GitHub issue #755. -(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Plain `backup sync` takes it too where it runs beside servers (PostgreSQL, MongoDB, Cassandra; on SQLite the serve lock already excludes purge), without the wait for running jobs: its insert-if-absent orders it against a CreateBackup, and the claim keeps it from adopting a prefix purge has un-rowed and not yet deleted (revision item 34). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. +(f) The maintenance claim. `backup purge`, `backup remove-target`, `backup migrate-legacy` (revision item 37) and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Plain `backup sync` takes it too where it runs beside servers (PostgreSQL, MongoDB, Cassandra; on SQLite the serve lock already excludes purge), without the wait for running jobs: its insert-if-absent orders it against a CreateBackup, and the claim keeps it from adopting a prefix purge has un-rowed and not yet deleted (revision item 34). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. -(g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and removes it only with `--prune-missing`, outright, as `--forget-store` removes rows: no DELETING or DELETED tombstone is left and no object is deleted, so on host-local roots a prune on the instance that lacks the bytes leaves them on the other, where a later sync adopts them again (revision item 34); without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root. +(g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and removes it only with `--prune-missing`, outright, as `--forget-store` removes rows: no DELETING or DELETED tombstone is left and no object is deleted, so on host-local roots a prune on the instance that lacks the bytes leaves them on the other, where a later sync adopts them again (revision item 34); without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size, nor one whose ARN, under either spelling, an in-catalog backup holds (it names `extenddb backup migrate-legacy `, revision item 37). Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules and `migrate-legacy`'s replacement, which writes the row of a prefix it wrote only after the same check (`verify_prefix`) passes on it (PR 10, revision item 37), are the only paths from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root. (h) Trust boundary of sync: a principal that can write under an account's prefix can forge a backup for that account. Checksums do not help, they are forgeable by the same principal. The deployment guide states that write access to an account's prefix is backup-administrator authority for that account, and gives per-account IAM prefix policies. A server-side MAC over manifests is not in v1; if hostile writers with bucket access are in scope for a deployment, the operator restricts write access rather than relying on ExtendDB to detect forgeries. @@ -169,7 +169,7 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. On SQLite the restore's write batch (`RESTORE_BATCH_ITEMS`, 10,000 items, or `RESTORE_BATCH_BYTES`, 32 MiB of item text) and the decode chunks in front of it grow with the item's size up to the batch bound, whatever the attribute types, so a restore of items of more than a few hundred bytes holds more: 253 MiB over idle for 100k items of about 2.5 KB of strings, 130 MiB under one arena (two runs each), and more for lists of numbers of the same text size, which decode larger (345 to 424 MiB, 184 to 195 MiB under one arena, at 100k items of 128 numbers, with or without a vector index); at 1M items of 128 numbers 552 MiB under the default arenas and 204 MiB under one arena, and 301 to 316 MiB under one arena for 20,000 items of 1,536 numbers (`15-backup-restore-bench-pr11.md`). A PostgreSQL vector index adds 27 to 53 MiB to the restore at 100k items (five pairs of runs with and without it). Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. `backup sync`, which is not a job, keeps of the configured root's rows (every status) each row's status, store id and table and the key of each AVAILABLE row of the configured store id, dropping the rest of each row as it is read, and holds the objects of the one prefix the listing is in with their sizes and the prefix of each manifest it listed; every backend's `list_backup_rows` collects its whole result before it yields the first row, so the read of the rows is sync's largest term on a catalog with many rows of the root (tombstones within their retention included): about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of `source_table_details_json`; 610 MiB at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none, and 148 MiB at 50,000 rows whose JSON is `{}`, so nearly all of it is the parsed description), measured on a SQLite debug build on a 16-core x86_64 Linux host under glibc's default arenas, the same peak `backup list` reaches over those rows; `backup sync --move-store` keeps the old root's rows without their parsed description and the configured root's (account, backup id) pairs, and reads one manifest and one prefix listing at a time, so its peak is that same read of the rows: 569 MiB at 20,000 rows of a table with five GSIs on a SQLite debug build and 500 MiB on a PostgreSQL debug build, against 51 and 49 MiB at 50 rows, 37 s each with filesystem roots on a local disk, on that host (revision item 36); the filesystem store builds its listing of the whole root in one `Vec` before it yields the first key (about 72 bytes plus the key per object, the store trait's doc), and the S3 store lists a page of 1,000 keys at a time. Measured on a filesystem store with 100 objects per prefix and 60-byte keys: 45 MiB of peak RSS for an empty root, 66 MiB for 100,000 objects and 98 MiB for 300,000, about 190 bytes per object, nearly all of it the store's listing (revision item 34). The report keeps an entry per prefix for the whole run (it lists every prefix the catalog already has a row for among the skipped ones), so at one object per prefix the per-prefix terms double that: about 350 bytes per prefix, 62 MiB of peak RSS for 50,000 such prefixes and 112 MiB for 200,000 against 45 MiB for an empty root (and about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build of PR 10 with a filesystem root and no rows. On an in-memory SQLite server that cost is paid in the serving process at every start. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. On SQLite the restore's write batch (`RESTORE_BATCH_ITEMS`, 10,000 items, or `RESTORE_BATCH_BYTES`, 32 MiB of item text) and the decode chunks in front of it grow with the item's size up to the batch bound, whatever the attribute types, so a restore of items of more than a few hundred bytes holds more: 253 MiB over idle for 100k items of about 2.5 KB of strings, 130 MiB under one arena (two runs each), and more for lists of numbers of the same text size, which decode larger (345 to 424 MiB, 184 to 195 MiB under one arena, at 100k items of 128 numbers, with or without a vector index); at 1M items of 128 numbers 552 MiB under the default arenas and 204 MiB under one arena, and 301 to 316 MiB under one arena for 20,000 items of 1,536 numbers (`15-backup-restore-bench-pr11.md`). A PostgreSQL vector index adds 27 to 53 MiB to the restore at 100k items (five pairs of runs with and without it). Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. `backup sync`, which is not a job, keeps of the configured root's rows (every status) each row's status, store id and table and the key of each AVAILABLE row of the configured store id, dropping the rest of each row as it is read, and holds the objects of the one prefix the listing is in with their sizes and the prefix of each manifest it listed; every backend's `list_backup_rows` collects its whole result before it yields the first row, so the read of the rows is sync's largest term on a catalog with many rows of the root (tombstones within their retention included): about 22.5 KiB of peak RSS per row of a table with five GSIs (1,452 bytes of `source_table_details_json`; 610 MiB at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none, and 148 MiB at 50,000 rows whose JSON is `{}`, so nearly all of it is the parsed description), measured on a SQLite debug build on a 16-core x86_64 Linux host under glibc's default arenas, the same peak `backup list` reaches over those rows; `backup sync --move-store` keeps the old root's rows without their parsed description and the configured root's (account, backup id) pairs, and reads one manifest and one prefix listing at a time, so its peak is that same read of the rows: 569 MiB at 20,000 rows of a table with five GSIs on a SQLite debug build and 500 MiB on a PostgreSQL debug build, against 51 and 49 MiB at 50 rows, 37 s each with filesystem roots on a local disk, on that host (revision item 36); the filesystem store builds its listing of the whole root in one `Vec` before it yields the first key (about 72 bytes plus the key per object, the store trait's doc), and the S3 store lists a page of 1,000 keys at a time. Measured on a filesystem store with 100 objects per prefix and 60-byte keys: 45 MiB of peak RSS for an empty root, 66 MiB for 100,000 objects and 98 MiB for 300,000, about 190 bytes per object, nearly all of it the store's listing (revision item 34). The report keeps an entry per prefix for the whole run (it lists every prefix the catalog already has a row for among the skipped ones), so at one object per prefix the per-prefix terms double that: about 350 bytes per prefix, 62 MiB of peak RSS for 50,000 such prefixes and 112 MiB for 200,000 against 45 MiB for an empty root (and about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build of PR 10 with a filesystem root and no rows. On an in-memory SQLite server that cost is paid in the serving process at every start. `backup migrate-legacy` runs the backup job's writer, so its terms are the backup's above, with the in-catalog read in place of the snapshot's (PostgreSQL one row at a time from a cursor in a `REPEATABLE READ` transaction held for the backup's whole copy, SQLite batches of at most 500 rows or 4 MiB, MongoDB a cursor batch under the 16 MiB reply bound, Cassandra pages of 8 rows of a 64-row bucket): 55 to 56 MiB of peak RSS for 1,000 and for 100,000 items of the job's 200-byte shape on PostgreSQL, MongoDB and Cassandra debug builds, and 55 and 122 MiB on SQLite, whose excess is its connections' page cache (`cache_size`), on the 16-core host above (revision item 37). A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. @@ -222,7 +222,7 @@ Which document and which row each PR changes; what each row says is in the ownin | `.github/workflows/matrix-nightly.yml` and `devtools/backup-restore-matrix` (the nightly's script) | PR 9b | | upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 (and PR 8 for the store-backed backups a restored catalog no longer lists) | | upgrade manual: CreateBackup without `[backup.store]`, `kind = "disabled"`, the SQLite default root, shared store roots and `--adopt-store` | PR 7 (and PR 10 for `--adopt-store`) | -| upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 (`migrate-legacy`: #821, item 34) | +| upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 (`migrate-legacy`: #821, items 34 and 37) | | upgrade manual: PR 14's migration refusal, `--force-drop-legacy-backups`, CreateBackup without a store after PR 14 | PR 14 | | upgrade manual: the #618 regression on the off-host path and the way out; the `[backup.store]` paragraph's asynchronous create and the restore of a store-backed backup deferred to PR 9a | PR 8 | | `docs/troubleshooting.md` DeleteTable entry and admin guide `control_plane_delay_seconds` row | #395 (in its diff) | @@ -471,10 +471,10 @@ Interfaces it adds: // the claim; a default of 0 for the backends without status entries; PR 10 (#820, revision item 36) ``` `backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; the store lists in key order, so sync holds one prefix's objects at a time and the prefix of each manifest it listed, beside what it keeps of the root's rows, whose read is its larger cost (section 7), the reserved `changelog` and `exports/` components skipped; a filesystem store builds its own listing of the root first, section 7), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. An ARN of another region is adopted and logged (requests resolve it by the ARN as written); one of another partition than `aws` is skipped, since every request refuses such an ARN before it reads a row (revision item 34). `--move-store ` re-points the rows of that root to the configured store. It needs the configured store healthy, refuses the configured root and a root with no rows, and its wait covers the jobs of every store id the catalog records for the old root as well as the configured store's (revision item 36). It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row whose owner is dead becomes AVAILABLE if a verified manifest is there, and otherwise stops the move with no status written, since a copy without the complete backup says nothing about the old root's prefix (revision item 36), and a DELETING row whose prefix there holds no verified manifest has its deletion finished in the configured store (a manifest that fails verification is a leftover the deletion removes). For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest and that its size and item total are the row's, and stops on a row whose copy is incomplete, naming it (every row that stops the move is named in one run, and none moves while any does; a row that would give an account two rows of one backup id stops it too); `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync), a FAILED row whose heartbeat is within `stale_job_after_secs` (its job may still write under the prefix) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then, after one re-read of the configured root's identity file (the deletion fence's `check_catalog`), it rewrites `store_kind`, `store_root` and `store_id` of the rows still of the old root and not CREATING, and lists the old root again, reporting a row still there; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts (the `backup_index` row PR 9b's matrix seeds by hand is the row this adoption writes); the previous owner acts on the root until a later check that can read its catalog finds the store unhealthy, at the earliest one reconciler interval after the cycle in progress ends, unless its deletion fence (section 4(c)) refuses a deletion or a CreateBackup first, so every server of the previous catalog is stopped before the adoption. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too once the previous process has exited (a new in-memory server is started only after the old one has exited; one started beside it adopts the root while the old one still acts on it until a later check or its fence); a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. -`migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). +`migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` (as implemented through `verify_prefix`, sync's check) and checks that the manifest's item count equals the legacy backup's; only then does it replace the legacy backup by its row, in one step per backend through `LegacyBackupReader::replace_with_index_row` rather than an insert and the reader's `delete` (revision item 37). It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4; that is every backup a 0.1.x release took), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). As implemented it also refuses a table name the store-prefix rules refuse, a backup name the manifest refuses, a MongoDB backup with no item collection, what the legacy restore refuses (a vector index, a multi-part base key) and an ARN a row of another store holds, and skips a CREATING or DELETING in-catalog backup (revision item 37). The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`; a command's take tries the exclusive lock a few times over about 10 ms on its connection, so a job's one-statement shared lock does not refuse it. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, and so is the holder of any maintenance claim row, which such a command takes over whatever its heartbeat says; before the wait it runs the reconciler's dead-owner resolutions itself, as the claim's holder (the cycle takes the held branch without reading the claim back through the zero window) (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 29 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. -State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; rows deleted outright by `--prune-missing` (AVAILABLE rows whose manifest is gone), `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE row by `migrate-legacy`. +State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; rows deleted outright by `--prune-missing` (AVAILABLE rows whose manifest is gone), `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE, unowned row by `migrate-legacy`, in one step per backend (revision item 37). Documentation rows it corrects: upgrade manual: on SQLite the backup CLI commands that write the catalog take the serve lock, so the server is stopped for `backup sync` and its options and for `purge` (section 4); legacy backups are carried forward with `backup migrate-legacy` (above), and one it refuses (an id with no instant, an item the validator refuses, a pre-0.0.4 SQLite definition) is carried forward by restoring it and taking a new backup, which has a new ARN; a store owned by another catalog is adopted only with `backup sync --adopt-store`. ### PR 11: CI and documentation @@ -494,8 +494,8 @@ ExtendDB's own ImportTable reads a single local file today; reading a store pref ### PR 14: legacy reader removal -- PR 14: legacy reader removal, one minor release after PR 6. Its migration refuses to run while any legacy backup row exists, naming the count and `extenddb backup migrate-legacy --all`, and `extenddb migrate --force-drop-legacy-backups` is the explicit opt-out. It then drops the legacy tables on PostgreSQL and SQLite; on Cassandra, whose three legacy tables also hold the new rows, it reads every row of the three tables and deletes by key those whose `store_kind` is null, then drops the per-account `backup_items`; it sweeps `_backup_` collections on MongoDB including orphans with no catalog document, and settles Cassandra's backup path (section 3). After PR 14, CreateBackup with no configured store returns a ValidationException naming `[backup.store]`. -The upgrade manual tells operators to migrate, restore or delete legacy backups before the release that removes the reader, whose migration refuses to run while any legacy backup row remains unless `--force-drop-legacy-backups` is given (PR 14). Invariants: section 3's Cassandra decision; the one-minor-release rule (decision 3). Documentation rows: upgrade manual: PR 14's migration refuses while any legacy backup row exists, naming the count and `backup migrate-legacy --all`, `extenddb migrate --force-drop-legacy-backups` is the opt-out, and after PR 14 CreateBackup without a store returns a ValidationException naming `[backup.store]`. +- PR 14: legacy reader removal, one minor release after PR 6. Its migration refuses to run while any legacy backup row that is not DELETED exists (every frozen `delete` and `migrate-legacy`'s replacement leave a DELETED row on PostgreSQL, SQLite and MongoDB, revision item 37), naming the count and `extenddb backup migrate-legacy --all`, and `extenddb migrate --force-drop-legacy-backups` is the explicit opt-out. It then drops the legacy tables on PostgreSQL and SQLite; on Cassandra, whose three legacy tables also hold the new rows, it reads every row of the three tables and deletes by key those whose `store_kind` is null, then drops the per-account `backup_items`; it sweeps `_backup_` collections on MongoDB including orphans with no catalog document, and settles Cassandra's backup path (section 3). After PR 14, CreateBackup with no configured store returns a ValidationException naming `[backup.store]`. +The upgrade manual tells operators to migrate, restore or delete legacy backups before the release that removes the reader, whose migration refuses to run while any legacy backup row that is not DELETED remains unless `--force-drop-legacy-backups` is given (PR 14). Invariants: section 3's Cassandra decision; the one-minor-release rule (decision 3). Documentation rows: upgrade manual: PR 14's migration refuses while any legacy backup row exists, naming the count and `backup migrate-legacy --all`, `extenddb migrate --force-drop-legacy-backups` is the opt-out, and after PR 14 CreateBackup without a store returns a ValidationException naming `[backup.store]`. - Live compatibility check, measured not asserted: import an ExtendDB backup's `data/` prefix into an Amazon DynamoDB table with ImportTable and compare item counts and a sampled item set; restore a real DynamoDB export into ExtendDB. Results recorded in the backup manual. @@ -608,3 +608,5 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 35. (2026-11-05, PR 11 as implemented, #747.) The CI filters: `--filter "not import_export"` came with the PostgreSQL integration job's first version (687e46ff) and was copied into the SQLite job (556c501e); neither commit gives a reason. A foreground server wrote a PID file in every tree up to e0d6a0af, which made foreground mode write none by default and added `--write-pid-file` to the job's start in the same commit; 556c501e came after e0d6a0af, and its SQLite job passes the flag. So the restart `devtools/run-tests` makes for the import and export tests could stop the job's server in every tree, and the filter was not working around it. The MongoDB and Cassandra jobs already run `tests/test_import_export.py` unfiltered (`--pytest --comprehensive --parallel`). The expression selects out the ten tests of that file and nothing else (none of the comprehensive suite, the CLI lifecycle list or the backup-store step); run the job's way on release builds of this tree they passed 10 of 10 on SQLite and on PostgreSQL, so both filters are dropped and no narrower one is kept. The open cells (testing summary) are measured in `15-backup-restore-bench-pr11.md`, in `-pr9.md`'s method, none left open; the Cassandra cells ran on a single 4.1 node of their own with the Cassandra integration workflow's service settings (`PasswordAuthenticator`, a 1 GiB heap), since a node running `AllowAllAuthenticator` cannot hold the role `extenddb init` creates. Later runs on the same host with other harnesses, on release builds of this change's sources (04c29155, an earlier version of this commit with the same Rust sources, and 30553b38, whose Rust sources differ only in the help text of two `init` options), reproduce the cells in kind and are in the record, with a control of items of 2.5 KB strings run on this commit's sources. Findings beyond the figures: SQLite's vector restore under glibc's default arenas is higher at 1M than at 100k (552 against 354 MiB over idle; the 1M run levelled off after about 150 s, and no other size was measured) and is 195 and 204 MiB under `MALLOC_ARENA_MAX=1` (flat within about 25 MiB over the measured runs), the allocator's term section 7 describes for large items; that memory is the restore's write batch (up to 10,000 items or 32 MiB of item text) and decode chunks, not the vector index, and grows with the item's size up to the batch bound whatever the attribute types (100k items of about 2.5 KB of strings held 253 MiB, 130 MiB under one arena), lists of numbers decoding larger at the same text size (345 to 424 MiB and 184 to 195 MiB at 100k items of 128 numbers, with or without the index; 20,000 items of 1,536 numbers 669 MiB and 301 to 316 MiB under one arena), while on PostgreSQL the index adds 27 to 53 MiB (five pairs of runs with and without it, each from one harness; a pair whose two runs came from different harnesses is left out); one arena cost the 1M restore 32% in duration and two 100k restores running at once 131% (57.5 against 24.9 s); the vector index is about 20% of the 100k restore's duration on SQLite and 33 to 47% on PostgreSQL (the same pairs); MongoDB's restore becomes `ACTIVE` at the backfill worker's next cycle after the load (`GSI_BACKFILL_INTERVAL`, 5 s, which the restore's end does not wake), so its duration is the load's plus up to 5 s, 3.9 to 8.9 s at 100k over five runs; the peaks of one cell varied by up to about 20% between runs; and a whole-table `SearchVectors` on SQLite is the exact scan of `crates/storage-sqlite/src/vector_search.rs` (5.5 to 15.8 s over 1M 128-dimension vectors in seven runs, 0.24 s on PostgreSQL). `docs/manuals/14-backup-and-restore.md` is the operator's account derived from this document, which remains the record (section 10's first row): the strategy, RPO and RTO per store kind with the three records' figures and their host class, the configuration (pointing at the admin guide's rows and the store crate), securing the store, what a running backup costs each database, the restore drill, catalog-loss recovery with `backup sync`, `--adopt-store` and the in-memory startup sync, legacy backups (with `migrate-legacy`, #821, not shipped, the restore-and-back-up-again path), what a restore preserves, and ImportTable and export interchange with Amazon DynamoDB (a store-prefix import is #748). Each PR 11 acceptance statement is in it. Figures measured for it on the same host class: a 25 s PostgreSQL backup beside 12,156 overwrites of 200-byte items in another table grew that table from 0.4 to 4.4 MiB, its dead row versions "not yet removable" by a `VACUUM` during the backup (0.5 MiB for the same writer with no backup); on SQLite the WAL grows by the pages each write changes, which depends on the indexes: a 22.8 s backup beside 15,472 overwrites of a table with no secondary index grew the `-wal` file from 4.0 to 61.1 MiB (about 3.9 KB per write), and 13,996 overwrites of a table with two GSIs during a 25.6 s backup from 4.1 to 1,294 MiB (82 to 97 KB per write over four runs, about 20 to 24 pages: an asynchronous GSI write goes through `gsi_pending` and is applied in later transactions); the same writers with no backup left it at 4.0 and 4.1 MiB, and the file keeps its grown size after the backup and after a stop. The record's "Figures measured for the manual" section carries these runs' method (the writer, the relation and file measured, the `VACUUM (VERBOSE)` 10 s in), build and load, with their reproductions on 04c29155. Its process-level statements were checked against release builds of this tree on SQLite and PostgreSQL: the root placement refusals and their texts and an absent root's, the drill, a re-created catalog's `owned by another catalog` health with `[backup.store]` configured and, on file-backed SQLite with the default root, the logged warning, no `backup_store` component and in-catalog `CreateBackup` until the adoption, the accounts re-created with `manage create-account --account-id` and the old key pairs kept with `manage import-access-key`, `--adopt-store` exiting 6 beside a SQLite server and adopting once it stopped, the in-memory startup sync line and, on the `sqlite-memory,dev-mode` build, the adopted backups answering `AccessDeniedException` until their account was re-created with its old id, the in-catalog create's warning and the restore of its backup through a store-configured server, an S3 store at moto with the default spool, `backup sync` exiting 6 beside a SQLite server, ImportTable refusing a gzip file and a `FileSource.Path` through a symbolic link and importing a decompressed backup data file, the export's `FilePath` and `S3Prefix`, and on PostgreSQL the claim session found by its lock key (`hashtextextended('backup:', RESTORE_LOCK_SEED)` in the catalog database, distinct from the `extenddb serve` registration session) whose termination stopped the job at its next heartbeat and left the row to the reconciler, which failed it once stale. Checked on release builds of 04c29155 (this change's Rust sources) as well: an off-host copy with loose modes refused by `backup sync` (exit 1, naming the directory) and its restore failing, the store unhealthy only when the copy is the newest backup; the registration session's termination failing every store-backed backup the server ran (reason `shutdown`); `--prune-missing` waiting on a backup that failed within `stale_job_after_secs`; a catalog database restored alone beside a newer data database (tables created since absent and empty when created again, tables deleted since listed and answering `ResourceNotFoundException`); both symbolic-link refusals of `FileSource.Path`; an on-demand table's GSIs reported with 0 and 0 throughput in `SourceTableFeatureDetails`; and the PostgreSQL pair dumped with the server stopped under `umask 077` (files 0600) and restored into new databases created with `createdb -O` by the admin role, after which `CreateTable` and `PutItem` of a new table work, while a restore over the existing databases restores nothing and one into databases the admin role owns breaks `CreateTable` ("permission denied for schema public"). On release builds of this commit's sources, on PostgreSQL 16 containers: the entry's procedure as it now stands (an older `pg_dump` refused, "aborting because of server version mismatch", exit 1, leaving an empty file, which `psql -v ON_ERROR_STOP=1` restores with exit 0, after which the server refuses the catalog as not initialized; both dumps exit 0 and non-empty; the old databases renamed aside, the restored server reading the table and its GSI and creating a table, then the old databases dropped), and on a new, empty server `createdb -O` refused until the application role was created with the dump's password, the restore, and the server, whose catalog still named the old server's address, connecting its data pool there: it put a new table's items into the old server's data database while that server ran, failed to start once it was gone, and served the restored table, with `CreateTable` and `PutItem` landing on the new server, once `data_database_connection_string` was set to the new address; and an access key imported through the management API from a request body read from a file. A catalog restored from a dump whose newest `AVAILABLE` backup was deleted after the dump keeps its store unhealthy, and this release has no command that clears that without a copy of the prefix (`--prune-missing` refuses while the store is unhealthy); the manual says to take the dump after deletions. The admin guide's Data Recovery entry named the PostgreSQL data database `extenddb_catalog_data`; `init` names it after the catalog without its `_catalog` suffix (`extenddb` by default) and records it as `data_database_name` (the connection string as `data_database_connection_string`), and the entry, the architecture, design, quickstart and security-model guides, `AGENTS.md`, the troubleshooting guide and the `extenddb` skill's runtime-symptoms reference (the `pg_stat_activity` query), the technical-debt register's A-1 entry, the storage-extension guide's database list and `init --help` now say so, with each backend's databases; the quickstart's `init` table gives `--pg-user` (the admin role, `$USER` then `postgres`), `--pg-pass`, `--extenddb-user` and `--extenddb-pass` with their defaults, and its sample lines show migration 7; a catalog name without the suffix needs `--data-db`, since `init` otherwise derives the catalog's own name, creates it once and fails on the second create, leaving an empty database (not changed here). On PostgreSQL the entry, the security model, the deployment guide's checklist and the manual treat the catalog and data databases as one copy (both dumps with the server stopped, or a base backup), with `umask 077` before the dumps, since the catalog dump holds the encryption key and the application role's password, and the restore goes into new, empty databases owned by the application role, with the old databases renamed aside until the restored server has read every table, the `pg_dump` of the server's major version and both dump files checked, and on a new PostgreSQL server the application role created first and, at another address, the recorded data-database address set; the entry's SQLite row gives the restore (the current `-wal` and `-shm` removed first), and the upgrade manual's dumps take `umask 077` and its rollback restores the pair; the manual's catalog-loss section says what a catalog older than the tables gives and that tables backed up since restore once `backup sync` lists their backups. ADR-0019 and the RFC section are superseded in part, not whole: CreateBackup on Cassandra still writes their in-catalog format, before it returns (section 3). ADR-0019 changes only its Status line, as `docs/adr/README.md` allows, to the "Superseded in part by [document]" form the README's process now names for a decision a later design replaces in part, and section 10's closing paragraph states what that Status covers, including that the reconciler now removes an interrupted in-catalog create; the RFC's section carries a note, and its section 4 and Known Gaps lines on synchronous operations and on restored indexes carry notes. Edited with this item: section 7's memory paragraph (the restore-memory and vector-index sentences), section 10's table row and closing paragraph, the PR 11 line and the testing summary's performance paragraph. 36. (2026-11-06, PR 10b as implemented, #820.) #820 ships `backup sync --move-store `, `--forget-store ` and `--forget-backup ` and plain sync's report of the rows of other stores, with three `BackupCatalog` methods on the four backends: `list_backup_rows_by_root(kind, root, status)` as the interfaces block had it; `rewrite_store(from_kind, from_root, keys, to)`, which rewrites only a row still of the old kind and root and not CREATING and counts the rows it rewrote, so a row another writer moved, or the CREATING row of a server still on the old root, is left rather than moved without its bytes (PostgreSQL one `UPDATE` over `unnest` of the keys in a transaction the client commits, under `SET LOCAL lock_timeout` 3 s and `statement_timeout` 30 s with up to three attempts on a lock timeout, so the statement of a command killed while it waits on a row lock commits nothing once the claim is gone; SQLite one `BEGIN IMMEDIATE` transaction, every key or none; MongoDB `update_many` in chunks of 1,000 keys that also `$inc` `lock_seq`, a convention: any write to the document, this one included, makes a transaction that writes it conflict and re-run, the re-run reads the row's new status, and a registration keeps the store its server is configured with; Cassandra per key; the trait doc says what a client's death in the call leaves on each); and `summarize_store_roots()`, the rows per store (kind, root, id) and status, which the design did not have: `list_backup_rows_by_root` takes a root, and nothing listed the roots (PostgreSQL and SQLite one `GROUP BY`, MongoDB a `$group`, Cassandra the status table's partitions with their entries counted and not re-read, which the report of other stores then checks against each store's listing, printing what no row stands behind as status entries). A partial rewrite (between MongoDB chunks, between Cassandra keys) leaves rows under both roots; every step is idempotent and the command exits 1 naming what is left. On Cassandra `rewrite_store` writes the new root's status entry before the row's lightweight transaction (`IF store_kind = ? AND store_root = ? AND backup_status = `) and deletes the old entry after it, the one exception to the module doc's row-first rule: a stop after the new entry leaves only an entry every reader skips, where the row first would leave a moved row that no listing of the configured store returns; an unknown outcome is settled by a `LOCAL_SERIAL` read, and `NotMatched` reads the row at `LOCAL_SERIAL` first, keeps the new entry when the row already names `to` (this call's write), and otherwise removes it and reads again, up to three attempts. `start_backup_delete`'s and `set_backup_deleted`'s lightweight transactions carry the store kind and root they read, so a DeleteBackup of a server on the old root that read the row before the rewrite does not apply, reads the row again and moves its status entry under the root the row now names: the moved row is never left without its own entry. The report of other stores reads `summarize_store_roots` and names each CREATING and DELETING row through `list_backup_rows` of that store's exact identity; a store of the configured root under another id counts as another store; it is printed after every sync that listed the root to its end, skipped when the run stopped or the listing failed, and a failed read of it exits 1. So 4(g)'s "rows for another store are reported" now holds for every such row, where item 34 had it for the prefixes whose backup id such a row holds (section 4, PR 10). `--move-store` needs the configured store healthy, refuses the configured root and a root with no rows (printing the stores the catalog holds rows for; a root matches as stored, byte for byte), takes the maintenance claim on every backend, SQLite included under its serve lock, and waits over the configured store and every store id the catalog records for the old root (`maintenance::wait_for_quiet` takes a slice of identities), since a CreateBackup of a server still on the old root has its row there. Each row of the old root is checked against the configured store as the PR 10 contract says, read thus: a CREATING row goes through the reconciler's per-row resolution (`resolve_dead_creating_row`, split out of `resolve_dead_creating_rows` so both callers share one body) with the configured store's calls and `OnIncomplete::Leave`: it is written AVAILABLE when the copy holds its complete backup, and any other probe result stops the move with no status written (the takeover's generation is the one change), since a copy without the complete backup proves nothing about the old root's prefix (a copy's order is arbitrary) and a FAILED row would have the configured store's reconciler delete the copy and a server still on the old root the original; the way out is a reconciler on the old root (a server, or on SQLite plain sync, configured with it), which resolves the row there, or `--forget-backup`; the PR 10 contract's sentence is edited with this item; a DELETING row whose prefix there holds no verified manifest is finished with `finish_backup_delete` behind the fence, and stays DELETED when the move then stops; an AVAILABLE row is probed by `probe_manifest` (now `pub(super)`) and its manifest's size and item total must be the row's, so another backup under the same ARN is reported, not moved; a row that would give an account two rows of one backup id (against the configured root's rows, and among the moved rows, which only Cassandra can hold) stops the move, checked before any row is, so such a row gets no resolution and no finished deletion (on Cassandra two spellings of an ARN share one prefix), and the configured root's rows are read before anything changes, so a failed read stops the move with nothing changed. Every stopping row is named in one pass and none moves while any does; with none, `StoreFence::check_catalog` re-reads the identity file once before `rewrite_store` (a refusal records the store unhealthy, as the fence does everywhere), the old root is listed again and a row still there is reported with exit 1 (a key counts as moved only when the configured store's listing then holds its row, and a row neither root lists, removed by another writer during the move, is reported as gone; a re-list that fails after a rewrite that counted every key takes every key as moved), and a plain sync of the configured root follows, in whose report of other stores a row of the old root (an insert that landed during that sync) is named and exits 1; a move that stops runs no sync. One `extenddb::audit` line is logged per row moved, resolved or finished (section 4, 5.2, PR 10). The removals of `--forget-store` and `--forget-backup` delete no object and read no store, so they run on a store in any state and with none configured (`kind = "disabled"`), under the claim on every backend, with the wait over the configured store when its identity is known and otherwise every store id the catalog records for the configured root (and the root's ids for `--forget-store`; with no store the wait covers the RUNNING restores). They open the store writing nothing (described as a read leaves it, so no default root is created and no `validate` probe file written; the identity file read, never created; when it agrees with the rows, `check_configured_store_read_only` runs once and is recorded), and on SQLite the command's own pass over what the stopped server left runs without the store's check (`ReconcilerState::for_maintenance_holder_without_store_check`). Each status is removed with a method that exists, no fourth one: AVAILABLE by `forget_available_backup_row` with its in-use rule, DELETED and FAILED by `remove_backup_row`, CREATING by `remove_dead_creating_backup_row` on the catalog's clock, DELETING by `set_backup_deleted` then `remove_backup_row` (no duty meets the DELETED state between them, by the order of the listings: one that listed rows before the claim listed this one DELETING, and one that lists it DELETED did so after the claim was taken and reads the claim held); a removal that meets another status reads the row once more and applies that status's removal. `remove_backup_row` takes the store kind and root the caller read and removes only a row still of them (Cassandra's lightweight transaction `IF backup_status = ? AND store_kind = ? AND store_root = ?`, `NotMatched` answering `false`), so a tombstone duty of a server on the old root does not remove a row the move rewrote after the duty read it. A DELETING row of the configured store is the exception to "delete no object": the forget finishes its deletion first, behind the fence (`finish_backup_delete`, the deletion its DeleteBackup asked for), which needs the store's check to have passed, and otherwise refuses the row; on SQLite the command's own pass may finish it first, and the removal of its DELETED row says so; a DELETING row of another store keeps its bytes, and a sync of that root after its return adopts the backup again. While the configured root has no identity file and the catalog holds rows for it (`FileMissingWithRows`: an unmounted volume), every row of that root is refused, since a server's check rewrites a missing file at a root with no rows and reports the store healthy. Both leave an in-use row, a CREATING row with a live owner and a FAILED row whose heartbeat is within the owner window, and exit 1. `--forget-backup` repeats (in order, stopping at the first ARN not removed; an ARN that no row, under either spelling, and no in-catalog backup carries is printed as having nothing to remove and counts as done, so a command line can be run again after a stop; every ARN's form is checked before the claim is taken), takes the ARN as the row carries it and names a row under the other spelling without removing it, reports an earlier release's in-catalog backup as one (exit 1, nothing changed), and then, when a store is configured, runs the store's check once and prints it, naming the next ARN whose manifest is missing; that check writes nothing (`check_configured_store_read_only`: the identity read without the create and without `validate`, the id compared with the one the command opened the store with, the manifest probe), and is not run when the root had no identity file as the command opened it. That is the remedy for a restored catalog whose newest AVAILABLE backup was deleted after the catalog's backup, which item 34 left with "put the prefix back"; `--prune-missing` keeps refusing on `ManifestMissing`, the host-local detector, and plain sync's refusal there names `--forget-backup` with the ARN. After its removals `--forget-store` deletes, under the claim, the root's status entries whose row is gone or names another store, each checked against its row at `LOCAL_SERIAL` (`remove_orphan_status_entries`; on Cassandra a deleter whose lightweight transaction lands before a rewrite of its row and whose entry batch lands after it leaves one under the old root, and so does a writer that died between its row and its batch), prints how many, and prints the store ids the catalog still records for the root with no row listed, without changing the exit (section 4, PR 10). The reconciler's `Duty::fenced` re-reads the row after the fence and skips one that is gone or whose status or store changed, then reads the maintenance claim again and starts nothing while it is held, two catalog reads per deletion; each FAILED-prefix and DELETED-prefix deletion and each tombstone removal that read the claim free must end by its deadline, `DUTY_DELETION_BOUND` (30 s, the S3 store's per-call bound, under the SDK's 60 s attempt timeout) after that read was sent: a read answered after the deadline starts nothing, the prefix deletion runs through the store's new `delete_prefix_before(prefix, deadline) -> PrefixDeletion { deleted, finished }` (the filesystem walk checks the deadline beside its cancel flag before each removal, the S3 loop before each `ListObjectsV2`, `DeleteObjects`, `ListMultipartUploads` and `AbortMultipartUpload`; the trait's default checks once, for a one-step store), and the call is also dropped at the deadline; past it the duty stops for the cycle, the row left for the next. Beside servers the forget options wait `FORGET_WAIT`, the bound plus one 30 s store call, from the claim before their first FAILED or DELETED removal and print the wait; on SQLite, under the serve lock, they do not wait. So what a deletion duty can still do to the bytes of a row a forget removes (4(a)) is the step in flight at its deadline, one removal on a filesystem root or one S3 request (up to 1,000 keys); the bound, like every lease in the design, assumes the server's process keeps running, and one paused past the wait between its last check and a call already sent (SIGSTOP, a frozen cgroup, a VM pause) runs that call when it resumes. A marker written per deletion, which the forget's removal would be conditioned on, is the full remedy, and is not taken: it adds a catalog write to every deletion for a case outside every other lease. The store-backed CreateBackup reads the claim again immediately before `claim_backup`, and the store-backed restore immediately before its registration as well as after it, so a claim taken before that read refuses the job with no row or record (`LimitExceededException`); what remains is the window from that read to the insert, however long the insert takes: a CreateBackup of a server still on the old root whose insert lands during the move's wait is listed by it, one that lands before the rewrite is left CREATING under the old root and reported by the re-list (the rewrite skips CREATING rows), one that lands during the move's sync of the configured root is named by that sync's closing report of other stores if it lands before it (exit 1), and one that lands after only by a later plain sync; the backup manual says to check plain sync's report of other stores before deleting the old root. `OTHER_STORE` now names the move ("or copy its tree under the configured root and move the backup's rows to the configured store with `extenddb backup sync --move-store `, and restore again"), as 5.3 step 7 says, and the reconciler's module doc with it. The three options conflict with each other, with `--adopt-store` and with `--prune-missing`. Memory: the move keeps the old root's rows without their parsed description, and its peak is the backends' whole read of those rows, 569 MiB at 20,000 rows of a five-GSI table on a SQLite debug build and 500 MiB on a PostgreSQL one, about 27 and 23 KiB per row above about 50 MiB (section 7). The PR 10 section's split line, interfaces block and sync contract, and section 7, are edited with this item; 4(c), 4(f), 4(g), 5.3 step 7, PR 8's claim-held paragraph and the PR 10 acceptance and transition lines state what was built and stay as written. The backup manual's procedure copies the old tree without its `extenddb-store.json` (the restarted servers gave the new root its own; a copied file turns every running server's store unhealthy until a restart, and, when a backup was taken under the new root meanwhile, needs that backup's row forgotten before the move, whose sync adopts it again). Not in #820: `migrate-legacy` (#821), a sweep of Cassandra's orphan status entries beyond `--forget-store`'s root, and any detection of a server left configured with the old root, which the backup manual's procedure reconfigures first. + +37. (2026-11-06, PR 10c as implemented, #821.) #821 ships `extenddb backup migrate-legacy |--all` (exactly one, a clap group) on the four backends, written in the engine's `backup/migrate_legacy.rs` against `BackupCatalog + LegacyBackupReader` and the configured store, so it runs on `Fake` and `MemStore`. The format gains `SnapshotDescriptor::KIND_LEGACY_IN_CATALOG` (`legacy_in_catalog`, the marker empty) with no format version change and no fixture byte changed; for that kind `source_backend` and `source_extenddb_version` name the binary that wrote the copy, since the in-catalog creator's version was never recorded. The manifest's `source_table` is, by one rule on every backend, the table the legacy restore creates (`source_table_from_legacy`, beside `restore_input`, pinned by an exhaustive property against `restore_input` and `restore_input_from_manifest` over the definition shapes): `restore_input` with no overrides (a backup with no definition keeps its keys, its key attribute definitions and the 5/5 fallback), plus the definition's table class, TTL and stream and the SSE description built from the specification in the backup ARN's region; the vector state is `[]` on PostgreSQL and SQLite with no names recorded (PostgreSQL with no definition and a NULL or zero snapshot too), `null` on MongoDB and Cassandra, and refused for a SQLite backup with no definition. So a specification with its own `KMSMasterKeyId` restores, after the migration, as `KMS` without the key id (item 33's general difference; the backup manual names it). `LegacyBackupReader` gains three members (section 3): `items(account, arn)`, the backup's name, recorded source, vector state and a `'static` item stream, each a frozen copy of its restore's read (PostgreSQL a `REPEATABLE READ READ ONLY` catalog transaction whose first statement re-checks AVAILABLE, owned by a spawned task that sends through a one-slot channel, held for the backup's whole copy as the restore holds it; SQLite the restore's keyset batches, each in its own read transaction re-checking AVAILABLE; MongoDB a cursor on `_backup_` with `batch_size` `DEFAULT_FETCH_BATCH` and a frozen copy of `document_to_item`, `Unsupported` for a document with no `backup_id`; Cassandra the buckets in order, paged by `item_index` 8 rows at a time, `LEGACY_ITEMS_PAGE`, where the restore's copy reads a whole bucket of 64); `list_all()`, the in-catalog backups of every account that are not DELETED, by creation time then ARN (Cassandra a range read filtered to `store_kind` null); and `replace_with_index_row(row) -> Replaced { Replaced, AlreadyReplaced, NotAvailable(String), InUse(String) }`. The replacement takes the place of the PR 10 paragraph's "insert the row and delete the legacy rows through the reader's `delete`": DeleteBackup does not read the maintenance claim, so in that order a DeleteBackup between the AVAILABLE check and the insert was answered DELETED for a backup that then came back as the migrated row, a restore target in CREATING made the `delete` refuse after the insert and ListBackups list the backup twice, and on Cassandra, where the legacy row and the index row share the `backups_by_arn` key, the frozen `delete` would remove the migrated row's list rows and leave a DELETED row that every later insert answers `false` for. Per backend: PostgreSQL one catalog transaction (the `backups` row `FOR UPDATE`, the CREATING-target check, the insert statement of `insert_backup_row` factored out, the items and definition deleted, DELETED); SQLite the same under the write lock and `BEGIN IMMEDIATE`; MongoDB one session transaction (`$inc lock_seq` while AVAILABLE, the CREATING-target check, the upsert, DELETING with `migrated_to_store: true`) and then the frozen drop of the collection and DELETED, a DELETING document with an index row of the same ARN and store being finished as `AlreadyReplaced`; Cassandra the row converted in place by one lightweight transaction `IF backup_status = 'AVAILABLE' AND store_kind = null AND active_restores = null` (after `creating_restore_target`, which clears stale registrations), then the index entries (`write_index_entries`, sent once more on a failure) and the buckets deleted; an unknown outcome is settled by a serial read of the index row, and `NotMatched` is read again, up to three attempts, and answered from what it finds. An index row of the account, ARN and store is `AlreadyReplaced` (what is left of the in-catalog backup is removed), one of the id under another ARN or store `AlreadyExists` with nothing written. The row is written unowned (`owner_generation` 0) with `completed_at` and `heartbeat_at` on the database's clock, as `insert_backup_row` writes one (4(e)). Three frozen paths change, each to order a DeleteBackup wholly before or after a replacement: PostgreSQL's legacy `delete` reads the status under its row lock and answers not found for anything but AVAILABLE; MongoDB's reads a DELETING document marked `migrated_to_store` as not found, in `delete` and in the lock's miss; Cassandra's DELETING claim carries `AND store_kind = null`. SQLite's is unchanged: the command holds the serve lock. Every frozen `delete` and the replacement leave a DELETED in-catalog row on PostgreSQL, SQLite and MongoDB and no items, definition or collection, and on Cassandra no in-catalog row and no buckets, so PR 14's refusal counts the rows that are not DELETED, and on Cassandra a migrated row carries `store_kind` and is kept (PR 14). The command takes the maintenance claim on every backend, SQLite included under the serve lock, for its whole run (4(f)), and waits as `purge` does over the configured store's identity, after `resolve_dead_owners` and with a zero window on SQLite (the PR 10 contract's SQLite rule); it needs the configured store healthy and its identity known. The backup job's writer runs under a `ClaimWatch` (`confirms_the_claim`, the loss token, the reason), the job's `HeartbeatHandle` or the maintenance claim's `HeldClaim`, whose pre-put check is that no heartbeat has reported the loss (purge's rule, with no "confirmed within two periods" clause) and whose timer cancels a `lost_signal` token when it records the loss, so a loss stops the copy at the writer's next await; the job's tests run unchanged. `--all` reads `list_all`; `` is matched among its keys as written and rebuilt in this server's region, in the ARN's account, and an `` no in-catalog backup has but a catalog row of the configured store holds is "already in the store" (exit 0) after a replacement call with that row, which on Cassandra finishes a conversion that stopped after its lightweight transaction (`--all` does not list such a row, an index row; the backup manual says to finish it with ``). Per backup, in this order: the id (no instant, every 0.1.x backup, refused before anything is read); the table name by the store-prefix rules and the partition; a catalog row under the ARN (the configured store's: finished; another store's: refused), before the status, so MongoDB's DELETING `migrated_to_store` document is finished, not skipped; the status (CREATING and DELETING skipped, named); the open (`Unsupported` refused, "the legacy restore cannot read it either"); the name (`is_valid_backup_name`, now exported from core); what `ensure_restorable` refuses ("the legacy restore refuses it too"); the unknown vector state. Then the copy: objects under the prefix with no row are an earlier run's and are deleted through the deletion fence first, always rewritten, one rule; the fence also runs before the first put (a refusal stops the run, 4(c)) and before each deletion of a copy (a refusal is a note; the next run deletes it). Item order `unordered`. The count is checked twice, the items written against the in-catalog count after the data files and the manifest's total against it after the re-read through `verify_prefix` (the sidecar, the digest, `from_json_bytes`, every data file at its listed size: sync's check, 4(g)); the manifest's `source_table.item_count` is the recorded count and `table_size_bytes` the sum of the copied items' sizes, so after a migration `BackupSizeBytes` is the store's bytes and the source table's size that sum, either of which can differ from the in-catalog backup's; the name, ARN, id, creation times and item count do not change. Outcomes: `migrated`, `finished`, already in the store, `refused` (reason and way out; `InUse` names `backup remove-target`), `skipped` (CREATING, DELETING, deleted or no longer AVAILABLE at the replacement) and `failed` (a store call, a stream error, a copy that does not read back); a refusal or failure after the first put deletes the copy through the fence. Exit 0 only when every backup reached is in the store and the run reached its end, so with `--all` a skipped backup makes it exit 1 (on Cassandra a create in flight does). Global stops, exit 1 at once with the claim released: the listing, a catalog call, a lost claim, a signal, the fence before a put, the writer's unhealthy store, and a replacement whose call fails (its outcome unknown: the copy is left, and the next run finishes it if the row landed, else deletes and rewrites it). A signal is acted on before each backup and between data files through the writer's shutdown token, the copy deleted; once the manifest is written the replacement is not interrupted. One `extenddb::audit` line per migration and per finish. `backup sync` gains `SkipReason::InCatalog { backup_arn, status }` ("an in-catalog backup (STATUS) holds this ARN; run extenddb backup migrate-legacy "), not transient, from `LegacyBackupReader::describe` under both spellings before the row read under them, so `sync` and `startup_sync` now need `LegacyBackupReader` (every caller's storage has it); on Cassandra it answers before the insert's `false`. That makes 4(g)'s "only path from a prefix to an AVAILABLE row" two paths, both behind `verify_prefix`, and gives sync a skip 4(g) now lists. ListBackups drops an in-catalog summary whose ARN a listed (CREATING or AVAILABLE) catalog row has and keeps the row's, the one state that lists a backup twice being an older build's crash between the two writes. Under `backup-test-hooks` the writer's `after_data_file` and the command's `before_manifest` and `after_manifest` pause it as they pause a job. So item 34's "shares nothing with sync but the insert" does not hold: the command shares `verify_prefix`, sync's row, the claim and the wait, and not the insert. Measured: SIGKILL at each of the three points on SQLite and PostgreSQL left the in-catalog backup whole and no row, and the rerun deleted the copy and migrated it (on PostgreSQL the killed command's advisory lock ended with its session, so the rerun took the claim at once); peak RSS 55 to 56 MiB for 1,000 and for 100,000 items on PostgreSQL, MongoDB and Cassandra debug builds and 122 MiB for 100,000 on SQLite, its page cache (section 7). The backup manual's legacy section now describes the command, where item 35 had it "not shipped". Sections 3, 4(c), 4(f), 4(g) and 7, the PR 10 paragraph and transitions, the PR 14 lines and section 10's upgrade-manual row are edited with this item. Not in #821: PR 14's removal of the reader (#749); carrying a backup with a 13-digit id into the store format, refused by design since the format's ids carry their instant; a Cassandra wire leg (its direct tests cover the stream, the listing, the conversion, its lost answer and its stop after the row). diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 49e3b96a5..b65fb8a9d 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -124,10 +124,10 @@ Behaviour stated as measured, and every number below, was observed with release | Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. For a store-backed restore the registration takes the same lock on the `backup_index` row instead of the legacy `backups` row (PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB a `$inc lock_seq` on the `backup_index` document, Cassandra the `active_restores` lightweight transaction on the backup's partition), the same ordering guarantee, so a legacy restore and a store-backed restore of two different backups never contend and a `DeleteBackup` of a store-backed backup meets only store-backed registrations. Two concurrent registrations of one backup leave one restore target on Cassandra as on MongoDB: the registration's lightweight transaction (`IF backup_status = 'AVAILABLE' AND active_restores = null`) is linearized on the backup's partition, so the second finds `active_restores` non-null and is `BackupInUseException`; the in-use pre-read is advisory *(storage-level tests on both)*. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | | Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. A store-backed restore (one run from a `[backup.store]`) is instead resumed, not discarded: its owner's death leaves a RUNNING restore job record with a stale heartbeat and the target `CREATING`, and the reconciler of any instance with the configured store takes the record over under a fresh owner generation once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), stamping a fresh heartbeat in the same write so no second instance takes it too, and resumes the load from its recorded progress (per data file on PostgreSQL, per batch on SQLite, MongoDB and Cassandra), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; the target reaches `ACTIVE` with the backup's full item count, its secondary indexes complete: a resumed Cassandra restore re-applies its last batch as idempotent upserts that rewrite the same base and index rows (the write path rewrites an unchanged item's GSI and LSI rows rather than deleting them, #424). A resume that finds no recorded progress and a target holding more items than the backup's total is `FAILED` with reason `target_not_empty` and the target is left in place: the rule protects a catalog restored from a dump that names a restore which finished and was written to since. A target holding no more than the total with no recorded progress is a first data file applied in part: on Cassandra, which records a batch's progress after its writes, an ordinary crash before the first progress record (the first 10,000 items or 32 MiB) leaves written items and no progress, and the resume re-applies the first file with idempotent upserts and reaches `ACTIVE` with the exact count *(from the code; the storage and engine cells)*. MongoDB records each batch's documents and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to `ACTIVE`; only a crash before the first batch commits leaves an empty target, which resumes from the start. A record that one instance's reconciler has taken over five times without the restore ending (a resume whose final count or progress read fails every time, or that stops with the record RUNNING) is failed on the next takeover with reason `owner dead: taken over 5 times by one instance without the restore ending` and its target removed, so a record that can never finish does not hold its name and its backup for as long as the server runs; the count is per instance and a restart starts it over *(from the code)*. The target stays `CREATING` with a `FAILED` record: `DeleteTable` on it is `ResourceInUseException`, `DeleteBackup` of its backup and a second restore of that backup are `BackupInUseException`, and the reconciler does not remove it; `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record, after which `DeleteBackup` and a new restore into the name succeed. A legacy (in-catalog) restore is still failed and its target removed. Measured *(binary-level, this release)*: SIGTERM during a store-backed restore left the record RUNNING and the restarted server resumed it to `ACTIVE` with every item (on PostgreSQL after the 45 s stale window, on SQLite at once); a SIGKILL at each restore kill point left the target to finish with the exact count (the base table's and the GSI's) or, for a failing restore, gone, except at the first reach of the Cassandra point that pauses before a batch's first progress record: on Cassandra (killed after the first batch's 10,000 upserts, before their record) the resume of that build answered `target_not_empty` and left the target `CREATING`, and `extenddb backup remove-target` then removed it, after which `DescribeTable` was `ResourceNotFoundException` and `DeleteBackup` succeeded; this release resumes that case with upserts to `ACTIVE` *(from the code; the nightly kill-point job runs it)*. On MongoDB a SIGKILL after a file's `insert_many` leaves the batch's items and its progress committed together, so the resume continues to `ACTIVE` rather than `target_not_empty`. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | | Point-in-time recovery | 35-day window: `UpdateContinuousBackups` enables continuous backups, and `RestoreTableToPointInTime` restores to any second in the window | Not supported; use on-demand backups (`CreateBackup` and `RestoreTableFromBackup`) instead. `DescribeContinuousBackups` reports `ContinuousBackupsStatus: ENABLED` (the service reports this unconditionally) with `PointInTimeRecoveryStatus: DISABLED` and no restorable-time window. `UpdateContinuousBackups` with `PointInTimeRecoveryEnabled: true` returns `ContinuousBackupsUnavailableException`; with `false` it returns the same description. `RestoreTableToPointInTime` checks the source table, then returns `PointInTimeRecoveryUnavailableException` ("Point in time recovery is not enabled for table ''"), the exception the service models on that operation. A table that does not exist is `TableNotFoundException` ("Table not found: ") on all three operations, as on the service. Same on all four backends | -| Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
//`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (`extenddb backup sync` gives a new catalog a row for each backup under the root, and `--adopt-store` first makes it the owner of a root another catalog owned; a root copied to a new place keeps its backups' ARNs once `--move-store` re-points their rows). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | +| Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
//`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (`extenddb backup sync` gives a new catalog a row for each backup under the root, and `--adopt-store` first makes it the owner of a root another catalog owned; a root copied to a new place keeps its backups' ARNs once `--move-store` re-points their rows). `extenddb backup migrate-legacy ` or `--all` copies in-catalog backups into the store under their own ARNs, refusing the ones a 0.1.x release took (their backup ids carry no creation instant). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` and `TableSizeBytes` in `SourceTableDetails` are the table's figures as last counted by the stats pass while `CREATING`, which is 0 and 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact figures once `AVAILABLE`: the items written and the sum of their DynamoDB sizes; the service reported 0 and 0 in every fresh-table backup measured). `CREATING` lasts the whole upload (the durations below), and while it lasts `DeleteBackup` and `RestoreTableFromBackup` of the backup answer `BackupInUseException` "Backup is being created: " on all three backends; the 16 service backups measured were `AVAILABLE` 12 to 90 ms after `CreateBackup` answered, so a client that deletes or restores a backup right after creating it polls `DescribeBackup` first here. The backup is the table as it stood when its snapshot opened, which is inside the request but after the request's checks (the table's state, the store's health, the maintenance claim) and after any wait for a job permit, up to 30 s (below): every write acknowledged before the open is in the backup and none acknowledged after it, so a `PutItem` acknowledged while the request waited for a permit is in the backup (measured: a write acknowledged 1.01 s after a `CreateBackup` that waited for a permit was in it, 2 of 2 runs), and `BackupCreationDateTime` is the open's millisecond. Measured on the service: a restored backup contained every write acknowledged before `CreateBackup` was sent and the 18 writes acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed, so the service takes the table at an instant inside the request as well. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). A `CreateBackup` still waiting in its snapshot open or row insert when the server stops writes no row; it is answered `ServiceUnavailable` "The server is shutting down. Retry the request." only after a registration loss, where the jobs' token is cancelled while the listener still serves, since on SIGTERM the listener's 5 s connection drain closes the connection before that token is cancelled (measured: the connection closed 8.2 s after the request was sent). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). Without a store, and on Cassandra, a table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s): the in-catalog format writes no store prefix. With a store, a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it with `ValidationException` before it opens a snapshot or writes a row ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"). Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | | `DeleteBackup` response | The full `DescribeBackup` description with `BackupStatus` `DELETED`, including `SourceTableDetails.ProvisionedThroughput` (a required member) and `SourceTableFeatureDetails` (measured) | The same on all four backends: the engine reads the backup's recorded definition before the delete and fills the description as `DescribeBackup` does, with `BackupStatus` `DELETED`. A backup whose definition this build cannot read or that does not parse is still deleted and described without feature details (logged at warn), which means a damaged `PROVISIONED` backup is described with on-demand `ProvisionedThroughput` `{0, 0}`: ExtendDB's choice, deletability over the throughput member of a row that is being removed, where `DescribeBackup` on the same row is an `InternalServerError`; a failure to reach the database fails the request and deletes nothing. | -| A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | +| A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` (or another backup maintenance command: `backup sync`, `backup migrate-legacy` or `backup remove-target`) holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | | A store-backed backup while the configured store is unhealthy | Not applicable: managed storage | `CreateBackup` on PostgreSQL, SQLite and MongoDB with a configured `[backup.store]`, and `DeleteBackup` and `RestoreTableFromBackup` of a backup in that store, while the store is unhealthy or its first check is pending, answer HTTP 400 `ValidationException` "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", the kind (`filesystem`, `s3`) and the category in parentheses being the store's (`/health` reports the same category); before the first check of a store checked after the listener: "The s3 backup store configured on this server has not passed a check yet (its first check is pending); the operation is refused until it does". `DescribeBackup` and `ListBackups` still describe such a backup. Once `DeleteBackup`'s DELETING write has committed the request answers the description with `BackupStatus` `DELETED` whatever the object deletion does; a deletion the store does not answer within its bound is left DELETING for the reconciler, logged, and the request still answers DELETED. `CreateBackup`'s refusal comes after the table's ACTIVE probe and the name checks and before the maintenance check and the job permit, so it takes no slot and writes no row; the in-catalog create (no store, Cassandra) touches no store and is not refused. | | `DescribeBackup` when the definition cannot be read | Not applicable | A definition row that does not parse, a renamed column, or a revoked privilege (`Internal` from the backend) is an `InternalServerError`, as it is for `RestoreTableFromBackup` on the same read; only a definition version this build does not know is described without `SourceTableFeatureDetails`. | | Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. That snapshot is the majority-committed one, which can trail `DescribeTable`'s `local` read by the majority-commit lag, so a CreateBackup sent as soon as `DescribeTable` shows a new table `ACTIVE` could find it missing; the open then waits, at most 2 s, until the snapshot shows the table, and answers `TableNotFoundException` only for a table a `local` read does not show `ACTIVE` either. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 72acb9444..dde7df1c2 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -173,7 +173,7 @@ Startup opens the configured store and, for a filesystem store, validates it and The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until `extenddb backup sync --adopt-store` makes the new catalog its owner (below), the file is removed or a new root configured (on the default SQLite root, that is the logged warning above for the server, which keeps CreateBackup on the in-catalog path, while `backup sync` handles that root as configured, so `--adopt-store` adopts it). A root that has to change (a new volume, a new bucket) keeps its backups' rows when its tree is copied under the new root and `extenddb backup sync --move-store ` re-points them (below). The default root, `backups` beside the database file, is shared by every database file in that directory: `--adopt-store` on it takes the root from whichever catalog wrote its file, so give each database its own `[backup.store]` when several share a directory. A server on an in-memory SQLite database writes the file with `ephemeral_catalog: true`; at its next start it adopts a root whose file names another catalog and carries that marker, and no other, so it never takes a durable catalog's root, and then syncs the root's backups into its new catalog (the `path` row above: until that sync's log line, the earlier backups answer `BackupNotFoundException`). A catalog whose root another catalog adopted (by `--adopt-store`, or a second in-memory server's start) keeps acting on the root until a later check that can read the catalog finds it owned by another catalog, at the earliest one reconciler interval after the cycle in progress ends (a check whose catalog read fails leaves the health as it was), with one exception, its deletion fence: right before it deletes a backup's objects (DeleteBackup, the reconciler's deletions, a failed backup job's cleanup, `purge`), removes a FAILED or DELETED row and writes a CreateBackup's first object, and also before DeleteBackup and `purge` change the row, a server or `purge` re-reads `extenddb-store.json`, and when the file names another catalog, names this catalog under another store id than the one the store's check settled (a root replaced by another root of this catalog), or the root has none, it deletes nothing and refuses the request; a file naming another catalog also marks the store unhealthy (`owned by another catalog`) at once, and one of another store id `identity mismatch`. DeleteBackup takes a file it cannot read, or a root with no file, at the read before the row is changed as it takes a store that does not answer: it writes DELETING and answers `DELETED`, the read before the deletion refuses it, and the row stays DELETING for the reconciler, whose deletion runs the fence again. `--prune-missing`'s removal of a row is not fenced: it deletes no object. A refusal at the read before the row is changed (the file names another catalog or another store id) leaves the row as it was, so DescribeBackup still answers AVAILABLE. One at the read before the deletion (an adoption landed between the two) leaves an AVAILABLE row DELETING with nothing deleted (DeleteBackup and DescribeBackup there answer `DELETED`), which this catalog's reconciler finishes only if this catalog owns the root again, and a dead CREATING row that `purge` already removed has its prefix kept. The read narrows the window to the gap between it and the deletion; on S3 that gap is a listing and a `DeleteObjects`, each under the SDK's retries, so seconds when the store throttles. A check that read the file before the adoption and ends after a fence's refusal records the store healthy again until the next cycle's check: each deletion stays fenced in that time, but a restore is accepted. A backup job of the previous owner that fails after the adoption leaves its partial prefix (no manifest, and no row of the new owner) under the root, which no duty or command of the new owner removes. A write that has already passed that read still lands, so stop every server of the catalog a file names before adopting its root. While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. -`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge`, and `sync` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup sync`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6). On a catalog `serve` would refuse they exit with `serve`'s code and message: 3 with migrations pending (with the `extenddb migrate --yes` advice), 4 on a catalog a later release migrated with a breaking migration, and 5 on one that needs attention (the upgrade manual's [catalog status](07-upgrade-manual.md#the-catalog-status) table). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge`, and `sync`, `migrate-legacy` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup sync`, `extenddb backup migrate-legacy`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6). On a catalog `serve` would refuse they exit with `serve`'s code and message: 3 with migrations pending (with the `extenddb migrate --yes` advice), 4 on a catalog a later release migrated with a breaking migration, and 5 on one that needs attention (the upgrade manual's [catalog status](07-upgrade-manual.md#the-catalog-status) table). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. `extenddb backup sync [--adopt-store] [--prune-missing]` (or with one of `--move-store `, `--forget-store ` and `--forget-backup `, below) rebuilds the catalog's rows from the configured store: after a lost or re-created catalog, or after a backup's prefix was copied under the root from another deployment or another backend. It reads the catalog's rows for the root (of every status and every store id they carry), then lists the whole root and handles each `/
//` prefix as the listing passes it, checking it against those rows before it reads anything under it. Sync holds one prefix's object list at a time and the prefix of each manifest it saw; a filesystem store builds its listing of the whole root in memory first (about 72 bytes plus the key per object), an S3 store lists a page at a time. The report keeps an entry per prefix for the whole run, and it lists every prefix the catalog already has a row for among the skipped ones, so on a root of small backups the per-prefix term dominates: about 350 bytes per prefix at one object per prefix (62 MiB of peak RSS at 50,000 such prefixes and 112 MiB at 200,000, against 45 MiB for an empty root; about 165 bytes per object at 100 objects per prefix), measured on a SQLite debug build with a filesystem root and no rows. Of the catalog's rows it keeps each row's status, store id and table, and the key of each AVAILABLE row of the configured store, but every backend reads the rows whole before sync sees the first, so the read is the larger cost and grows with the rows of the root, tombstones within their retention included: about 22.5 KiB per row of a table with five GSIs (1,452 bytes of source table description per row; 610 MiB of peak RSS at 25,000 rows and 1,144 MiB at 50,000, against 47 MiB with none; 148 MiB at 50,000 rows whose description is `{}`), measured on a SQLite debug build on a 16-core x86_64 Linux host with glibc's default arenas. `backup list` peaks the same over those rows. A prefix with no row gets an AVAILABLE row under the configured store's identity when its manifest and sidecar verify, every data file the manifest lists is under the prefix at its recorded size, and the manifest's ARN names the prefix's account, table and backup id; an ARN of another region is adopted and logged (requests find it by its ARN as written), and one of another partition than `aws` is skipped, since every request refuses such an ARN. The row carries the manifest's backup name, size, item count and source table, and the backup id's instant as its creation time. It never adopts a prefix with a row of any status (CREATING, DELETING, DELETED, FAILED or AVAILABLE), one whose manifest, sidecar or data files are missing or do not match, one whose manifest names another backup, or anything under `exports/` or a table's `changelog/`, and prints each with its reason. Before the insert it reads the row under both spellings of the ARN (this server's region and the manifest's), and the insert happens only if no row exists: a CreateBackup whose row landed after sync read the rows keeps its row, as does a row of another root, account or table holding the backup id, and sync changes no existing row. It also prints the AVAILABLE rows whose manifest the listing does not hold, and, once the listing has reached its end, the rows of the other stores (below). It exits 0 once it ran, skipped prefixes included, and 1 when a prefix was skipped because a store read or a catalog call failed (the run is incomplete; a second run adopts it), when the listing failed part-way (the backups adopted before the failure keep their rows), when a signal stopped it, when the rows of the other stores could not be read, or when `--prune-missing` left a row it may remove on a later run (a store or catalog call failed, its manifest's `head` included, or a restore is reading the backup: run the command again once the restore ends). A row `--prune-missing` leaves because its manifest is back or another writer moved it is reported, and the exit stays 0: @@ -217,6 +217,19 @@ Took the maintenance claim as "extenddb backup sync on (pid )". The configured store's check passes (store id ); each server finds it so at its next reconciler cycle. ``` +`extenddb backup migrate-legacy ` or `--all` (exactly one) copies in-catalog backups into the configured store under their own ARNs, names, backup ids and creation times, and replaces each by its catalog row, AVAILABLE and unowned as a row `backup sync` inserts (the backup manual's [Legacy backups](14-backup-and-restore.md#legacy-backups) section gives what it refuses and the way out for each). `` is matched as written and rebuilt in this server's region; `--all` takes every in-catalog backup of every account that is not deleted, oldest first. It needs the configured store healthy, opens the catalog as `serve` does (on SQLite under the serve lock: stop the server first; beside it the command exits 6), takes the maintenance claim on every backend for its whole run and waits as `purge` does, so CreateBackup and RestoreTableFromBackup answer `LimitExceededException` on every instance until it ends. Per backup it refuses an id with no creation instant (every backup a 0.1.x release took) before it reads anything; streams the items through the backup job's writer, each checked with the item validator under this instance's `[limits]`, into data files under the backup's prefix, then the DynamoDB manifests and `extenddb-manifest.json` (`snapshot.kind` `legacy_in_catalog`, no marker, `item_order` `unordered`, `source_backend` and `source_extenddb_version` this binary's); runs the deletion fence before its first put and before each deletion of a copy; reads the prefix back as sync does and checks the item count against the in-catalog backup's twice; and only then replaces the in-catalog backup with its row in one step. Objects under a backup's prefix with no row are an earlier run's and are deleted through the fence before the copy is written again. Each line is `migrated`, `finished` (an earlier run had written its row; what was left of the in-catalog backup is removed), already in the store, `refused` (with its reason and way out; the in-catalog backup is kept), `skipped` (CREATING, DELETING, or deleted meanwhile) or `failed` (a later run tries again); each migration is logged under `extenddb::audit` with the account and the ARN. `--all` goes on past a backup it does not migrate and exits 1 at the end with the counts; a lost claim, a store the writer finds unhealthy, a failed listing or catalog call and a signal (acted on between backups and between data files, the copy in progress deleted) stop it at once, exit 1, the claim released. It exits 0 when every backup it reached is in the store. While an in-catalog backup holds an ARN, `backup sync` skips a prefix under that ARN, naming the command. Its peak memory was 55 to 56 MiB for 1,000 and for 100,000 items on PostgreSQL, MongoDB and Cassandra debug builds and 122 MiB for 100,000 on SQLite (the page cache of its connections), on the 16-core host above: + +``` +=== extenddb backup migrate-legacy === +Store: filesystem (store id ) +Took the maintenance claim as "extenddb backup migrate-legacy on (pid )": until the command ends every backup and restore job of this catalog answers LimitExceededException. +migrated (account ): 303 items, 3158 bytes +refused (account ): 3 item(s) cannot be restored; first: key {"pk":{"S":"a"},"sk":{"N":"1"}}, attribute amount: Number overflow. Attempting to store a number with magnitude larger than supported range (#618); restore it (RestoreTableFromBackup), correct the items the log names, and take a new backup of the restored table, which has a new ARN. Its in-catalog backup is kept + +=== extenddb backup migrate-legacy complete: 1 migrated, 0 finished, 1 refused, 0 skipped, 0 failed === +Error: 1 backup(s) were not migrated; see above +``` + Every `extenddb backup` command refuses an in-memory SQLite database (`[storage.sqlite] path = ":memory:"`, or no path on a `sqlite-memory` build) before it opens the catalog, exit 1: it would open a new, empty catalog that ends with the command. A server on such a database syncs with its store at every start instead (the `path` row above). `extenddb backup remove-target --account
` removes a restore target that `DeleteTable` refuses (a `CREATING` table) together with its restore record: the target of a `FAILED` restore whatever the reason (`target_not_empty` included: the state a resume leaves when it finds no recorded progress and more items in the target than the backup holds, which a catalog restored from a dump can produce when it names a restore that finished and was written to since; an ordinary crash before the first progress record on Cassandra leaves no more items than the backup holds and is resumed, not failed; until the command runs a `target_not_empty` target keeps its backup in use, so `DeleteBackup` and a second restore of the backup are `BackupInUseException` and the name is held; `owner dead: taken over 5 times by one instance without the restore ending` is the reason of a record one server's reconciler resumed five times without the restore ending, whose target the reconciler removes itself), the target of a `RUNNING` restore whose owner is dead (no heartbeat within `stale_job_after_secs` on the database's clock, the listing the reconciler uses; on SQLite under the serve lock every owner is dead), and a legacy (in-catalog) restore's target the same way. It refuses a `RUNNING` restore with a live owner ("stop it or wait for it"), a `DONE` restore and a table no restore record names, the latter two as "not a restore target", and exits 1. It needs no store (it runs whether or not `[backup.store]` is configured, and creates neither the default root nor its identity file), takes the catalog-level maintenance claim first as `purge` does and releases it before it returns, logs the removal under `extenddb::audit` with the account, the table and the backup's ARN, and prints what it did: diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 4ad610031..dd5e7244a 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -6,9 +6,9 @@ This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 7, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)), and migration 7 adds the `restore_progress` table to the PostgreSQL data database (it does not apply on the other backends). -Configure `[backup.store]` before the release that stops creating in-catalog backups. The [Backup and Restore manual](14-backup-and-restore.md) covers choosing and securing a store, the recovery figures, the restore drill, catalog-loss recovery and what to do with in-catalog backups; this section says what the upgrade changes. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 7 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 7 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, and a catalog re-created with `init` finds the old root's `extenddb-store.json` naming the old catalog: the [Backup and Restore manual](14-backup-and-restore.md) gives both rules and the recovery with `extenddb backup sync --adopt-store`. +Configure `[backup.store]` before the release that stops creating in-catalog backups. The [Backup and Restore manual](14-backup-and-restore.md) covers choosing and securing a store, the recovery figures, the restore drill, catalog-loss recovery and what to do with in-catalog backups; this section says what the upgrade changes. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 7 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 7 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. In-catalog backups are carried into the configured store under their own ARNs with `extenddb backup migrate-legacy ` or `--all` (the Backup and Restore manual's Legacy backups section): on SQLite the server is stopped for it, and on every backend each backup and restore job answers `LimitExceededException` while it runs. A backup it refuses (a backup id with no creation instant, which is every backup a 0.1.x release took; an item the item check refuses; a SQLite backup taken before catalog 0.0.4) is carried forward by restoring it and taking a new backup of the restored table, which has a new ARN. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, and a catalog re-created with `init` finds the old root's `extenddb-store.json` naming the old catalog: the [Backup and Restore manual](14-backup-and-restore.md) gives both rules and the recovery with `extenddb backup sync --adopt-store`. -A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. +A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. `extenddb backup migrate-legacy` refuses these backups on their ids, which carry no creation instant, so a restore and a new backup is also their way into the store. The [general upgrade procedure](#general-upgrade-procedure) gives the sequence. The [version history](#version-history) records the 0.1.x catalog versions and what the 0.1.x binaries print, and [Behavior Changes by Release](#behavior-changes-by-release) what each release changes in behavior. @@ -49,7 +49,7 @@ A backup of a table that holds an `N` value the wire did not validate, or an att Migration 6 adds the catalog index of backups kept in a backup store, turns each restore registration into a restore job record, adds the maintenance claim record (`backup_maintenance`) and writes the catalog's random id (`settings.catalog_id`). On SQLite and PostgreSQL the index is the `backup_index` table and the job record is new columns of `table_restores`; on MongoDB the index is the `backup_index` collection and the job record a `restore_job` field of each restore target's `tables` document; on Cassandra the index is new columns of `backups_by_arn` and the `backup_index_by_status` table, and the job record new `restore_*` columns of `tables`. Stop every server built before this migration before running `extenddb migrate --yes`: such a server neither reads the index nor marks the restore registrations it writes, so a restore it ran during the upgrade would be owned by nobody. A restore in flight at that stop is left as a RUNNING job record whose owner is dead; the first reconciler pass of the new server (within `backup.reconcile_interval_secs`, default 60 s) marks it FAILED and removes its CREATING target, so the name is free again and the table has to be restored anew. On Cassandra the backfill reads `tables` with a filtered scan and writes each registration with a lightweight transaction, which takes seconds on a catalog of ordinary size. On Cassandra and MongoDB, which have no transaction around a body, every write of the migration is safe to send again: a `migrate --yes` that fails part way leaves migration 6 unrecorded, `serve` keeps refusing the catalog, and the next `migrate --yes` runs the body again and finishes. -A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). The pre-upgrade backup predates every backup this release wrote to the store, so a restored catalog has no `backup_index` row for them while the store keeps their bytes under its root; the previous binary serves neither (it does not read the store). After the upgrade is run again, `extenddb backup sync --adopt-store` gives them rows: the store's identity file names the pre-rollback catalog, so the re-upgraded catalog finds the store owned by another catalog until the command makes it the owner (on SQLite's default root the server instead logs a warning and leaves the store unconfigured, so `/health` has no `backup_store` component until then), and the command then adopts every backup under the root. On SQLite `backup sync`, with or without its options, takes the serve lock as `backup purge` does, so the server is stopped while it runs; on PostgreSQL, MongoDB and Cassandra it runs beside the servers under the maintenance claim. In-catalog backups taken after it are lost with the rows, as before. +A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). The pre-upgrade backup predates every backup this release wrote to the store, so a restored catalog has no `backup_index` row for them while the store keeps their bytes under its root; the previous binary serves neither (it does not read the store). After the upgrade is run again, `extenddb backup sync --adopt-store` gives them rows: the store's identity file names the pre-rollback catalog, so the re-upgraded catalog finds the store owned by another catalog until the command makes it the owner (on SQLite's default root the server instead logs a warning and leaves the store unconfigured, so `/health` has no `backup_store` component until then), and the command then adopts every backup under the root. On SQLite `backup sync`, with or without its options, takes the serve lock as `backup purge` does, so the server is stopped while it runs; on PostgreSQL, MongoDB and Cassandra it runs beside the servers under the maintenance claim. In-catalog backups taken after it are lost with the rows, as before. A backup that `migrate-legacy` carried into the store after the pre-upgrade backup is in-catalog again in the restored catalog, which the previous binary serves; once the upgrade is run again, sync skips its prefix and names `migrate-legacy`, which copies it again. ## Writing a New Migration diff --git a/docs/manuals/14-backup-and-restore.md b/docs/manuals/14-backup-and-restore.md index 62cc74d4b..96a41bdeb 100644 --- a/docs/manuals/14-backup-and-restore.md +++ b/docs/manuals/14-backup-and-restore.md @@ -25,8 +25,9 @@ PostgreSQL, SQLite and MongoDB. The catalog keeps a row per backup as the index Cassandra has no table snapshot. `CreateBackup` there writes the in-catalog format (inside the Cassandra cluster, in the catalog keyspace and the account's keyspace) whatever the store, and that copy pages the live table, so it is not point-in-time. A configured store serves restores into Cassandra of backups that another -backend wrote to the store. For off-host copies of a Cassandra deployment use the cluster's own snapshot -tooling. +backend wrote to the store, and `extenddb backup migrate-legacy` copies Cassandra's own backups into it +([Legacy backups](#legacy-backups)). For off-host copies of a Cassandra deployment use the cluster's own +snapshot tooling. What a table backup is not: @@ -505,7 +506,8 @@ id>); the catalog has no backup rows for it, so the store is unconfigured and Cr backups", and `/health` shows no `backup_store` component (`{"components":{},"status":"healthy"}`). Until the adoption below, `CreateBackup` answers `AVAILABLE` at once and writes an in-catalog backup into the database file, the failure domain this manual's first paragraph warns about; such a backup stays in-catalog -after the adoption ([Legacy backups](#legacy-backups)). Pause scheduled backups until the adoption, or +after the adoption until `extenddb backup migrate-legacy` copies it into the store ([Legacy +backups](#legacy-backups)). Pause scheduled backups until the adoption, or configure `[backup.store]` explicitly, which gives the unhealthy store above instead. In both shapes plain `backup sync` refuses (exit 1, "owned by another catalog"). Then: @@ -702,11 +704,76 @@ in-catalog format is deprecated: without a store `CreateBackup` logs a warning o in-catalog backup format, which a later release stops creating; configure [backup.store] ..."), and a later release stops creating it. -This release has no command that converts a legacy backup to the store format (`backup migrate-legacy` is -planned, #821). To move one into the store today, with the store configured: +### Migrating legacy backups into the store + +`extenddb backup migrate-legacy ` copies one legacy backup into the configured store, and `extenddb +backup migrate-legacy --all` every legacy backup of every account that is not deleted, oldest first. Give +exactly one of the two. Each backup keeps its ARN, backup id, name and `BackupCreationDateTime`: the items +are read through the in-catalog reader, checked one by one as a store-backed `CreateBackup` checks them, +written as data files under the backup's prefix, and the manifest is written last with `snapshot.kind` +`legacy_in_catalog`. The command then reads the prefix back as `backup sync` does (the sidecar, the digest, +the manifest, every data file at its listed size) and compares the item count with the one the legacy +backup recorded. Only then does it replace the legacy backup by the backup's catalog row, in one step: on +PostgreSQL and SQLite one transaction inserts the row, deletes the backup's items and definition and marks +the legacy row deleted; on MongoDB one transaction inserts the row and marks the document, after which the +item collection is dropped; on Cassandra the legacy row becomes the catalog row in place and the item +buckets are deleted. `DescribeBackup`, `ListBackups`, `RestoreTableFromBackup` and `DeleteBackup` then +serve it from the store, and `extenddb backup list` lists it. `` may be spelt with another region, as +`DescribeBackup` accepts it. + +What it costs. The command opens the catalog as `serve` does and takes the maintenance claim on every +backend for its whole run, so: + +- Every backup and restore job of the catalog answers `LimitExceededException` until the command ends, + on every server, and a second maintenance command (`backup purge --confirm`, the writing forms of `backup + sync`) fails at once. Before its first copy the command waits, as `backup purge` does, until no backup + or restore job of the store is running. +- On SQLite the server is stopped for it: beside a running server the command exits 6 and writes nothing. +- On PostgreSQL each backup's items are read in one `REPEATABLE READ` transaction on the catalog database, + held for as long as that backup's copy takes, which holds back vacuum of the catalog database meanwhile. + The legacy restore reads the same way. + +To keep the window short, migrate in pieces with ``, one backup at a time, between scheduled backups. + +The command prints one line per backup and a summary: + +``` +migrated arn:aws:dynamodb:us-east-1:123456789012:table/orders/backup/01793991641206-db64fc83 (account 123456789012): 303 items, 3158 bytes +refused arn:aws:dynamodb:us-east-1:123456789012:table/orders/backup/1793254932797-4116998c (account 123456789012): its backup id 1793254932797-4116998c carries no instant (an id a 0.1.x release issued), and a store prefix carries only the 14-digit shape; restore it (RestoreTableFromBackup) and take a new backup of the restored table, which has a new ARN. Its in-catalog backup is kept + +=== extenddb backup migrate-legacy complete: 1 migrated, 0 finished, 1 refused, 0 skipped, 0 failed === +``` + +Each migration is also logged under `extenddb::audit` with the account and the ARN. `finished` names a +backup an earlier run had copied and given its row, whose legacy rows the command now removed; `skipped` +names a legacy backup that is `CREATING` or `DELETING` (`DeleteBackup` again finishes a deletion that +stopped) or that was deleted while the command ran; `failed` names a store or catalog call that failed, +which a later run tries again. With `--all` the command goes on past a refused, skipped or failed backup and +exits 1 at the end; it exits 0 when every backup it reached is in the store. A lost claim, a store found +unhealthy, a failed catalog call or a signal stop it at once with exit 1; a first `SIGINT` or `SIGTERM` is +acted on between backups and between data files, and the copy in progress is deleted. + +What it refuses, keeping the legacy backup as it is and leaving nothing under its prefix: + +- **Every backup a 0.1.x release took.** Their backup ids (13 digits, or 13 bare digits) carry no creation + instant, and a store prefix carries only the 14-digit shape. Only legacy backups taken by this release + line migrate: those of a server without a store, with `kind = "disabled"`, and on Cassandra. +- A backup holding an item the item check refuses: a value the wire accepted before #618 was fixed (for + example an `N` of `1E126`), or an item over this instance's `max_item_size_bytes` or key limits. The + command logs each item's key and attribute, as a failed `CreateBackup` does. +- A SQLite backup taken before catalog 0.0.4, which recorded no table definition, so its vector indexes + cannot be stated. +- What the legacy restore refuses too (a vector index, a multi-part key); a table name that cannot name a + store prefix (only dots, or over 255 bytes); a backup name outside 3 to 255 characters of `a-z`, `A-Z`, + `0-9`, `_`, `-` and `.`; a MongoDB backup with no item collection; an ARN that a catalog row of another + store holds; and, at the replacement, a backup a restore target in `CREATING` still reads (`extenddb + backup remove-target` removes the target first). + +The way out for a refused backup, with the store configured: 1. Restore it under a new name: `aws dynamodb restore-table-from-backup --backup-arn - --target-table-name orders-moved --endpoint-url https://:18443`, and wait for `ACTIVE`. + --target-table-name orders-moved --endpoint-url https://:18443`, and wait for `ACTIVE`. For a + refused item, correct the items the command's log names in the restored table. 2. `aws dynamodb create-backup --table-name orders-moved --backup-name --endpoint-url https://:18443` and wait for `AVAILABLE`. This is a store-backed backup with a new ARN and a new `BackupCreationDateTime`, of a table named `orders-moved`. @@ -718,7 +785,35 @@ room loses every write since the legacy backup. While the table exists, `create- store-backed backup of that name with its current items; keep the legacy backup only for its older contents. A backup taken by an earlier release may restore without its secondary indexes (the upgrade manual's Current Status section); take a new backup of each table after an upgrade rather than -moving the old ones. A table with a vector index has no legacy backup: the in-catalog format refuses it. +moving the old ones. A table with a vector index has no legacy backup this release took (the in-catalog +format refuses it); an earlier release's is refused by the command, as the legacy restore refuses it. + +What a migrated backup reports. `BackupSizeBytes` becomes the store's bytes, and `SourceTableDetails`' +`TableSizeBytes` the sum of the copied items' sizes, so both can differ from what the legacy backup +reported; the ARN, the name, the backup id, the creation times and the item count do not change. The +source table recorded is the table the legacy restore would have created, with the table class, TTL and +stream settings it recorded. One setting does not carry over: an `SSESpecification` with its own +`KMSMasterKeyId` is recorded as `SSEType` `KMS` without the key id, so a table restored from the migrated +backup reports KMS encryption without that key id. + +A run that stops after it wrote some of a backup's objects and before its row (a crash, a lost claim, a +replacement whose answer did not come) leaves the legacy backup in place and the objects under its prefix. +`backup sync` skips such a prefix ("an in-catalog backup (AVAILABLE) holds this ARN; run extenddb backup +migrate-legacy "), and the next run deletes the objects and writes the copy again. A Cassandra +migration that stopped after the row was converted is finished by `migrate-legacy ` (`--all` does not +list it, since it is no longer a legacy row); until then the backup is served by `DescribeBackup`, +`ListBackups` and `RestoreTableFromBackup` but `extenddb backup list` does not show it. + +On Cassandra, `CreateBackup` writes the in-catalog format whether or not a store is configured, so a +Cassandra deployment with a store gains legacy backups every day: run `migrate-legacy --all` after the +backup schedule to keep its backups off the cluster, until a later release writes store-backed backups +there. + +Memory. The command's peak resident memory did not grow with the backup's size on PostgreSQL, MongoDB +and Cassandra: 55 to 56 MiB for 1,000 and for 100,000 items of the job's measurement shape (a debug build, +16 cores, local NVMe; 8.6 to 12.4 s for 100,000 items). On SQLite it rose from 55 to 122 MiB (7.6 s), the +page cache of the command's connections, which `cache_size` bounds at 32 MB each whatever the backup's +size. ## What a restore preserves diff --git a/tests/test_backup_job.py b/tests/test_backup_job.py index 0a64cbb95..79e2f714f 100644 --- a/tests/test_backup_job.py +++ b/tests/test_backup_job.py @@ -122,6 +122,25 @@ RUNNING restore exist proceeds once a server's held-claim cycle has finished the one and failed the other. +`backup migrate-legacy`, on the three deployments (the catalog reads on +SQLite and PostgreSQL): + +- two in-catalog backups (taken with `[backup.store] kind = "disabled"`) + of a table with a GSI and items of every type, the store then + configured: `` (spelt with another region) and `--all` copy each + into the store under its ARN, a manifest of kind `legacy_in_catalog`; + the in-catalog rows are left DELETED with no item or definition, a rerun + finds nothing, and the server lists each once, describes it as before, + restores it with the table's items and deletes it from the store; the + command needs exactly one of `` and `--all`; +- `--all` goes on past a backup holding items #618 let in and (SQLite, + PostgreSQL) one under a 13-digit id, refusing each with its reason and + keeping its in-catalog rows, and exits 1; the 13-digit one restores and + a new backup of the restored table is store-backed; +- (PostgreSQL) `migrate-legacy` fails at once while another maintenance + command holds the claim, and, holding it, waits for a live job and makes + `backup purge --confirm` fail at once. + The binary at `EXTENDDB_BINARY` decides which deployment runs: a SQLite build, a PostgreSQL build (with `EXTENDDB_TEST_PG_CONNECTION_STRING`) or a MongoDB build (with `EXTENDDB_TEST_MONGODB_URI`, a replica set of this suite's own: the @@ -1995,14 +2014,15 @@ def test_a_second_maintenance_command_fails_at_once(deployment, tmp_path): for a CREATING row whose owner beat just now (the row edited at the catalog, its heartbeat the database's now); while it waits, `backup purge --confirm`, `backup sync --adopt-store`, and `--move-store`, - `--forget-store` (of a root a second backup's row was edited to name) - and `--forget-backup` each fail within seconds, exit 1, naming the - one-at-a-time rule, and change nothing. The + `--forget-store` (of a root a second backup's row was edited to name), + `--forget-backup` and `backup migrate-legacy --all` each fail within + seconds, exit 1, naming the one-at-a-time rule, and change nothing. The first command, signalled, stops before any change and releases the claim. `--adopt-store` on the root this catalog owns then takes the claim and waits for the same live job ("Waiting for running jobs to - finish"), and a signal stops it before any change; after that a plain - sync and `--prune-missing` run. SQLite has + finish"), and a signal stops it before any change; so does `backup + migrate-legacy --all`, beside which `backup purge --confirm` fails at + once; after that a plain sync and `--prune-missing` run. SQLite has no second command to refuse: every writing command takes the serve lock first, which the sync cell shows.""" _sync_cell(deployment) @@ -2045,6 +2065,7 @@ def test_a_second_maintenance_command_fails_at_once(deployment, tmp_path): ("sync", "--move-store", elsewhere), ("sync", "--forget-store", elsewhere), ("sync", "--forget-backup", moved), + ("migrate-legacy", "--all"), ): started = time.monotonic() res = d.run("backup", *args) @@ -2091,6 +2112,35 @@ def test_a_second_maintenance_command_fails_at_once(deployment, tmp_path): adopting.kill() adopting.wait(timeout=10) assert _rows(d)[busy] == "CREATING" + # `migrate-legacy` takes the claim and waits for the same live job, as + # purge does; while it waits a second command fails at once, and a + # signal stops it before any change. + migrate_path = str(tmp_path / "migrate.out") + with open(migrate_path, "wb") as out: + migrating = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "migrate-legacy", "--all"], + cwd=d.cwd, stdout=out, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + ) + try: + deadline = time.monotonic() + 60 + while "Waiting for running jobs to finish" not in open(migrate_path).read(): + assert migrating.poll() is None, open(migrate_path).read() + assert time.monotonic() < deadline, open(migrate_path).read() + time.sleep(0.2) + text = open(migrate_path).read() + assert 'Took the maintenance claim as "extenddb backup migrate-legacy on ' in text, text + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 1 and "one maintenance command runs at a time" in res.stdout + res.stderr, res + migrating.send_signal(15) + assert migrating.wait(timeout=30) == 1 + text = open(migrate_path).read() + assert "stopped before any change: SIGTERM received while waiting" in text, text + assert "migrated " not in text, text + finally: + if migrating.poll() is None: + migrating.kill() + migrating.wait(timeout=10) + assert _rows(d)[busy] == "CREATING" _set_status(d, busy, "AVAILABLE") res = d.run("backup", "sync", "--prune-missing") assert res.returncode == 0, res.stdout + res.stderr @@ -3517,3 +3567,338 @@ def test_a_maintenance_command_waits_out_a_deleting_row_and_a_dead_restore(deplo assert _rows(d) == {b2: "DELETED"} assert not os.path.exists(_prefix(d, name, b2)), "the DELETING row's deletion was not finished" assert os.path.isdir(root) + + +def _catalog_rows(d: _Deployment, statement: str, params: tuple) -> list[tuple]: + """The rows one query on the deployment's catalog returns (`?` + placeholders, which become `%s` on PostgreSQL).""" + if d.backend == "sqlite": + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + return conn.execute(statement, params).fetchall() + finally: + conn.close() + import psycopg2 + + conn = psycopg2.connect(d.catalog) + try: + with conn.cursor() as cur: + cur.execute(statement.replace("?", "%s"), params) + return cur.fetchall() + finally: + conn.close() + + +def _in_catalog_rows(d: _Deployment, arn: str) -> tuple[str, int, int]: + """An in-catalog backup's rows at the catalog: its `backups` status + (`""` when it has no row), and its `backup_items` and + `backup_definitions` row counts.""" + status = _catalog_rows(d, "SELECT backup_status FROM backups WHERE backup_arn = ?", (arn,)) + items = _catalog_rows(d, "SELECT count(*) FROM backup_items WHERE backup_arn = ?", (arn,))[0][0] + definitions = _catalog_rows(d, "SELECT count(*) FROM backup_definitions WHERE backup_arn = ?", (arn,))[0][0] + return (status[0][0] if status else "", items, definitions) + + +def _copy_in_catalog_backup(d: _Deployment, arn: str, new_arn: str) -> None: + """Copy an in-catalog backup's `backups`, `backup_items` and + `backup_definitions` rows under `new_arn`, every column but the ARN (and + SQLite's item `id`) as it is: a backup an earlier release took under an + id of another shape, restorable through the in-catalog reader.""" + for table in ("backups", "backup_items", "backup_definitions"): + if d.backend == "sqlite": + columns = [r[1] for r in _catalog_rows(d, f"PRAGMA table_info({table})", ())] + else: + columns = [ + r[0] + for r in _catalog_rows( + d, + "SELECT column_name FROM information_schema.columns WHERE table_name = ? " + "AND table_schema = current_schema() ORDER BY ordinal_position", + (table,), + ) + ] + columns = [c for c in columns if c != "id"] + chosen = ", ".join("?" if c == "backup_arn" else c for c in columns) + n = _catalog( + d, + f"INSERT INTO {table} ({', '.join(columns)}) SELECT {chosen} FROM {table} WHERE backup_arn = ?", + (new_arn, arn), + ) + assert n >= 1, f"{table}: {n} rows copied" + + +def _typed_canonical(item: dict) -> str: + """`_canonical` for items holding binaries and sets: a binary as hex, + each set's members sorted.""" + + def norm(value): + if isinstance(value, dict): + return { + k: sorted(norm(v) for v in vs) if k in ("NS", "SS", "BS") else norm(vs) + for k, vs in value.items() + } + if isinstance(value, list): + return [norm(v) for v in value] + if isinstance(value, (bytes, bytearray)): + return bytes(value).hex() + return value + + return json.dumps(norm(item), sort_keys=True, separators=(",", ":")) + + +def _migrate(d: _Deployment, *args): + res = d.run("backup", "migrate-legacy", *args) + print(f"migrate-legacy {' '.join(args)}: exit {res.returncode}\n{res.stdout}{res.stderr}") + return res + + +# Items the in-catalog copy has to carry whole: a number set, a binary, a +# map holding a list, a boolean and a null beside the keys. +TYPED_ITEMS = [ + { + "pk": {"S": f"typed-{i}"}, + "sk": {"N": str(i)}, + "g": {"S": "typed"}, + "ns": {"NS": ["1", "2.5", str(10**20)]}, + "b": {"B": bytes([0, 255, i])}, + "m": {"M": {"l": {"L": [{"S": "x"}, {"N": "-1E-5"}]}, "t": {"BOOL": True}, "z": {"NULL": True}}}, + } + for i in range(3) +] + + +def test_migrate_legacy_carries_in_catalog_backups_into_the_store_under_their_arns(deployment, tmp_path): + """Two in-catalog backups of a table with a GSI (taken with `[backup.store] + kind = "disabled"`), then the store configured and the server started on + it once. On PostgreSQL and MongoDB the command runs beside the server; on + SQLite the server is stopped, the serve lock being the command's. + `migrate-legacy` needs exactly one of `` and `--all` (exit 2). The + first backup by its ARN spelt with another region migrates under its own + ARN: exit 0, its line with the item count, the claim line, the summary; + `--all` then migrates the second, and again finds nothing to migrate; the + first by ARN again is "already in the store". Each prefix holds a + manifest of kind `legacy_in_catalog` with no marker, unordered, under the + backup's name and the item count. `backup list` shows both rows + AVAILABLE; on SQLite and PostgreSQL each in-catalog backup's `backups` + row is DELETED with no item and no definition row left. The server then + lists each backup once, describes it with its name, creation time and + GSI as before, restores it with the table's items and its GSI answering, + and DeleteBackup of one deletes its prefix. A plain sync adopts + nothing.""" + d = deployment + _set_store_disabled(d) + server = _Server(d.config, str(tmp_path / "serve-disabled.log")) + try: + _wait_health(d.port, lambda h: True, server=server) + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + name = f"legacy-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load(d, creds, name, 300) + for item in TYPED_ITEMS: + ddb.put_item(TableName=name, Item=item) + source = sorted(_typed_canonical(i) for i in _scan_items(ddb, name)) + assert len(source) == 303 + first = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, first, timeout=60) + second = ddb.create_backup(TableName=name, BackupName="weekly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, second, timeout=60) + before = {arn: ddb.describe_backup(BackupArn=arn)["BackupDescription"] for arn in (first, second)} + assert "is writing the in-catalog backup format" in server.log_text() + finally: + server.stop() + if not os.path.isdir(d.root): + os.mkdir(d.root, 0o700) + _set_root(d, d.root) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + store_id = _store_id(d.root) + if d.backend == "sqlite": + server.stop() + print(f"branch: {d.backend}, the server {'stopped' if d.backend == 'sqlite' else 'running'}") + for args in ((), ("--all", first)): + res = _migrate(d, *args) + assert res.returncode == 2, res.stdout + res.stderr + assert "migrate-legacy " in res.stderr, res.stderr + + region = first.split(":")[3] + respelt = first.replace(f":{region}:", ":eu-west-3:", 1) + res = _migrate(d, respelt) + assert res.returncode == 0, res.stdout + res.stderr + assert f"Store: filesystem {os.path.realpath(d.root)} (store id {store_id})" in res.stdout, res.stdout + assert 'Took the maintenance claim as "extenddb backup migrate-legacy on ' in res.stdout, res.stdout + assert "every backup and restore job of this catalog answers LimitExceededException" in res.stdout + assert f"migrated {first} (account {ACCOUNT}): 303 items, " in res.stdout, res.stdout + assert "1 migrated, 0 finished, 0 refused, 0 skipped, 0 failed ===" in res.stdout, res.stdout + plain = re.sub(r"\x1b\[[0-9;]*m", "", res.stdout) + assert re.search( + r"INFO extenddb::audit: copied an in-catalog backup into the store and replaced it by its catalog " + rf'row action="backup migrate-legacy" account_id={ACCOUNT} backup_arn={re.escape(first)}$', + plain, + re.M, + ), plain + + res = _migrate(d, "--all") + assert res.returncode == 0, res.stdout + res.stderr + assert f"migrated {second} (account {ACCOUNT}): 303 items, " in res.stdout, res.stdout + assert first not in res.stdout, res.stdout + assert "1 migrated, 0 finished, 0 refused, 0 skipped, 0 failed ===" in res.stdout, res.stdout + res = _migrate(d, "--all") + assert res.returncode == 0, res.stdout + res.stderr + assert "No in-catalog backup to migrate." in res.stdout, res.stdout + res = _migrate(d, first) + assert res.returncode == 0, res.stdout + res.stderr + assert f"{first} (account {ACCOUNT}) is already in the store: nothing to migrate" in res.stdout, res.stdout + + for arn, backup_name in ((first, "nightly"), (second, "weekly")): + with open(os.path.join(_prefix(d, name, arn), "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["backup_arn"] == arn, manifest["backup_arn"] + assert manifest["backup_name"] == backup_name + assert manifest["snapshot"] == {"kind": "legacy_in_catalog", "marker": ""}, manifest["snapshot"] + assert manifest["item_order"] == "unordered" + assert manifest["source_backend"] == d.backend, manifest["source_backend"] + assert manifest["source_table"]["item_count"] == 303 + assert [g["IndexName"] for g in manifest["source_table"]["global_secondary_indexes"]] == ["by-g"] + assert sum(f["item_count"] for f in manifest["data_files"]) == 303 + rows = _rows(d) + assert rows == {first: "AVAILABLE", second: "AVAILABLE"}, rows + if d.catalog: + for arn in (first, second): + assert _in_catalog_rows(d, arn) == ("DELETED", 0, 0), arn + if d.backend == "sqlite": + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + + listed = [s["BackupArn"] for s in ddb.list_backups(TableName=name)["BackupSummaries"]] + assert sorted(listed) == sorted([first, second]), listed + for arn in (first, second): + after = ddb.describe_backup(BackupArn=arn)["BackupDescription"] + was = before[arn] + for member in ("BackupArn", "BackupName", "BackupStatus", "BackupCreationDateTime"): + assert after["BackupDetails"][member] == was["BackupDetails"][member], (member, after, was) + assert after["SourceTableDetails"]["TableName"] == name + assert after["SourceTableDetails"]["ItemCount"] == 303 + gsis = after["SourceTableFeatureDetails"]["GlobalSecondaryIndexes"] + assert [g["IndexName"] for g in gsis] == ["by-g"], gsis + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=first)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target, timeout=120) + assert sorted(_typed_canonical(i) for i in _scan_items(ddb, target)) == source + hits = ddb.query( + TableName=target, + IndexName="by-g", + KeyConditionExpression="g = :g", + ExpressionAttributeValues={":g": {"S": "typed"}}, + )["Items"] + assert len(hits) == 3, hits + ddb.delete_backup(BackupArn=second) + deadline = time.monotonic() + 60 + while os.path.exists(_prefix(d, name, second)): + assert time.monotonic() < deadline, "the deleted backup's prefix is still there" + time.sleep(0.2) + listed = [s["BackupArn"] for s in ddb.list_backups(TableName=name)["BackupSummaries"]] + assert listed == [first], listed + finally: + server.stop() + res = d.run("backup", "sync") + assert res.returncode == 0, res.stdout + res.stderr + assert "0 backup(s) adopted" in res.stdout, res.stdout + + +def test_migrate_legacy_refuses_a_backup_with_no_instant_or_an_item_618_let_in_and_keeps_it(deployment, tmp_path): + """Three in-catalog backups: one of a table holding three items with an + `N` value #618 let in (BatchWriteItem), one of a table that restores, + and (SQLite, PostgreSQL) a copy of the second's rows under a 13-digit + backup id, the shape every 0.1.x release issued. `migrate-legacy --all` + migrates the second and goes on past the other two, each refused with + its reason and its way out (the item count, the first item and #618; + the id with no instant), and exits 1 naming the two. The refused + backups keep their in-catalog rows and get no prefix; the server + describes each AVAILABLE, and the way out works: the 13-digit backup + restores through the in-catalog reader and a new backup of the restored + table is store-backed under a new ARN.""" + d = deployment + _set_store_disabled(d) + server = _Server(d.config, str(tmp_path / "serve-disabled.log")) + try: + _wait_health(d.port, lambda h: True, server=server) + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + bad_table = f"n618-legacy-{uuid.uuid4().hex[:6]}" + bad = _write_batch(d, ddb, creds, bad_table) + bad_arn = ddb.create_backup(TableName=bad_table, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, bad_arn, timeout=60) + name, good = _small_backed_up_table(ddb, "legacy") + finally: + server.stop() + old = "" + if d.catalog: + millis, suffix = _backup_id(good).split("-") + old = good.rsplit("/", 1)[0] + f"/{int(millis)}-{suffix}" + assert len(_backup_id(old).split("-")[0]) == 13, old + _copy_in_catalog_backup(d, good, old) + if not os.path.isdir(d.root): + os.mkdir(d.root, 0o700) + _set_root(d, d.root) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + if d.backend == "sqlite": + server.stop() + res = _migrate(d, "--all") + out = res.stdout + res.stderr + assert res.returncode == 1, out + assert f"migrated {good} (account {ACCOUNT}): {HAND_BUILT_ITEMS} items, " in res.stdout, out + first = sorted(bad, key=lambda i: (i["pk"]["S"], float(i["sk"]["N"])))[0] + refused = ( + f"refused {bad_arn} (account {ACCOUNT}): {len(bad)} item(s) cannot be restored; first: key " + f"{_key_json(first)}, attribute " + ) + assert refused in res.stdout, out + line = res.stdout.split(refused, 1)[1].split("\n", 1)[0] + assert " (#618); restore it (RestoreTableFromBackup), correct the items the log names, and take a new " \ + "backup of the restored table, which has a new ARN. Its in-catalog backup is kept" in line, line + refusals = 2 if old else 1 + if old: + assert ( + f"refused {old} (account {ACCOUNT}): its backup id {_backup_id(old)} carries no instant (an id a " + "0.1.x release issued), and a store prefix carries only the 14-digit shape; restore it " + "(RestoreTableFromBackup) and take a new backup of the restored table, which has a new ARN. " + "Its in-catalog backup is kept" + ) in res.stdout, out + logged = re.findall(rf"backup {re.escape(bad_arn)}: item cannot be restored: key ", res.stdout) + assert len(logged) == len(bad), out + assert f"1 migrated, 0 finished, {refusals} refused, 0 skipped, 0 failed ===" in res.stdout, out + assert f"{refusals} backup(s) were not migrated; see above" in res.stderr, out + assert not os.path.exists(_prefix(d, bad_table, bad_arn)), "the refused copy was left" + assert sorted(os.listdir(os.path.join(d.root, ACCOUNT))) == [name], os.listdir(os.path.join(d.root, ACCOUNT)) + assert os.listdir(os.path.join(d.root, ACCOUNT, name)) == [_backup_id(good)] + assert _rows(d) == {good: "AVAILABLE"} + if d.catalog: + assert _in_catalog_rows(d, bad_arn) == ("AVAILABLE", len(bad) + len(GOOD_ITEMS), 1) + assert _in_catalog_rows(d, old) == ("AVAILABLE", HAND_BUILT_ITEMS, 1) + assert _in_catalog_rows(d, good) == ("DELETED", 0, 0) + res = _migrate(d, bad_arn) + assert res.returncode == 1 and refused in res.stdout, res.stdout + res.stderr + + if d.backend == "sqlite": + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + for arn in [bad_arn] + ([old] if old else []): + details = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert details["BackupStatus"] == "AVAILABLE", details + if old: + listed = [s["BackupArn"] for s in ddb.list_backups(TableName=name)["BackupSummaries"]] + assert sorted(listed) == sorted([good, old]), listed + target = f"{name}-r" + ddb.restore_table_from_backup(TargetTableName=target, BackupArn=old) + _wait_active(ddb, target, timeout=120) + assert _count(ddb, target) == HAND_BUILT_ITEMS + again = ddb.create_backup(TableName=target, BackupName="carried")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, again, timeout=60) + assert os.path.isfile(os.path.join(_prefix(d, target, again), "extenddb-manifest.json")) + finally: + server.stop() diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py index 0910d87e3..bc58885ac 100644 --- a/tests/test_backup_store_config.py +++ b/tests/test_backup_store_config.py @@ -28,7 +28,8 @@ `extenddb-store.json` in it, and a relative root resolved against the shell's directory in daemon mode; - `extenddb backup list` and `backup purge --confirm` on an empty store, and - `backup purge` refused while a server holds the SQLite serve lock (exit 6); + `backup purge` refused while a server holds the SQLite serve lock (exit 6), + as are `backup sync`'s row-changing options and `backup migrate-legacy`; - an in-memory SQLite server restarted onto the root it used: the second start adopts the root the first start's catalog owned and syncs before it listens, so the first start's backup is described and restored; every @@ -1535,11 +1536,13 @@ def test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock(self, sqlite assert res.returncode == 6, res.stdout + res.stderr assert "another extenddb process" in (res.stdout + res.stderr), res.stdout + res.stderr # So is every option of `backup sync` that moves or removes rows, - # with or without a store to read. + # with or without a store to read, and `backup migrate-legacy`. for args in ( ("sync", "--move-store", "/srv/elsewhere"), ("sync", "--forget-store", "/srv/elsewhere"), ("sync", "--forget-backup", "arn:aws:dynamodb:us-east-1:123456789012:table/t/backup/01700000000000-00000000"), + ("migrate-legacy", "--all"), + ("migrate-legacy", "arn:aws:dynamodb:us-east-1:123456789012:table/t/backup/01700000000000-00000000"), ): res = d.run("backup", *args) assert res.returncode == 6, (args, res.stdout + res.stderr)