docs(backup): design revision for off-host backup and restore [backup/restore 1/4] by robinnsc · Pull Request #396 · ExtendDB/extenddb

robinnsc · 2026-10-13T03:25:25Z

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine (stacked on #394) · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #625 (part of #624).

What

Revision of docs/design/15-backup-restore.md from #348, written against main abf8082 (#384 in its merged form, the Cassandra backend #338, the PostgreSQL GSI build #401) and the companion PRs #393, #394, #395, reviewed over eight rounds and the maintainer's reviews, and then split: the document is the overview of the series (the invariants, the per-operation contracts and a short section per later PR), and the companions' own documentation is the record of what their code does. One commit, squashed with #348's work (Lee is co-author); supersedes #348. The document carries a revision log (items 1–25); the entries that change behavior are summarized below, as the design states them; the mechanisms are in the sections of the PRs that implement them.

Snapshot semantics. SnapshotEngine returns the table definition together with the items from one operation (TableSnapshot). The snapshot is opened in the CreateBackup request, so the backup holds every write acknowledged before the request was received, and the backup id is minted from the instant the open returned, just after the pin, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. Indexes are reported in whatever build state they had and only an index being deleted is dropped; a PostgreSQL GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open uses one connection and one transaction; on the separate layout the tables row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool (LimitExceeded over the bound), so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a pre-#361 table (key columns without COLLATE "C") unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (55P03, 40P01, 57014), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, retrying a data commit that fails after the catalog commit on a fresh connection for a bounded time before the request returns, so an orphan survives only a process death or a data database down past that bound; the control-plane poller probes held tables and retries each about once per active_poll, at once on a snapshot's or an in-catalog backup's release, and starts active after a restart; a drop DeleteTable deferred is due at once; DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail (#395's MongoDB module doc gives the server's error). Cassandra implements the trait as Unsupported, and the engine routes CreateBackup there to LegacyBackupReader::create until PR 14 whether or not [backup.store] is set, the path every backend takes while no store is configured; the off-host path is never taken there.

Coordination. Per-job claims with fencing tokens replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's RESTORE_LOCK_SEED and lock key; a transaction-mode pooler named as unsupported), the row itself on MongoDB (born owned through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets only, coordinator clock). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim, zero rows matched included, aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read, so a lost response cannot turn a backup a client has seen AVAILABLE into a deleted one. A job deletes a prefix, or a restore target, only after its FAILED write matched a row; otherwise the claim holder owns it. The reconciler's duties are listed once (PR 8's section), including DELETED rows that still hold bytes, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root, store_id); a CREATING or DELETING row of another store is reported and left alone, and DeleteBackup and Restore of another store's row return a ValidationException that names no root. The engine reaches the catalog through a BackupCatalog trait with a JobClaim (claim, heartbeat, conditional status write, re-read), listings for ListBackups, the store-id check and the root-matched commands, the in-use read and remove_restore_target, the backend-internal removal of a restore target that DeleteTable refuses (a FAILED restore left for reason target_not_empty keeps its target for the operator, who removes it with extenddb backup remove-target <table> under the maintenance claim), added in PR 6 and extended by the later PRs; new rows live in backup_index (Cassandra reuses its tables with store_kind set and adds backup_index_by_status), MongoDB's catalog gains the versioned migration step it lacks, PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch (and PostgreSQL's staleness test also selects NULL), so the first reconciler pass fails them, and every pre-PR-6 server stops before extenddb migrate. Row methods take account-scoped keys (backup_index keyed by account and backup id, with no stored key prefix), restore-target methods the target's account, name and table id (which #384 function each backend wraps is PR 9a's design). The PostgreSQL claim connection sets TCP keepalives. While the store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. ListBackups accepts a table ARN in TableName from PR 6.

DeleteBackup and sync. The row is tombstoned DELETING, the manifest is deleted first, and the row goes to DELETED when the prefix lists empty. backup sync lists every object, is scoped to the configured store, never adopts a CREATING, DELETING, DELETED or FAILED prefix or one whose listing lacks a data file the manifest lists at its size, and marks a row whose manifest head is gone DELETING only with --prune-missing, so the reconciler removes what remains. It checks extenddb-store.json, which records the store id and the owning catalog's catalog_id; another catalog's file, rows without the file, rows with two ids or a missing newest manifest mark the store unhealthy instead of stopping startup, and --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way; instances of one catalog must share one root. --move-store resolves the old root's CREATING and DELETING rows, compares every data file and refuses to carry a tombstone over a verified manifest (offering --forget-backup); --forget-store deletes a root's rows; purge, remove-target, move, forget, prune and adopt take a catalog maintenance claim that every job checks (while it is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty), and max_concurrent_jobs is per instance. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. --forget-backup runs under the claim and refuses a row a live job uses and a backup a CREATING restore target was registered from; one tombstone rule covers --move-store; on SQLite every writing backup command takes the serve lock; purge deletes each row's prefix and the row, never a whole account prefix, and keeps extenddb-store.json; the store-identity check runs at server startup around validate, at every sync and at every reconciler cycle, and creates extenddb-store.json with put_if_absent (If-None-Match: * on S3, O_EXCL and link on the filesystem), so the loser of a race reads the winner's id; migrate-legacy carries in-catalog backups into the store after re-reading and validating what it wrote, and refuses, keeping the legacy rows, ids with no instant, items the validator refuses and pre-0.0.4 SQLite definitions. DeleteBackup's in-use check and DELETING write and a restore's registration (AVAILABLE and in-use) run under the backup row's lock, so they serialize as on main; main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup; on MongoDB the registration's write is an $inc, on Cassandra main's active_restores lightweight transactions serve new rows too. Two live catalogs writing one root are a stated non-goal.

Restore. The request creates the target table and a restore_jobs record atomically in a detached task and returns CREATING (TableAlreadyExistsException or TableInUseException on a name race, as #384 returns); the job spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and restore_backfill_pending in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through remove_restore_target; the record is the table's child, so a FAILED record goes with its table and a DONE record stays for the operator; RestoreSummary is reported only while CREATING, as main does. Multi-part-key tables are restored (a change from #384: bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one); on PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Vector tables are backed up and restored on the off-host path (refuse_backup_of_vector_table leaves that path in PR 8 and stays in front of the legacy create; the restore refuses only a target without vector support from PR 9a; one #384 engine test is rewritten and the wire test follows the configured path). Both requests check overrides, ARN, target name, backup row and, for a restore, whether the backup is in use before taking the job permit, so #384's concurrent-restore test keeps passing; max_concurrent_jobs defaults to 4. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A key-ordered restore runs #394's per-file key-range check; an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed or its target deleted only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. A backup is in use while a restore target registered from it is CREATING, as on main: a second restore and DeleteBackup return BackupInUseException, and the registration re-checks it. A restore stops on shutdown leaving its row RUNNING; download transport errors are retried for 15 minutes; a takeover rewrites spool_host; the job row records its store and kind; an ARN of another account carrying a backup's id is BackupNotFound.

Format, store and snapshot. Sections 1, 2 and 3 point at the companions' documentation and keep only the rules later PRs build on: the manifest is the commit record and a prefix without one is not a backup; a bad sidecar is refused by restore and skipped by sync; the version gate accepts 1 to the current version; key_ordered is a promise the writer makes and the reader checks per file; the 32 MiB line cap is the floor of data_file_target_bytes; vector_indexes has three states and a target without vector support refuses a non-empty one; backup_creation_date_time is the snapshot instant and the id's; duplicate keys are refused on every target, by the count where nothing else detects them; N keeps 38 digits and a MongoDB sort key beyond 34 is refused. From the store: put and delete_prefix are unordered (the heartbeat check before each put and the emptiness check before a tombstone row is removed are the protections), runtime shutdown sends no abort (the server awaits each job's failure path), listing has no delimiter form; put_if_absent is PR 7's. From the snapshot: Cassandra returns Unsupported and CreateBackup there goes to the legacy create until PR 14's decision; DEFAULT_FETCH_BATCH is one constant. Section 8 keeps the account-prefixed keys, verify-before-write, plaintext contents and the audit log, and points at the README's IAM example and config.rs's transport and proxy rules.

Configuration. Store settings under [backup.store]. The default root is per backend: beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error; without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 (Cassandra always), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced; a store transport failure at startup degrades the component instead of stopping the server. The memory formula is in item text bytes and PR 8 bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; PR 6 adds the [backup] struct with the job keys and PR 7 adds [backup.store]; the pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path; purge takes the maintenance claim and deletes each row's prefix and the row.

Phasing. The PR series: PR 0 to 5 one line each naming the module and where its documentation lives (PR 3: crates/backup-store/README.md and module docs; PR 4: crates/core/src/backup_format module docs and the testdata README; PR 5: crates/storage/src/snapshot.rs, the four snapshot_engine.rs, docs/troubleshooting.md), then one section each for PR 1, 6, 7, 8, 9a, 9b, 10, 11, 13 and 14 (ten PRs after the four open: the reconciler ships with the CreateBackup job as PR 8, CI and documentation together as PR 11) with the same five parts: goal; the invariants of sections 4 and 5 it honours, by letter and step; the interfaces it adds with signatures and row keys (the BackupCatalog and JobClaim signatures split by the PR that adds each method, bulk_load, the backup_index and restore_jobs schema, the migration numbers with their version-literal-ok markers for PR 1 and PR 6); the state transitions it owns with their conditions; and its acceptance list kept whole, with its kill points, the main tests that must pass unchanged and the documentation rows of section 10 it corrects. Parallelism, the critical path and Phase 3 are unchanged.

Split. The design stopped describing code the companions document. Of the 67 companion facts the design stated, 56 were also stated in the companion's own README or module docs; the split removes the design's copies and leaves a pointer per section (section 1 at crates/core/src/backup_format and its testdata README, section 2 at crates/backup-store/README.md and its config.rs, key.rs and lib.rs docs, section 3 at crates/storage/src/snapshot.rs and the four snapshot_engine.rs). The five facts only the design stated moved into the companions (#393: the filesystem list's whole-root Vec, the empty-prefix IAM example; #395: the reason the fetch size is a constant, the pre-#361 sort cost; the half-pool measurement was already in #395's module doc) or were dropped (the dependency-change facts, whose record is #393's description and Cargo.lock; the vacuum and WAL growth figures, which belong to the backup manual). Each companion gains one sentence saying the overview states the rules its callers honour and its own docs are the record of what the code does. The detailed design of PR 6 to 14 (per-backend SQL, lock sequences, the page query, the step-by-step job) is written and reviewed with each PR's code; the overview keeps every rule the past review Majors rest on (the report maps all 30) and the acceptance lists whole. Heading numbers and step numbers are unchanged, so revision log items 1–24 still resolve; item 25 records the split. The file goes from 279,584 to 187,722 bytes.

Also adds the missing rows for designs 12 and 13 to docs/design/README.md; the row for 15 describes the document as the overview of the series and names the crates whose documentation holds the format, store and snapshot details.

Why

The draft was written on 2026-09-10 against 4fd2c5f8. Since then #384 fixed two of its seven motivating defects and landed more than the draft reviewed (the backup id and ARN parser, detached restores with per-backend owners and sweeps, slot semaphores, the full ListBackups request, every override applied, table class deliberately not restored), and #401 changed how PostgreSQL builds a new GSI. Eight review rounds and the maintainer's reviews against this revision and the companion PRs found design-level problems (snapshot/definition atomicity, delete ordering, non-idempotent resume, a fenced-out job deleting a prefix or a table a successor owned, a lost AVAILABLE response deleting a finished backup, DELETED rows over bytes nothing removed, orphaned multipart uploads, a restore record that could not outlive its table), statements about #384, #393, #394 and #395 that were false as merged or as proposed, a reconciler that would delete the copy of a moved store, a Cassandra write the database refuses as one transaction, the removal of CreateBackup on sqlite-memory, an IAM example that grants no list action, and undefined pieces (the Cassandra restore claim, the default store root on backends without a data directory, Cassandra backup creation after PR 6). The fifth round found: a reconciler that resolves a row against a store that does not hold its bytes; --forget-store followed by --move-store deleting recovered backups; sync inferring deletion from an unmounted store; maintenance commands with no cross-instance exclusion; a PostgreSQL resume trusting a free lock; no duplicate detection on restores into Cassandra; a memory formula sized by item size rather than item text; a CreateBackup on Cassandra the text refused although the engine routes it to the legacy create; and the version-literal-ok markers exempting nothing. The sixth round found: no engine-to-backend interface for the catalog operations the text required; an undefined MongoDB restore terminal that the reconciler would have resumed mid-backfill; a MongoDB catalog version with no migration path; "restores hold their own half-pool slots" describing a state PR 6 removes; a byte budget with no constant and two mechanisms; vector-table backups the text promised and #384's kept refusals forbade; DescribeBackup members the manifest could not carry; a request order that broke #384's concurrent-restore test at the default job limit; a SQLite default root that could stop an upgraded server; in-flight #384 restores the migration left unspecified. The maintainer's review of 2026-10-28 found: a migration backfill PostgreSQL's staleness test never selects; a second restore and a DeleteBackup of a backup in use that main refuses and the text allowed; sync adopting an incomplete copy after --forget-backup; catalog listings with no interface; a PostgreSQL page with no byte bound; two required manifest members missing from the reader's list; two store ids from one empty root; a reconciler that left the states a maintenance command waits on; the vector refusal removed from the legacy create too; a refused ListBackups ARN form the service accepts; CreateBackup silent on values #618 lets in; a failed restore's target removed through a DeleteTable that refuses it; and #393 behaviours stated before they were implemented. The seventh round found: a backup refusal of values #618 let in that regresses main with no way to find the items; restore-target removal keyed so that it cannot address a Cassandra row and wrapped round functions with other contracts; a PostgreSQL page query with no key columns or outer order and an unbounded read past the cut; a MongoDB registration write that does not conflict; a DeleteBackup/restore race main does not have; backup rows read by a bare id across accounts; migrate-legacy deleting backups it could not carry; PR 14 dropping unmigrated backups; store identity that could not tell two catalogs or two host-local roots apart; and restored index rows and vector index states left to a queue. The design's companion facts were then checked against #393, #394 and #395 as fixed in that round, which corrected the drop order, the proxy test count, the endpoint sources, the restore's prehashed reader and the manifest parse figure. The eighth round, a sentence-level verification of that text, found: a claim that main's two PostgreSQL race tests pin the new path, which exercise the legacy path; three Cassandra statements the schema or main's code contradicts (a full-key reason citing ALLOW FILTERING where a secondary index exists, a removal described as unconditional, an index flip no lightweight transaction can carry); a target_not_empty exception stated once and contradicted three times, with no operator path for such a target; two sqlite-memory catalogs able to own one root; a false SDK virtual-hosting example; a CQL IS NULL predicate that does not exist; and wording and date slips. All are closed in this revision. The implementation PRs in this series follow the revised text, so it should land with or before them.

Testing done

Documentation only. cargo test -p extenddb-storage-postgres --test doc_version_literals passes; deleting any one of the three version-literal-ok markers (section 4's Authority line, PR 1's and PR 6's sections) makes it fail, checked for each. Every sentence of the split overview is a sentence of the pre-split text, a one-sentence pointer at a companion's documentation, a template heading, or a listed rewording that keeps its meaning (.review/reviews/r8/DONE-SPLIT.md); the build is a script (.review/pr396-split.py) that asserts the first of those. Claims about #384 and #401 were checked against main abf8082; the pointers at #393, #394 and #395 name files and doc items at their heads (aef17ddd, 3f7107dc, 069c4b17). The companions' doc builds (cargo doc --no-deps) and cargo fmt --check pass for the touched crates, with no new warning.

Not changed here

Sections 6 (PITR) and 9 (observability) beyond the items above.
The backup manual and the deployment-guide statements are PR 11; section 10's table names the owning PR of every other documentation row (the upgrade manual's rows belong to PR 6, 7, 8, 10 and 14).
The RFC-0003 backup criterion the trait implies is a separate docs change, not part of feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality (documentation only)
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: this document is the design record for the series. It specifies a Storage trait change, an on-disk format and new CLI subcommands, for which CONTRIBUTING.md asks for an RFC; its Decision 5 records that later PRs reference this document instead of a separate RFC. A maintainer should confirm that decision or ask for the document to be filed under docs/rfcs/.

Breaking changes

None (documentation). The design names the operator-visible changes it commits later PRs to: CreateBackup without [backup.store] keeps the in-catalog create with a deprecation warning until PR 14 removes it; vector tables become backable and restorable on the off-host path, the legacy create keeping main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path); ListBackups accepts a table ARN in TableName; on the off-host path a backup of a table holding a value #618 let in fails where main backs it up; PR 14's migration refuses while legacy backup rows remain; instances that share a catalog must share one store root; every pre-PR-6 server must stop before extenddb migrate; on SQLite the backup CLI commands that write need the server stopped; a defaulted SQLite store root that fails validation logs a warning instead of being used; a FAILED restore left for reason target_not_empty is removed with extenddb backup remove-target, a new CLI subcommand (PR 10).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

LeeroyHannigan

I checked the document against origin/main at 744ea29, which now carries #384, and its DynamoDB claims against the API reference. The statements about current code hold for the migration and catalog version reservations, the lock seed, the legacy artifacts, the PITR surface, /health, the import skip behaviour, the ListBackups time bounds, and the version literal test. Most findings come from #384 changing between 13 October, when this revision described it, and its merge on 23 October. Two rules conflict with what merged and block. Three more items are not inline. The reconciler resumes a dead restore after 3600 seconds where main clears an abandoned target after 60. Four code references, among them manifest.rs and hash_key_text, name code that is not on main. The S3 example endpoint is plain HTTP and section 8 does not cover transport.

LeeroyHannigan

I checked this revision against main at abf8082, the heads of #393, #394 and #395, and my first review.

From the first review, C1 to C7 are fixed. The restore recovery window is now 300 s and the document records it against #384's 60 s. Three items remain: the sqlite-memory removal (reply on that thread), line 94, which still names manifest.rs and testdata/extenddb-manifest/v1.json as if they were on main, and section 8, which still says nothing about transport for a custom endpoint.

The other inline comments are on text added since the first review. Once these and the three above are addressed, I will approve.

New crate extenddb-backup-store: one trait, BackupStore, with put, get, head, list, delete_prefix and validate, and two implementations. The backup job and the restore path (later PRs) write and read portable backups through it; see the design in #396 (docs/design/15-backup-restore.md once merged). Keys are /-separated component paths bounded at 1024 bytes, each component at most 255 bytes (the filesystem's NAME_MAX); empty components, ".", "..", backslashes, control characters and U+FFFE/U+FFFF (which S3's XML listings cannot carry) are rejected. That rule is the upper bound both stores accept; the key module doc is the one list of where each store narrows it, and a backup reads through either store only if its keys avoid those. Prefix matching is component-aligned (prefix a/b does not match key a/bc). put() returns ObjectMeta (bytes written, ETag when the store has one) so the manifest writer does not need a second head() per data file. After a failed put the key holds nothing, the previous object, or the new complete object (a put that stored the object and then failed: the filesystem's directory sync, an S3 complete that timed out client-side), never a partial one. Filesystem store (FilesystemStore): - Every operation resolves its path with one component walk (symlink_metadata per component, owner-only check on each directory, a symlink in an intermediate component refused). put, get and head also refuse a symlink at the final component; for list and delete_prefix a symlink at the prefix itself is nothing there, and symlinks below the prefix are skipped by list and removed as links (never followed) by delete_prefix. list reports only names that pass the key rule; a non-UTF-8 name, a backslash or a temporary is skipped and still deleted. - The symlink checks are path-based and can be raced by a process that can write under the root, so the store requires that only the owner can reach the tree and enforces it on Unix: open() and validate() refuse a root that grants any permission to group or others or is owned by another uid, and a root whose ancestor is owned by a third user or is writable by group or others without the sticky bit. Every operation re-checks the root, refuses it when its canonical path has changed since open (an ancestor moved or replaced by a symlink), and applies the same owner-only rule to every existing directory it descends into, on the way to the prefix as well as below it. Directories are created 0700 and objects 0600 at creation. Errors name the path and the fix (chown / chmod 0700). - Writes go to a temporary sibling (.put-<uuid>.tmp), flush, fsync, rename, fsync of the directory and of each ancestor up to the root. The flush is what surfaces the last write's error: tokio's sync_all waits for the last background write but does not return its error (tokio keeps only its kind for a later write or flush), so without a flush before the drop a short file would be renamed into place. A put that does not reach the rename (an error, its future dropped by a caller's timeout or a cancelled job, a panicking body) removes the temporary from a drop guard; a failure after the rename leaves the new, complete object. Only a process killed mid-put leaves a .put-*.tmp. Directories a failed put created stay (empty, invisible to list, removed by delete_prefix): removing them raced a concurrent put into the same directory. A put onto a key that is a directory is refused before the body is read. get() opens with O_NOFOLLOW and reports a file removed under it as NotFound. - delete_prefix checks every directory under the prefix against the owner-only rule before it removes anything, so a refusal leaves the prefix whole; it then removes the prefix directory and everything under it, leaves directories above the prefix, and treats an entry another deleter removed first as already gone (not counted), so two reconcilers retrying the same prefix do not fail each other. The trait defines the count once: objects in the call's own listing it removed or found already gone; concurrent filesystem counts partition the objects, concurrent S3 counts may overlap. - validate() re-applies the owner-only and ancestor rules, checks the root's canonical path, and writes, flushes, syncs and removes a probe under the root. - Paths only a race or a crash reaches (a file removed between the walk and the open, an entry vanishing mid-listing, O_NOFOLLOW after the walk, the canonical check in prepare_parent, the re-checks of directories a concurrent process created or changed, a symlink ancestor in check_root_ownership, the fsyncs of the temporary and of the ancestors) are kept as defence in depth; the module doc lists them as untested. The redundant prefix filter in list and a repeated owner check in delete_prefix("") are removed. S3 store (S3Store), on aws-sdk-s3 with credentials from the SDK chain: - open() refuses an empty bucket, a part size outside 5 MiB to 5 GiB, an endpoint that is not an absolute http:// or https:// URI (a bare host:port, a stray character; the message names the problem and never the value, which may carry user information), and an http:// endpoint whose host is not a recognised loopback form (127.x.y.z, [::1], or localhost with force_path_style), unless allow_insecure_endpoint is set, which logs a warning. The rule applies to the endpoint the client uses: the config file's, or one the SDK reads from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config, which open reads and sets on the client explicitly; without the override an interceptor also refuses any http:// request to another host before it is sent. - Multipart upload above the configured part size (default 8 MiB) with bounded concurrency, at most 10,000 parts; a failure anywhere after CreateMultipartUpload, including a failed CompleteMultipartUpload, aborts the upload, and a drop guard, armed until the inline abort has returned, aborts it when the put's future is dropped (a cancelled job). CreateMultipartUpload runs in a spawned task that arms the guard before handing it back, so a put dropped while that request is in flight still aborts the upload it created. A CreateMultipartUpload the SDK retried after the server created the first, and a validate dropped during its create, leave an empty upload only the lifecycle rule removes (README). delete_prefix also aborts in-progress uploads under the prefix (component-aligned, as for objects), which sweeps what a killed process left; if the bucket policy denies ListMultipartUploads it logs a warning and leaves them to the lifecycle rule, and any other listing error fails the call. Requests carry a 60 s attempt and 300 s operation timeout; the attempt includes sending a part, so the docs size upload_part_size_bytes at about the link's bytes/s × 60. A GetObject body, which the SDK streams after those timeouts have ended, is bounded per chunk (60 s between chunks fails the read as a transport error naming the key), beside the SDK's own stalled-stream protection (under 1 B/s for 5 s). - Paginated listing that skips keys failing the key rule (console folder markers); prefix deletion in DeleteObjects batches of 1000 over the raw listing, with per-key errors surfaced (AccessDenied as PermissionDenied, keys reported without the store prefix). - A key that fits the shared rule but exceeds S3's 1024-byte limit once the configured bucket prefix is added is refused as InvalidKey; a bucket prefix over 512 bytes is refused at open. - Service errors, denials included, are reduced to code and service message in Display (no request or credential material; Debug of a transport error carries the SDK error's Debug, headers included), and a response without an S3 code carries its HTTP status instead ("s3 UploadPart failed: HTTP 500 (no error code in the response)"); NoSuchBucket is a bucket problem and NoSuchUpload an upload problem, neither NotFound; InvalidToken, ExpiredToken and TokenRefreshRequired are PermissionDenied. Transport errors display their source chain, so "dispatch failure" carries its cause (no region, no credentials, connection refused). validate() lists one key under the prefix rather than calling HeadBucket, because a HEAD error has no body: the 403 keeps S3's code (AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch), a 404 names the missing bucket, a 301 names the bucket's region. It then probes each other grant on a random key under the prefix (DeleteObject and GetObject of the absent key, CreateMultipartUpload then AbortMultipartUpload, ListMultipartUploads); a denial names the missing grant, the create's names kms:GenerateDataKey too, and the GetObject one explains that S3 answers a missing key with 403 when s3:ListBucket does not cover it. kms:Decrypt is not exercised. The probe list is documented once, on S3Store's validate. head() on a denied bucket can only report the status ("HTTP 403 (no error code in the response)"), as the README states. The README covers the lifecycle rules a bucket needs (AbortIncompleteMultipartUpload, and on a versioned or Object Lock bucket NoncurrentVersionExpiration and Expiration with ExpiredObjectDeleteMarker, since delete_prefix and each validate leave delete markers there). - open_with_interceptor is a test-util-gated failure-injection seam; test-util also exposes the client config and settable part and page limits so the tests can check them. Configuration: BackupStoreConfig models the [backup.store] table (kind = "filesystem" | "s3", default filesystem) with [backup.store.filesystem] and [backup.store.s3] sub-tables. The server will nest it under its own key rather than flatten it, because deny_unknown_fields does not survive serde(flatten). Only the section matching kind is read; open() fails at startup when it is missing or a value is unusable. The struct has no Default: the server supplies the default root. Tests: 31 unit tests (key rule, configuration and open refusals including malformed endpoints refused without echoing them, the loopback rule including localhost without path style and the request-time check, a table pinning map_sdk_error with and without an S3 code) and a conformance suite run against both stores (round trip, overwrite, empty body, listing and prefix rules, list order by key bytes, delete counts, invalid keys, multipart-sized bodies, failed-put cleanup, put/head/list metadata agreement). Filesystem tests (59) cover the ownership and symlink rules (root modes with any group or other bit refused, 0700 and 0500 accepted, world-writable parent and grandparent refused, sticky parent accepted, permissive subdirectory refused through and below the prefix and before delete_prefix removes anything, root widened after open refused by every operation, in-root and escaping symlinks at and above the prefix, symlinks below the prefix removed without following, an ancestor moved and replaced by a symlink refused by every operation), a full disk (a child process with a lowered RLIMIT_FSIZE: the put fails with EFBIG and the previous object survives; validate fails), concurrent delete_prefix, temporaries hidden, delete_prefix("") keeps the root, delete_prefix keeps the directories above the prefix, directory keys not found, a put onto a directory refused before the body is read, failed puts leave only empty directories, a directory-sync failure after the rename keeps the new object, names failing the key rule (non-UTF-8 file and directory names among them) hidden and deleted, a root replaced by a symlink, and a put dropped mid-body and a put whose body panics leave no temporary. Three of them (a root, subdirectory or ancestor owned by another uid) need chown and skip unless run as root; with EXTENDDB_TEST_REQUIRE_ROOT set they fail instead. S3 tests (44) against moto cover the multipart abort paths (a failed part, a failed complete, NoSuchUpload at complete, a dropped put, a put dropped while CreateMultipartUpload is in flight, driven through a delaying TCP proxy, a put dropped during its inline abort and a validate dropped during its probe abort, through a proxy that swallows the abort, and an orphaned upload), the part-count limit, the concurrency bound, pagination of objects, and of uploads through an interceptor that cuts pages (moto never truncates ListMultipartUploads), the upload sweep's component-aligned filter, per-key DeleteObjects errors, foreign keys hidden and deleted, every validate probe (GetObject included) and the grant each denial names, a denied upload listing in delete_prefix and a failing one (500, ExpiredToken) failing it, 301 and 403 (responses rewritten through an interceptor, since moto never produces them), a GetObject body cut after its headers by a TCP proxy failing within the body idle timeout, a transport failure's cause, missing-bucket errors, the prefix and prefix-plus-key length caps, the client's timeouts, region and addressing style (those three run in a child with the suite's own credentials, so no S3 test needs anything from the environment), an SSO profile reaching the SSO provider, an endpoint from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config refused at open (child processes), and an http:// request to another host refused before it is sent. 52 hand mutants of the rules fixed in review (34 in round 4, 14 in round 5, 4 in round 6) were each killed by the suite. CI: test.yml installs moto from requirements-moto.txt (moto 5.2.4 and its direct dependencies pinned at the versions the last green run resolved; the pip cache is keyed on the file), starts it in server mode, runs the workspace with the store crate excluded, then runs the store crate with the S3 variables scoped to that step and EXTENDDB_TEST_S3_REQUIRED set so the S3 suite fails rather than skips when the endpoint is missing. Both steps use --locked. On the stable leg a further step runs the already-built conformance binary's filesystem tests as root with EXTENDDB_TEST_REQUIRE_ROOT set, so the uid cells run and fail rather than skip if the step is not root; the step sets pipefail. A final step prints moto.log when the job fails. Dependencies: aws-config and aws-sdk-s3 are pinned to the newest releases that build on the workspace rust-version (1.88); their transitive aws-* and aws-smithy-* crates are held in Cargo.lock at 1.88-compatible versions, and .cargo/config.toml sets the resolver's incompatible-rust-versions fallback so a plain cargo update keeps them there. aws-config enables the sso feature (adding aws-sdk-sso and aws-sdk-ssooidc, and linking ring, hex and zeroize, a second cryptography library beside aws-lc-rs) so a server whose credentials come from an SSO profile can reach the bucket. http 1 (already in the graph) is a direct dependency, to parse the endpoint at open; it adds no package. aws-types (=1.3.11, already in the graph through aws-config) is named directly so the store can read the S3-specific endpoint setting; it adds no package. One shipped crate moves: crc 3.4.0 to 3.3.0, because crc-fast 1.9.0 (1.10.0 needs Rust 1.89) requires crc ~3.3 and the workspace has one crc (sqlx-core uses it). License notices regenerated for the four feature sets; the crc line is the only content change. Cargo.lock gains duplicate versions none of which is avoidable at this SDK version: foldhash 0.1 and 0.2, hashbrown 0.16 (a fourth version, via lru), http 0.2 and http-body 0.4 beside their 1.x (the SDK's legacy http dependency), sha1 0.10 beside 0.11 (aws-smithy-checksums), spin 0.9 and 0.10 (crc-fast). Not done: list() collects the filesystem walk into a Vec before streaming. The trait promises lexicographic order and a directory walk cannot provide that incrementally; listings under the backup layout are small. The three bare `cargo build --release` jobs in integration.yml now build the SDK into the workspace build (Build release 5:03 -> 7:51, 4:14 -> 8:07, 4:54 -> 5:56); building `-p extenddb` there, as the Dockerfile does, is a workspace-wide CI change outside this crate. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan

I checked this revision against main at abf8082, the heads of #393, #394 and #395 as of this morning, and my two earlier reviews. The thirteen open items are fixed, the store-scoping one included: the copy-A-to-B, --forget-store A sequence deletes nothing under either root, and --move-store refuses to carry a tombstone over a verified manifest. Removing any version-literal-ok marker fails the test, and a stale literal elsewhere in its paragraph fails it too. Every Cassandra shape the text names applies on 4.1 and the cross-table conditional batch is refused. A role carrying the IAM example in section 8 passed every request validate sends on Amazon S3, with the missing-key read answering 404.

The inline comments are on text added since. Two of them are measured: the migration writes heartbeat_at NULL for interrupted restores, and the PostgreSQL staleness test the document states never selects a NULL row, so those targets stay CREATING; ListBackups with a table ARN answers 200 on the service. The rest are sequences written out against the text: a second restore of a backup in use is not refused where main refuses it, and the #384 test cited as passing asserts that refusal; --forget-backup routes an incomplete copy into sync, which adopts a prefix on its sidecar alone; BackupCatalog has no listing for ListBackups, the startup id check or the root-matched commands; the byte budget does not bound a PostgreSQL fetch (27.7 GB in one page at the worst-case line); the reader's required members omit two that #394 requires; two instances on one empty root can end with two store ids; and removing the vector refusal in PR 8a removes it from the legacy create too. Four smaller items, among them three #393 behaviours that sections 2 and 8 state as closed and my review of #393 this morning found open.

Two things my earlier reviews did not raise, and that are not conditions: 5.3 step 7 and the reconciler remove a failed restore's target through DeleteTable, which main refuses for a restore target (crates/storage-postgres/src/delete_table.rs:50-71); and PR 6 uses [backup] keys whose struct PR 7 adds.

Once the inline items are addressed, I will approve.

New crate extenddb-backup-store: one trait, BackupStore, with put, get, head, list, delete_prefix and validate, and two implementations. The backup job and the restore path (later PRs) write and read portable backups through it; see the design in #396 (docs/design/15-backup-restore.md once merged). Keys are /-separated component paths bounded at 1024 bytes, each component at most 255 bytes (the filesystem's NAME_MAX); empty components, ".", "..", backslashes, control characters and U+FFFE/U+FFFF (which S3's XML listings cannot carry) are rejected. That rule is the upper bound both stores accept; the key module doc is the one list of where each store narrows it, and a backup reads through either store only if its keys avoid those. Prefix matching is component-aligned (prefix a/b does not match key a/bc). put() returns ObjectMeta (bytes written, ETag when the store has one) so the manifest writer does not need a second head() per data file. After a failed put the key holds nothing, the previous object, or the new complete object (a put that stored the object and then failed: the filesystem's directory sync, an S3 complete that timed out client-side), never a partial one. Filesystem store (FilesystemStore): - Every operation resolves its path with one component walk (symlink_metadata per component, owner-only check on each directory, a symlink in an intermediate component refused). put, get and head also refuse a symlink at the final component; for list and delete_prefix a symlink at the prefix itself is nothing there, and symlinks below the prefix are skipped by list and removed as links (never followed) by delete_prefix. list reports only names that pass the key rule; a non-UTF-8 name, a backslash or a temporary is skipped and still deleted. - The symlink checks are path-based and can be raced by a process that can write under the root, so the store requires that only the owner can reach the tree and enforces it on Unix: open() and validate() refuse a root that grants any permission to group or others or is owned by another uid, and a root whose ancestor is owned by a third user or is writable by group or others without the sticky bit. Every operation re-checks the root, refuses it when its canonical path has changed since open (an ancestor moved or replaced by a symlink), and applies the same owner-only rule to every existing directory it descends into, on the way to the prefix as well as below it. Directories are created 0700 and objects 0600 at creation. Errors name the path and the fix (chown / chmod 0700). - Writes go to a temporary sibling (.put-<uuid>.tmp), flush, fsync, rename, fsync of the directory and of each ancestor up to the root. The flush is what surfaces the last write's error: tokio's sync_all waits for the last background write but does not return its error (tokio keeps only its kind for a later write or flush), so without a flush before the drop a short file would be renamed into place. A put that does not reach the rename (an error, its future dropped by a caller's timeout or a cancelled job at any point before the rename, the temporary's open on the blocking pool included, a panicking body) removes the temporary from a drop guard created with the file inside the blocking open; a failure after the rename leaves the new, complete object. Only a process that ends without running the guard (killed, or exiting without waiting for the runtime's blocking threads) leaves a .put-*.tmp; validate's probe has the same guard. Directories a failed put created stay (empty, invisible to list, removed by delete_prefix): removing them raced a concurrent put into the same directory. A put onto a key that is a directory is refused before the body is read. get() opens with O_NOFOLLOW and reports a file removed under it as NotFound. - delete_prefix checks every directory under the prefix against the owner-only rule before it removes anything, so a refusal leaves the prefix whole; it then removes the prefix directory and everything under it, leaves directories above the prefix, and treats an entry another deleter removed first as already gone (not counted), so two reconcilers retrying the same prefix do not fail each other. A delete_prefix whose future is dropped stops before its next entry removal (the blocking walk checks a flag the dropped future sets), as the S3 store's stops at its next request. The trait defines the count once: objects in the call's own listing it removed or found already gone; concurrent filesystem counts partition the objects, concurrent S3 counts may overlap. - validate() re-applies the owner-only and ancestor rules, checks the root's canonical path, and writes, flushes, syncs and removes a probe under the root (a probe a concurrent delete_prefix("") removed first counts as removed). - Paths only a race or a crash reaches (a file removed between the walk and the open, an entry vanishing mid-listing, O_NOFOLLOW after the walk, the canonical check in prepare_parent, the re-checks of directories a concurrent process created or changed, a symlink ancestor in check_root_ownership, the fsyncs of the temporary and of the ancestors) are kept as defence in depth; the module doc lists them as untested. The redundant prefix filter in list and a repeated owner check in delete_prefix("") are removed. S3 store (S3Store), on aws-sdk-s3 with credentials from the SDK chain: - open() refuses an empty bucket, an S3 Express directory bucket name (--x-s3), a part size outside 5 MiB to 5 GiB, an endpoint that is not an absolute http:// or https:// URI or that carries user information, a query, a fragment or an IPv6 zone id, or that http::Uri or url::Url refuses (the message names the source the endpoint came from and the problem class, never the value), and an http:// endpoint whose host is not a recognised loopback form (127.x.y.z, an IPv6 address equal to ::1, or localhost) or that is used without force_path_style (the SDK picks virtual-hosted addressing by bucket name: a dotted name, an access point ARN, any bucket on localhost), unless allow_insecure_endpoint is set, which logs a warning. The client is built with S3 Express session auth disabled, so it sends no CreateSession. The rules apply to every endpoint setting the SDK reads for a client that sends requests for the store: the S3 client's (the config file's, or one the SDK reads from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config, which open reads and sets on the client explicitly), and those of the STS, SSO and SSO OIDC clients the credential providers build (AWS_ENDPOINT_URL_STS, _SSO, _SSO_OIDC, AWS_ENDPOINT_URL, the profile's endpoint_url and services entries), which are checked whether or not the configured credentials use them; a refusal names the client and the setting. Without the override an interceptor also refuses any http:// request the S3 client would send to another host before it is sent; the credential providers' requests are checked only through their endpoints at open. An http:// endpoint is reached through an HTTP client built with aws-smithy-http-client's Builder (rustls with aws-lc, as the SDK's default) and no proxy, so HTTP_PROXY, http_proxy, ALL_PROXY and all_proxy cannot carry a signed clear-text S3 request off the host; https:// keeps the SDK's proxy behaviour (HTTP CONNECT only, so a socks5:// ALL_PROXY fails it). The credential chain's own http:// requests (container credentials, IMDS) use the environment's proxy; the docs say NO_PROXY must then list 169.254.169.254, 169.254.170.2 and 169.254.170.23. - Multipart upload above the configured part size (default 8 MiB) with bounded concurrency, at most 10,000 parts; a failure anywhere after CreateMultipartUpload, including a failed CompleteMultipartUpload, aborts the upload, and a drop guard, armed until the inline abort has returned, aborts it when the put's future is dropped while its runtime runs (a cancelled job). Runtime shutdown drops the guard's spawned abort unpolled, an in-flight CreateMultipartUpload task included, so uploads of puts in flight at shutdown, parts included, stay until a delete_prefix covering them or the lifecycle rule removes them; the trait doc says a caller that must leave nothing awaits its puts first. CreateMultipartUpload runs in a spawned task that arms the guard before handing it back, so a put dropped while that request is in flight still aborts the upload it created; validate's probe create runs the same way. A CreateMultipartUpload the SDK retried after the server created the first leaves an empty upload only the lifecycle rule removes (README). delete_prefix also aborts in-progress uploads under the prefix (component-aligned, as for objects), which sweeps what a killed process left; a denied ListMultipartUploads, or a denied AbortMultipartUpload of one upload (the sweep goes on to the next), is logged and left to the lifecycle rule, so a retry succeeds once the objects are gone; any other listing or abort error fails the call. Requests carry a 60 s attempt and 300 s operation timeout; the attempt includes sending a part, so the docs size upload_part_size_bytes at about the link's bytes/s × 60. A GetObject body, which the SDK streams after those timeouts have ended, is bounded per chunk (60 s between chunks fails the read as a transport error naming the key), beside the SDK's own stalled-stream protection (under 1 B/s for 5 s). - Paginated listing that skips keys failing the key rule (console folder markers); prefix deletion in DeleteObjects batches of 1000 over the raw listing, with per-key errors surfaced (AccessDenied as PermissionDenied, keys reported without the store prefix). - A key that fits the shared rule but exceeds S3's 1024-byte limit once the configured bucket prefix is added is refused as InvalidKey; a bucket prefix over 512 bytes is refused at open. - Service errors, denials included, are reduced to code and service message in Display (no request or credential material; Debug of a transport error carries the SDK error's Debug, headers included), and a response without an S3 code carries its HTTP status instead ("s3 UploadPart failed: HTTP 500 (no error code in the response)"); NoSuchBucket is a bucket problem and NoSuchUpload an upload problem, neither NotFound; InvalidToken, ExpiredToken and TokenRefreshRequired are PermissionDenied. Transport errors display their source chain, so "dispatch failure" carries its cause (no region, no credentials, connection refused). validate() lists one key under the prefix rather than calling HeadBucket, because a HEAD error has no body: the 403 keeps S3's code (AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch), a 404 names the missing bucket, a 301 names the bucket's region. It then probes each other grant on a random key under the prefix (DeleteObject and GetObject of the absent key, CreateMultipartUpload then AbortMultipartUpload, ListMultipartUploads); a denial names the missing grant, the create's names kms:GenerateDataKey too, and the GetObject one explains that S3 answers a missing key with 403 when s3:ListBucket does not cover it. A probe abort answered NoSuchUpload passes (a concurrent delete_prefix("") sweeps the probe upload). kms:Decrypt is not exercised. The probe list is documented once, on S3Store's validate. head() on a denied bucket can only report the status ("HTTP 403 (no error code in the response)"), as the README states. The README covers the lifecycle rules a bucket needs (AbortIncompleteMultipartUpload, and on a versioned or Object Lock bucket NoncurrentVersionExpiration and Expiration with ExpiredObjectDeleteMarker, since delete_prefix and each validate leave delete markers there). - open_with_interceptor is a test-util-gated failure-injection seam; test-util also exposes the client config and settable part and page limits so the tests can check them. On both stores, a put that overlaps a delete_prefix covering its key may fail or may leave its complete object after the deletion returns; the store gives no ordering. Configuration: BackupStoreConfig models the [backup.store] table (kind = "filesystem" | "s3", default filesystem) with [backup.store.filesystem] and [backup.store.s3] sub-tables. The server will nest it under its own key rather than flatten it, because deny_unknown_fields does not survive serde(flatten). Only the section matching kind is read; open() fails at startup when it is missing or a value is unusable. The struct has no Default: the server supplies the default root. Tests: 33 unit tests (key rule, configuration and open refusals including every endpoint shape the syntax rule refuses, refused without echoing it (the scheme included) and naming its client and source, the clear-text rule including an http:// endpoint without path style (localhost, a dotted bucket on an IP host, an access point ARN), Debug of the config without the endpoint's user information, the request-time check, a table pinning map_sdk_error with and without an S3 code) and a conformance suite run against both stores (round trip, overwrite, empty body, listing and prefix rules, list order by key bytes, delete counts, invalid keys, multipart-sized bodies, failed-put cleanup, put/head/list metadata agreement). Filesystem tests (62) cover the ownership and symlink rules (root modes with any group or other bit refused, 0700 and 0500 accepted, world-writable parent and grandparent refused, sticky parent accepted, permissive subdirectory refused through and below the prefix and before delete_prefix removes anything, root widened after open refused by every operation, in-root and escaping symlinks at and above the prefix, symlinks below the prefix removed without following, an ancestor moved and replaced by a symlink refused by every operation), a full disk (a child process with a lowered RLIMIT_FSIZE: the put fails with EFBIG and the previous object survives; validate fails), concurrent delete_prefix, temporaries hidden, delete_prefix("") keeps the root, delete_prefix keeps the directories above the prefix, directory keys not found, a put onto a directory refused before the body is read, failed puts leave only empty directories, a directory-sync failure after the rename keeps the new object, names failing the key rule (non-UTF-8 file and directory names among them) hidden and deleted, a root replaced by a symlink, a dropped delete_prefix stopping at its next entry, validate beside delete_prefix(""), and no temporary left by a put dropped mid-body once its temporary exists, a put whose body panics, or 200 puts and validates aborted after 0 to 400 µs (most during their open). Three of them (a root, subdirectory or ancestor owned by another uid) need chown and skip unless run as root; with EXTENDDB_TEST_REQUIRE_ROOT set they fail instead. S3 tests (51) against moto cover the multipart abort paths (a failed part, a failed complete, NoSuchUpload at complete, a dropped put, a put or a validate dropped while CreateMultipartUpload is in flight (dropped once moto lists the upload, while a delaying TCP proxy holds the response), a put dropped during its inline abort and a validate dropped during its probe abort, through a proxy that swallows the abort, each dropped once the proxy reports the abort held, and an orphaned upload), the part-count limit, the concurrency bound, pagination of objects, and of uploads through an interceptor that cuts pages (moto never truncates ListMultipartUploads), the upload sweep's component-aligned filter, per-key DeleteObjects errors, foreign keys hidden and deleted, every validate probe (GetObject included) and the grant each denial names, a denied upload listing or upload abort in delete_prefix succeeding, and a failing one (500, ExpiredToken) failing it, 301 and 403 (responses rewritten through an interceptor, since moto never produces them), a GetObject body cut after its headers by a TCP proxy failing within the body idle timeout, a transport failure's cause, missing-bucket errors, the prefix and prefix-plus-key length caps, the client's timeouts, region and addressing style (those three run in a child with the suite's own credentials and no proxy variables, so no S3 test needs anything from the environment), an SSO profile reaching the SSO provider, an endpoint from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config refused at open naming its source, endpoint winning over each of them, and eight refused endpoint shapes through endpoint and each SDK source (49 child processes), an http:// endpoint reached without the environment's proxy and an https:// one through HTTPS_PROXY, a validate whose probe abort answers NoSuchUpload passing, runtime shutdown with puts in flight leaving uploads that delete_prefix sweeps, the credential providers' endpoints refused at open (eight child cells; a listener on a non-loopback address of the host receives nothing), a directory bucket refused at open, and an http:// request to another host refused before it is sent. The five tests that put a TCP proxy in front of the endpoint skip, with a printed reason, when it is not http://. 80 hand mutants of the rules fixed in review (34 in round 4, 14 in round 5, 4 in round 6, 13 for the maintainer's review of 2026-10-28, 15 in round 7); 78 are each killed by the suite. Two are not: the one that lets a child inherit the parent's proxy variables fails two child tests only together with the mutant that leaves the proxy enabled, and the one that leaves S3 Express session auth enabled has no effect while open refuses directory bucket names (the s3 module doc lists it as untested). CI: test.yml installs moto from requirements-moto.txt (moto 5.2.4 and its direct dependencies pinned at the versions the last green run resolved; the pip cache is keyed on the file), starts it in server mode, runs the workspace with the store crate excluded, then runs the store crate with the S3 variables scoped to that step and EXTENDDB_TEST_S3_REQUIRED set so the S3 suite fails rather than skips when the endpoint is missing. Both steps use --locked. On the stable leg a further step runs the already-built conformance binary's filesystem tests as root with EXTENDDB_TEST_REQUIRE_ROOT set, so the uid cells run and fail rather than skip if the step is not root; the step sets pipefail. A final step prints moto.log when the job fails. Dependencies: aws-config and aws-sdk-s3 are pinned to the newest releases that build on the workspace rust-version (1.88); their transitive aws-* and aws-smithy-* crates are held in Cargo.lock at 1.88-compatible versions, and .cargo/config.toml sets the resolver's incompatible-rust-versions fallback so a plain cargo update keeps them there. aws-config enables the sso feature (adding aws-sdk-sso and aws-sdk-ssooidc, and linking ring, hex and zeroize, a second cryptography library beside aws-lc-rs) so a server whose credentials come from an SSO profile can reach the bucket. http 1 (already in the graph) is a direct dependency, to parse the endpoint at open; it adds no package. aws-types (=1.3.11, already in the graph through aws-config) is named directly so the store can read the S3-specific endpoint setting; it adds no package. url (=2.5.8) and aws-smithy-http-client (=1.1.9, rustls-aws-lc), both already in the graph through the SDK, are direct dependencies, to apply the SDK's URL parser at open and to build the proxy-less client for an http:// endpoint; they add no package. One shipped crate moves: crc 3.4.0 to 3.3.0, because crc-fast 1.9.0 (1.10.0 needs Rust 1.89) requires crc ~3.3 and the workspace has one crc (sqlx-core uses it). License notices regenerated for the four feature sets; the crc line is the only content change. Cargo.lock gains duplicate versions none of which is avoidable at this SDK version: foldhash 0.1 and 0.2, hashbrown 0.16 (a fourth version, via lru), http 0.2 and http-body 0.4 beside their 1.x (the SDK's legacy http dependency), sha1 0.10 beside 0.11 (aws-smithy-checksums), spin 0.9 and 0.10 (crc-fast). Not done: list() collects the filesystem walk into a Vec before streaming. The trait promises lexicographic order and a directory walk cannot provide that incrementally; listings under the backup layout are small. The three bare `cargo build --release` jobs in integration.yml now build the SDK into the workspace build (Build release 5:03 -> 7:51, 4:14 -> 8:07, 4:54 -> 5:56); building `-p extenddb` there, as the Dockerfile does, is a workspace-wide CI change outside this crate. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

robinnsc · 2026-10-29T11:30:11Z

Pushed 7cbbbfb. Two changes beyond the inline replies:

The design is split (revision log item 25). Sections 1 to 3 are now pointers at feat(backup-store): store trait with filesystem and S3 implementations [backup/restore 2/4] #393, feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394 and feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395's own docs plus the rules later PRs rest on; sections 4 and 5 keep the state machines, identities, claim rules and per-operation contracts; PR 1, 6, 7, 8, 9a, 9b, 10, 11, 13 and 14 each have a short section (goal, invariants, interfaces with signatures and row keys, transitions, acceptance). Their detailed design comes with their code. The file went from 280 KB to 188 KB; 40 KB of what remains is the revision log. The companions each gain two or three doc sentences that invert the pointer direction (their docs are the record of what the code does; the overview is the record of the rules callers honour).
The series is condensed from twelve remaining PRs to ten (item 24): the reconciler ships with the CreateBackup job as PR 8, and CI and documentation ship together as PR 11.

If you would rather the revision log moved to its own file (another 40 KB off), say so; it is a one-line change to the split script.

LeeroyHannigan

I checked 7cbbbfb against main at cf321f0 and the current heads of #393, #394 and #395, and on PostgreSQL 16 the staleness test selects a heartbeat at the epoch, which is what the migration now writes.

RestoreTableFromBackup refuses a second restore of a backup in use before the permit, sync does not adopt a prefix whose listing lacks a data file or has it at another size, PR 8's PostgreSQL page query stops at the byte budget, PR 7 creates the store-id file with put_if_absent, and the reconciler finishes DELETING rows and resolves dead owners' rows while a maintenance claim is held. BackupCatalog has the three listings, the legacy create keeps the vector refusal, ListBackups takes a table ARN, and the backup job fails a backup that holds a value #618 let in and logs the item's key.

The format and store details point at #394's docs, which require stream_specification and time_to_live, and at #393's, which describe the endpoint, proxy and abort behaviour its code has now.

LeeroyHannigan

LGTM 🚀

…ackend Adds the one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code keeps its own copy of this logic for now; the engine-side backup job (next PR) is the first caller. Stacked on #394: SnapshotInfo is the manifest's SnapshotDescriptor (one definition, with the three KIND_* constants), and key_schema_is_orderable is KeyOrder::resolve, so a stream a backend marks ordered is one the manifest parser accepts and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. Trait (crates/storage/src/snapshot.rs): open_table_snapshot(account_id, table_name) returns TableSnapshot { description: TableDescription, info: SnapshotInfo, key_ordered: bool, items: BoxStream<Result<Item>> }. The description is the storage-level one at the snapshot instant, not the API response (restore_summary may be set, table_throughput_mode_summary is not, the counts are the stats pass's); items are yielded as stored (valid N normalized, invalid raw, TTL-expired unswept items included), and any error means start over, SnapshotExpired being the one worth retrying. The stream owns the backend transaction or session; dropping it releases it. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. The trait doc names what each backend holds while a stream is open and which errors each returns (Cassandra answers Unsupported before it resolves the table), the idle timeouts that end a stream as Internal, and why the trait has no default body and StorageError stays exhaustive; PostgreSQL and SQLite bound open snapshots per server (below) and refuse over the bound with LimitExceeded. StorageError gains SnapshotExpired for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient. snapshot.rs's module doc says that the design overview (docs/design/15-backup-restore.md, #396) states the rules the trait's callers honour and that this file's and each backend's snapshot_engine.rs module docs are the record of what the snapshots do; DEFAULT_FETCH_BATCH's doc says why the size is a constant (over a 1 ms link the larger fetch would save about a second per million rows), and the PostgreSQL module doc's pre-#361 paragraph carries the sort's cost (7 MB on disk for 100k rows at work_mem 4 MB). Indexes: the description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since #401 UpdateTable creates the data table before committing the row, so this is a guard, not an expected state). A restore recreates the indexes in the description and builds them from the items, so a CREATING index with a data table comes back complete; one left out would not come back at all. PostgreSQL, one shared database: the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. Separate data database: the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key. A fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit); the byte budget is the backup job PR's. A table whose text key columns do not have a collation named "C" or "POSIX" (created before the key columns were declared with it; "default" on a C database also counts as not bytewise, which streams unordered and complete) is streamed unordered, with a warning: ordering it would turn the read into a sort of the whole table before the first row. SQLSTATE 40001 (recovery conflict on a standby) and 72000 (snapshot_too_old on a primary, PostgreSQL 16 and earlier) map to SnapshotExpired, pinned by a test that raises each code on the live server. The stream holds ACCESS SHARE on the data table until its last fetch, so a DROP TABLE waits for it; DeleteTable takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that). On both layouts the synchronous path takes every drop's lock before the catalog removal commits. It deletes the table's pending queue rows first, waiting for in-flight claims with no timeout, as #401 intends. It then drops the base, GSI, LSI and vector-index data tables, base first, within one 1 s budget for the group (SYNC_DROP_LOCK_TIMEOUT; before each DROP, lock_timeout is the time left). On the shared layout that runs in the catalog transaction behind a savepoint. With a separate data database it runs in a data transaction opened first, and a task the request awaits commits the catalog and then that transaction, so a cancelled caller cannot stop between the two. A lock wait that fails (55P03, a deadlock victim 40P01, or a statement_timeout's 57014) rolls the drops back and takes the asynchronous path with the row still held. A write transaction, a SHARE holder, an index build or a reader that starts after the probe therefore costs the request at most the 1 s, and no data table is left without its row. With a separate data database a data commit that fails after the catalog commit has the drop retried on a fresh connection for up to 60 s, so a failover, a dropped connection or a network blip no longer orphans the data tables; only a process that dies, or a data database unreachable for that whole window, between the catalog commit and the last retry leaves them without a row, and then they stay until an operator drops the relations the error log names (docs/troubleshooting.md, "Data tables without a catalog row"). The base table goes first because a write holds it before it writes an LSI or a synchronous GSI: with the index tables first, a write in progress waited about 1 s on the drop, and the drop went asynchronous. On the shared layout a DeleteTable probing between an open's snapshot and its LOCK does not see the open; if the LOCK is granted first, the drop times out after 1 s, is rolled back with the removal, and DeleteTable returns DELETING (in 80 unaided open-and-delete pairs, the delete first in half of them, 0 left a data table without a row on either layout; 39 to 40 deletes ran synchronously and 40 to 41 asynchronously). The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred, also after a restart: startup recovery runs the same pass and wakes the poller when it deferred a drop. A drop is deferred when the probe finds another session holding any lock on the table's relations, or when the drop fails in a lock wait (55P03, 40P01 or 57014); any other failure is logged and left to the normal window and the 60 s idle pass. Each pass probes a held table and skips it without waiting, and drops a free one within the same 1 s budget, so every held table is retried about once a second however many are held, and a CreateTable that is due is activated on the same cadence (1.1 s with four drops deferred). A drop that DeleteTable deferred is due at once, not after control_plane_delay_seconds. The active loop waits on the wake as well as on its timer, so a snapshot's or an in-catalog backup's release has the drop retried at once, and any holder's release is followed within about a second: 108 to 166 ms in the tests, with a 10 s test-only active poll for the wake cases. The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the default (0.25 on every backend), the bound and the retry cadence, the same-name CreateTable refusal until then, and a two-step query, over every relation of the table, that finds any session holding a lock on them and its lock mode. A DeleteTable of a table that is already DELETING returns its description and changes nothing. SQLite: one deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened and the WAL grows until the stream ends (a TRUNCATE checkpoint waits out the busy timeout and reports busy; the module doc gives the measured reply). #384's create_backup_from_wal_snapshot holds the writer lock while it pins because it also inserts the CREATING row; this path writes nothing. build_table_description_on reads the definition, including #384's restore_summary from table_restores, on that transaction. The stream holds a backup slot (backup_server_slots, shared with the in-catalog backup) until its read transaction ends (the short page that shows the end, or the first error) or it is dropped, so the pool arithmetic that keeps half the pool free counts snapshots; with no slot free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them. The pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index, so key_ordered is true for single-range-key tables; multi-part keys fall back to rowid order. A page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it has yielded its last item or is dropped, so the slots bound how many decoded copies exist, as #384's create_backup_single_connection does for the in-catalog backup. SQLite DeleteTable of a DELETING table (control_plane_delay_seconds of 1 or more) returns its description too. Cassandra: open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog BackupEngine path stays its backup path. A direct test asserts the refusal naming the backend on an existing table and a missing one. MongoDB: a session started with snapshot=true reads the tables document, the indexes documents (doc_to_table_description_in now takes an optional session) and the base collection at one cluster time; the marker is that time. Every read sets readConcern: snapshot explicitly. The production catalog handle carries readConcern: local (as a URI with readConcernLevel= would), and on it a read without the level fails: the first read reports no snapshot time, which the open refuses naming the read concern, and a later read in the pinned session is refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported; each of the two refusals names its own cause. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired, pinned by a unit test that builds the command error from a reply document; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes, CursorNotFound mapped to Internal). Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. key_ordered is false: for a composite key _id is a netstring of the key text, which is not the format's order for any key type. Test hooks (extenddb_storage::snapshot::paging): a page-size knob, a round-trip counter, and a pause at two points of the open (AfterDefinition, BeforeItems) that the conformance check holds while it commits writes. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with the test-hooks feature of extenddb-storage, which the test-hooks features of storage-postgres and storage-sqlite and the new snapshot-test-hooks of storage-mongodb enable (storage-mongodb's existing test-hooks, the integration binary's coordination hooks, is unchanged). The backend crates' tests enable them through a dev-dependency on themselves. Production builds carry neither the hooks nor the harness. PostgreSQL also gains test-hooks entries spawn_control_plane_poller_for_tests and spawn_control_plane_poller_with_active_poll_for_tests (the production poller, the second with its active retry interval as a parameter), recover_control_plane_at_startup_for_tests, control_plane_drop_attempts_for_tests (a per-engine count of drop attempts, a held table the probe skipped counted), arm_delete_table_pause_at (holds DeleteTable at BeforeDrops, between its probe and its drops, or at BetweenCommits, between the catalog commit and the data commit on the separate layout; arm_delete_table_pause is the BeforeDrops form), shorten_data_drop_retry_bound_for_tests (a process-global shorter bound for the retried data drop), and re-exports of the data relation names and of apply_all_migrations, so tests/table_snapshot.rs binds the crate's lists instead of copying them. The two pauses (the open's and DeleteTable's) are one generic type, extenddb_storage::test_pause::Pause, and snapshot_conformance::PageSize is the one page-size guard the three backends' tests use. tests/table_snapshot.rs fails instead of skipping without a connection string when EXTENDDB_TEST_PG_REQUIRED is set, which integration.yml sets. Tests: extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared page size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open (so a backend that ignored the knob, or a MongoDB cursor without batch_size, fails here instead of passing vacuously), then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. Asserts: 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, the counter advanced after the writer committed (a stream that read every page during the open fails), key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare (the format's comparator) when ordered, and a second snapshot taken after the writer stops shows the final round. Then, once per pause point, on a fresh table: with the open held, it commits W1, an UpdateTable adding a GSI, and W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the new GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts the snapshot's description must equal the storage layer's describe_table with the running counts and table_throughput_mode_summary cleared; restore_summary is compared, and each backend's suite also compares a table given a restore record (assert_snapshot_describes_the_table). Every drain is under a 120 s deadline naming the stage and the items read, and a test waiting for the hooks lock gives up when one holder has kept it 600 s, naming the wait. READ COMMITTED or no FOR SHARE on PostgreSQL, the data cursor or the index read outside the session on MongoDB, and the pauses moved outside the read window on SQLite or MongoDB, fail it, as do the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion), a description built from a second code path (the DescribeTable comparison), and a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3 with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. The suites pass on PostgreSQL 16 (both layouts), file-backed SQLite and MongoDB 7 (replica set; the standalone refusal against a standalone mongod), and the Cassandra test on Cassandra 4.1. PG: tests/table_snapshot.rs (81 tests and two ignored race measurements: consistency, marker and key order, tables with legacy-collation key columns streaming unordered and complete on a scratch database whose collation is checked not to be C, five held snapshots on a ten-connection pool leaving GetItem and DescribeTable unblocked with the sixth LimitExceeded and an open succeeding after one is dropped, a GSI without its data table left out with its attribute, DeleteTable during a snapshot returning DELETING, with the production poller at a 10 s active poll the table gone within 3 s of a release during the pause after the first drop attempt (for a dropped stream, a drained stream, a fetch error, an open cancelled at BeforeItems and the in-catalog backup), for a plain reader holding ACCESS SHARE that sends no wake, held 10 s past the poller's window, the table gone within 3 s of its release, a reader across a restart (the table gone within 3 s of its release), a drop failing on a dependent view making at most 3 attempts in the 15 s after the window, a restored table's restore_summary in the snapshot description, DeleteTable with only a GSI or vector-index data table locked taking the async path, an unparseable row ending the stream and freeing its permit, twenty concurrent DeleteTables on a ten-connection pool, an open completing with nine of ten connections held, an open held while nine DeleteTables queue, DeleteTable at BeforeItems taking the async path, a fetch error freeing the permit, a drained stream out of its transaction before its end is polled, a ROW EXCLUSIVE and a SHARE holder sending DeleteTable down the asynchronous path within about 1 s with no data table left without its row, a DeleteTable cancelled during that wait leaving the table ACTIVE with its data, write holders on the base and three GSI tables released 0.8 s apart taking it there at about 1 s, a write holding the base and then writing the GSI table finishing first with the table removed in the request, a claimed queue row holding DeleteTable until it ends, a probed reader sending it asynchronous in milliseconds, a statement timeout and a deadlock victim taking the asynchronous path, DeleteTable of a DELETING table returning its description and changing nothing, a deferred drop due at once, four held tables probed with no lock wait (the first released gone within 2 s, a CreateTable ACTIVE within 2 s), all on both pool layouts; on the separate layout, a data commit that fails after the catalog commit (its backend terminated between the two) retried until the drop goes through with the request returning DELETING and no relation left, a data database refusing connections past a shortened bound returning Internal naming every leaked relation with the row gone, and a caller cancelled between the two commits leaving no orphan; on the shared layout, a snapshot open or a GSI-only reader taking its lock after DeleteTable's probe sending it down the asynchronous path with the row and both data tables kept, and a table dropped before the shared-layout LOCK being TableNotFound; the SQLSTATE mapping; the two ignored race measurements, which delay the delete or the open in alternate pairs and assert both outcomes occurred) with a Drop guard that drops the scratch databases on success or panic, added to the direct-storage step in integration.yml; the full storage-postgres suite is 259 tests. SQLite: in-module tests (consistency, key order, the in-memory copy releasing the connection, in-memory copies holding their slot, a damaged attribute definition streaming unordered and complete, the rowid fallback at a page of seven with no duplicates and at least six fetches, slots shared with CreateBackup and its refusal text, the WAL checkpoint reply while a snapshot is held, a restored table's restore_summary compared on a file-backed and an in-memory engine, a drained stream and a stream that hit an error freeing their slot while still held, and an in-memory copy freeing its slot with its last item) with the same guard for the scratch files, created under the test binary's target directory. MongoDB: in-module tests (consistency, an undecodable document ending the stream, reads without the snapshot level failing with code 72 and the error text, the code mapping, a restored table's restore fields compared, the round-trip counter counting only the snapshot stream (a GSI backfill leaves it unchanged), and a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod container it starts (and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it)) on per-test scratch databases built without the bootstrapper, as #384's live tests do, the catalog handle with the production option set, dropped by a guard on success or panic, so a URI in the environment never makes cargo test touch another instance's databases; gated on EXTENDDB_TEST_MONGODB_URI (failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, which the CI job does), holding the snapshot hooks lock, run by a new integration-mongodb.yml job that starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod (both on 127.0.0.1 only). tests/test_table_operations.py: DeleteTable of a DELETING table returns its description, on PostgreSQL and SQLite (the state written into the catalog), skipped on MongoDB and Cassandra, which never leave a table DELETING. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

ExtendDB#393) New crate extenddb-backup-store: one trait, BackupStore, with put, get, head, list, delete_prefix and validate, and two implementations. The backup job and the restore path (later PRs) write and read portable backups through it. The README and the lib.rs module doc say that the design overview (docs/design/15-backup-restore.md, ExtendDB#396) states the rules the store's callers honour and that this crate's docs are the record of what the store does. Keys are /-separated component paths bounded at 1024 bytes, each component at most 255 bytes (the filesystem's NAME_MAX); empty components, ".", "..", backslashes, control characters and U+FFFE/U+FFFF (which S3's XML listings cannot carry) are rejected. That rule is the upper bound both stores accept; the key module doc is the one list of where each store narrows it, and a backup reads through either store only if its keys avoid those. Prefix matching is component-aligned (prefix a/b does not match key a/bc). put() returns ObjectMeta (bytes written, ETag when the store has one) so the manifest writer does not need a second head() per data file. After a failed put the key holds nothing, the previous object, or the new complete object (a put that stored the object and then failed: the filesystem's directory sync, an S3 complete that timed out client-side), never a partial one. Filesystem store (FilesystemStore): - Every operation resolves its path with one component walk (symlink_metadata per component, owner-only check on each directory, a symlink in an intermediate component refused). put, get and head also refuse a symlink at the final component; for list and delete_prefix a symlink at the prefix itself is nothing there, and symlinks below the prefix are skipped by list and removed as links (never followed) by delete_prefix. list reports only names that pass the key rule; a non-UTF-8 name, a backslash or a temporary is skipped and still deleted. - The symlink checks are path-based and can be raced by a process that can write under the root, so the store requires that only the owner can reach the tree and enforces it on Unix: open() and validate() refuse a root that grants any permission to group or others or is owned by another uid, and a root whose ancestor is owned by a third user or is writable by group or others without the sticky bit. Every operation re-checks the root, refuses it when its canonical path has changed since open (an ancestor moved or replaced by a symlink), and applies the same owner-only rule to every existing directory it descends into, on the way to the prefix as well as below it. Directories are created 0700 and objects 0600 at creation. Errors name the path and the fix (chown / chmod 0700). - Writes go to a temporary sibling (.put-<uuid>.tmp), flush, fsync, rename, fsync of the directory and of each ancestor up to the root. The flush is what surfaces the last write's error: tokio's sync_all waits for the last background write but does not return its error (tokio keeps only its kind for a later write or flush), so without a flush before the drop a short file would be renamed into place. A put that does not reach the rename (an error, its future dropped by a caller's timeout or a cancelled job at any point before the rename, the temporary's open on the blocking pool included, a panicking body) removes the temporary from a drop guard created with the file inside the blocking open; a failure after the rename leaves the new, complete object. Only a process that ends without running the guard (killed, or exiting without waiting for the runtime's blocking threads) leaves a .put-*.tmp; validate's probe has the same guard. Directories a failed put created stay (empty, invisible to list, removed by delete_prefix): removing them raced a concurrent put into the same directory. A put onto a key that is a directory is refused before the body is read. get() opens with O_NOFOLLOW and reports a file removed under it as NotFound. - delete_prefix checks every directory under the prefix against the owner-only rule before it removes anything, so a refusal leaves the prefix whole; it then removes the prefix directory and everything under it, leaves directories above the prefix, and treats an entry another deleter removed first as already gone (not counted), so two reconcilers retrying the same prefix do not fail each other. A delete_prefix whose future is dropped stops before its next entry removal (the blocking walk checks a flag the dropped future sets), as the S3 store's stops at its next request. The trait defines the count once: objects in the call's own listing it removed or found already gone; concurrent filesystem counts partition the objects, concurrent S3 counts may overlap. - validate() re-applies the owner-only and ancestor rules, checks the root's canonical path, and writes, flushes, syncs and removes a probe under the root (a probe a concurrent delete_prefix("") removed first counts as removed). - Paths only a race or a crash reaches (a file removed between the walk and the open, an entry vanishing mid-listing, O_NOFOLLOW after the walk, the canonical check in prepare_parent, the re-checks of directories a concurrent process created or changed, a symlink ancestor in check_root_ownership, the fsyncs of the temporary and of the ancestors) are kept as defence in depth; the module doc lists them as untested. The redundant prefix filter in list and a repeated owner check in delete_prefix("") are removed. S3 store (S3Store), on aws-sdk-s3 with credentials from the SDK chain: - open() refuses an empty bucket, an S3 Express directory bucket or directory-bucket access point name (--x-s3, --xa-s3; the two names the SDK routes to S3 Express), a part size outside 5 MiB to 5 GiB, an endpoint that is not an absolute http:// or https:// URI or that carries user information, a query, a fragment or an IPv6 zone id, or that http::Uri or url::Url refuses (the message names the source the endpoint came from and the problem class, never the value), and an http:// endpoint whose host is not a recognised loopback form (127.x.y.z, an IPv6 address equal to ::1, or localhost) or that is used without force_path_style (the SDK picks virtual-hosted addressing by bucket name: a virtual-hostable name on localhost, an access point ARN on any custom endpoint; on an IP host every plain name is path-style, so the rule is keyed on the scheme because that choice is the SDK's and may change), unless allow_insecure_endpoint is set, which logs a warning. The client is built with S3 Express session auth disabled, so a client that reaches an S3 Express name sends no CreateSession (tested through a test-util open that skips the name refusal). The rules apply to every endpoint setting the SDK reads for a client that sends requests for the store: the S3 client's (the config file's, or one the SDK reads from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config, which open reads and sets on the client explicitly), those of the STS, SSO and SSO OIDC clients the credential providers build (AWS_ENDPOINT_URL_STS, _SSO, _SSO_OIDC, AWS_ENDPOINT_URL, the profile's endpoint_url and services entries), which are checked whether or not the configured credentials use them, and the IMDS endpoint setting (AWS_EC2_METADATA_SERVICE_ENDPOINT or the profile's ec2_metadata_service_endpoint), whose http:// form may name only loopback, 169.254.169.254 or [fd00:ec2::254]; a refusal names the client and the setting. The IMDS and general settings are read from the environment and the selected profile directly, as the SDK's IMDS client and endpoint provider read them (the service-key lookup is kept for S3, STS, SSO and SSO OIDC, whose service-specific variables the SDK does read), so a variable the SDK gives no meaning to, such as AWS_EC2_METADATA_SERVICE_ENDPOINT_ with a trailing underscore, cannot shadow the one it reads. AWS_CONTAINER_CREDENTIALS_FULL_URI keeps the SDK's own rule and is read only for the proxy rule. The settings are read from a first load given a fixed region, so a refused configuration sends nothing; the second load resolves the region (IMDS, off EC2 without region or AWS_EC2_METADATA_DISABLED) and builds the clients. Without the override an interceptor also refuses any http:// request the S3 client would send to another host before it is sent; the credential providers' requests are checked only through their endpoints at open. An http:// endpoint is reached through an HTTP client built with aws-smithy-http-client's Builder (rustls with aws-lc, as the SDK's default) and no proxy, so HTTP_PROXY, http_proxy, ALL_PROXY and all_proxy cannot carry a signed clear-text S3 request off the host; https:// keeps the SDK's proxy behaviour (HTTP CONNECT only, so a socks5:// ALL_PROXY fails it). When any endpoint of the credential chain is http:// (an STS, SSO, SSO OIDC or general setting, AWS_CONTAINER_CREDENTIALS_FULL_URI, or the IMDS endpoint setting), the chain's clients get the same proxy-less client, so a credential request to a loopback emulator never reaches the proxy and its https:// requests bypass HTTPS_PROXY (the S3 client's do not). Without such a setting the chain uses the SDK's default client, and the docs say NO_PROXY must then list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2. - Multipart upload above the configured part size (default 8 MiB) with bounded concurrency, at most 10,000 parts; a failure anywhere after CreateMultipartUpload, including a failed CompleteMultipartUpload, aborts the upload, and a drop guard, armed until the inline abort has returned, aborts it when the put's future is dropped while its runtime runs (a cancelled job). Runtime shutdown drops the guard's spawned abort unpolled, an in-flight CreateMultipartUpload task included, so uploads of puts in flight at shutdown, parts included, stay until a delete_prefix covering them or the lifecycle rule removes them; the trait doc says a caller that must leave nothing awaits its puts first. CreateMultipartUpload runs in a spawned task that arms the guard before handing it back, so a put dropped while that request is in flight still aborts the upload it created; validate's probe create runs the same way. A CreateMultipartUpload the SDK retried after the server created the first leaves an empty upload only the lifecycle rule removes (README). delete_prefix also aborts in-progress uploads under the prefix (component-aligned, as for objects), which sweeps what a killed process left; a denied ListMultipartUploads, or a denied AbortMultipartUpload of one upload (the sweep goes on to the next), is logged and left to the lifecycle rule, so a retry succeeds once the objects are gone; any other listing or abort error fails the call. Requests carry a 60 s attempt and 300 s operation timeout; the attempt includes sending a part, so the docs size upload_part_size_bytes at about the link's bytes/s × 60. A GetObject body, which the SDK streams after those timeouts have ended, is bounded per chunk (60 s between chunks fails the read as a transport error naming the key), beside the SDK's own stalled-stream protection (under 1 B/s for 5 s). Resolving credentials (the whole chain, every AssumeRole hop and an SSO refresh included) is bounded separately by the SDK's 5 s identity-cache load timeout and fails the request with "identity resolver timed out after 5s"; the SDK's retry settings reach the S3 client only, the provider clients keep the standard three attempts. - Paginated listing that skips keys failing the key rule (console folder markers); prefix deletion in DeleteObjects batches of 1000 over the raw listing, with per-key errors surfaced (AccessDenied as PermissionDenied, keys reported without the store prefix). - A key that fits the shared rule but exceeds S3's 1024-byte limit once the configured bucket prefix is added is refused as InvalidKey; a bucket prefix over 512 bytes is refused at open. - Service errors, denials included, are reduced to code and service message in Display (no request or credential material; Debug of a transport error carries the SDK error's Debug, headers included), and a response without an S3 code carries its HTTP status instead ("s3 UploadPart failed: HTTP 500 (no error code in the response)"); NoSuchBucket is a bucket problem and NoSuchUpload an upload problem, neither NotFound; InvalidToken, ExpiredToken and TokenRefreshRequired are PermissionDenied. Transport errors display their source chain, so "dispatch failure" carries its cause (no region, no credentials, connection refused). validate() lists one key under the prefix rather than calling HeadBucket, because a HEAD error has no body: the 403 keeps S3's code (AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch), a 404 names the missing bucket, a 301 names the bucket's region. It then probes each other grant on a random key under the prefix (DeleteObject and GetObject of the absent key, CreateMultipartUpload then AbortMultipartUpload, ListMultipartUploads); a denial names the missing grant, the create's names kms:GenerateDataKey too, and the GetObject one explains that S3 answers a missing key with 403 when s3:ListBucket does not cover it. A probe abort answered NoSuchUpload passes (a concurrent delete_prefix("") sweeps the probe upload). kms:Decrypt is not exercised. The probe list is documented once, on S3Store's validate. head() on a denied bucket can only report the status ("HTTP 403 (no error code in the response)"), as the README states. The README covers the lifecycle rules a bucket needs (AbortIncompleteMultipartUpload, and on a versioned or Object Lock bucket NoncurrentVersionExpiration and Expiration with ExpiredObjectDeleteMarker, since delete_prefix and each validate leave delete markers there), and its IAM section says the example drops the s3:prefix condition when the prefix is empty. - open_with_interceptor and open_accepting_s3_express_names are test-util-gated seams; test-util also exposes the client config, settable part and page limits, and on the filesystem store a hook run before each removal of a delete_prefix walk, so the tests can hold the walk at a chosen entry. On both stores, a put that overlaps a delete_prefix covering its key may fail or may leave its complete object after the deletion returns; the store gives no ordering. Configuration: BackupStoreConfig models the [backup.store] table (kind = "filesystem" | "s3", default filesystem) with [backup.store.filesystem] and [backup.store.s3] sub-tables. The server will nest it under its own key rather than flatten it, because deny_unknown_fields does not survive serde(flatten). Only the section matching kind is read; open() fails at startup when it is missing or a value is unusable. The struct has no Default: the server supplies the default root. Tests: 33 unit tests (key rule, configuration and open refusals including every endpoint shape the syntax rule refuses, refused without echoing it (the scheme included) and naming its client and source, the clear-text rule including an http:// endpoint without path style (localhost, an access point ARN, and a dotted bucket on an IP host, which the SDK addresses path-style and the rule refuses all the same), Debug of the config without the endpoint's user information, the request-time check, a table pinning map_sdk_error with and without an S3 code) and a conformance suite run against both stores (round trip, overwrite, empty body, listing and prefix rules, list order by key bytes, delete counts, invalid keys, multipart-sized bodies, failed-put cleanup, put/head/list metadata agreement). Filesystem tests (62) cover the ownership and symlink rules (root modes with any group or other bit refused, 0700 and 0500 accepted, world-writable parent and grandparent refused, sticky parent accepted, permissive subdirectory refused through and below the prefix and before delete_prefix removes anything, root widened after open refused by every operation, in-root and escaping symlinks at and above the prefix, symlinks below the prefix removed without following, an ancestor moved and replaced by a symlink refused by every operation), a full disk (a child process with a lowered RLIMIT_FSIZE: the put fails with EFBIG and the previous object survives; validate fails), concurrent delete_prefix, temporaries hidden, delete_prefix("") keeps the root, delete_prefix keeps the directories above the prefix, directory keys not found, a put onto a directory refused before the body is read, failed puts leave only empty directories, a directory-sync failure after the rename keeps the new object, names failing the key rule (non-UTF-8 file and directory names among them) hidden and deleted, a root replaced by a symlink, a dropped delete_prefix removing at most one entry after the drop (the walk held at a chosen removal through the test hook: a file with a sibling after it, the last file of a directory, a directory), validate beside delete_prefix(""), and no temporary left by a put dropped mid-body once its temporary exists, a put whose body panics, or 200 puts and validates aborted after 0 to 400 µs (most during their open). Three of them (a root, subdirectory or ancestor owned by another uid) need chown and skip unless run as root; with EXTENDDB_TEST_REQUIRE_ROOT set they fail instead. S3 tests (54) against moto cover the multipart abort paths (a failed part, a failed complete, NoSuchUpload at complete, a dropped put, a put or a validate dropped while CreateMultipartUpload is in flight (dropped once moto lists the upload, while a delaying TCP proxy holds the response), a put dropped during its inline abort and a validate dropped during its probe abort, through a proxy that swallows the abort, each dropped once the proxy reports the abort held, and an orphaned upload), the part-count limit, the concurrency bound, pagination of objects, and of uploads through an interceptor that cuts pages (moto never truncates ListMultipartUploads), the upload sweep's component-aligned filter, per-key DeleteObjects errors, foreign keys hidden and deleted, every validate probe (GetObject included) and the grant each denial names, a denied upload listing or upload abort in delete_prefix succeeding, and a failing one (500, ExpiredToken) failing it, 301 and 403 (responses rewritten through an interceptor, since moto never produces them), a GetObject body cut after its headers by a TCP proxy failing within the body idle timeout, a transport failure's cause, missing-bucket errors, the prefix and prefix-plus-key length caps, the client's timeouts, region and addressing style (those three run in a child with the suite's own credentials and no proxy variables, so no S3 test needs anything from the environment), an SSO profile reaching the SSO provider, an endpoint from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config refused at open naming its source, endpoint winning over each of them, and eight refused endpoint shapes through endpoint and each SDK source (51 child processes, two of them under AWS_IGNORE_CONFIGURED_ENDPOINT_URLS, which neither lifts a refusal nor drops a loopback setting), an http:// endpoint reached without the environment's proxy and an https:// one through HTTPS_PROXY, a validate whose probe abort answers NoSuchUpload passing, runtime shutdown with puts in flight leaving uploads that delete_prefix sweeps, the credential providers' endpoints refused at open (nine child cells, one with AWS_ENDPOINT_URL shadowed by an https AWS_ENDPOINT_URL_ under https variables for every credential client; a listener on a non-loopback address of the host receives nothing), the IMDS endpoint setting (seven child cells: a non-loopback http:// one from the variable, from the variable beside an https AWS_EC2_METADATA_SERVICE_ENDPOINT_, or from the profile refused with nothing sent, user information refused without the secret, loopback and the two IMDS addresses accepted), the credential providers' requests reaching a loopback http:// endpoint and never the proxy (seven child cells: STS through its own setting and through AWS_ENDPOINT_URL on localhost, the container URI, SSO, an SSO OIDC refresh, the IMDS setting, and an https:// S3 endpoint still through HTTPS_PROXY beside an http:// STS setting), both S3 Express names refused at open, the client sending no CreateSession to an S3 Express name, and an http:// request to another host refused before it is sent. The five tests that put a TCP proxy in front of the endpoint skip, with a printed reason, when it is not http://. 89 hand mutants of the rules fixed in review (34 in round 4, 14 in round 5, 4 in round 6, 13 for the maintainer's review of 2026-10-28, 15 in round 7, 7 in round 8, 2 in round 9); 88 are each killed by the suite. One is not: the one that lets a child inherit the parent's proxy variables fails two child tests only together with the mutant that leaves the proxy enabled. CI: test.yml installs moto from requirements-moto.txt (moto 5.2.4 and its direct dependencies pinned at the versions the last green run resolved; the pip cache is keyed on the file), starts it in server mode, runs the workspace with the store crate excluded, then runs the store crate with the S3 variables scoped to that step and EXTENDDB_TEST_S3_REQUIRED set so the S3 suite fails rather than skips when the endpoint is missing. Both steps use --locked. On the stable leg a further step runs the already-built conformance binary's filesystem tests as root with EXTENDDB_TEST_REQUIRE_ROOT set, so the uid cells run and fail rather than skip if the step is not root; the step sets pipefail. A final step prints moto.log when the job fails. Dependencies: aws-config and aws-sdk-s3 are pinned to the newest releases that build on the workspace rust-version (1.88); their transitive aws-* and aws-smithy-* crates are held in Cargo.lock at 1.88-compatible versions, and .cargo/config.toml sets the resolver's incompatible-rust-versions fallback so a plain cargo update keeps them there. aws-config enables the sso feature (adding aws-sdk-sso and aws-sdk-ssooidc, and linking ring, hex and zeroize, a second cryptography library beside aws-lc-rs) so a server whose credentials come from an SSO profile can reach the bucket. http 1 (already in the graph) is a direct dependency, to parse the endpoint at open; it adds no package. aws-types (=1.3.11, already in the graph through aws-config) is named directly so the store can read the S3-specific endpoint setting; it adds no package. aws-runtime (=1.6.0, already in the graph through aws-config) is named directly so the store can read the selected profile with the SDK's own parser and file set; it adds no package. url (=2.5.8) and aws-smithy-http-client (=1.1.9, rustls-aws-lc), both already in the graph through the SDK, are direct dependencies, to apply the SDK's URL parser at open and to build the proxy-less client for an http:// endpoint; they add no package. One shipped crate moves: crc 3.4.0 to 3.3.0, because crc-fast 1.9.0 (1.10.0 needs Rust 1.89) requires crc ~3.3 and the workspace has one crc (sqlx-core uses it). License notices regenerated for the four feature sets; the crc line is the only content change. Cargo.lock gains duplicate versions none of which is avoidable at this SDK version: foldhash 0.1 and 0.2, hashbrown 0.16 (a fourth version, via lru), http 0.2 and http-body 0.4 beside their 1.x (the SDK's legacy http dependency), sha1 0.10 beside 0.11 (aws-smithy-checksums), spin 0.9 and 0.10 (crc-fast). Not done: list() collects the filesystem walk into a Vec before streaming. The trait promises lexicographic order and a directory walk cannot provide that incrementally; listings under the backup layout are small. The README's list/delete_prefix bullet and the BackupStore::list doc state it, with the cost (about 72 bytes plus the key per object) and that the S3 store's listing is paged. The three bare `cargo build --release` jobs in integration.yml now build the SDK into the workspace build (Build release 5:03 -> 7:51, 4:14 -> 8:07, 4:54 -> 5:56); building `-p extenddb` there, as the Dockerfile does, is a workspace-wide CI change outside this crate. Signed-off-by: Scott Robinson <robinnsc@amazon.com> Co-authored-by: Lee Hannigan <lhnng@amazon.com>

…hash, typed backup ARN (ExtendDB#394) Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from ExtendDB#337's draft with the changes the design review asked for before v1 freezes. The module doc says that the design overview (docs/design/15-backup-restore.md, ExtendDB#396) states the rules the format's writers and readers honour, and that this module's docs and the testdata README are the record of the format itself. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES and read into one buffer the decoder reuses across items, as the encoder reuses its own; it keeps the longest line's capacity, at most the cap. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in ` at line 3 column 4` sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. verified() keeps the slice path that checks digests before the first item, and then decodes without hashing again. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling a download or over a file in place under an owner-only store root; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read that source without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding) and keep every size, count and line check; the digests are not compared again, so a file replaced by one of another length is refused by the compressed-size check and a same-length replacement is not detected. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value rendered it, so a message ending in ` at line 0 column 0` loses one such suffix per level as before (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins thirteen nested texts, seven of that class at one to three levels with main's output, one of them with a suffix more than the levels strip). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. validation/number.rs (main's code, which every format parser and the request path reach) does its exponent arithmetic in i64. An exponent at either end of i32 combined with the digits no longer overflows: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse. An invalid N key's InvalidKey message carries the number validation's reason. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (written by BackupDocuments::build, checked by verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. verify_manifest_checksum takes the sidecar as the bytes the store returned and accepts one shape only, the line the writer stores, "<64 lowercase hex> extenddb-manifest.json", with or without its final \n. Bytes that are not UTF-8 (refused with their own reason, the bytes not shown), a bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; table_creation_date_time finite, since JSON has no NaN or infinity and serde_json writes them as null; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). BackupDocuments::build runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, under 10% of the cost, once for the members); the doc states the cost per data file: about 6 to 12 µs and 1.6 KB retained per file with short keys. A key_range endpoint naming an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files. ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an item whose key KeyOrder::endpoint_key refuses, and an empty file; with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs one tracker over the items of every file it encodes, so the endpoint check and the strict-ascent rule exist once: next_file closes a file with its KeyRange and keeps its last key, so a boundary that does not ascend (the next file starting with that key or one before it) is refused at the next file's first item, naming the previous file's last key, with nothing of that file encoded; finish closes the last file. With one tracker per file the same boundary is found only at build, after every file is stored. The tracker checks each key in place and keeps the first and latest keys, with an N range key parsed once; the check parses the recorded endpoints once. Per 318-byte item the check went from 1.07 µs and 11.5 allocations to 0.66 µs and 5 (one partition: 1.64 µs and 19.3 to 0.69 µs and 5.3). - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules (the exponent a signed 32-bit integer), the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. It states which writers store an invalid N key as written (BatchWriteItem and TransactWriteItems, ExtendDB#618; UpdateItem on an absent item and ImportTable run no number check on the key) and the rule for a key-ordered writer that meets one: the backup is refused with an error naming the key, through KeyRangeTracker::push, the one call per item, which checks the key as KeyOrder::endpoint_key does. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: an OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled stream specification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. A new member omitted when absent leaves the bytes unchanged, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes, and fails it until the list changes. A new or renamed enum value that no fixture uses changes no byte either, so enum_values_are_pinned pins the values of every enum in the manifest: the derived ones through serde's variant list, and KeyType, ScalarAttributeType and DistanceFunction, whose readers are hand-written, through tables filled from a match with no wildcard, and AttributeValue, the value type of every key_range endpoint and data-file line, through its ten type descriptors the same way, plus a check that the descriptors its reader accepts among every string of one to four upper-case letters are exactly those ten. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag and s3SseAlgorithm are Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file> for a full export to a prefix, measured; the documentation's full-export example omits the prefix; the module doc states both and what ExtendDB writes). dataFileS3Key and manifestFilesS3Key are built by one function, so an empty prefix gives data/<file> and manifest-files.json, without a leading /. Every leading and trailing / of the prefix is dropped once, so a// and /a/ give a/data/<file> and a/manifest-files.json, as a does; the store drops a leading /, one trailing / and refuses an empty component, so the two rules agree on every prefix the store accepts. DataFileEntry::from_checksums builds each entry from the encoder's result. BackupDocuments::build(header, data_files, store_key_prefix, bucket), with each data file paired with the ETag its put returned (None on a filesystem store), assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix with leading and trailing / dropped, then BackupArn::store_prefix) once and runs validate once. It builds each manifest-files.json line from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the summary from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table). Its doc gives the write order: documents_before_manifest() returns manifest-files.json, manifest-summary.json and the sidecar, the job re-checks its claim, and manifest_document() is the manifest, written last; documents() is the four in that order. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are private to backup_format. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 175 in the module (174 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals (a repeated key and an invalid stored N key among them) and two non-overlapping files each holding every key, through ExtenddbManifest::reader as well, a first item that is not the range's first and a file with no items through reader, reader_prehashed reading a wrong-digest file that reader refuses, the tracker refusing a boundary that does not ascend at the next file's first item with nothing encoded, the summary's itemCount as the sum when source_table.item_count differs, a non-UTF-8 sidecar, each etag on its file, the three documents before the manifest and the manifest document, the size sum refused at 2^63, the prehashed decoder's uncompressed count one too high, AttributeValue's ten descriptors and no other, the writer's tracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare for S, N and B keys, the prehashed decoder keeping every check but the digest, BackupDocuments' four documents agreeing on one prefix, the pinned enum values, N exponents at both ends of i32, a forged position in a data-line error, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses (non-finite timestamps included), endpoints repeating a type descriptor, the pinned member list of every manifest type, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed); 13 of the fixes for the maintainer's review of 2026-10-28 were killed, none equivalent; 15 of the round-7 fixes were killed, none equivalent; 18 of the round-8 fixes were killed at the test each targets, among them the four that survived round 8 (an AttributeValue descriptor alias, reader_prehashed hashing anyway, the summary's itemCount taken from source_table, build's size sum wrapping), none equivalent. Signed-off-by: Scott Robinson <robinnsc@amazon.com> Co-authored-by: Lee <leeroyhannigan@yahoo.ie>

…ackend (ExtendDB#395) Adds the one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code keeps its own copy of this logic for now; the engine-side backup job (next PR) is the first caller. Stacked on ExtendDB#394: SnapshotInfo is the manifest's SnapshotDescriptor (one definition, with the three KIND_* constants), and key_schema_is_orderable is KeyOrder::resolve, so a stream a backend marks ordered is one the manifest parser accepts and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. Trait (crates/storage/src/snapshot.rs): open_table_snapshot(account_id, table_name) returns TableSnapshot { description: TableDescription, info: SnapshotInfo, key_ordered: bool, items: BoxStream<Result<Item>> }. The description is the storage-level one at the snapshot instant, not the API response (restore_summary may be set, table_throughput_mode_summary is not, the counts are the stats pass's); items are yielded as stored (valid N normalized, invalid raw, TTL-expired unswept items included), and any error means start over, SnapshotExpired being the one worth retrying. The stream owns the backend transaction or session; dropping it releases it. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. The trait doc names what each backend holds while a stream is open and which errors each returns (Cassandra answers Unsupported before it resolves the table), the idle timeouts that end a stream as Internal, and why the trait has no default body and StorageError stays exhaustive; PostgreSQL and SQLite bound open snapshots per server (below) and refuse over the bound with LimitExceeded. StorageError gains SnapshotExpired for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient. snapshot.rs's module doc says that the design overview (docs/design/15-backup-restore.md, ExtendDB#396) states the rules the trait's callers honour and that this file's and each backend's snapshot_engine.rs module docs are the record of what the snapshots do; DEFAULT_FETCH_BATCH's doc says why the size is a constant (over a 1 ms link the larger fetch would save about a second per million rows), and the PostgreSQL module doc's pre-ExtendDB#361 paragraph carries the sort's cost (7 MB on disk for 100k rows at work_mem 4 MB). Indexes: the description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since ExtendDB#401 UpdateTable creates the data table before committing the row, so this is a guard, not an expected state). A restore recreates the indexes in the description and builds them from the items, so a CREATING index with a data table comes back complete; one left out would not come back at all. PostgreSQL, one shared database: the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. Separate data database: the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key. A fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit); the byte budget is the backup job PR's. A table whose text key columns do not have a collation named "C" or "POSIX" (created before the key columns were declared with it; "default" on a C database also counts as not bytewise, which streams unordered and complete) is streamed unordered, with a warning: ordering it would turn the read into a sort of the whole table before the first row. SQLSTATE 40001 (recovery conflict on a standby) and 72000 (snapshot_too_old on a primary, PostgreSQL 16 and earlier) map to SnapshotExpired, pinned by a test that raises each code on the live server. The stream holds ACCESS SHARE on the data table until its last fetch, so a DROP TABLE waits for it; DeleteTable takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that). On both layouts the synchronous path takes every drop's lock before the catalog removal commits. It deletes the table's pending queue rows first, waiting for in-flight claims with no timeout, as ExtendDB#401 intends. It then drops the base, GSI, LSI and vector-index data tables, base first, within one 1 s budget for the group (SYNC_DROP_LOCK_TIMEOUT; before each DROP, lock_timeout is the time left). On the shared layout that runs in the catalog transaction behind a savepoint. With a separate data database it runs in a data transaction opened first, and a task the request awaits commits the catalog and then that transaction, so a cancelled caller cannot stop between the two. A lock wait that fails (55P03, a deadlock victim 40P01, or a statement_timeout's 57014) rolls the drops back and takes the asynchronous path with the row still held. A write transaction, a SHARE holder, an index build or a reader that starts after the probe therefore costs the request at most the 1 s, and no data table is left without its row. With a separate data database a data commit that fails after the catalog commit has the drop retried on a fresh connection for up to 60 s, so a failover, a dropped connection or a network blip no longer orphans the data tables; only a process that dies, or a data database unreachable for that whole window, between the catalog commit and the last retry leaves them without a row, and then they stay until an operator drops the relations the error log names (docs/troubleshooting.md, "Data tables without a catalog row"). The base table goes first because a write holds it before it writes an LSI or a synchronous GSI: with the index tables first, a write in progress waited about 1 s on the drop, and the drop went asynchronous. On the shared layout a DeleteTable probing between an open's snapshot and its LOCK does not see the open; if the LOCK is granted first, the drop times out after 1 s, is rolled back with the removal, and DeleteTable returns DELETING (in 80 unaided open-and-delete pairs, the delete first in half of them, 0 left a data table without a row on either layout; 39 to 40 deletes ran synchronously and 40 to 41 asynchronously). The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred, also after a restart: startup recovery runs the same pass and wakes the poller when it deferred a drop. A drop is deferred when the probe finds another session holding any lock on the table's relations, or when the drop fails in a lock wait (55P03, 40P01 or 57014); any other failure is logged and left to the normal window and the 60 s idle pass. Each pass probes a held table and skips it without waiting, and drops a free one within the same 1 s budget, so every held table is retried about once a second however many are held, and a CreateTable that is due is activated on the same cadence (1.1 s with four drops deferred). A drop that DeleteTable deferred is due at once, not after control_plane_delay_seconds. The active loop waits on the wake as well as on its timer, so a snapshot's or an in-catalog backup's release has the drop retried at once, and any holder's release is followed within about a second: 108 to 166 ms in the tests, with a 10 s test-only active poll for the wake cases. The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the default (0.25 on every backend), the bound and the retry cadence, the same-name CreateTable refusal until then, and a two-step query, over every relation of the table, that finds any session holding a lock on them and its lock mode. A DeleteTable of a table that is already DELETING returns its description and changes nothing. SQLite: one deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened and the WAL grows until the stream ends (a TRUNCATE checkpoint waits out the busy timeout and reports busy; the module doc gives the measured reply). ExtendDB#384's create_backup_from_wal_snapshot holds the writer lock while it pins because it also inserts the CREATING row; this path writes nothing. build_table_description_on reads the definition, including ExtendDB#384's restore_summary from table_restores, on that transaction. The stream holds a backup slot (backup_server_slots, shared with the in-catalog backup) until its read transaction ends (the short page that shows the end, or the first error) or it is dropped, so the pool arithmetic that keeps half the pool free counts snapshots; with no slot free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them. The pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index, so key_ordered is true for single-range-key tables; multi-part keys fall back to rowid order. A page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it has yielded its last item or is dropped, so the slots bound how many decoded copies exist, as ExtendDB#384's create_backup_single_connection does for the in-catalog backup. SQLite DeleteTable of a DELETING table (control_plane_delay_seconds of 1 or more) returns its description too. Cassandra: open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog BackupEngine path stays its backup path. A direct test asserts the refusal naming the backend on an existing table and a missing one. MongoDB: a session started with snapshot=true reads the tables document, the indexes documents (doc_to_table_description_in now takes an optional session) and the base collection at one cluster time; the marker is that time. Every read sets readConcern: snapshot explicitly. The production catalog handle carries readConcern: local (as a URI with readConcernLevel= would), and on it a read without the level fails: the first read reports no snapshot time, which the open refuses naming the read concern, and a later read in the pinned session is refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported; each of the two refusals names its own cause. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired, pinned by a unit test that builds the command error from a reply document; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes, CursorNotFound mapped to Internal). Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. key_ordered is false: for a composite key _id is a netstring of the key text, which is not the format's order for any key type. Test hooks (extenddb_storage::snapshot::paging): a page-size knob, a round-trip counter, and a pause at two points of the open (AfterDefinition, BeforeItems) that the conformance check holds while it commits writes. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with the test-hooks feature of extenddb-storage, which the test-hooks features of storage-postgres and storage-sqlite and the new snapshot-test-hooks of storage-mongodb enable (storage-mongodb's existing test-hooks, the integration binary's coordination hooks, is unchanged). The backend crates' tests enable them through a dev-dependency on themselves. Production builds carry neither the hooks nor the harness. PostgreSQL also gains test-hooks entries spawn_control_plane_poller_for_tests and spawn_control_plane_poller_with_active_poll_for_tests (the production poller, the second with its active retry interval as a parameter), recover_control_plane_at_startup_for_tests, control_plane_drop_attempts_for_tests (a per-engine count of drop attempts, a held table the probe skipped counted), arm_delete_table_pause_at (holds DeleteTable at BeforeDrops, between its probe and its drops, or at BetweenCommits, between the catalog commit and the data commit on the separate layout; arm_delete_table_pause is the BeforeDrops form), shorten_data_drop_retry_bound_for_tests (a process-global shorter bound for the retried data drop), and re-exports of the data relation names and of apply_all_migrations, so tests/table_snapshot.rs binds the crate's lists instead of copying them. The two pauses (the open's and DeleteTable's) are one generic type, extenddb_storage::test_pause::Pause, and snapshot_conformance::PageSize is the one page-size guard the three backends' tests use. tests/table_snapshot.rs fails instead of skipping without a connection string when EXTENDDB_TEST_PG_REQUIRED is set, which integration.yml sets. Tests: extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared page size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open (so a backend that ignored the knob, or a MongoDB cursor without batch_size, fails here instead of passing vacuously), then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. Asserts: 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, the counter advanced after the writer committed (a stream that read every page during the open fails), key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare (the format's comparator) when ordered, and a second snapshot taken after the writer stops shows the final round. Then, once per pause point, on a fresh table: with the open held, it commits W1, an UpdateTable adding a GSI, and W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the new GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts the snapshot's description must equal the storage layer's describe_table with the running counts and table_throughput_mode_summary cleared; restore_summary is compared, and each backend's suite also compares a table given a restore record (assert_snapshot_describes_the_table). Every drain is under a 120 s deadline naming the stage and the items read, and a test waiting for the hooks lock gives up when one holder has kept it 600 s, naming the wait. READ COMMITTED or no FOR SHARE on PostgreSQL, the data cursor or the index read outside the session on MongoDB, and the pauses moved outside the read window on SQLite or MongoDB, fail it, as do the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion), a description built from a second code path (the DescribeTable comparison), and a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3 with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. The suites pass on PostgreSQL 16 (both layouts), file-backed SQLite and MongoDB 7 (replica set; the standalone refusal against a standalone mongod), and the Cassandra test on Cassandra 4.1. PG: tests/table_snapshot.rs (81 tests and two ignored race measurements: consistency, marker and key order, tables with legacy-collation key columns streaming unordered and complete on a scratch database whose collation is checked not to be C, five held snapshots on a ten-connection pool leaving GetItem and DescribeTable unblocked with the sixth LimitExceeded and an open succeeding after one is dropped, a GSI without its data table left out with its attribute, DeleteTable during a snapshot returning DELETING, with the production poller at a 10 s active poll the table gone within 3 s of a release during the pause after the first drop attempt (for a dropped stream, a drained stream, a fetch error, an open cancelled at BeforeItems and the in-catalog backup), for a plain reader holding ACCESS SHARE that sends no wake, held 10 s past the poller's window, the table gone within 3 s of its release, a reader across a restart (the table gone within 3 s of its release), a drop failing on a dependent view making at most 3 attempts in the 15 s after the window, a restored table's restore_summary in the snapshot description, DeleteTable with only a GSI or vector-index data table locked taking the async path, an unparseable row ending the stream and freeing its permit, twenty concurrent DeleteTables on a ten-connection pool, an open completing with nine of ten connections held, an open held while nine DeleteTables queue, DeleteTable at BeforeItems taking the async path, a fetch error freeing the permit, a drained stream out of its transaction before its end is polled, a ROW EXCLUSIVE and a SHARE holder sending DeleteTable down the asynchronous path within about 1 s with no data table left without its row, a DeleteTable cancelled during that wait leaving the table ACTIVE with its data, write holders on the base and three GSI tables released 0.8 s apart taking it there at about 1 s, a write holding the base and then writing the GSI table finishing first with the table removed in the request, a claimed queue row holding DeleteTable until it ends, a probed reader sending it asynchronous in milliseconds, a statement timeout and a deadlock victim taking the asynchronous path, DeleteTable of a DELETING table returning its description and changing nothing, a deferred drop due at once, four held tables probed with no lock wait (the first released gone within 2 s, a CreateTable ACTIVE within 2 s), all on both pool layouts; on the separate layout, a data commit that fails after the catalog commit (its backend terminated between the two) retried until the drop goes through with the request returning DELETING and no relation left, a data database refusing connections past a shortened bound returning Internal naming every leaked relation with the row gone, and a caller cancelled between the two commits leaving no orphan; on the shared layout, a snapshot open or a GSI-only reader taking its lock after DeleteTable's probe sending it down the asynchronous path with the row and both data tables kept, and a table dropped before the shared-layout LOCK being TableNotFound; the SQLSTATE mapping; the two ignored race measurements, which delay the delete or the open in alternate pairs and assert both outcomes occurred) with a Drop guard that drops the scratch databases on success or panic, added to the direct-storage step in integration.yml; the full storage-postgres suite is 259 tests. SQLite: in-module tests (consistency, key order, the in-memory copy releasing the connection, in-memory copies holding their slot, a damaged attribute definition streaming unordered and complete, the rowid fallback at a page of seven with no duplicates and at least six fetches, slots shared with CreateBackup and its refusal text, the WAL checkpoint reply while a snapshot is held, a restored table's restore_summary compared on a file-backed and an in-memory engine, a drained stream and a stream that hit an error freeing their slot while still held, and an in-memory copy freeing its slot with its last item) with the same guard for the scratch files, created under the test binary's target directory. MongoDB: in-module tests (consistency, an undecodable document ending the stream, reads without the snapshot level failing with code 72 and the error text, the code mapping, a restored table's restore fields compared, the round-trip counter counting only the snapshot stream (a GSI backfill leaves it unchanged), and a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod container it starts (and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it)) on per-test scratch databases built without the bootstrapper, as ExtendDB#384's live tests do, the catalog handle with the production option set, dropped by a guard on success or panic, so a URI in the environment never makes cargo test touch another instance's databases; gated on EXTENDDB_TEST_MONGODB_URI (failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, which the CI job does), holding the snapshot hooks lock, run by a new integration-mongodb.yml job that starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod (both on 127.0.0.1 only). tests/test_table_operations.py: DeleteTable of a DELETING table returns its description, on PostgreSQL and SQLite (the state written into the catalog), skipped on MongoDB and Cassandra, which never leave a table DELETING. Signed-off-by: Scott Robinson <robinnsc@amazon.com>

robinnsc requested review from LeeroyHannigan, amrith, c33howard, jcshepherd, pdf-amzn and yesyayen as code owners October 13, 2026 03:25

robinnsc mentioned this pull request Oct 13, 2026

docs(design): add the backup and restore design #348

Closed

6 of 8 tasks

robinnsc changed the title ~~docs(design): revise the backup and restore design against the current code~~ docs(backup): design revision for off-host backup and restore [backup/restore 1/4] Oct 13, 2026

robinnsc force-pushed the docs/backup-restore-design-revised branch from d843f33 to 95fec9a Compare October 13, 2026 06:50

robinnsc mentioned this pull request Oct 16, 2026

fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384

Merged

5 of 8 tasks

LeeroyHannigan reviewed Oct 24, 2026

View reviewed changes

robinnsc force-pushed the docs/backup-restore-design-revised branch 3 times, most recently from 7ce3b1b to 3d6b4e9 Compare October 26, 2026 20:02

LeeroyHannigan reviewed Oct 27, 2026

View reviewed changes

robinnsc force-pushed the docs/backup-restore-design-revised branch 2 times, most recently from 1e798df to 7d9c72d Compare October 28, 2026 00:02

navareamol assigned robinnsc Oct 28, 2026

navareamol added this to the 1.0 milestone Oct 28, 2026

This was referenced Oct 28, 2026

Off-host backup and restore (tracking) #624

Open

Backup/restore: design document #625

Closed

robinnsc force-pushed the docs/backup-restore-design-revised branch from 7d9c72d to c56d274 Compare October 28, 2026 06:31

robinnsc requested a review from LeeroyHannigan October 28, 2026 07:58

LeeroyHannigan reviewed Oct 28, 2026

View reviewed changes

LeeroyHannigan added the v1 Needed for the 1.0 release label Oct 28, 2026

robinnsc force-pushed the docs/backup-restore-design-revised branch from 166ade8 to 7cbbbfb Compare October 29, 2026 11:28

LeeroyHannigan reviewed Oct 29, 2026

View reviewed changes

LeeroyHannigan approved these changes Oct 29, 2026

View reviewed changes

robinnsc added this pull request to the merge queue Oct 29, 2026

Merged via the queue into main with commit e75acf7 Oct 29, 2026
27 checks passed

robinnsc deleted the docs/backup-restore-design-revised branch October 29, 2026 17:51

This was referenced Oct 30, 2026

feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671

Merged

feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672

Merged

This was referenced Oct 31, 2026

docs: add the documentation site #491

Open

docs: architecture dash sweep, one feature status owner, one affiliation statement #530

Open

This was referenced Oct 31, 2026

Backup/restore: CreateBackup job, worker and reconciler duties #699

Open

feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700

Open

LeeroyHannigan mentioned this pull request Nov 1, 2026

fix(backup): report point-in-time recovery as unsupported #335

Merged

8 tasks done

navareamol added the release-blocker Must ship for its milestone's release label Nov 3, 2026

This was referenced Nov 4, 2026

feat(backup): backup sync, --adopt-store and --prune-missing [backup/restore 10] #825

Draft

docs(backup): the backup and restore manual, the open measurements and the CI filters [backup/restore 11] #855

Draft

Conversation

robinnsc commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing done

Not changed here

Checklist

Breaking changes

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robinnsc commented Oct 29, 2026

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

robinnsc commented Oct 13, 2026 •

edited

Loading