From a317b99c181ca137d9ccf7363625986fd3baedd5 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Wed, 4 Nov 2026 09:49:56 +0000 Subject: [PATCH 1/3] feat(backup): [backup.store], the store identity file, backup purge The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df3: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee Signed-off-by: Scott Robinson --- .github/workflows/integration.yml | 26 + Cargo.lock | 10 + SOFTWARE-LICENSE-NOTICES-CASSANDRA.html | 1161 +++++- SOFTWARE-LICENSE-NOTICES-DEV.html | 1190 +++++- SOFTWARE-LICENSE-NOTICES-MONGODB.html | 936 ++++- SOFTWARE-LICENSE-NOTICES.html | 1189 +++++- crates/app/Cargo.toml | 2 + crates/app/src/cmd_backup.rs | 927 +++++ crates/app/src/cmd_destroy.rs | 14 +- crates/app/src/cmd_serve.rs | 19 +- crates/app/src/init_helpers.rs | 25 +- crates/app/src/lib.rs | 97 +- crates/backup-store/README.md | 20 + crates/backup-store/src/config.rs | 34 +- crates/backup-store/src/fs.rs | 321 +- crates/backup-store/src/lib.rs | 77 +- crates/backup-store/src/s3.rs | 442 ++- crates/backup-store/tests/conformance.rs | 535 +++ crates/config/Cargo.toml | 4 + crates/config/src/backup.rs | 674 +++- crates/config/src/lib.rs | 15 +- crates/engine/Cargo.toml | 5 + crates/engine/src/backup.rs | 189 +- crates/engine/src/backup/maintenance.rs | 648 ++++ crates/engine/src/backup/reconciler.rs | 242 +- crates/engine/src/backup/runtime.rs | 856 ++++- crates/engine/src/backup/store_identity.rs | 664 ++++ crates/engine/src/backup/tests.rs | 3373 ++++++++++++++++- crates/engine/src/lib.rs | 6 +- crates/server/Cargo.toml | 3 + crates/server/src/backup_store.rs | 1468 +++++++ crates/server/src/lib.rs | 40 +- crates/server/src/serve.rs | 87 +- .../storage-cassandra/src/backup_catalog.rs | 568 ++- .../storage-cassandra/src/cassandra_util.rs | 33 + crates/storage-cassandra/src/lib.rs | 22 +- .../storage-cassandra/src/migrations/mod.rs | 54 +- .../src/migrations/registrations.rs | 72 +- crates/storage-cassandra/src/test_hooks.rs | 53 + .../tests/direct/backup_engine.rs | 648 +++- .../tests/migrations/lock.rs | 92 +- crates/storage-mongodb/src/backup_catalog.rs | 382 +- .../storage-mongodb/src/backup_live_tests.rs | 556 ++- .../src/data_engine_key_repair.rs | 3 +- .../src/data_engine_live_tests_repair_pass.rs | 64 + crates/storage-mongodb/src/lib.rs | 45 +- crates/storage-mongodb/src/migrations/mod.rs | 22 +- .../src/migrations/registrar.rs | 25 +- .../src/migrations/registrations.rs | 42 +- .../storage-mongodb/tests/migrations_locks.rs | 70 +- crates/storage-postgres/Cargo.toml | 5 + crates/storage-postgres/src/backup_catalog.rs | 399 +- crates/storage-postgres/src/backup_engine.rs | 39 +- crates/storage-postgres/src/lib.rs | 53 +- .../storage-postgres/src/migrations/locks.rs | 38 +- crates/storage-postgres/src/migrations/mod.rs | 2 +- .../src/migrations/registrar.rs | 67 +- .../storage-postgres/tests/backup_restore.rs | 1226 +++++- .../tests/migrations_serving.rs | 63 +- crates/storage-sqlite/src/backup.rs | 507 ++- crates/storage-sqlite/src/backup_catalog.rs | 393 +- crates/storage-sqlite/src/config.rs | 4 + crates/storage-sqlite/src/lib.rs | 246 +- crates/storage/src/backup_catalog.rs | 120 +- crates/storage/src/config.rs | 10 + crates/storage/src/lib.rs | 4 +- crates/storage/src/server_components.rs | 30 + devtools/catalog_checks.py | 19 +- devtools/check-cassandra-catalog-lock | 53 +- devtools/check-mongodb-catalog-lock | 29 +- devtools/check-postgres-catalog-lock | 42 +- devtools/run-tests | 39 +- docs/design/13-storage-mongodb.md | 7 + docs/design/15-backup-restore.md | 39 +- docs/differences-from-dynamodb.md | 2 + docs/dynamodb-limits.md | 3 +- docs/manuals/05-admin-guide.md | 57 +- docs/manuals/07-upgrade-manual.md | 4 +- docs/manuals/11-deployment-guide.md | 3 +- extenddb.sample.toml | 25 +- tests/test_backup_store_config.py | 1519 ++++++++ 81 files changed, 22141 insertions(+), 956 deletions(-) create mode 100644 crates/app/src/cmd_backup.rs create mode 100644 crates/engine/src/backup/maintenance.rs create mode 100644 crates/engine/src/backup/store_identity.rs create mode 100644 crates/server/src/backup_store.rs create mode 100644 tests/test_backup_store_config.py diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 7fd08685e..b9acb4b03 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -246,11 +246,33 @@ jobs: - name: Install Python dependencies run: pip install -r requirements.txt + # Local S3 endpoint for tests/test_backup_store_config.py's S3 cells + # (an S3 store's /health against an endpoint that refuses, stalls and + # recovers), which devtools/run-tests runs on a SQLite build after the + # main suite. Same moto as the unit-test workflow, pinned in + # requirements-moto.txt, installed into a virtualenv of its own: its + # boto3 and botocore pins differ from requirements.txt's, and the + # suite keeps running on the environment installed above. On port + # 9200, clear of the server's. + - name: Start S3 mock + run: | + python -m venv "$RUNNER_TEMP/moto-venv" + "$RUNNER_TEMP/moto-venv/bin/pip" install -r requirements-moto.txt + nohup "$RUNNER_TEMP/moto-venv/bin/moto_server" -p 9200 > moto.log 2>&1 & + for _ in $(seq 1 30); do + if curl -sf http://127.0.0.1:9200/ > /dev/null; then exit 0; fi + sleep 1 + done + echo "moto_server did not start"; cat moto.log; exit 1 + - name: Run integration tests env: EXTENDDB_TEST_ENDPOINT: https://127.0.0.1:18443 EXTENDDB_ADMIN_USER: admin EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} + # The backup store wire tests' S3 cells; the file sets the AWS_* + # variables for the servers it starts, so none leaks here. + EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9200 run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" # Same rationale as the postgres job: the daemonized server logs to @@ -259,6 +281,10 @@ jobs: if: failure() run: sudo journalctl -t extenddb --no-pager -n 500 || true + - name: Print the S3 mock's log + if: failure() + run: cat moto.log || true + run-integration-dev-mode: # dev-mode was shipped with no CI coverage at all, which is how the batch and # transaction authorization regression reached main: the build compiled, so diff --git a/Cargo.lock b/Cargo.lock index c033bcd24..b836d650e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1660,11 +1660,13 @@ dependencies = [ "clap", "daemonize", "extenddb-auth", + "extenddb-backup-store", "extenddb-config", "extenddb-core", "extenddb-engine", "extenddb-server", "extenddb-storage", + "futures", "libc", "rcgen", "rustls", @@ -1740,9 +1742,11 @@ version = "0.1.14" dependencies = [ "anyhow", "config", + "extenddb-backup-store", "extenddb-core", "extenddb-storage", "serde", + "tempfile", "toml", "tracing", ] @@ -1770,6 +1774,7 @@ dependencies = [ "base64 0.22.1", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-core", "extenddb-storage", "futures", @@ -1795,6 +1800,7 @@ dependencies = [ "bcrypt", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-cache", "extenddb-config", "extenddb-core", @@ -1810,6 +1816,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "syslog-tracing", + "tempfile", "time", "tokio", "tower", @@ -1913,7 +1920,9 @@ dependencies = [ "bcrypt", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-core", + "extenddb-engine", "extenddb-storage", "extenddb-storage-postgres", "futures", @@ -1923,6 +1932,7 @@ dependencies = [ "serde", "serde_json", "sqlx", + "tempfile", "time", "tokio", "toml", diff --git a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html index 80174e5eb..8a5f1b144 100644 --- a/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html +++ b/SOFTWARE-LICENSE-NOTICES-CASSANDRA.html @@ -49,14 +49,14 @@

ExtendDB Software License Notices

Overview of licenses:

All license text:

@@ -1741,6 +1741,424 @@

Used by:

See the License for the specific language governing permissions and limitations under the License. + + +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2017 Juniper Networks, Inc.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
     
  • @@ -1767,7 +2185,220 @@

    Used by:

  • toml_edit 0.22.27
  • toml_write 0.1.2
  • -
                                     Apache License
    +                
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    + +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
                                Version 2.0, January 2004
                             http://www.apache.org/licenses/
     
    @@ -1955,7 +2586,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1975,10 +2606,227 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
    +
    +Copyright [yyyy] [name of copyright owner]
    +
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
    +
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -2165,7 +3013,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2184,15 +3033,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2382,8 +3223,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2402,7 +3242,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2592,7 +3432,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2604,14 +3444,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2801,7 +3640,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2813,13 +3652,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3009,7 +3850,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3028,7 +3869,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3218,7 +4059,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3237,7 +4078,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3427,7 +4268,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4503,6 +5344,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4521,15 +5363,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4544,6 +5389,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4554,9 +5400,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4574,6 +5423,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -5420,6 +6270,7 @@ 

    Used by:

  • hmac 0.12.1
  • inout 0.1.4
  • md-5 0.10.6
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  •                               Apache License
    @@ -6950,6 +7801,7 @@ 

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -7951,6 +8841,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -8038,6 +8985,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8355,6 +9335,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8564,6 +9572,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8597,7 +9633,9 @@

    Used by:

    MIT License

    Used by:

    MIT License
     
    @@ -8736,6 +9774,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -9107,6 +10173,34 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
  • Unicode License v3

    @@ -9231,6 +10325,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES-DEV.html b/SOFTWARE-LICENSE-NOTICES-DEV.html
    index 9ff5a037d..1aa37ca78 100644
    --- a/SOFTWARE-LICENSE-NOTICES-DEV.html
    +++ b/SOFTWARE-LICENSE-NOTICES-DEV.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1537,25 +1537,216 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                     Apache License
                                Version 2.0, January 2004
    @@ -1745,30 +1936,687 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2017 Juniper Networks, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
     
    -   Licensed under the Apache License, Version 2.0 (the "License");
    -   you may not use this file except in compliance with the License.
    -   You may obtain a copy of the License at
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
     
    -       http://www.apache.org/licenses/LICENSE-2.0
    +Copyright [yyyy] [name of copyright owner]
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -1955,7 +2803,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1974,15 +2823,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2172,8 +3013,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2192,7 +3032,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2382,7 +3222,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2394,14 +3234,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2591,7 +3430,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2603,13 +3442,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2799,7 +3640,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2818,7 +3659,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3008,7 +3849,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3027,7 +3868,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3217,7 +4058,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4083,6 +4924,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4100,15 +4942,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4121,6 +4966,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4131,8 +4977,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • +
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4148,6 +4998,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -4992,6 +5843,7 @@ 

    Used by:

  • hmac 0.12.1
  • inout 0.1.4
  • md-5 0.10.6
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  •                               Apache License
    @@ -6870,6 +7722,7 @@ 

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -7608,6 +8500,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7695,6 +8644,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8012,6 +8994,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8226,7 +9236,37 @@

    Used by:

    MIT License

    Used by:

    +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    +
    MIT License
     
    @@ -8370,6 +9410,7 @@ 

    Used by:

    MIT License

    Used by:

    The MIT License (MIT)
    @@ -8646,6 +9687,34 @@ 

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
  • Unicode License v3

    @@ -8770,6 +9839,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES-MONGODB.html b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    index 6f1570169..7920a20c8 100644
    --- a/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    +++ b/SOFTWARE-LICENSE-NOTICES-MONGODB.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1534,6 +1534,215 @@

    Used by:

    See the License for the specific language governing permissions and limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
     
  • @@ -2177,24 +2386,454 @@

    Used by:

    you may not use this file except in compliance with the License. You may obtain a copy of the License at - https://www.apache.org/licenses/LICENSE-2.0 + https://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
    +
    +Copyright [yyyy] [name of copyright owner]
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
    +
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -2381,7 +3020,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2400,15 +3040,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2598,8 +3230,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2618,7 +3249,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2808,7 +3439,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2820,14 +3451,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3017,7 +3647,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3029,13 +3659,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3225,7 +3857,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3244,7 +3876,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3434,7 +4066,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3453,7 +4085,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3643,7 +4275,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4509,6 +5141,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4528,15 +5161,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4549,6 +5185,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4559,10 +5196,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • ring 0.17.14
  • ron 0.8.1
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • serde_with 3.21.0
  • @@ -4583,6 +5222,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -5437,6 +6077,7 @@ 

    Used by:

  • md-5 0.10.6
  • md-5 0.11.0
  • pbkdf2 0.13.0
  • +
  • sha1 0.10.6
  • sha1 0.11.0
  • sha2 0.10.9
  • sha2 0.11.0
  • @@ -6899,6 +7540,7 @@

    Used by:

    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7724,6 +8440,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -8068,6 +8817,34 @@

    Used by:

    DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8454,6 +9231,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8487,7 +9292,9 @@

    Used by:

    MIT License

    Used by:

    MIT License
     
    @@ -8626,6 +9433,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -9027,6 +9862,34 @@

    Used by:

    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    © 2016 Bertram Truong
    +© 2021 Kornel Lesiński
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
     
  • @@ -9152,6 +10015,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/SOFTWARE-LICENSE-NOTICES.html b/SOFTWARE-LICENSE-NOTICES.html
    index 47ee4fe1c..9103a0534 100644
    --- a/SOFTWARE-LICENSE-NOTICES.html
    +++ b/SOFTWARE-LICENSE-NOTICES.html
    @@ -49,14 +49,14 @@ 

    ExtendDB Software License Notices

    Overview of licenses:

    All license text:

    @@ -1538,25 +1538,216 @@

    Used by:

    Apache License 2.0

    Used by:

    +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "[]"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright [yyyy] [name of copyright owner]
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    +
                                     Apache License
                                Version 2.0, January 2004
    @@ -1746,30 +1937,687 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2017 Juniper Networks, Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                     Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright {yyyy} {name of copyright owner}
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                    Apache License
    +                           Version 2.0, January 2004
    +                        http://www.apache.org/licenses/
    +
    +   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +   1. Definitions.
    +
    +      "License" shall mean the terms and conditions for use, reproduction,
    +      and distribution as defined by Sections 1 through 9 of this document.
    +
    +      "Licensor" shall mean the copyright owner or entity authorized by
    +      the copyright owner that is granting the License.
    +
    +      "Legal Entity" shall mean the union of the acting entity and all
    +      other entities that control, are controlled by, or are under common
    +      control with that entity. For the purposes of this definition,
    +      "control" means (i) the power, direct or indirect, to cause the
    +      direction or management of such entity, whether by contract or
    +      otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +      outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +      "You" (or "Your") shall mean an individual or Legal Entity
    +      exercising permissions granted by this License.
    +
    +      "Source" form shall mean the preferred form for making modifications,
    +      including but not limited to software source code, documentation
    +      source, and configuration files.
    +
    +      "Object" form shall mean any form resulting from mechanical
    +      transformation or translation of a Source form, including but
    +      not limited to compiled object code, generated documentation,
    +      and conversions to other media types.
    +
    +      "Work" shall mean the work of authorship, whether in Source or
    +      Object form, made available under the License, as indicated by a
    +      copyright notice that is included in or attached to the work
    +      (an example is provided in the Appendix below).
    +
    +      "Derivative Works" shall mean any work, whether in Source or Object
    +      form, that is based on (or derived from) the Work and for which the
    +      editorial revisions, annotations, elaborations, or other modifications
    +      represent, as a whole, an original work of authorship. For the purposes
    +      of this License, Derivative Works shall not include works that remain
    +      separable from, or merely link (or bind by name) to the interfaces of,
    +      the Work and Derivative Works thereof.
    +
    +      "Contribution" shall mean any work of authorship, including
    +      the original version of the Work and any modifications or additions
    +      to that Work or Derivative Works thereof, that is intentionally
    +      submitted to Licensor for inclusion in the Work by the copyright owner
    +      or by an individual or Legal Entity authorized to submit on behalf of
    +      the copyright owner. For the purposes of this definition, "submitted"
    +      means any form of electronic, verbal, or written communication sent
    +      to the Licensor or its representatives, including but not limited to
    +      communication on electronic mailing lists, source code control systems,
    +      and issue tracking systems that are managed by, or on behalf of, the
    +      Licensor for the purpose of discussing and improving the Work, but
    +      excluding communication that is conspicuously marked or otherwise
    +      designated in writing by the copyright owner as "Not a Contribution."
    +
    +      "Contributor" shall mean Licensor and any individual or Legal Entity
    +      on behalf of whom a Contribution has been received by Licensor and
    +      subsequently incorporated within the Work.
    +
    +   2. Grant of Copyright License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      copyright license to reproduce, prepare Derivative Works of,
    +      publicly display, publicly perform, sublicense, and distribute the
    +      Work and such Derivative Works in Source or Object form.
    +
    +   3. Grant of Patent License. Subject to the terms and conditions of
    +      this License, each Contributor hereby grants to You a perpetual,
    +      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +      (except as stated in this section) patent license to make, have made,
    +      use, offer to sell, sell, import, and otherwise transfer the Work,
    +      where such license applies only to those patent claims licensable
    +      by such Contributor that are necessarily infringed by their
    +      Contribution(s) alone or by combination of their Contribution(s)
    +      with the Work to which such Contribution(s) was submitted. If You
    +      institute patent litigation against any entity (including a
    +      cross-claim or counterclaim in a lawsuit) alleging that the Work
    +      or a Contribution incorporated within the Work constitutes direct
    +      or contributory patent infringement, then any patent licenses
    +      granted to You under this License for that Work shall terminate
    +      as of the date such litigation is filed.
    +
    +   4. Redistribution. You may reproduce and distribute copies of the
    +      Work or Derivative Works thereof in any medium, with or without
    +      modifications, and in Source or Object form, provided that You
    +      meet the following conditions:
    +
    +      (a) You must give any other recipients of the Work or
    +          Derivative Works a copy of this License; and
    +
    +      (b) You must cause any modified files to carry prominent notices
    +          stating that You changed the files; and
    +
    +      (c) You must retain, in the Source form of any Derivative Works
    +          that You distribute, all copyright, patent, trademark, and
    +          attribution notices from the Source form of the Work,
    +          excluding those notices that do not pertain to any part of
    +          the Derivative Works; and
    +
    +      (d) If the Work includes a "NOTICE" text file as part of its
    +          distribution, then any Derivative Works that You distribute must
    +          include a readable copy of the attribution notices contained
    +          within such NOTICE file, excluding those notices that do not
    +          pertain to any part of the Derivative Works, in at least one
    +          of the following places: within a NOTICE text file distributed
    +          as part of the Derivative Works; within the Source form or
    +          documentation, if provided along with the Derivative Works; or,
    +          within a display generated by the Derivative Works, if and
    +          wherever such third-party notices normally appear. The contents
    +          of the NOTICE file are for informational purposes only and
    +          do not modify the License. You may add Your own attribution
    +          notices within Derivative Works that You distribute, alongside
    +          or as an addendum to the NOTICE text from the Work, provided
    +          that such additional attribution notices cannot be construed
    +          as modifying the License.
    +
    +      You may add Your own copyright statement to Your modifications and
    +      may provide additional or different license terms and conditions
    +      for use, reproduction, or distribution of Your modifications, or
    +      for any such Derivative Works as a whole, provided Your use,
    +      reproduction, and distribution of the Work otherwise complies with
    +      the conditions stated in this License.
    +
    +   5. Submission of Contributions. Unless You explicitly state otherwise,
    +      any Contribution intentionally submitted for inclusion in the Work
    +      by You to the Licensor shall be under the terms and conditions of
    +      this License, without any additional terms or conditions.
    +      Notwithstanding the above, nothing herein shall supersede or modify
    +      the terms of any separate license agreement you may have executed
    +      with Licensor regarding such Contributions.
    +
    +   6. Trademarks. This License does not grant permission to use the trade
    +      names, trademarks, service marks, or product names of the Licensor,
    +      except as required for reasonable and customary use in describing the
    +      origin of the Work and reproducing the content of the NOTICE file.
    +
    +   7. Disclaimer of Warranty. Unless required by applicable law or
    +      agreed to in writing, Licensor provides the Work (and each
    +      Contributor provides its Contributions) on an "AS IS" BASIS,
    +      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +      implied, including, without limitation, any warranties or conditions
    +      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +      PARTICULAR PURPOSE. You are solely responsible for determining the
    +      appropriateness of using or redistributing the Work and assume any
    +      risks associated with Your exercise of permissions under this License.
    +
    +   8. Limitation of Liability. In no event and under no legal theory,
    +      whether in tort (including negligence), contract, or otherwise,
    +      unless required by applicable law (such as deliberate and grossly
    +      negligent acts) or agreed to in writing, shall any Contributor be
    +      liable to You for damages, including any direct, indirect, special,
    +      incidental, or consequential damages of any character arising as a
    +      result of this License or out of the use or inability to use the
    +      Work (including but not limited to damages for loss of goodwill,
    +      work stoppage, computer failure or malfunction, or any and all
    +      other commercial damages or losses), even if such Contributor
    +      has been advised of the possibility of such damages.
    +
    +   9. Accepting Warranty or Additional Liability. While redistributing
    +      the Work or Derivative Works thereof, You may choose to offer,
    +      and charge a fee for, acceptance of support, warranty, indemnity,
    +      or other liability obligations and/or rights consistent with this
    +      License. However, in accepting such obligations, You may act only
    +      on Your own behalf and on Your sole responsibility, not on behalf
    +      of any other Contributor, and only if You agree to indemnify,
    +      defend, and hold each Contributor harmless for any liability
    +      incurred by, or claims asserted against, such Contributor by reason
    +      of your accepting any such warranty or additional liability.
    +
    +   END OF TERMS AND CONDITIONS
    +
    +   APPENDIX: How to apply the Apache License to your work.
    +
    +      To apply the Apache License to your work, attach the following
    +      boilerplate notice, with the fields enclosed by brackets "{}"
    +      replaced with your own identifying information. (Don't include
    +      the brackets!)  The text should be enclosed in the appropriate
    +      comment syntax for the file format. We also recommend that a
    +      file or class name and description of purpose be included on the
    +      same "printed page" as the copyright notice for easier
    +      identification within third-party archives.
    +
    +   Copyright 2018-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
    +
    +   Licensed under the Apache License, Version 2.0 (the "License");
    +   you may not use this file except in compliance with the License.
    +   You may obtain a copy of the License at
    +
    +       http://www.apache.org/licenses/LICENSE-2.0
    +
    +   Unless required by applicable law or agreed to in writing, software
    +   distributed under the License is distributed on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +   See the License for the specific language governing permissions and
    +   limitations under the License.
    +
    +
    +
  • +
  • +

    Apache License 2.0

    +

    Used by:

    + +
                                  Apache License
    +                        Version 2.0 January 2004
    +                     http://www.apache.org/licenses/
    +
    +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    +
    +1. Definitions.
    +
    +   "License" shall mean the terms and conditions for use, reproduction,
    +   and distribution as defined by Sections 1 through 9 of this document.
    +
    +   "Licensor" shall mean the copyright owner or entity authorized by
    +   the copyright owner that is granting the License.
    +
    +   "Legal Entity" shall mean the union of the acting entity and all
    +   other entities that control, are controlled by, or are under common
    +   control with that entity. For the purposes of this definition,
    +   "control" means (i) the power, direct or indirect, to cause the
    +   direction or management of such entity, whether by contract or
    +   otherwise, or (ii) ownership of fifty percent (50%) or more of the
    +   outstanding shares, or (iii) beneficial ownership of such entity.
    +
    +   "You" (or "Your") shall mean an individual or Legal Entity
    +   exercising permissions granted by this License.
    +
    +   "Source" form shall mean the preferred form for making modifications,
    +   including but not limited to software source code, documentation
    +   source, and configuration files.
    +
    +   "Object" form shall mean any form resulting from mechanical
    +   transformation or translation of a Source form, including but
    +   not limited to compiled object code, generated documentation,
    +   and conversions to other media types.
    +
    +   "Work" shall mean the work of authorship, whether in Source or
    +   Object form, made available under the License, as indicated by a
    +   copyright notice that is included in or attached to the work
    +   (an example is provided in the Appendix below).
    +
    +   "Derivative Works" shall mean any work, whether in Source or Object
    +   form, that is based on (or derived from) the Work and for which the
    +   editorial revisions, annotations, elaborations, or other modifications
    +   represent, as a whole, an original work of authorship. For the purposes
    +   of this License, Derivative Works shall not include works that remain
    +   separable from, or merely link (or bind by name) to the interfaces of,
    +   the Work and Derivative Works thereof.
    +
    +   "Contribution" shall mean any work of authorship, including
    +   the original version of the Work and any modifications or additions
    +   to that Work or Derivative Works thereof, that is intentionally
    +   submitted to Licensor for inclusion in the Work by the copyright owner
    +   or by an individual or Legal Entity authorized to submit on behalf of
    +   the copyright owner. For the purposes of this definition, "submitted"
    +   means any form of electronic, verbal, or written communication sent
    +   to the Licensor or its representatives, including but not limited to
    +   communication on electronic mailing lists, source code control systems,
    +   and issue tracking systems that are managed by, or on behalf of, the
    +   Licensor for the purpose of discussing and improving the Work, but
    +   excluding communication that is conspicuously marked or otherwise
    +   designated in writing by the copyright owner as "Not a Contribution."
    +
    +   "Contributor" shall mean Licensor and any individual or Legal Entity
    +   on behalf of whom a Contribution has been received by Licensor and
    +   subsequently incorporated within the Work.
    +
    +2. Grant of Copyright License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   copyright license to reproduce, prepare Derivative Works of,
    +   publicly display, publicly perform, sublicense, and distribute the
    +   Work and such Derivative Works in Source or Object form.
    +
    +3. Grant of Patent License. Subject to the terms and conditions of
    +   this License, each Contributor hereby grants to You a perpetual,
    +   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
    +   (except as stated in this section) patent license to make, have made,
    +   use, offer to sell, sell, import, and otherwise transfer the Work,
    +   where such license applies only to those patent claims licensable
    +   by such Contributor that are necessarily infringed by their
    +   Contribution(s) alone or by combination of their Contribution(s)
    +   with the Work to which such Contribution(s) was submitted. If You
    +   institute patent litigation against any entity (including a
    +   cross-claim or counterclaim in a lawsuit) alleging that the Work
    +   or a Contribution incorporated within the Work constitutes direct
    +   or contributory patent infringement, then any patent licenses
    +   granted to You under this License for that Work shall terminate
    +   as of the date such litigation is filed.
    +
    +4. Redistribution. You may reproduce and distribute copies of the
    +   Work or Derivative Works thereof in any medium, with or without
    +   modifications, and in Source or Object form, provided that You
    +   meet the following conditions:
    +
    +   (a) You must give any other recipients of the Work or
    +       Derivative Works a copy of this License; and
    +
    +   (b) You must cause any modified files to carry prominent notices
    +       stating that You changed the files; and
    +
    +   (c) You must retain, in the Source form of any Derivative Works
    +       that You distribute, all copyright, patent, trademark, and
    +       attribution notices from the Source form of the Work,
    +       excluding those notices that do not pertain to any part of
    +       the Derivative Works; and
    +
    +   (d) If the Work includes a "NOTICE" text file as part of its
    +       distribution, then any Derivative Works that You distribute must
    +       include a readable copy of the attribution notices contained
    +       within such NOTICE file, excluding those notices that do not
    +       pertain to any part of the Derivative Works, in at least one
    +       of the following places: within a NOTICE text file distributed
    +       as part of the Derivative Works; within the Source form or
    +       documentation, if provided along with the Derivative Works; or,
    +       within a display generated by the Derivative Works, if and
    +       wherever such third-party notices normally appear. The contents
    +       of the NOTICE file are for informational purposes only and
    +       do not modify the License. You may add Your own attribution
    +       notices within Derivative Works that You distribute, alongside
    +       or as an addendum to the NOTICE text from the Work, provided
    +       that such additional attribution notices cannot be construed
    +       as modifying the License.
    +
    +   You may add Your own copyright statement to Your modifications and
    +   may provide additional or different license terms and conditions
    +   for use, reproduction, or distribution of Your modifications, or
    +   for any such Derivative Works as a whole, provided Your use,
    +   reproduction, and distribution of the Work otherwise complies with
    +   the conditions stated in this License.
    +
    +5. Submission of Contributions. Unless You explicitly state otherwise,
    +   any Contribution intentionally submitted for inclusion in the Work
    +   by You to the Licensor shall be under the terms and conditions of
    +   this License, without any additional terms or conditions.
    +   Notwithstanding the above, nothing herein shall supersede or modify
    +   the terms of any separate license agreement you may have executed
    +   with Licensor regarding such Contributions.
    +
    +6. Trademarks. This License does not grant permission to use the trade
    +   names, trademarks, service marks, or product names of the Licensor,
    +   except as required for reasonable and customary use in describing the
    +   origin of the Work and reproducing the content of the NOTICE file.
    +
    +7. Disclaimer of Warranty. Unless required by applicable law or
    +   agreed to in writing, Licensor provides the Work (and each
    +   Contributor provides its Contributions) on an "AS IS" BASIS,
    +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
    +   implied, including, without limitation, any warranties or conditions
    +   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
    +   PARTICULAR PURPOSE. You are solely responsible for determining the
    +   appropriateness of using or redistributing the Work and assume any
    +   risks associated with Your exercise of permissions under this License.
    +
    +8. Limitation of Liability. In no event and under no legal theory,
    +   whether in tort (including negligence), contract, or otherwise,
    +   unless required by applicable law (such as deliberate and grossly
    +   negligent acts) or agreed to in writing, shall any Contributor be
    +   liable to You for damages, including any direct, indirect, special,
    +   incidental, or consequential damages of any character arising as a
    +   result of this License or out of the use or inability to use the
    +   Work (including but not limited to damages for loss of goodwill,
    +   work stoppage, computer failure or malfunction, or any and all
    +   other commercial damages or losses), even if such Contributor
    +   has been advised of the possibility of such damages.
    +
    +9. Accepting Warranty or Additional Liability. While redistributing
    +   the Work or Derivative Works thereof, You may choose to offer,
    +   and charge a fee for, acceptance of support, warranty, indemnity,
    +   or other liability obligations and/or rights consistent with this
    +   License. However, in accepting such obligations, You may act only
    +   on Your own behalf and on Your sole responsibility, not on behalf
    +   of any other Contributor, and only if You agree to indemnify,
    +   defend, and hold each Contributor harmless for any liability
    +   incurred by, or claims asserted against, such Contributor by reason
    +   of your accepting any such warranty or additional liability.
    +
    +END OF TERMS AND CONDITIONS
    +
    +APPENDIX: How to apply the Apache License to your work.
    +
    +   To apply the Apache License to your work, attach the following
    +   boilerplate notice, with the fields enclosed by brackets "[]"
    +   replaced with your own identifying information. (Don't include
    +   the brackets!)  The text should be enclosed in the appropriate
    +   comment syntax for the file format. We also recommend that a
    +   file or class name and description of purpose be included on the
    +   same "printed page" as the copyright notice for easier
    +   identification within third-party archives.
     
    -   Licensed under the Apache License, Version 2.0 (the "License");
    -   you may not use this file except in compliance with the License.
    -   You may obtain a copy of the License at
    +Copyright [yyyy] [name of copyright owner]
     
    -       http://www.apache.org/licenses/LICENSE-2.0
    +Licensed under the Apache License, Version 2.0 (the "License");
    +you may not use this file except in compliance with the License.
    +You may obtain a copy of the License at
     
    -   Unless required by applicable law or agreed to in writing, software
    -   distributed under the License is distributed on an "AS IS" BASIS,
    -   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    -   See the License for the specific language governing permissions and
    -   limitations under the License.
    +	http://www.apache.org/licenses/LICENSE-2.0
     
    +Unless required by applicable law or agreed to in writing, software
    +distributed under the License is distributed on an "AS IS" BASIS,
    +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    +See the License for the specific language governing permissions and
    +limitations under the License.
     
  • Apache License 2.0

    Used by:

                                  Apache License
    -                        Version 2.0 January 2004
    +                        Version 2.0, January 2004
                          http://www.apache.org/licenses/
     
     TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
    @@ -1956,7 +2804,8 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright [yyyy] [name of copyright owner] +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -1975,15 +2824,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2173,8 +3014,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2016 Alex Crichton -Copyright (c) 2017 The Tokio Authors +Copyright (c) 2019 Matthias Einwag Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2193,7 +3033,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2383,7 +3223,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright (c) 2019 Matthias Einwag +Copyright 2014 Paho Lurie-Gregg Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2395,14 +3235,13 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License. -
    +limitations under the License.
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2592,7 +3431,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2014 Paho Lurie-Gregg +Copyright 2017 The Rust Project Developers Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2604,13 +3443,15 @@

    Used by:

    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and -limitations under the License.
    +limitations under the License. +
  • Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -2800,7 +3641,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 The Rust Project Developers +Copyright 2017 http-rs authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -2819,7 +3660,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3009,7 +3850,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 http-rs authors +Copyright 2017 quininer kel Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -3028,7 +3869,7 @@

    Used by:

    Apache License 2.0

    Used by:

                                  Apache License
                             Version 2.0, January 2004
    @@ -3218,7 +4059,7 @@ 

    Used by:

    same "printed page" as the copyright notice for easier identification within third-party archives. -Copyright 2017 quininer kel +Copyright 2018 The pin-utils authors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -4084,6 +4925,7 @@

    Used by:

  • base64 0.21.7
  • base64 0.22.1
  • bitflags 2.11.1
  • +
  • bytes-utils 0.1.4
  • cc 1.2.62
  • cfg-if 1.0.4
  • cmake 0.1.58
  • @@ -4101,15 +4943,18 @@

    Used by:

  • errno 0.3.14
  • event-listener-strategy 0.5.4
  • event-listener 5.4.2
  • +
  • fastrand 2.5.0
  • find-msvc-tools 0.1.9
  • flate2 1.1.9
  • fnv 1.0.7
  • form_urlencoded 1.2.2
  • hashbrown 0.14.5
  • hashbrown 0.15.5
  • +
  • hashbrown 0.16.1
  • hashbrown 0.17.1
  • heck 0.5.0
  • httparse 1.10.1
  • +
  • hyper-rustls 0.27.10
  • idna 1.1.0
  • idna_adapter 1.2.2
  • indexmap 2.14.0
  • @@ -4122,6 +4967,7 @@

    Used by:

  • num-integer 0.1.46
  • num-traits 0.2.19
  • once_cell 1.21.4
  • +
  • openssl-probe 0.2.1
  • parking 2.2.1
  • parking_lot 0.12.5
  • parking_lot_core 0.9.12
  • @@ -4132,8 +4978,12 @@

    Used by:

  • pest_meta 2.8.6
  • pkg-config 0.3.33
  • regex-automata 0.4.14
  • +
  • regex-lite 0.1.9
  • regex-syntax 0.8.10
  • +
  • ring 0.17.14
  • ron 0.8.1
  • +
  • rustc_version 0.4.1
  • +
  • rustls-native-certs 0.8.4
  • rustls 0.23.46
  • scopeguard 1.2.0
  • signal-hook-registry 1.4.8
  • @@ -4149,6 +4999,7 @@

    Used by:

  • url 2.5.8
  • uuid 1.23.1
  • version_check 0.9.5
  • +
  • xmlparser 0.13.6
  • yasna 0.6.0
  •                               Apache License
    @@ -4993,6 +5844,7 @@ 

    Used by:

  • hmac 0.12.1
  • inout 0.1.4
  • md-5 0.10.6
  • +
  • sha1 0.10.6
  • sha2 0.10.9
  •                               Apache License
    @@ -6452,6 +7304,7 @@ 

    Used by:

    +
  • +
  • +

    ISC License

    +

    Used by:

    + +
    Copyright 2015-2025 Brian Smith.
    +
    +Permission to use, copy, modify, and/or distribute this software for any
    +purpose with or without fee is hereby granted, provided that the above
    +copyright notice and this permission notice appear in all copies.
    +
    +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
    +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
    +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
    +SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
    +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
    +OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
    +CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
     
  • @@ -7163,6 +8055,63 @@

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2019 Sean McArthur
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
    +
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2018-2023 Sean McArthur
    +Copyright (c) 2016 Alex Crichton
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in
    +all copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    +THE SOFTWARE.
    +
     
  • @@ -7250,6 +8199,39 @@

    Used by:

    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    Copyright (c) 2019 Hyper Contributors
    +
    +Permission is hereby granted, free of charge, to any
    +person obtaining a copy of this software and associated
    +documentation files (the "Software"), to deal in the
    +Software without restriction, including without
    +limitation the rights to use, copy, modify, merge,
    +publish, distribute, sublicense, and/or sell copies of
    +the Software, and to permit persons to whom the Software
    +is furnished to do so, subject to the following
    +conditions:
    +
    +The above copyright notice and this permission notice
    +shall be included in all copies or substantial portions
    +of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
    +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
    +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
    +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
    +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
    +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
    +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    +DEALINGS IN THE SOFTWARE.
     
  • @@ -7567,6 +8549,34 @@

    Used by:

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    MIT License
    +
    +Copyright (c) 2016 Jerome Froelich
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -7781,7 +8791,37 @@

    Used by:

    MIT License

    Used by:

    +
    MIT License
    +
    +Copyright (c) 2022 Nugine
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    +
    MIT License
     
    @@ -7920,6 +8960,34 @@ 

    Used by:

    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    +
  • +
  • +

    MIT License

    +

    Used by:

    + +
    The MIT License (MIT)
    +
    +Copyright (c) 2014 Mathijs van de Nes
    +
    +Permission is hereby granted, free of charge, to any person obtaining a copy
    +of this software and associated documentation files (the "Software"), to deal
    +in the Software without restriction, including without limitation the rights
    +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    +copies of the Software, and to permit persons to whom the Software is
    +furnished to do so, subject to the following conditions:
    +
    +The above copyright notice and this permission notice shall be included in all
    +copies or substantial portions of the Software.
    +
    +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    +SOFTWARE.
  • MIT License

    @@ -8325,6 +9393,7 @@

    zlib License

    Used by:

    Copyright (c) 2024 Orson Peters
     
    diff --git a/crates/app/Cargo.toml b/crates/app/Cargo.toml
    index a1e66490f..099ea21cf 100644
    --- a/crates/app/Cargo.toml
    +++ b/crates/app/Cargo.toml
    @@ -21,6 +21,8 @@ extenddb-engine = { workspace = true }
     extenddb-storage = { workspace = true }
     extenddb-config = { workspace = true }
     extenddb-server = { workspace = true }
    +extenddb-backup-store = { workspace = true }
    +futures = { workspace = true }
     tokio = { workspace = true }
     anyhow = { workspace = true }
     clap = { workspace = true }
    diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs
    new file mode 100644
    index 000000000..1896cd56e
    --- /dev/null
    +++ b/crates/app/src/cmd_backup.rs
    @@ -0,0 +1,927 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! `extenddb backup`: the operator's commands over the configured backup
    +//! store.
    +//!
    +//! `backup list` prints the catalog's rows for the configured store's kind
    +//! and root, every status and every store id the rows carry, with `--failed`
    +//! narrowing to the FAILED tombstones the reconciler keeps for the operator;
    +//! each line carries the `store_root` and the `store_id` so an operator sees
    +//! which root and which id a row names. It is a read: it opens the catalog
    +//! beside a running server (on SQLite without the serve lock and without the
    +//! startup recovery a server runs), reads the root's identity file without
    +//! creating one, and runs on an unhealthy store, printing the store's
    +//! problem in its header, since an operator whose store is unhealthy is the
    +//! one who most wants to see which rows name the root.
    +//!
    +//! `backup purge --confirm` deletes every backup of the configured store: the
    +//! prefix of each row (`///`, derived from the
    +//! ARN, never a whole account prefix, so a prefix with no row survives) and
    +//! the row itself, outright, leaving no tombstone. It leaves
    +//! `extenddb-store.json`, which is under no account prefix, so the root
    +//! keeps its identity, and the `exports/` component. It needs the store
    +//! healthy, its identity known: a purge over an unhealthy store would act on
    +//! rows it cannot trust. Before it deletes it takes the catalog-level
    +//! maintenance claim, which it heartbeats on a timer for the claim's whole
    +//! life, at the period the jobs heartbeat their rows at
    +//! (`extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL`, so the
    +//! `stale_job_after_secs` floor that keeps a live job from being declared
    +//! dead keeps this claim held too), so every reader of the claim sees it
    +//! held from the take to the release, through a deletion phase of any
    +//! length; a heartbeat that finds the claim lost, or fails, stops the
    +//! deletion loop before its next row. It then waits, up to
    +//! `2 * stale_job_after_secs`, until no CREATING or FAILED backup row of the
    +//! store has a live owner (a heartbeat within `stale_job_after_secs` on the
    +//! catalog's clock, as the backend's `list_backup_rows_older_than` reads it; a
    +//! FAILED row's heartbeat may still be refreshed by a job that has not yet
    +//! seen its claim lost, and its prefix must not be deleted under a manifest
    +//! it may still write) and no restore job is RUNNING; it refuses and
    +//! releases the claim if they have not gone by then, printing the wait. A
    +//! CREATING row the wait saw no live owner for is deleted too: the row
    +//! first (`remove_dead_creating_backup_row`, on the catalog's clock, which
    +//! leaves a row whose heartbeat is young after all, and its objects with
    +//! it), then its prefix. A DELETING row is not
    +//! waited on: nothing heartbeats one, the reconciler finishes any it finds,
    +//! and the deletion is the same idempotent sequence purge runs on every row
    +//! (two deleters of one prefix do not fail each other, and the row's moves
    +//! are conditional), so purge finishes it itself; for the same reason a row
    +//! a DeleteBackup moved to DELETING or DELETED, or removed, between the
    +//! command's read and its turn is finished and counted as deleted, not left
    +//! (`extenddb_engine::maintenance::purge_row` states each state's step).
    +//! Every backup and restore
    +//! job checks the claim before it starts and refuses while it is held
    +//! (`LimitExceededException`; the engine's backup module states where in
    +//! the job's order the check runs). The wait ends early, with the claim
    +//! released and nothing deleted, when a heartbeat finds the claim lost, when
    +//! the rows cannot be read, or on `SIGINT` (Ctrl-C) or `SIGTERM`; the same
    +//! signal during the deletion ends the loop before its next row, never
    +//! inside one, so a stopped command leaves no half-deleted row and no claim
    +//! for the next command or server to wait out. Without `--confirm` the
    +//! command is a
    +//! read like `list` (no serve lock, nothing created): it prints what it
    +//! would delete, changes nothing and exits 0, with no store configured as
    +//! with one. Each deletion is logged under
    +//! the `extenddb::audit` target with the account and the ARN.
    +//!
    +//! On SQLite purge opens the database the way `serve` does, which takes the
    +//! serve lock: it refuses while a server runs on the file, and a server
    +//! refuses to start while it runs. Under that lock every job owner is dead
    +//! (one process serves the file, and this is it), and no reconciler runs
    +//! while the command holds the file, so before the wait the command runs
    +//! the reconciler's duties itself with a zero stale window, as the holder
    +//! of the maintenance claim (the cycle takes its held branch and does not
    +//! read the claim back through that window): a legacy restore
    +//! the stopped server left RUNNING is failed and its target removed, a
    +//! target a FAILED restore left CREATING is removed, an in-catalog CREATING
    +//! backup it left is removed, and a DELETING row is finished; the wait and
    +//! the CREATING-row removal then also use a zero window, so a CREATING or
    +//! FAILED row whose heartbeat is young (the stopped server's job wrote it)
    +//! is neither waited on nor left, and the wait has nothing to wait for. The
    +//! server's own startup under the same lock clears a maintenance claim a
    +//! stopped command left, for the same reason, and the command's own take
    +//! of the claim takes over a row a killed command left. On the other
    +//! backends purge runs beside live servers; the maintenance claim is the
    +//! exclusion.
    +//!
    +//! Every command opens the catalog through the backend's factory, under its
    +//! own name (`extenddb backup list`, `extenddb backup purge`), and none runs
    +//! the startup recovery a server runs (`ServerComponentsOptions::startup_recovery`
    +//! off): that work belongs to the servers the command runs beside, and a
    +//! command that ran it could exit with it half done. On
    +//! PostgreSQL, MongoDB and Cassandra the command holds a serving
    +//! registration for its run, as a server does, so a migration waits for it,
    +//! and the registration carries the command's name, which the migration's
    +//! wait prints (PostgreSQL's session `application_name`, MongoDB's
    +//! `extenddb_servers` document, Cassandra's `extenddb_servers` row). The
    +//! command releases the registration when it ends, whatever the outcome, so
    +//! a migration does not wait for a row or document to expire. A catalog the
    +//! factory refuses (a migration running, the catalog at another migration
    +//! than this binary's) ends the command with the message and the exit code
    +//! `serve` gives for it: 6 while a migration runs, 3 for a catalog behind
    +//! this binary. A registration lost during the command does not stop it:
    +//! no supervisor runs in a command, so a command that outlasts a lost
    +//! registration runs beside a migration that started after the loss. The
    +//! signal listener is up before the open in every subcommand, so a Ctrl-C
    +//! during the open or the identity read ends the command once the open
    +//! returns, with the registration released.
    +//!
    +//! `extenddb destroy` removes the databases and leaves the store untouched:
    +//! the store is the copy that survives destroy, and this command is the one
    +//! that deletes it.
    +
    +use std::sync::Arc;
    +use std::time::{Duration, Instant};
    +
    +use clap::{Args, Subcommand};
    +use extenddb_config as config;
    +use extenddb_engine::maintenance::{
    +    Interrupt, Stop, Wait, WaitOutcome, purge_rows, wait_for_quiet,
    +};
    +use extenddb_engine::{BackupRuntime, ConfiguredStore, HeldClaim, StoreHealth, StoreProblem};
    +use extenddb_storage::StorageEngine;
    +use extenddb_storage::backup_catalog::{
    +    BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, StoreIdentity,
    +};
    +use extenddb_storage::error::StorageError;
    +use extenddb_storage::hooks::ServingRegistrar;
    +use futures::TryStreamExt;
    +
    +#[derive(Args)]
    +#[allow(clippy::doc_markdown)] // Clap help text, not rustdoc
    +pub struct BackupArgs {
    +    /// Path to configuration file (accepted before or after the subcommand)
    +    #[arg(short, long, default_value = "extenddb.toml", global = true)]
    +    config: String,
    +
    +    #[command(subcommand)]
    +    command: BackupCommand,
    +}
    +
    +#[derive(Subcommand)]
    +#[allow(clippy::doc_markdown)] // Clap help text, not rustdoc
    +enum BackupCommand {
    +    /// List the catalog's backup rows for the configured backup store. A read that runs
    +    /// beside a server, on an unhealthy store too.
    +    List {
    +        /// Only the FAILED rows (backups whose creation failed, kept for the operator)
    +        #[arg(long)]
    +        failed: bool,
    +    },
    +    /// Delete every backup of the configured backup store: each backup's objects and its
    +    /// catalog row. Leaves extenddb-store.json, exports/ and any prefix with no row.
    +    /// Takes the maintenance claim first and waits for running jobs to finish.
    +    Purge {
    +        /// Confirm the deletion (required; without it the command prints what it would delete
    +        /// and exits 0)
    +        #[arg(long)]
    +        confirm: bool,
    +    },
    +}
    +
    +/// How often the waiting purge re-reads the rows.
    +const WAIT_POLL: Duration = Duration::from_secs(2);
    +
    +/// The commands' names, as the catalog's other sessions see them: on
    +/// PostgreSQL, MongoDB and Cassandra every command registers as the
    +/// catalog's server does, so a migration waits for it, and the registration
    +/// carries this name, which the migration prints while it waits.
    +const LIST: &str = "extenddb backup list";
    +const PURGE: &str = "extenddb backup purge";
    +
    +pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> {
    +    if !std::path::Path::new(&args.config).exists() {
    +        anyhow::bail!(
    +            "Config file '{}' not found; use --config  to specify its location.",
    +            args.config,
    +        );
    +    }
    +    let app_config = config::load(&args.config)?;
    +    // The signal listener is up before the catalog is opened, in every
    +    // subcommand: a signal during the open or the identity read is acted on
    +    // once the open returns, with the serving registration released, where
    +    // a listener started after the open would leave the registration for
    +    // `migrate --yes` to wait out. During the purge the same handle ends the
    +    // wait at once and the deletion loop before its next row.
    +    let interrupt = listen_for_signals();
    +    let (command, mode) = match args.command {
    +        BackupCommand::List { .. } => (LIST, Mode::Read),
    +        BackupCommand::Purge { confirm: false } => (PURGE, Mode::Read),
    +        BackupCommand::Purge { confirm: true } => (PURGE, Mode::Write),
    +    };
    +    let opening = open(&app_config, &args.config, release, command, mode).await?;
    +    if let Some(signal) = interrupt.signalled() {
    +        if let Opening::Store(opened) = opening {
    +            opened.close().await;
    +        }
    +        anyhow::bail!("{signal} received while opening the catalog; nothing was changed");
    +    }
    +    match (args.command, opening) {
    +        (BackupCommand::List { failed }, Opening::Store(opened)) => {
    +            let done = list(&opened, failed).await;
    +            opened.close().await;
    +            done
    +        }
    +        (BackupCommand::List { .. }, Opening::NoStore(reason)) => {
    +            println!("=== backups in the configured store: none ({reason}) ===");
    +            println!("  (none)");
    +            Ok(())
    +        }
    +        // The dry run is a read, as `list` is: no serve lock, nothing
    +        // created, the plan from the rows of the root; with no store, the
    +        // plan is empty and says why, as `list` does.
    +        (BackupCommand::Purge { confirm: false }, Opening::Store(opened)) => {
    +            let done = dry_run(&opened).await;
    +            opened.close().await;
    +            done
    +        }
    +        (BackupCommand::Purge { confirm: false }, Opening::NoStore(reason)) => {
    +            println!("=== extenddb backup purge ===");
    +            println!("Store:  none ({reason})");
    +            println!();
    +            println!("--- Backups to delete (each backup's objects and its catalog row):");
    +            println!("  (none)");
    +            println!();
    +            println!("Dry run: nothing was deleted.");
    +            Ok(())
    +        }
    +        (BackupCommand::Purge { confirm: true }, Opening::Store(opened)) => {
    +            let done = purge(&opened, &app_config, interrupt).await;
    +            opened.close().await;
    +            done
    +        }
    +        (BackupCommand::Purge { confirm: true }, Opening::NoStore(reason)) => {
    +            anyhow::bail!("{reason}")
    +        }
    +    }
    +}
    +
    +/// What `open` found: the configured store, or why there is none.
    +enum Opening {
    +    Store(Opened),
    +    NoStore(String),
    +}
    +
    +/// How a command opens the catalog.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +enum Mode {
    +    /// A read beside a running server: no serve lock, no startup recovery,
    +    /// the identity file read and never written.
    +    Read,
    +    /// A write: on SQLite under the serve lock, as `serve` opens it, but
    +    /// without the startup recovery a server runs (on PostgreSQL the vector
    +    /// index rebuild, the control-plane recovery and the GSI queue workers,
    +    /// on MongoDB the key-repair pass, on Cassandra the control-plane
    +    /// transitions), which belongs to the servers the command runs beside
    +    /// and which the command must not exit with half done; the store's
    +    /// check may write the identity file of an empty root.
    +    Write,
    +}
    +
    +/// The storage engine, the backup runtime with the configured store, the
    +/// store's identity as the command read it, and the serving registration
    +/// the command holds for its run.
    +struct Opened {
    +    storage: Arc,
    +    runtime: Arc,
    +    backend: String,
    +    identity: Identity,
    +    registration: Option>,
    +}
    +
    +/// The store's identity as the command read it.
    +enum Identity {
    +    /// The check passed, or the file was read and agrees with the rows.
    +    Known(StoreIdentity),
    +    /// The root has no identity file and the catalog no rows for it: a
    +    /// fresh root nothing has written to.
    +    Empty,
    +    /// The check failed (a listing still runs; a purge refuses).
    +    Problem(StoreProblem),
    +}
    +
    +impl Opened {
    +    /// End the command's use of the catalog: release its serving
    +    /// registration.
    +    async fn close(self) {
    +        release_registration(self.registration).await;
    +    }
    +
    +    fn store(&self) -> &ConfiguredStore {
    +        self.runtime
    +            .configured_store()
    +            .expect("open returns only with a configured store")
    +    }
    +}
    +
    +/// Open the catalog as `command`. A catalog the factory refuses (on
    +/// PostgreSQL: a migration running, or the catalog at another migration than
    +/// this binary's) is an [`extenddb_server::CatalogRefusal`], with the
    +/// message and exit code `serve` gives for the same status.
    +async fn open(
    +    app_config: &config::AppConfig,
    +    config_path: &str,
    +    release: &'static str,
    +    command: &'static str,
    +    mode: Mode,
    +) -> anyhow::Result {
    +    let mut options = extenddb_storage::server_components::ServerComponentsOptions::default();
    +    options.release = release;
    +    options.max_concurrent_jobs = app_config.backup.max_concurrent_jobs;
    +    options.beside_server = mode == Mode::Read;
    +    options.startup_recovery = false;
    +    options.command = Some(command);
    +    let mut storage_config = app_config.storage.clone();
    +    storage_config.set_instance_id(&format!(
    +        "{}:backup-command",
    +        extenddb_server::backup_store::hostname()
    +    ));
    +    let components = extenddb_storage::create_server_components(
    +        storage_config.as_trait(),
    +        &app_config.server.region,
    +        options,
    +    )
    +    .await;
    +    let components = match components {
    +        Ok(components) => components,
    +        Err(extenddb_storage::server_components::BackendError::Catalog(error)) => {
    +            return Err(extenddb_server::catalog_refusal(
    +                &error,
    +                release,
    +                app_config,
    +                Some(config_path),
    +            )
    +            .await
    +            .into());
    +        }
    +        Err(e) => anyhow::bail!("cannot open the catalog: {e}"),
    +    };
    +    // The registration is the runtime hooks', which the command does not
    +    // otherwise keep: held here, it lasts until `Opened::close`.
    +    let registration = components
    +        .runtime_hooks
    +        .as_ref()
    +        .and_then(|hooks| hooks.serving_registrar());
    +    let storage = components.engine;
    +    let backend = app_config.storage.backend.clone();
    +    let setup = extenddb_server::backup_store::configure_described(
    +        &app_config.backup,
    +        app_config.storage.database_file().as_deref(),
    +        &backend,
    +        &*storage,
    +        match mode {
    +            Mode::Read => extenddb_server::backup_store::Caller::ReadOnlyCommand,
    +            Mode::Write => extenddb_server::backup_store::Caller::Command,
    +        },
    +    )
    +    .await;
    +    let setup = match setup {
    +        Ok(setup) => setup,
    +        Err(e) => {
    +            release_registration(registration).await;
    +            return Err(e);
    +        }
    +    };
    +    let runtime = Arc::new(setup.runtime);
    +    let Some(store) = runtime.configured_store() else {
    +        // Why there is none, as `configure_described` found it: the section
    +        // absent or disabled, the default root not yet created, or the
    +        // default root failed its check with no rows for it.
    +        let why = setup
    +            .not_configured
    +            .expect("a runtime without a store carries the reason");
    +        release_registration(registration).await;
    +        return Ok(Opening::NoStore(why.to_string()));
    +    };
    +    // The server's startup leaves an S3 store's check to its reconciler;
    +    // the command runs it here. A read leaves the root as it finds it: the
    +    // identity file is read, never created, under the store's check bound as
    +    // the server's check is, so an endpoint that accepts and never answers
    +    // or a mount that has stopped answering costs the command the bound and
    +    // not the store's retries.
    +    let identity = match (mode, store.health()) {
    +        (Mode::Read, _) => match extenddb_engine::read_identity_bounded(store, &*storage).await {
    +            Ok(Some(identity)) => Identity::Known(identity),
    +            Ok(None) => Identity::Empty,
    +            Err(problem) => Identity::Problem(problem),
    +        },
    +        (Mode::Write, StoreHealth::Checking) => {
    +            let outcome = extenddb_engine::check_configured_store(store, &*storage, &backend).await;
    +            store.record_check(outcome.clone());
    +            match outcome {
    +                Ok(identity) => Identity::Known(identity),
    +                Err(problem) => Identity::Problem(problem),
    +            }
    +        }
    +        (Mode::Write, StoreHealth::Healthy) => match store.identity() {
    +            Some(identity) => Identity::Known(identity),
    +            None => Identity::Problem(StoreProblem::Catalog(
    +                "the store's id is unknown".to_owned(),
    +            )),
    +        },
    +        (Mode::Write, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem),
    +    };
    +    Ok(Opening::Store(Opened {
    +        storage,
    +        runtime,
    +        backend,
    +        identity,
    +        registration,
    +    }))
    +}
    +
    +/// How long a command waits for its registration's release: a row or
    +/// document whose delete does not answer in time expires on its own (45 s
    +/// after its last renewal on Cassandra and on MongoDB), and PostgreSQL's
    +/// session ends with the process.
    +const RELEASE_TIMEOUT: Duration = Duration::from_secs(10);
    +
    +/// Release a command's serving registration, if it holds one, so a
    +/// migration that waits for the registrations does not wait for it to
    +/// expire. Bounded by [`RELEASE_TIMEOUT`].
    +async fn release_registration(registration: Option>) {
    +    release_registration_within(registration, RELEASE_TIMEOUT).await;
    +}
    +
    +async fn release_registration_within(
    +    registration: Option>,
    +    bound: Duration,
    +) {
    +    if let Some(registration) = registration
    +        && tokio::time::timeout(bound, registration.release())
    +            .await
    +            .is_err()
    +    {
    +        tracing::warn!(
    +            "the command's serving registration was not released within {} s; it expires on \
    +             its own",
    +            bound.as_secs()
    +        );
    +    }
    +}
    +
    +async fn rows_of(
    +    storage: &dyn StorageEngine,
    +    identity: &StoreIdentity,
    +    status: Option,
    +) -> anyhow::Result> {
    +    storage
    +        .list_backup_rows(status, identity)
    +        .try_collect()
    +        .await
    +        .map_err(|e| anyhow::anyhow!("cannot list the catalog's backup rows: {e}"))
    +}
    +
    +fn print_row(row: &BackupRow) {
    +    println!(
    +        "{:<9} {:<12} {:<32} {:>12} {:>10}  {}  store_root={} store_id={}{}",
    +        row.status,
    +        row.account_id,
    +        row.table_name,
    +        row.size_bytes,
    +        row.item_count,
    +        row.backup_arn,
    +        row.store.store_root,
    +        row.store.store_id,
    +        row.failure_reason
    +            .as_deref()
    +            .map(|r| format!("  failure_reason={r:?}"))
    +            .unwrap_or_default()
    +    );
    +}
    +
    +/// Every row of the store's kind and root, whatever id it carries: the ids
    +/// the rows record, plus the file's when the check read one.
    +async fn rows_of_root(
    +    opened: &Opened,
    +    status: Option,
    +) -> anyhow::Result<(Vec, Vec)> {
    +    let store = opened.store();
    +    let mut ids = opened
    +        .storage
    +        .read_store_ids(store.kind, &store.root)
    +        .await
    +        .map_err(|e| anyhow::anyhow!("cannot read the catalog's store ids: {e}"))?;
    +    if let Identity::Known(identity) = &opened.identity
    +        && !ids.contains(&identity.store_id)
    +    {
    +        ids.push(identity.store_id.clone());
    +    }
    +    let mut rows = Vec::new();
    +    for id in &ids {
    +        let identity = StoreIdentity {
    +            store_kind: store.kind,
    +            store_root: store.root.clone(),
    +            store_id: id.clone(),
    +        };
    +        rows.extend(rows_of(&*opened.storage, &identity, status).await?);
    +    }
    +    Ok((ids, rows))
    +}
    +
    +async fn list(opened: &Opened, failed: bool) -> anyhow::Result<()> {
    +    let status = failed.then_some(BackupStatus::Failed);
    +    let store = opened.store();
    +    let (ids, rows) = rows_of_root(opened, status).await?;
    +    match &opened.identity {
    +        Identity::Known(identity) => println!(
    +            "=== backups in the {} store {} (store id {}) ===",
    +            store.kind, store.root, identity.store_id
    +        ),
    +        Identity::Empty => println!(
    +            "=== backups in the {} store {} (the root has no {} yet; a server's first check \
    +             creates it) ===",
    +            store.kind,
    +            store.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => println!(
    +            "=== backups in the {} store {} (the store is unhealthy: {problem}; rows of every \
    +             id the catalog records for the root are listed) ===",
    +            store.kind, store.root
    +        ),
    +    }
    +    if ids.len() > 1 {
    +        println!(
    +            "  the catalog's rows for this root carry {} store ids: {}",
    +            ids.len(),
    +            ids.join(", ")
    +        );
    +    }
    +    if rows.is_empty() {
    +        println!("  (none)");
    +        return Ok(());
    +    }
    +    for row in &rows {
    +        print_row(row);
    +    }
    +    println!("{} row(s)", rows.len());
    +    Ok(())
    +}
    +
    +/// The plan a purge prints before it deletes, and the dry run's whole
    +/// output: the store, its rows, and what stays.
    +async fn print_plan(opened: &Opened) -> anyhow::Result<()> {
    +    let configured = opened.store();
    +    println!("=== extenddb backup purge ===");
    +    match &opened.identity {
    +        Identity::Known(identity) => println!(
    +            "Store:  {} {} (store id {})",
    +            identity.store_kind, identity.store_root, identity.store_id
    +        ),
    +        Identity::Empty => println!(
    +            "Store:  {} {} (no {} yet)",
    +            configured.kind,
    +            configured.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => println!(
    +            "Store:  {} {} (unhealthy: {problem})",
    +            configured.kind, configured.root
    +        ),
    +    }
    +    let (_, planned) = rows_of_root(opened, None).await?;
    +    println!();
    +    println!("--- Backups to delete (each backup's objects and its catalog row):");
    +    if planned.is_empty() {
    +        println!("  (none)");
    +    }
    +    for row in &planned {
    +        print_row(row);
    +    }
    +    println!();
    +    println!(
    +        "Left in place: {}, exports/, and any prefix the catalog has no row for.",
    +        extenddb_engine::STORE_IDENTITY_KEY
    +    );
    +    println!();
    +    Ok(())
    +}
    +
    +/// `backup purge` without `--confirm`: the plan, nothing changed, exit 0.
    +/// On an unhealthy store the plan still lists the rows, and the closing
    +/// line says that `--confirm` refuses until the store's check passes.
    +async fn dry_run(opened: &Opened) -> anyhow::Result<()> {
    +    print_plan(opened).await?;
    +    match &opened.identity {
    +        Identity::Problem(problem) => println!(
    +            "Dry run: nothing was deleted. The store is unhealthy ({}); --confirm refuses until \
    +             the store's check passes.",
    +            problem.category()
    +        ),
    +        Identity::Known(_) | Identity::Empty => println!(
    +            "Dry run: nothing was deleted. Run again with --confirm to permanently delete every \
    +             backup of the configured store listed above."
    +        ),
    +    }
    +    Ok(())
    +}
    +
    +async fn purge(
    +    opened: &Opened,
    +    app_config: &config::AppConfig,
    +    mut interrupt: Interrupt,
    +) -> anyhow::Result<()> {
    +    let backup = &app_config.backup;
    +    let storage = &*opened.storage;
    +    let configured = opened.store();
    +    let identity = match &opened.identity {
    +        Identity::Known(identity) => identity,
    +        Identity::Empty => anyhow::bail!(
    +            "the {} backup store {} has no {} and the catalog has no rows for it: nothing to purge",
    +            configured.kind,
    +            configured.root,
    +            extenddb_engine::STORE_IDENTITY_KEY
    +        ),
    +        Identity::Problem(problem) => anyhow::bail!(
    +            "the {} backup store {} is unhealthy: {problem}; fix that before purging its backups",
    +            configured.kind,
    +            configured.root
    +        ),
    +    };
    +    let store = &*configured.store;
    +    print_plan(opened).await?;
    +
    +    let stale_after = Duration::from_secs(backup.stale_job_after_secs);
    +    let holder = format!(
    +        "extenddb backup purge on {} (pid {})",
    +        extenddb_server::backup_store::hostname(),
    +        std::process::id()
    +    );
    +    // The signal listener has been up since before the open, so a signal
    +    // from here on ends the command with the claim released: at once during
    +    // the wait, before the next row during the deletion.
    +    let claim = storage
    +        .maintenance_claim(&holder, stale_after)
    +        .await
    +        .map_err(|e| match e {
    +            StorageError::AlreadyExists(m) => anyhow::anyhow!(
    +                "{m}; one maintenance command runs at a time, and a claim whose holder died \
    +                 expires after stale_job_after_secs ({} s)",
    +                backup.stale_job_after_secs
    +            ),
    +            other => anyhow::anyhow!("cannot take the maintenance claim: {other}"),
    +        })?;
    +    println!("Took the maintenance claim as {holder:?}.");
    +    let claim = HeldClaim::start(claim, JOB_HEARTBEAT_INTERVAL);
    +
    +    // On SQLite the command holds the file alone: every job owner is dead
    +    // and no reconciler runs, so their resolution is this command's, and the
    +    // owner of any row is dead whatever its heartbeat says, so the window a
    +    // live owner is told apart by is zero. On the other backends purge runs
    +    // beside live servers, whose jobs heartbeat within `stale_after`.
    +    let owner_window = if opened.backend == "sqlite" {
    +        resolve_dead_owners(opened).await;
    +        Duration::ZERO
    +    } else {
    +        stale_after
    +    };
    +
    +    // Wait for the jobs the claim excludes from starting to finish.
    +    let wait_limit = stale_after * 2;
    +    let started = Instant::now();
    +    let outcome = wait_for_quiet(
    +        storage,
    +        identity,
    +        &claim,
    +        Wait {
    +            owner_window,
    +            limit: wait_limit,
    +            poll: WAIT_POLL,
    +        },
    +        &mut interrupt,
    +        |remaining| {
    +            println!(
    +                "Waiting for running jobs to finish: {}",
    +                remaining.join("; ")
    +            )
    +        },
    +    )
    +    .await;
    +    match outcome {
    +        WaitOutcome::Quiet => {}
    +        WaitOutcome::Busy(remaining) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "jobs were still running after waiting {} s (2 * stale_job_after_secs): {}. The \
    +                 maintenance claim is released; retry once they finish",
    +                started.elapsed().as_secs(),
    +                remaining.join("; ")
    +            );
    +        }
    +        WaitOutcome::Lost(reason) => {
    +            // The release is conditioned on the generation: another holder's
    +            // claim is left to it.
    +            release(claim).await;
    +            anyhow::bail!("stopped before any deletion: {reason}; retry once the claim is free");
    +        }
    +        WaitOutcome::CatalogError(e) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "stopped before any deletion: {e}. The maintenance claim is released; retry once \
    +                 the catalog answers"
    +            );
    +        }
    +        WaitOutcome::Interrupted(signal) => {
    +            release(claim).await;
    +            anyhow::bail!(
    +                "stopped before any deletion: {signal} received while waiting. The maintenance \
    +                 claim is released"
    +            );
    +        }
    +    }
    +
    +    // Re-read under the claim: the plan printed before it may be stale.
    +    let rows = match rows_of(storage, identity, None).await {
    +        Ok(rows) => rows,
    +        Err(e) => {
    +            release(claim).await;
    +            return Err(e);
    +        }
    +    };
    +    let outcome = purge_rows(
    +        storage,
    +        store,
    +        &rows,
    +        owner_window,
    +        &claim,
    +        &interrupt,
    +        |row| {
    +            tracing::info!(
    +                target: "extenddb::audit",
    +                action = "backup purge",
    +                account_id = %row.account_id,
    +                backup_arn = %row.backup_arn,
    +                store_root = %row.store.store_root,
    +                "deleted backup objects and catalog row"
    +            );
    +            println!("deleted {} ({})", row.backup_arn, row.status);
    +        },
    +    )
    +    .await;
    +    release(claim).await;
    +    let deleted = outcome.deleted.len();
    +    println!();
    +    println!(
    +        "=== extenddb backup purge complete: {deleted} backup(s) deleted, {} left ===",
    +        rows.len() - deleted
    +    );
    +    match outcome.stopped {
    +        Some(Stop::Lost(reason)) => {
    +            anyhow::bail!(
    +                "stopped before the remaining rows: {reason}; retry once the claim is free"
    +            )
    +        }
    +        Some(Stop::Interrupted(signal)) => anyhow::bail!(
    +            "stopped before the remaining rows: {signal} received. The maintenance claim is \
    +             released; run the command again for the rest"
    +        ),
    +        None => {}
    +    }
    +    if !outcome.skipped.is_empty() {
    +        for line in &outcome.skipped {
    +            println!("  left: {line}");
    +        }
    +        anyhow::bail!(
    +            "{} backup(s) could not be deleted; see above",
    +            outcome.skipped.len()
    +        );
    +    }
    +    Ok(())
    +}
    +
    +/// The handle every subcommand reads a signal from, fed by a task that waits
    +/// for `SIGINT` (Ctrl-C) or `SIGTERM`. The first signal stops the command at
    +/// its next safe point (after the open, the wait at once, the deletion loop
    +/// before its next row) with its serving registration and, in a purge, the
    +/// claim released; a second one exits the process at once, the registration
    +/// then expiring on its own and the claim by the stale rule (on SQLite the
    +/// next server start clears it).
    +fn listen_for_signals() -> Interrupt {
    +    let (tx, interrupt) = Interrupt::new();
    +    tokio::spawn(async move {
    +        let name = signal_name().await;
    +        println!(
    +            "{name} received; finishing the current step and releasing what the command holds (a \
    +             second {name} exits at once)."
    +        );
    +        let _ = tx.send(Some(name));
    +        let again = signal_name().await;
    +        println!(
    +            "{again} received again; exiting now. The serving registration expires on its own and \
    +             a maintenance claim by the stale rule."
    +        );
    +        std::process::exit(130);
    +    });
    +    interrupt
    +}
    +
    +#[cfg(unix)]
    +async fn signal_name() -> &'static str {
    +    use tokio::signal::unix::{SignalKind, signal};
    +    let ctrl_c = tokio::signal::ctrl_c();
    +    let Ok(mut sigterm) = signal(SignalKind::terminate()) else {
    +        let _ = ctrl_c.await;
    +        return "SIGINT";
    +    };
    +    tokio::select! {
    +        _ = ctrl_c => "SIGINT",
    +        _ = sigterm.recv() => "SIGTERM",
    +    }
    +}
    +
    +#[cfg(not(unix))]
    +async fn signal_name() -> &'static str {
    +    let _ = tokio::signal::ctrl_c().await;
    +    "Ctrl-C"
    +}
    +
    +/// The reconciler's duties, run once with a zero stale window: on SQLite
    +/// under the serve lock every owner is dead. The command holds the
    +/// maintenance claim, and its state says so, so the cycle takes the held
    +/// branch without reading the claim (read through a zero window, the
    +/// command's own claim would read as held only within one instant of its
    +/// heartbeat). What was resolved is printed; a failure is printed and the
    +/// wait decides.
    +async fn resolve_dead_owners(opened: &Opened) {
    +    match extenddb_engine::reconciler::cycle(
    +        &*opened.storage,
    +        &opened.runtime,
    +        Duration::ZERO,
    +        &opened.backend,
    +        &mut extenddb_engine::reconciler::ReconcilerState::for_maintenance_holder(),
    +    )
    +    .await
    +    {
    +        Ok(report) => {
    +            for table in &report.legacy_restores_removed {
    +                println!("removed the restore target {table}: its restore did not finish");
    +            }
    +            for table in &report.failed_targets_removed {
    +                println!("removed the restore target {table}: its restore had failed");
    +            }
    +            for table in &report.targets_left_to_owner {
    +                println!(
    +                    "left the table {table}: it is no longer the restore target its record names"
    +                );
    +            }
    +            for arn in &report.legacy_backups_removed {
    +                println!("removed the in-catalog backup {arn}: its create did not finish");
    +            }
    +            for arn in &report.deletions_finished {
    +                println!("finished the deletion of {arn}");
    +            }
    +        }
    +        Err(e) => println!("could not resolve the jobs the stopped server left: {e}"),
    +    }
    +}
    +
    +/// Release the claim; a failure is printed, since the claim then expires by
    +/// the stale rule and the command's own outcome is what the operator needs.
    +async fn release(claim: HeldClaim) {
    +    if let Err(e) = claim.release().await {
    +        println!("could not release the maintenance claim ({e}); it expires on its own");
    +    }
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use std::sync::Arc;
    +    use std::sync::atomic::{AtomicUsize, Ordering};
    +    use std::time::Duration;
    +
    +    use async_trait::async_trait;
    +    use extenddb_storage::hooks::{Reregistration, ServingRegistrar};
    +
    +    use super::{release_registration, release_registration_within};
    +
    +    /// A registrar that counts its releases, each taking `takes`.
    +    struct Counting {
    +        released: AtomicUsize,
    +        takes: Duration,
    +    }
    +
    +    #[async_trait]
    +    impl ServingRegistrar for Counting {
    +        async fn lost(&self) -> String {
    +            std::future::pending().await
    +        }
    +        async fn register_again(&self) -> Reregistration {
    +            Reregistration::Registered
    +        }
    +        async fn release(&self) {
    +            tokio::time::sleep(self.takes).await;
    +            self.released.fetch_add(1, Ordering::SeqCst);
    +        }
    +    }
    +
    +    /// A command's registration is released when the command ends, so a
    +    /// migration does not wait for its row or document to expire; a release
    +    /// that does not answer is abandoned at the bound.
    +    #[tokio::test]
    +    async fn a_commands_registration_is_released_within_the_bound() {
    +        let quick = Arc::new(Counting {
    +            released: AtomicUsize::new(0),
    +            takes: Duration::ZERO,
    +        });
    +        release_registration(Some(quick.clone() as Arc)).await;
    +        assert_eq!(quick.released.load(Ordering::SeqCst), 1);
    +
    +        let stalled = Arc::new(Counting {
    +            released: AtomicUsize::new(0),
    +            takes: Duration::from_secs(3600),
    +        });
    +        let bound = Duration::from_millis(100);
    +        let started = std::time::Instant::now();
    +        release_registration_within(Some(stalled.clone() as Arc), bound)
    +            .await;
    +        assert_eq!(stalled.released.load(Ordering::SeqCst), 0);
    +        assert!(
    +            started.elapsed() < Duration::from_secs(5),
    +            "{:?}",
    +            started.elapsed()
    +        );
    +
    +        release_registration(None).await;
    +    }
    +}
    diff --git a/crates/app/src/cmd_destroy.rs b/crates/app/src/cmd_destroy.rs
    index a9f95e023..3966d929b 100755
    --- a/crates/app/src/cmd_destroy.rs
    +++ b/crates/app/src/cmd_destroy.rs
    @@ -3,7 +3,10 @@
     
     //! `extenddb destroy` — tear down a extenddb deployment (REQ-CAT-012).
     //!
    -//! Reads config, enumerates tables, requires `--yes` to confirm, drops both databases.
    +//! Reads config, enumerates tables, requires `--yes` to confirm, drops both
    +//! databases. The backup store (`[backup.store]`) is not touched: it is the
    +//! copy that survives destroy, and `extenddb backup purge --confirm` is the
    +//! command that deletes it.
     
     use clap::Args;
     
    @@ -25,7 +28,8 @@ pub struct DestroyArgs {
         #[arg(long)]
         pg_pass: Option,
     
    -    /// Confirm destruction (required, no interactive prompt)
    +    /// Confirm destruction (required, no interactive prompt). The backup store is left
    +    /// untouched; use `extenddb backup purge --confirm` to delete its backups.
         #[arg(long)]
         yes: bool,
     }
    @@ -90,6 +94,12 @@ pub async fn run(args: DestroyArgs) -> anyhow::Result<()> {
     
         println!();
         println!("WARNING: This will permanently destroy ALL data in both databases.");
    +    println!(
    +        "The backup store ([backup.store]) is left untouched; `extenddb backup purge --confirm` \
    +         deletes its backups. Its extenddb-store.json names this catalog: a catalog re-created \
    +         with `init` needs that file removed (or a new root), or the store is reported owned \
    +         by another catalog."
    +    );
         println!();
     
         if !args.yes {
    diff --git a/crates/app/src/cmd_serve.rs b/crates/app/src/cmd_serve.rs
    index a754aa4c2..4fd7ad61d 100755
    --- a/crates/app/src/cmd_serve.rs
    +++ b/crates/app/src/cmd_serve.rs
    @@ -174,6 +174,13 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
             backend,
             config::redact_password(app_config.storage.connection_config()),
         );
    +    let banner_line3 = format!(
    +        "  {}",
    +        extenddb_server::backup_store::banner_line(
    +            &app_config.backup,
    +            app_config.storage.database_file().as_deref(),
    +        )
    +    );
         let mut warnings = app_config.storage.startup_warnings();
         warnings.extend(app_config.server.startup_warnings());
         let startup_warnings = render_startup_warnings(&warnings);
    @@ -181,11 +188,13 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
             eprint!("{wordmark}");
             eprintln!("{banner_line1}");
             eprintln!("{banner_line2}");
    +        eprintln!("{banner_line3}");
             eprint!("{startup_warnings}");
         } else {
             print!("{wordmark}");
             println!("{banner_line1}");
             println!("{banner_line2}");
    +        println!("{banner_line3}");
             print!("{startup_warnings}");
         }
         if cfg!(feature = "dev-mode") {
    @@ -275,10 +284,14 @@ pub fn run(args: &ServeArgs, build: BuildInfo) -> anyhow::Result<()> {
         if config_exists {
             params = params.with_config_path(&args.config);
         }
    -    let result = tokio::runtime::Builder::new_multi_thread()
    +    // The runtime is ended with a bound (`run_then_shut_down`): a backup
    +    // store check abandoned on a mount that has stopped answering leaves a
    +    // blocking thread parked there, and a plain drop of the runtime would
    +    // join it, holding the process open after the drain until SIGKILL.
    +    let runtime = tokio::runtime::Builder::new_multi_thread()
             .enable_all()
    -        .build()?
    -        .block_on(extenddb_server::serve(params));
    +        .build()?;
    +    let result = crate::run_then_shut_down(runtime, extenddb_server::serve(params));
         // `serve` has printed the refusal; end with its code, which `migrate`
         // and `verify` use for the same status.
         if let Err(e) = &result
    diff --git a/crates/app/src/init_helpers.rs b/crates/app/src/init_helpers.rs
    index 15e700f3f..8be891e82 100755
    --- a/crates/app/src/init_helpers.rs
    +++ b/crates/app/src/init_helpers.rs
    @@ -312,7 +312,30 @@ key_path = "{tls_key}"
     # failed_row_retention_secs = 604800   # FAILED tombstones kept for the operator (7 days)
     # reconcile_interval_secs = 60         # Reconciler cycle
     # max_concurrent_jobs = 4              # Backup and restore jobs running at once on this instance
    -# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store
    +# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store; required
    +#                                        # with an S3 store except on file-backed SQLite, where
    +#                                        # it defaults to backup-spool/ beside the database file
    +
    +# [backup.store]
    +# Where CreateBackup's backups will live, from a later release (this release's
    +# CreateBackup writes in-catalog backups with any store, and the store serves
    +# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem
    +# store at backups/ beside the database file. Absent on PostgreSQL, MongoDB,
    +# Cassandra and an in-memory SQLite database: no store, and CreateBackup
    +# writes in-catalog backups with a deprecation warning. kind = "disabled"
    +# opts out on any backend. Instances that share a catalog must share one
    +# store root (an S3 bucket and prefix, or a shared mount).
    +# kind = "filesystem"                  # "filesystem" | "s3" | "disabled"
    +# [backup.store.filesystem]
    +# path = "/var/lib/extenddb/backups"   # Must exist, owned by the extenddb user, mode 0700
    +# [backup.store.s3]
    +# bucket = "my-extenddb-backups"
    +# prefix = "extenddb/"                 # Optional key prefix
    +# region = "us-east-1"                 # Optional; the AWS SDK's chain otherwise
    +# endpoint = "http://127.0.0.1:9000"   # Optional, for an S3-compatible server
    +# force_path_style = true              # Required with an http:// endpoint
    +# upload_part_size_bytes = 8388608     # 5 MiB to 5 GiB
    +# max_concurrent_uploads = 4
     
     # [import]
     # paths = []                     # Allowed directories for import operations
    diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs
    index 157f29d9b..d2ea052cc 100644
    --- a/crates/app/src/lib.rs
    +++ b/crates/app/src/lib.rs
    @@ -3,9 +3,9 @@
     
     //! ExtendDB application/CLI library.
     //!
    -//! Owns the command-line interface (`serve`, `init`, `destroy`, `verify`,
    -//! `migrate`, `status`, `stop`, `settings`, `manage`, `catalog-check`,
    -//! `healthcheck`) and the
    +//! Owns the command-line interface (`serve`, `init`, `destroy`, `backup`,
    +//! `verify`, `migrate`, `status`, `stop`, `settings`, `manage`,
    +//! `catalog-check`, `healthcheck`) and the
     //! subcommand dispatch. It is backend-agnostic: a backend's thin `main`
     //! installs its backend with
     //! [`set_backend`](extenddb_storage::set_backend) and then calls [`run`]:
    @@ -22,6 +22,7 @@
     //! }
     //! ```
     
    +mod cmd_backup;
     mod cmd_catalog_check;
     mod cmd_destroy;
     mod cmd_healthcheck;
    @@ -66,8 +67,10 @@ enum Command {
         Serve(cmd_serve::ServeArgs),
         /// Initialize a new extenddb deployment
         Init(cmd_init::InitArgs),
    -    /// Tear down a extenddb deployment
    +    /// Tear down a extenddb deployment (the databases; the backup store is left untouched)
         Destroy(cmd_destroy::DestroyArgs),
    +    /// List or purge the backups of the configured backup store
    +    Backup(cmd_backup::BackupArgs),
         /// Validate a extenddb deployment
         Verify(cmd_verify::VerifyArgs),
         /// Apply catalog migrations (exit 0 up to date or applied, 1 error, 3 pending without --yes,
    @@ -116,6 +119,19 @@ pub fn run(build: BuildInfo) -> anyhow::Result<()> {
                 Ok(())
             }
             Command::Destroy(args) => run_interactive(cmd_destroy::run(args)),
    +        Command::Backup(args) => {
    +            let result = run_interactive(cmd_backup::run(args, build.version));
    +            // A catalog the backend refused: the message `serve` prints for
    +            // the status, and its exit code (3 behind, 6 while a migration
    +            // runs).
    +            if let Err(e) = &result
    +                && let Some(refusal) = e.downcast_ref::()
    +            {
    +                eprintln!("Error: cannot open the catalog: {refusal}");
    +                std::process::exit(i32::from(refusal.exit_code));
    +            }
    +            result
    +        }
             Command::Verify(args) => {
                 let code = run_interactive(cmd_verify::run(args, build.version))?;
                 if code != 0 {
    @@ -191,8 +207,75 @@ fn run_interactive(
         tracing_subscriber::fmt()
             .try_init()
             .unwrap_or_else(|e| eprintln!("Warning: logging init failed: {e}"));
    -    tokio::runtime::Builder::new_current_thread()
    +    let runtime = tokio::runtime::Builder::new_current_thread()
             .enable_all()
    -        .build()?
    -        .block_on(future)
    +        .build()?;
    +    run_then_shut_down(runtime, future)
    +}
    +
    +/// How long the process waits at exit for the runtime's blocking threads
    +/// after the command's future returns. A thread parked in a filesystem call
    +/// on a mount that has stopped answering (a backup store check the engine
    +/// abandoned at its bound) returns only when the mount does; past this bound
    +/// the thread is detached and the process exits without it.
    +pub(crate) const RUNTIME_SHUTDOWN_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
    +
    +/// Run `future` to completion on `runtime`, then end the runtime within
    +/// [`RUNTIME_SHUTDOWN_TIMEOUT`]. Dropping the runtime instead joins every
    +/// blocking thread, so one parked syscall held the process open past its
    +/// shutdown until SIGKILL.
    +pub(crate) fn run_then_shut_down(
    +    runtime: tokio::runtime::Runtime,
    +    future: impl std::future::Future,
    +) -> T {
    +    let output = runtime.block_on(future);
    +    runtime.shutdown_timeout(RUNTIME_SHUTDOWN_TIMEOUT);
    +    output
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use super::{RUNTIME_SHUTDOWN_TIMEOUT, run_then_shut_down};
    +
    +    /// A blocking task that never returns (a filesystem call on a mount that
    +    /// has stopped answering) whose handle the caller dropped does not hold
    +    /// the runtime's end past the bound, on both runtimes the binary builds
    +    /// (`serve`'s multi-thread one and the commands' current-thread one).
    +    #[test]
    +    fn a_parked_blocking_thread_does_not_hold_the_runtime_past_its_bound() {
    +        let builders = [
    +            tokio::runtime::Builder::new_multi_thread(),
    +            tokio::runtime::Builder::new_current_thread(),
    +        ];
    +        for mut builder in builders {
    +            let runtime = builder.enable_all().build().unwrap();
    +            let (done, finished) = std::sync::mpsc::channel();
    +            let started = std::time::Instant::now();
    +            std::thread::spawn(move || {
    +                run_then_shut_down(runtime, async {
    +                    // Parked for good; the thread is leaked on purpose and
    +                    // dies with the test process.
    +                    drop(tokio::task::spawn_blocking(|| {
    +                        loop {
    +                            std::thread::park();
    +                        }
    +                    }));
    +                    // Let the blocking pool pick the task up before the
    +                    // runtime ends, so a thread is parked when it does.
    +                    tokio::time::sleep(std::time::Duration::from_millis(50)).await;
    +                });
    +                let _ = done.send(());
    +            });
    +            // The test's own guard, well past the bound: a shutdown that
    +            // joins the parked thread fails here instead of hanging.
    +            finished
    +                .recv_timeout(RUNTIME_SHUTDOWN_TIMEOUT * 6)
    +                .expect("the runtime ends within its shutdown bound");
    +            let elapsed = started.elapsed();
    +            assert!(
    +                elapsed < RUNTIME_SHUTDOWN_TIMEOUT * 3,
    +                "the runtime's end took {elapsed:?}"
    +            );
    +        }
    +    }
     }
    diff --git a/crates/backup-store/README.md b/crates/backup-store/README.md
    index dd6251534..ae741ecfa 100644
    --- a/crates/backup-store/README.md
    +++ b/crates/backup-store/README.md
    @@ -77,6 +77,26 @@ store's callers honour; this README is the record of what the store does.
     - `put` returns the stored object's metadata (bytes written and, on S3, the
       ETag with quotes stripped; a multipart ETag is not an MD5) so a manifest
       can be written from the put alone.
    +- `put_if_absent` creates the object only if no object is under the key and
    +  returns `None`, leaving the existing object untouched, if one is; of two
    +  concurrent calls on one key exactly one stores its body. The filesystem
    +  store writes and syncs its temporary as `put` does (created
    +  `O_CREAT | O_EXCL`) and then `link`s it to the key, which fails if
    +  anything is there, so the object appears complete or not at all; the root
    +  must therefore be on a filesystem that supports hard links (some CIFS and
    +  FUSE mounts do not, and the call fails there with `EPERM`, which the
    +  server reports as the store having failed validation, with the `link`
    +  error in the log line). The S3
    +  store sends `PutObject` (or `CompleteMultipartUpload` for a body over one
    +  part) with `If-None-Match: *`: a 412 is the key present, and a 409
    +  `ConditionalRequestConflict` from a concurrent conditional write is
    +  re-sent after a short back-off, up to eight times. An S3-compatible
    +  server that ignores `If-None-Match` on `PutObject` turns the call into a
    +  plain put, last writer wins; moto 5.2 honours it (the conformance suite's
    +  race cell checks that exactly one of eight concurrent callers stores its
    +  body), other servers have not been run. The caller that depends on the
    +  guarantee re-reads the object after its call and compares, which is what
    +  catches a server that lost the race silently.
     - The S3 store takes credentials from the AWS SDK's default chain (the
       `s3` module doc names its members, among them `credential_process`,
       which runs a command named in the shared config file, and the SSO
    diff --git a/crates/backup-store/src/config.rs b/crates/backup-store/src/config.rs
    index 002a43e59..927329603 100644
    --- a/crates/backup-store/src/config.rs
    +++ b/crates/backup-store/src/config.rs
    @@ -11,7 +11,9 @@
     //!
     //! ```toml
     //! [backup.store]
    -//! kind = "filesystem"                   # "filesystem" | "s3"; filesystem when absent
    +//! kind = "filesystem"                   # "filesystem" | "s3" | "disabled"; filesystem when absent
    +//!                                       # ("disabled" configures no store on purpose; the server
    +//!                                       # then creates in-catalog backups and logs that)
     //!
     //! [backup.store.filesystem]
     //! path = "/backups"           # required when kind = "filesystem"; the server defaults it
    @@ -167,6 +169,21 @@ pub enum BackupStoreKind {
         Filesystem,
         /// S3 bucket store.
         S3,
    +    /// No store, on purpose: the opt-out of the server's per-backend default
    +    /// root. [`crate::open`] refuses it; a caller checks the kind first.
    +    Disabled,
    +}
    +
    +impl BackupStoreKind {
    +    /// The value `kind` takes in the configuration.
    +    #[must_use]
    +    pub const fn as_str(self) -> &'static str {
    +        match self {
    +            Self::Filesystem => "filesystem",
    +            Self::S3 => "s3",
    +            Self::Disabled => "disabled",
    +        }
    +    }
     }
     
     /// The `[backup.store]` table.
    @@ -175,7 +192,7 @@ pub enum BackupStoreKind {
     /// survive `serde(flatten)`, so the server nests this struct under its own
     /// key (`store`) in its `[backup]` model rather than flattening it; that
     /// keeps one definition of these fields and unknown-field rejection on both.
    -#[derive(Debug, Clone, Deserialize)]
    +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct BackupStoreConfig {
         /// Selected store kind; `filesystem` when absent.
    @@ -190,7 +207,7 @@ pub struct BackupStoreConfig {
     }
     
     /// `[backup.store.filesystem]` settings.
    -#[derive(Debug, Clone, Deserialize)]
    +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct FilesystemStoreConfig {
         /// Root directory for backup objects. Must exist at startup.
    @@ -201,7 +218,7 @@ pub struct FilesystemStoreConfig {
     /// from the standard AWS SDK chain. `Debug` prints the endpoint without its
     /// user information, which `open` refuses but a log of the configuration
     /// written before `open` would otherwise carry.
    -#[derive(Clone, Deserialize)]
    +#[derive(Clone, PartialEq, Eq, Deserialize)]
     #[serde(deny_unknown_fields)]
     pub struct S3StoreConfig {
         /// Destination bucket.
    @@ -375,6 +392,15 @@ mod tests {
             assert!(toml::from_str::("kind = \"gcs\"").is_err());
         }
     
    +    #[tokio::test]
    +    async fn disabled_parses_and_does_not_open() {
    +        let config: BackupStoreConfig = toml::from_str("kind = \"disabled\"").unwrap();
    +        assert_eq!(config.kind, BackupStoreKind::Disabled);
    +        assert_eq!(BackupStoreKind::Disabled.as_str(), "disabled");
    +        let err = open_err(&config).await;
    +        assert!(err.to_string().contains("disabled"), "{err}");
    +    }
    +
         #[tokio::test]
         async fn open_requires_the_matching_section() {
             let missing_fs = BackupStoreConfig {
    diff --git a/crates/backup-store/src/fs.rs b/crates/backup-store/src/fs.rs
    index 48bcb5895..fff5547dc 100644
    --- a/crates/backup-store/src/fs.rs
    +++ b/crates/backup-store/src/fs.rs
    @@ -80,6 +80,16 @@
     //! concurrent put into the same new directory, and could not cover a
     //! failure inside the creation walk itself.)
     //!
    +//! `put_if_absent` writes and syncs its temporary the same way, created
    +//! `O_CREAT | O_EXCL` as every temporary is, and then `link`s it to the key
    +//! instead of renaming: `link(2)` fails with `EEXIST` when anything is at
    +//! the new path, so of two concurrent calls on one key exactly one links and
    +//! the other reads `None`, the first body stays, and the object appears
    +//! under its key complete or not at all. The temporary is unlinked after the
    +//! link on every path (the two names are one inode, so the object stays). A
    +//! regular file already at the destination is reported `None` before the
    +//! body is read.
    +//!
     //! Where this store narrows the shared key space (temporary names, a file
     //! and a directory at one path, the filesystem's own name limit and case
     //! folding) is listed in the `key` module doc.
    @@ -142,10 +152,15 @@ impl FilesystemStore {
         ///
         /// # Errors
         ///
    -    /// Returns an error when the root does not exist, cannot be resolved, or
    -    /// is not a directory.
    +    /// Returns an error when the root path is empty, the root does not exist,
    +    /// cannot be resolved, or is not a directory.
         pub async fn open(root: impl AsRef) -> Result {
             let root = root.as_ref();
    +        if root.as_os_str().is_empty() {
    +            return Err(StoreError::Other(
    +                "the backup root path is empty; name the directory".to_owned(),
    +            ));
    +        }
             let canonical = tokio::fs::canonicalize(root).await.map_err(|e| {
                 StoreError::io(format!("backup root {} is not usable", root.display()), e)
             })?;
    @@ -165,6 +180,14 @@ impl FilesystemStore {
             })
         }
     
    +    /// The root label a catalog row records for this store: the canonical
    +    /// root directory. Two servers configured for one directory (by any
    +    /// path that resolves to it) spell the same label.
    +    #[must_use]
    +    pub fn store_root(&self) -> String {
    +        self.root.display().to_string()
    +    }
    +
         /// Run `hook` on the blocking thread of every `delete_prefix` walk
         /// right before each removal (a file, a symlink or a directory), with
         /// the number of removals that walk has made so far. The crate's own
    @@ -313,7 +336,14 @@ impl FilesystemStore {
             Ok(current)
         }
     
    -    async fn put_impl(&self, key: &str, mut body: ByteStream) -> Result {
    +    /// Resolve `key` to the directory its object is written into and the
    +    /// object's final path, creating the directories (`prepare_parent`) and
    +    /// refusing a symlink or a directory at the destination: a put never
    +    /// resolves through a link, and a directory there is a key-space
    +    /// collision (`a/b` exists, `a` is put), refused before the body is read
    +    /// rather than after the rename or link fails. Reports whether a regular
    +    /// file is already at the destination, which `put_if_absent` reads.
    +    async fn destination(&self, key: &str) -> Result {
             validate_key(key)?;
             let components: Vec<&str> = key.split('/').collect();
             let (file_name, parent_components) = components
    @@ -321,12 +351,7 @@ impl FilesystemStore {
                 .expect("a validated key has at least one component");
             let parent = self.prepare_parent(parent_components).await?;
             let final_path = parent.join(file_name);
    -
    -        // A symlink at the destination, in-root or escaping, is refused: a
    -        // put never resolves through a link. A directory there is a key-space
    -        // collision (`a/b` exists, `a` is put), refused before the body is
    -        // read rather than after the rename fails.
    -        match tokio::fs::symlink_metadata(&final_path).await {
    +        let exists = match tokio::fs::symlink_metadata(&final_path).await {
                 Ok(meta) if meta.is_symlink() => {
                     return Err(symlink_refused(&final_path));
                 }
    @@ -339,13 +364,101 @@ impl FilesystemStore {
                         ),
                     ));
                 }
    -            Ok(_) | Err(_) => {}
    -        }
    +            Ok(_) => true,
    +            Err(_) => false,
    +        };
    +        Ok(Destination {
    +            parent,
    +            final_path,
    +            exists,
    +        })
    +    }
     
    +    /// Write `body` to a new temporary sibling of `final_path`, flushed and
    +    /// synced, and return its path, the guard that removes it, and the
    +    /// number of bytes written. The temporary is created `O_CREAT | O_EXCL`
    +    /// with mode `0600` (`create_private_file`).
    +    ///
    +    /// The temporary and its guard come back together. The guard removes
    +    /// the temporary on every exit the caller does not disarm it for: an
    +    /// error, the put's future dropped (a caller's timeout, a cancelled
    +    /// job), including while the open itself is in flight on the blocking
    +    /// pool, or a body that panics.
    +    async fn write_temporary(
    +        parent: &Path,
    +        mut body: ByteStream,
    +    ) -> Result<(PathBuf, RemoveOnDrop, u64), StoreError> {
             let temp_path = parent.join(format!(
                 "{TEMP_PREFIX}{}{TEMP_SUFFIX}",
                 uuid::Uuid::new_v4()
             ));
    +        let (mut file, temp_guard) = create_private_file(&temp_path).await?;
    +        let mut size: u64 = 0;
    +        while let Some(chunk) = body.next().await {
    +            let chunk = chunk?;
    +            file.write_all(&chunk)
    +                .await
    +                .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    +            size += chunk.len() as u64;
    +        }
    +        // tokio's `File` hands each write to a background task and
    +        // reports its error on the next operation. `sync_all` waits for
    +        // the last background write but does not return its error (tokio
    +        // keeps only its kind for a later write or flush), so without a
    +        // flush before the drop a short file would be renamed into place
    +        // on a full disk (or past RLIMIT_FSIZE). `flush` returns it.
    +        file.flush()
    +            .await
    +            .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    +        // Defence in depth, no test: the fsync of the temporary matters
    +        // only across a crash or power loss, which the suite cannot
    +        // inject.
    +        file.sync_all()
    +            .await
    +            .map_err(|e| StoreError::io(format!("sync {}", temp_path.display()), e))?;
    +        drop(file);
    +        Ok((temp_path, temp_guard, size))
    +    }
    +
    +    /// Sync `parent` and every directory between it and the root, so the
    +    /// entry a rename or link just made, and the directories a put created
    +    /// on the way, are durable. Syncing every ancestor is a few extra
    +    /// fsyncs per put (one per path component) against a bounded depth;
    +    /// cheaper than tracking which ones this put created, and correct when a
    +    /// concurrent put created them. Defence in depth, no test: it matters
    +    /// only across a crash.
    +    async fn sync_parent_chain(&self, parent: &Path) -> Result<(), StoreError> {
    +        sync_dir(parent).await?;
    +        let mut dir = parent.to_path_buf();
    +        while let Some(p) = dir.parent() {
    +            if !p.starts_with(&self.root) || p == self.root {
    +                sync_dir(&self.root).await?;
    +                break;
    +            }
    +            sync_dir(p).await?;
    +            dir = p.to_path_buf();
    +        }
    +        Ok(())
    +    }
    +
    +    /// The metadata of the object now at `final_path`.
    +    async fn stored_meta(key: &str, final_path: &Path, size: u64) -> ObjectMeta {
    +        let last_modified = tokio::fs::symlink_metadata(final_path)
    +            .await
    +            .ok()
    +            .and_then(|m| m.modified().ok());
    +        ObjectMeta {
    +            key: key.to_owned(),
    +            size,
    +            last_modified,
    +            etag: None,
    +        }
    +    }
    +
    +    async fn put_impl(&self, key: &str, body: ByteStream) -> Result {
    +        let Destination {
    +            parent, final_path, ..
    +        } = self.destination(key).await?;
     
             // Durability contract: the temp file is synced before the rename and
             // the parent directory is synced after it, so an Ok return means both
    @@ -353,37 +466,9 @@ impl FilesystemStore {
             // directory sync the rename itself could be lost on power failure
             // even though the file data had been written.
             async {
    -            // The temporary and its guard come back together. The guard
    -            // removes it on every exit but the rename: an error (`?` below),
    -            // the put's future dropped (a caller's timeout, a cancelled
    -            // job), including while the open itself is in flight on the
    -            // blocking pool, or a body that panics. It lives inside this
    -            // block so dropping the block's future drops it.
    -            let (mut file, mut temp_guard) = create_private_file(&temp_path).await?;
    -            let mut size: u64 = 0;
    -            while let Some(chunk) = body.next().await {
    -                let chunk = chunk?;
    -                file.write_all(&chunk)
    -                    .await
    -                    .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    -                size += chunk.len() as u64;
    -            }
    -            // tokio's `File` hands each write to a background task and
    -            // reports its error on the next operation. `sync_all` waits for
    -            // the last background write but does not return its error (tokio
    -            // keeps only its kind for a later write or flush), so without a
    -            // flush before the drop a short file would be renamed into place
    -            // on a full disk (or past RLIMIT_FSIZE). `flush` returns it.
    -            file.flush()
    -                .await
    -                .map_err(|e| StoreError::io(format!("write {}", temp_path.display()), e))?;
    -            // Defence in depth, no test: the fsync of the temporary matters
    -            // only across a crash or power loss, which the suite cannot
    -            // inject.
    -            file.sync_all()
    -                .await
    -                .map_err(|e| StoreError::io(format!("sync {}", temp_path.display()), e))?;
    -            drop(file);
    +            // The guard lives inside this block so dropping the block's
    +            // future drops it (`write_temporary`).
    +            let (temp_path, mut temp_guard, size) = Self::write_temporary(&parent, body).await?;
                 tokio::fs::rename(&temp_path, &final_path)
                     .await
                     .map_err(|e| {
    @@ -400,32 +485,56 @@ impl FilesystemStore {
                 // before the put into one that holds nothing, which is worse
                 // than either complete object (the trait contract is "no partial
                 // object", not "no object").
    -            sync_dir(&parent).await?;
    -            // The directories between the root and the parent are durable
    -            // only once their parents' entries are synced too. Syncing every
    -            // ancestor is a few extra fsyncs per put (one per path component)
    -            // against a bounded depth; cheaper than tracking which ones this
    -            // put created, and correct when a concurrent put created them.
    -            // Defence in depth, no test: it matters only across a crash.
    -            let mut dir = parent.clone();
    -            while let Some(p) = dir.parent() {
    -                if !p.starts_with(&self.root) || p == self.root {
    -                    sync_dir(&self.root).await?;
    -                    break;
    +            self.sync_parent_chain(&parent).await?;
    +            Ok(Self::stored_meta(key, &final_path, size).await)
    +        }
    +        .await
    +    }
    +
    +    /// `put_if_absent`: the temporary is written and synced as `put` writes
    +    /// it, then `link`ed to the key. `link(2)` fails with `EEXIST` when
    +    /// anything is at the new path (a file, a directory, a symlink, whatever
    +    /// it points at), so the object appears under its key complete or not at
    +    /// all, and an object that was there first is untouched. A regular file
    +    /// already at the destination skips the body: the link would fail, and
    +    /// writing the body first would only cost disk. The temporary is removed
    +    /// after the link on every path; the link and the temporary are one
    +    /// inode, so the object stays.
    +    async fn put_if_absent_impl(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +    ) -> Result, StoreError> {
    +        let Destination {
    +            parent,
    +            final_path,
    +            exists,
    +        } = self.destination(key).await?;
    +        if exists {
    +            return Ok(None);
    +        }
    +        async {
    +            let (temp_path, temp_guard, size) = Self::write_temporary(&parent, body).await?;
    +            let linked = tokio::fs::hard_link(&temp_path, &final_path).await;
    +            // The temporary's name is surplus from here on whether the
    +            // link landed or not; the guard removes it when it drops at the
    +            // end of this block (and on every early return below).
    +            match linked {
    +                Ok(()) => {}
    +                Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {
    +                    drop(temp_guard);
    +                    return Ok(None);
    +                }
    +                Err(e) => {
    +                    return Err(StoreError::io(
    +                        format!("link {} to {}", temp_path.display(), final_path.display()),
    +                        e,
    +                    ));
                     }
    -                sync_dir(p).await?;
    -                dir = p.to_path_buf();
                 }
    -            let last_modified = tokio::fs::symlink_metadata(&final_path)
    -                .await
    -                .ok()
    -                .and_then(|m| m.modified().ok());
    -            Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified,
    -                etag: None,
    -            })
    +            drop(temp_guard);
    +            self.sync_parent_chain(&parent).await?;
    +            Ok(Some(Self::stored_meta(key, &final_path, size).await))
             }
             .await
         }
    @@ -512,15 +621,22 @@ impl FilesystemStore {
         }
     
         async fn validate_impl(&self) -> Result<(), StoreError> {
    -        let meta = tokio::fs::symlink_metadata(&self.root).await.map_err(|e| {
    -            StoreError::io(
    -                format!("backup root {} is not usable", self.root.display()),
    -                e,
    -            )
    -        })?;
    -        check_root_is_directory(&self.root, &meta)?;
    -        check_root_location(&self.root)?;
    -        check_root_ownership(&self.root, &meta)?;
    +        // The root's own checks are synchronous filesystem calls (`lstat` of
    +        // the root and each ancestor, `realpath`); they run on the blocking
    +        // pool so a root on a mount that has stopped answering parks a
    +        // blocking thread and not the runtime, and a caller's timeout on
    +        // this future can fire.
    +        let root = self.root.clone();
    +        tokio::task::spawn_blocking(move || {
    +            let meta = std::fs::symlink_metadata(&root).map_err(|e| {
    +                StoreError::io(format!("backup root {} is not usable", root.display()), e)
    +            })?;
    +            check_root_is_directory(&root, &meta)?;
    +            check_root_location(&root)?;
    +            check_root_ownership(&root, &meta)
    +        })
    +        .await
    +        .map_err(|e| StoreError::Other(format!("root check task failed: {e}")))??;
             let probe = self.root.join(format!(
                 "{TEMP_PREFIX}validate-{}{TEMP_SUFFIX}",
                 uuid::Uuid::new_v4()
    @@ -574,6 +690,15 @@ impl FilesystemStore {
         }
     }
     
    +/// Where a put writes: the directory the object goes into (created and
    +/// checked by `prepare_parent`), the object's final path, and whether a
    +/// regular file is already there.
    +struct Destination {
    +    parent: PathBuf,
    +    final_path: PathBuf,
    +    exists: bool,
    +}
    +
     impl BackupStore for FilesystemStore {
         fn put<'a>(
             &'a self,
    @@ -583,6 +708,14 @@ impl BackupStore for FilesystemStore {
             self.put_impl(key, body).boxed()
         }
     
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        self.put_if_absent_impl(key, body).boxed()
    +    }
    +
         fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
             self.get_impl(key).boxed()
         }
    @@ -624,6 +757,28 @@ impl BackupStore for FilesystemStore {
     /// cannot detect. Messages name the path and the fix. No check is made on
     /// other platforms.
     fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), StoreError> {
    +    check_private_tree(root, meta, "backup root")
    +}
    +
    +/// The owner-only rule of a directory the server keeps private, by its
    +/// canonical path: `check_root_ownership`'s rule (the directory owned by
    +/// the effective user with no group or other permission bits; every ancestor
    +/// a directory owned by the effective user or by root, not writable by group
    +/// or others unless sticky) for a directory that is not a store root, named
    +/// `what` in the messages (the server's restore spool). No check is made on
    +/// other platforms.
    +///
    +/// # Errors
    +///
    +/// `PermissionDenied` naming the directory or ancestor and the mode it
    +/// needs; `Io` when a directory on the way cannot be read.
    +pub fn check_private_directory(dir: &Path, what: &str) -> Result<(), StoreError> {
    +    let meta = std::fs::symlink_metadata(dir)
    +        .map_err(|e| StoreError::io(format!("stat {}", dir.display()), e))?;
    +    check_private_tree(dir, &meta, what)
    +}
    +
    +fn check_private_tree(root: &Path, meta: &std::fs::Metadata, what: &str) -> Result<(), StoreError> {
         #[cfg(unix)]
         {
             use std::os::unix::fs::MetadataExt;
    @@ -637,8 +792,8 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
                 // only a swap between the two reaches this (module doc).
                 if m.is_symlink() {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) is a symlink; configure the backup \
    -                     root by its resolved path",
    +                    "{} (an ancestor of the {what}) is a symlink; configure the {what} by its \
    +                     resolved path",
                         dir.display()
                     )));
                 }
    @@ -646,17 +801,17 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
                 let sticky = mode & 0o1000 != 0;
                 if m.uid() != euid && m.uid() != 0 {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) is owned by uid {}; the backup root \
    -                     must sit under directories owned by root or by the extenddb user",
    +                    "{} (an ancestor of the {what}) is owned by uid {}; the {what} must sit \
    +                     under directories owned by root or by the extenddb user",
                         dir.display(),
                         m.uid()
                     )));
                 }
                 if mode & 0o022 != 0 && !sticky {
                     return Err(StoreError::PermissionDenied(format!(
    -                    "{} (an ancestor of the backup root) has mode {mode:04o} and is writable by \
    -                     group or others without the sticky bit; another user could replace the \
    -                     root. chmod it to remove group and world write, or move the backup root",
    +                    "{} (an ancestor of the {what}) has mode {mode:04o} and is writable by group \
    +                     or others without the sticky bit; another user could replace the {what}. \
    +                     chmod it to remove group and world write, or move the {what}",
                         dir.display()
                     )));
                 }
    @@ -665,7 +820,7 @@ fn check_root_ownership(root: &Path, meta: &std::fs::Metadata) -> Result<(), Sto
         }
         #[cfg(not(unix))]
         {
    -        let _ = (root, meta);
    +        let _ = (root, meta, what);
         }
         Ok(())
     }
    diff --git a/crates/backup-store/src/lib.rs b/crates/backup-store/src/lib.rs
    index 5793dce10..23e7526ad 100644
    --- a/crates/backup-store/src/lib.rs
    +++ b/crates/backup-store/src/lib.rs
    @@ -33,8 +33,11 @@ mod s3;
     
     pub use config::{BackupStoreConfig, BackupStoreKind, FilesystemStoreConfig, S3StoreConfig};
     pub use error::StoreError;
    -pub use fs::FilesystemStore;
    -pub use s3::{MAX_PART_SIZE_BYTES, MAX_STORE_PREFIX_LEN, MIN_PART_SIZE_BYTES, S3Store};
    +pub use fs::{FilesystemStore, check_private_directory};
    +pub use s3::{
    +    ATTEMPT_TIMEOUT as S3_ATTEMPT_TIMEOUT, MAX_PART_SIZE_BYTES, MAX_STORE_PREFIX_LEN,
    +    MIN_PART_SIZE_BYTES, S3Store,
    +};
     
     /// Byte stream flowing into and out of a store.
     ///
    @@ -106,6 +109,31 @@ pub trait BackupStore: Send + Sync {
             body: ByteStream,
         ) -> BoxFuture<'a, Result>;
     
    +    /// Store `body` under `key` only if no object is there, and return the
    +    /// stored object's metadata; return `None`, leaving the existing object
    +    /// untouched and reading none of `body` the store did not already read,
    +    /// if one is. Of two concurrent calls on one key exactly one stores its
    +    /// body. The object appears under its key complete or not at all, as
    +    /// with [`BackupStore::put`], and a call that is dropped or fails before
    +    /// the object is in place leaves nothing (the same guards as `put`); a
    +    /// filesystem call whose `link` succeeded and whose parent-directory
    +    /// sync then fails returns `Err` with the object in place, as `put`
    +    /// does after its `rename`. Written for the store identity file; the
    +    /// mechanics are the stores': S3 sends `PutObject` (or, for a body over
    +    /// one part, `CompleteMultipartUpload`) with `If-None-Match: *`, where a
    +    /// 412 means the key is present and a 409 `ConditionalRequestConflict`
    +    /// from a concurrent conditional write is retried; the filesystem store
    +    /// writes and syncs its temporary as `put` does, created
    +    /// `O_CREAT | O_EXCL`, then `link`s it to the key, which fails if the
    +    /// key exists. An S3-compatible server that ignores `If-None-Match` on
    +    /// `PutObject` makes this a plain put, last writer wins; the README
    +    /// names it, and the caller's re-read is what catches it.
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>>;
    +
         /// Stream the object stored under `key`.
         ///
         /// Returns [`StoreError::NotFound`] when no object exists.
    @@ -183,13 +211,37 @@ pub trait BackupStore: Send + Sync {
     ///
     /// # Errors
     ///
    -/// Returns an error when the section matching the selected store kind is
    +/// Returns an error when `kind` is `disabled` (no store to open; the caller
    +/// checks the kind first), when the section matching the selected store kind
    +/// is
     /// absent, a value fails validation (an empty bucket, an S3 Express bucket
     /// name, a part size outside 5 MiB to 5 GiB, zero concurrency, a malformed
     /// prefix, an endpoint setting that breaks the rules in the `config` module
     /// doc, the SDK's own and its credential providers' and IMDS client's
     /// included), or the filesystem root is unusable.
     pub async fn open(config: &BackupStoreConfig) -> Result, StoreError> {
    +    open_described(config).await.map(|opened| opened.store)
    +}
    +
    +/// A store [`open_described`] built, with the two facts a catalog row records
    +/// about it.
    +pub struct OpenedStore {
    +    /// The store.
    +    pub store: Arc,
    +    /// Which kind it is (never [`BackupStoreKind::Disabled`]).
    +    pub kind: BackupStoreKind,
    +    /// The root label: the canonical directory of a filesystem store
    +    /// ([`FilesystemStore::store_root`]), `s3:///` with an
    +    /// optional `?endpoint=` for an S3 store ([`S3Store::store_root`]).
    +    pub root: String,
    +}
    +
    +/// [`open`], also returning the store's kind and root label.
    +///
    +/// # Errors
    +///
    +/// As [`open`].
    +pub async fn open_described(config: &BackupStoreConfig) -> Result {
         match config.kind {
             BackupStoreKind::Filesystem => {
                 let fs_config = config.filesystem.as_ref().ok_or_else(|| {
    @@ -198,7 +250,13 @@ pub async fn open(config: &BackupStoreConfig) -> Result, St
                             .to_owned(),
                     )
                 })?;
    -            Ok(Arc::new(FilesystemStore::open(&fs_config.path).await?))
    +            let store = FilesystemStore::open(&fs_config.path).await?;
    +            let root = store.store_root();
    +            Ok(OpenedStore {
    +                store: Arc::new(store),
    +                kind: BackupStoreKind::Filesystem,
    +                root,
    +            })
             }
             BackupStoreKind::S3 => {
                 let s3_config = config.s3.as_ref().ok_or_else(|| {
    @@ -206,7 +264,16 @@ pub async fn open(config: &BackupStoreConfig) -> Result, St
                         "[backup.store.s3] with a bucket is required when kind = \"s3\"".to_owned(),
                     )
                 })?;
    -            Ok(Arc::new(S3Store::open(s3_config).await?))
    +            let store = S3Store::open(s3_config).await?;
    +            let root = store.store_root().to_owned();
    +            Ok(OpenedStore {
    +                store: Arc::new(store),
    +                kind: BackupStoreKind::S3,
    +                root,
    +            })
             }
    +        BackupStoreKind::Disabled => Err(StoreError::Other(
    +            "[backup.store] kind = \"disabled\" configures no store; nothing to open".to_owned(),
    +        )),
         }
     }
    diff --git a/crates/backup-store/src/s3.rs b/crates/backup-store/src/s3.rs
    index 64c7e2a9d..465a23f32 100644
    --- a/crates/backup-store/src/s3.rs
    +++ b/crates/backup-store/src/s3.rs
    @@ -166,7 +166,10 @@ const _: () = {
     /// would otherwise hang a put for as long as the server keeps the socket.
     /// The attempt timeout includes sending the request body, so each part must
     /// upload within 60 s (see `upload_part_size_bytes` in the config module).
    -const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
    +/// Exported as the crate's `S3_ATTEMPT_TIMEOUT`: a caller that bounds a
    +/// sequence of small requests (the engine's store check) stays under one
    +/// attempt, so its bound fires before the SDK starts a retry.
    +pub const ATTEMPT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
     const OPERATION_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(300);
     
     /// Longest wait for the next chunk of a `GetObject` body. The two timeouts
    @@ -232,6 +235,8 @@ pub struct S3Store {
         /// Longest wait for the next chunk of a `GetObject` body;
         /// [`BODY_IDLE_TIMEOUT`] except in the crate's own tests.
         body_idle_timeout: std::time::Duration,
    +    /// The root label catalog rows record for this store ([`Self::store_root`]).
    +    root: String,
     }
     
     impl S3Store {
    @@ -436,6 +441,14 @@ impl S3Store {
                 // this keeps a client that reaches one (the test-only `open`)
                 // from sending the request at all.
                 .disable_s3_express_session_auth(true);
    +        // The label names the endpoint the client uses, whichever source
    +        // supplied it, so two servers reaching one bucket through one
    +        // endpoint spell one label however each was told about it.
    +        let root = store_root_label(
    +            &config.bucket,
    +            &prefix,
    +            endpoint.as_ref().map(|(endpoint, _)| endpoint.as_str()),
    +        );
             builder.set_endpoint_url(endpoint.map(|(endpoint, _)| endpoint));
             // The SDK's default client sends through the proxy named in
             // `HTTP_PROXY`, `http_proxy`, `ALL_PROXY` or `all_proxy`, which
    @@ -472,9 +485,24 @@ impl S3Store {
                 max_parts: MAX_PARTS,
                 page_size: None,
                 body_idle_timeout: BODY_IDLE_TIMEOUT,
    +            root,
             })
         }
     
    +    /// The root label a catalog row records for this store:
    +    /// `s3:///`, with `?endpoint=` appended when the
    +    /// client uses an endpoint other than the SDK's default (from `endpoint`
    +    /// or from the SDK's environment and shared config), the URL in one
    +    /// spelling: lowercase scheme and host, no default port, no `/` for an
    +    /// empty path, no user information, query or fragment. Two servers
    +    /// reaching the same bucket and prefix through the same endpoint spell
    +    /// the same label however each was told about the endpoint, and so
    +    /// address the same rows.
    +    #[must_use]
    +    pub fn store_root(&self) -> &str {
    +        &self.root
    +    }
    +
         /// Bucket-side key for a store key. S3 bounds the whole object key at
         /// [`crate::key::MAX_KEY_LEN`] bytes, prefix included, so a key that fits
         /// the shared rule can still be too long under a long prefix; refuse it
    @@ -493,6 +521,36 @@ impl S3Store {
         }
     
         async fn put_impl(&self, key: &str, body: ByteStream) -> Result {
    +        // `put_inner` answers `None` only to a conditional put.
    +        self.put_inner(key, body, false)
    +            .await?
    +            .ok_or_else(|| StoreError::Other(format!("s3 PutObject of {key} returned no object")))
    +    }
    +
    +    async fn put_if_absent_impl(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +    ) -> Result, StoreError> {
    +        self.put_inner(key, body, true).await
    +    }
    +
    +    /// The one upload path behind `put` and `put_if_absent`. With
    +    /// `if_none_match`, the request that creates the object (`PutObject` for
    +    /// a body of one part, `CompleteMultipartUpload` otherwise) carries
    +    /// `If-None-Match: *`: a 412 means an object is under the key and `None`
    +    /// is returned (the multipart upload is aborted first, since the parts
    +    /// would otherwise stay); a 409 `ConditionalRequestConflict` means a
    +    /// concurrent conditional write on the key was in flight, and the request
    +    /// is re-sent after a short back-off, up to [`CONDITIONAL_RETRIES`] times
    +    /// (the parts already uploaded are reused), after which the last answer
    +    /// is reported as the error it is.
    +    async fn put_inner(
    +        &self,
    +        key: &str,
    +        body: ByteStream,
    +        if_none_match: bool,
    +    ) -> Result, StoreError> {
             validate_key(key)?;
             let full_key = self.full_key(key)?;
             let mut chunks = std::pin::pin!(chunk_stream(body, self.part_size));
    @@ -506,21 +564,38 @@ impl S3Store {
             let Some(second) = second else {
                 // The whole body fits in one part: a single PutObject.
                 let size = first.len() as u64;
    -            let output = self
    -                .client
    -                .put_object()
    -                .bucket(&self.bucket)
    -                .key(&full_key)
    -                .body(first.into())
    -                .send()
    -                .await
    -                .map_err(|e| map_sdk_error("PutObject", e))?;
    -            return Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified: None,
    -                etag: output.e_tag().map(strip_etag_quotes),
    -            });
    +            let mut attempt = 0;
    +            loop {
    +                let mut request = self
    +                    .client
    +                    .put_object()
    +                    .bucket(&self.bucket)
    +                    .key(&full_key)
    +                    .body(first.clone().into());
    +                if if_none_match {
    +                    request = request.if_none_match("*");
    +                }
    +                match request.send().await {
    +                    Ok(output) => {
    +                        return Ok(Some(ObjectMeta {
    +                            key: key.to_owned(),
    +                            size,
    +                            last_modified: None,
    +                            etag: output.e_tag().map(strip_etag_quotes),
    +                        }));
    +                    }
    +                    Err(e) => match conditional_outcome(&e) {
    +                        Some(Conditional::Present) if if_none_match => return Ok(None),
    +                        Some(Conditional::Conflict)
    +                            if if_none_match && attempt < CONDITIONAL_RETRIES =>
    +                        {
    +                            attempt += 1;
    +                            tokio::time::sleep(conditional_backoff(attempt)).await;
    +                        }
    +                        _ => return Err(map_sdk_error("PutObject", e)),
    +                    },
    +                }
    +            }
             };
     
             // Every failure after CreateMultipartUpload must reach the abort,
    @@ -537,38 +612,62 @@ impl S3Store {
             // is in flight.
             let mut abort_guard = self.create_upload_guarded(&full_key).await?;
             let upload_id = abort_guard.upload_id.clone();
    -        let outcome: Result = async {
    +        let outcome: Result, StoreError> = async {
                 let (parts, size) = self
                     .upload_parts(&full_key, &upload_id, first, second, chunks)
                     .await?;
    -            let output = self
    -                .client
    -                .complete_multipart_upload()
    -                .bucket(&self.bucket)
    -                .key(&full_key)
    -                .upload_id(&upload_id)
    -                .multipart_upload(
    -                    CompletedMultipartUpload::builder()
    -                        .set_parts(Some(parts))
    -                        .build(),
    -                )
    -                .send()
    -                .await
    -                .map_err(|e| map_sdk_error("CompleteMultipartUpload", e))?;
    -            Ok(ObjectMeta {
    -                key: key.to_owned(),
    -                size,
    -                last_modified: None,
    -                etag: output.e_tag().map(strip_etag_quotes),
    -            })
    +            let mut attempt = 0;
    +            loop {
    +                let mut request = self
    +                    .client
    +                    .complete_multipart_upload()
    +                    .bucket(&self.bucket)
    +                    .key(&full_key)
    +                    .upload_id(&upload_id)
    +                    .multipart_upload(
    +                        CompletedMultipartUpload::builder()
    +                            .set_parts(Some(parts.clone()))
    +                            .build(),
    +                    );
    +                if if_none_match {
    +                    request = request.if_none_match("*");
    +                }
    +                match request.send().await {
    +                    Ok(output) => {
    +                        return Ok(Some(ObjectMeta {
    +                            key: key.to_owned(),
    +                            size,
    +                            last_modified: None,
    +                            etag: output.e_tag().map(strip_etag_quotes),
    +                        }));
    +                    }
    +                    Err(e) => match conditional_outcome(&e) {
    +                        Some(Conditional::Present) if if_none_match => return Ok(None),
    +                        Some(Conditional::Conflict)
    +                            if if_none_match && attempt < CONDITIONAL_RETRIES =>
    +                        {
    +                            attempt += 1;
    +                            tokio::time::sleep(conditional_backoff(attempt)).await;
    +                        }
    +                        _ => return Err(map_sdk_error("CompleteMultipartUpload", e)),
    +                    },
    +                }
    +            }
             }
             .await;
     
             // The guard stays armed until the inline abort has returned: a put
             // dropped while that abort is in flight (a caller's timeout) still
    -        // has the guard abort the upload.
    +        // has the guard abort the upload. A conditional complete that found
    +        // the key present leaves its upload open too, so it is aborted the
    +        // same way.
             let meta = match outcome {
    -            Ok(meta) => meta,
    +            Ok(Some(meta)) => meta,
    +            Ok(None) => {
    +                abort_upload(&self.client, &self.bucket, &full_key, &upload_id).await;
    +                abort_guard.disarm();
    +                return Ok(None);
    +            }
                 Err(e) => {
                     // Best effort: the original failure is what the caller needs.
                     abort_upload(&self.client, &self.bucket, &full_key, &upload_id).await;
    @@ -577,7 +676,7 @@ impl S3Store {
                 }
             };
             abort_guard.disarm();
    -        Ok(meta)
    +        Ok(Some(meta))
         }
     
         /// Send `CreateMultipartUpload` for the bucket-side key `full_key` and
    @@ -1061,10 +1160,7 @@ impl S3Store {
                              ({}; {reason})",
                             self.bucket, PROBE_LIST_OBJECTS.grant
                         )),
    -                    other => StoreError::Other(format!(
    -                        "backup bucket {} failed validation: {other}",
    -                        self.bucket
    -                    )),
    +                    other => validation_failure(&self.bucket, other),
                     }
                 })?;
             self.validate_write_grants().await
    @@ -1080,10 +1176,7 @@ impl S3Store {
                     "the configured credentials lack {grant} on backup bucket {} ({reason}){extra}",
                     self.bucket
                 )),
    -            other => StoreError::Other(format!(
    -                "backup bucket {} failed validation: {other}",
    -                self.bucket
    -            )),
    +            other => validation_failure(&self.bucket, other),
             };
             self.client
                 .delete_object()
    @@ -1183,6 +1276,14 @@ impl BackupStore for S3Store {
             self.put_impl(key, body).boxed()
         }
     
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        self.put_if_absent_impl(key, body).boxed()
    +    }
    +
         fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
             self.get_impl(key).boxed()
         }
    @@ -1258,6 +1359,59 @@ impl BackupStore for S3Store {
         }
     }
     
    +/// [`S3Store::store_root`]'s rule, over the normalized prefix and the
    +/// endpoint the client uses (the configured one, or the one the SDK's
    +/// environment and shared config supplied). The endpoint is spelled one
    +/// way whatever the operator typed: scheme and host lowercased, a default
    +/// port (`80` for `http`, `443` for `https`) dropped, an empty path spelled
    +/// without its `/`, and (defence in depth for a label built before `open`'s
    +/// checks ran) user information, query and fragment removed. An endpoint
    +/// `http::Uri` accepts but `url` does not parse is spelled as typed, minus
    +/// those three parts.
    +fn store_root_label(bucket: &str, prefix: &str, endpoint: Option<&str>) -> String {
    +    let mut label = format!("s3://{bucket}/{prefix}");
    +    if let Some(endpoint) = endpoint {
    +        label.push_str("?endpoint=");
    +        label.push_str(&normalized_endpoint(endpoint.trim()));
    +    }
    +    label
    +}
    +
    +/// The one spelling of an endpoint URL the root label carries
    +/// (`store_root_label`).
    +fn normalized_endpoint(endpoint: &str) -> String {
    +    if let Ok(url) = url::Url::parse(endpoint)
    +        && let Some(host) = url.host_str()
    +    {
    +        // `Url` lowercases the scheme and the host (a domain, or an IPv6
    +        // address's hex digits), and reports no port when it is the scheme's
    +        // default.
    +        let mut spelled = format!("{}://{host}", url.scheme());
    +        if let Some(port) = url.port() {
    +            spelled.push(':');
    +            spelled.push_str(&port.to_string());
    +        }
    +        if url.path() != "/" {
    +            spelled.push_str(url.path());
    +        }
    +        return spelled;
    +    }
    +    let authority_start = endpoint.find("//").map_or(0, |at| at + 2);
    +    let rest = &endpoint[authority_start..];
    +    let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
    +    let authority = &rest[..authority_end];
    +    let authority = authority
    +        .rfind('@')
    +        .map_or(authority, |at| &authority[at + 1..]);
    +    let path_end = rest.find(['?', '#']).unwrap_or(rest.len());
    +    let path = &rest[authority_end..path_end];
    +    format!(
    +        "{}{}{path}",
    +        endpoint[..authority_start].to_ascii_lowercase(),
    +        authority.to_ascii_lowercase()
    +    )
    +}
    +
     /// Normalize the configured bucket prefix: strip leading `/`, require a
     /// trailing `/` when non-empty, and hold it to the same component rules as
     /// keys.
    @@ -1955,6 +2109,46 @@ fn is_no_such_upload(err: &SdkError) -> bool {
         matches!(err, SdkError::ServiceError(ctx) if ctx.err().code() == Some("NoSuchUpload"))
     }
     
    +/// Why a request sent with `If-None-Match: *` did not store its object.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +enum Conditional {
    +    /// 412 `PreconditionFailed`: an object is under the key.
    +    Present,
    +    /// 409 `ConditionalRequestConflict`: another conditional write on the
    +    /// key was in flight; S3 asks the caller to retry.
    +    Conflict,
    +}
    +
    +/// Classify a conditional write's refusal, or `None` for any other error.
    +/// S3 answers a failed `If-None-Match` with 412 and the code
    +/// `PreconditionFailed`, and a collision of conditional writes with 409 and
    +/// `ConditionalRequestConflict`; either the status or the code identifies
    +/// the case, so a response carrying only one of them still classifies.
    +fn conditional_outcome(err: &SdkError) -> Option {
    +    let SdkError::ServiceError(ctx) = err else {
    +        return None;
    +    };
    +    match (ctx.raw().status().as_u16(), ctx.err().code()) {
    +        (412, _) | (_, Some("PreconditionFailed")) => Some(Conditional::Present),
    +        (409, Some("ConditionalRequestConflict")) | (_, Some("ConditionalRequestConflict")) => {
    +            Some(Conditional::Conflict)
    +        }
    +        _ => None,
    +    }
    +}
    +
    +/// How many times a conditional write answered `ConditionalRequestConflict`
    +/// is re-sent before the conflict is reported. Each retry waits
    +/// [`conditional_backoff`]; the eight waits sum to 7.9 s.
    +const CONDITIONAL_RETRIES: u32 = 8;
    +
    +/// The wait before retry `attempt` (1-based) of a conflicting conditional
    +/// write: 100 ms, doubling each retry (100, 200, 400, 800 ms), capped at
    +/// 1.6 s from the fifth retry on.
    +fn conditional_backoff(attempt: u32) -> std::time::Duration {
    +    std::time::Duration::from_millis(50u64.saturating_mul(1u64 << attempt.min(5)))
    +}
    +
     /// Best-effort abort of a multipart upload, after a put failure or from a
     /// drop guard; the original failure (if any) is what the caller needs, so
     /// an abort error is only logged. `NoSuchUpload` means the upload is no
    @@ -2152,6 +2346,42 @@ where
         }
     }
     
    +/// A `validate` failure that is not a denial, with the bucket named. A
    +/// transport failure keeps its class: the caller tells a store it could not
    +/// reach (which degrades the store) from a configuration it refuses (which
    +/// stops startup), so the bucket goes in as a link of the error chain rather
    +/// than flattening the error into text.
    +fn validation_failure(bucket: &str, error: StoreError) -> StoreError {
    +    match error {
    +        StoreError::Transport { source } => StoreError::Transport {
    +            source: Box::new(ValidationContext {
    +                context: format!("backup bucket {bucket} failed validation"),
    +                source,
    +            }),
    +        },
    +        other => StoreError::Other(format!("backup bucket {bucket} failed validation: {other}")),
    +    }
    +}
    +
    +/// The context link `validation_failure` adds to a transport error's chain.
    +#[derive(Debug)]
    +struct ValidationContext {
    +    context: String,
    +    source: Box,
    +}
    +
    +impl std::fmt::Display for ValidationContext {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str(&self.context)
    +    }
    +}
    +
    +impl std::error::Error for ValidationContext {
    +    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +        Some(self.source.as_ref())
    +    }
    +}
    +
     /// The SDK's non-service errors display as a bare category ("dispatch
     /// failure"); the cause (no region configured, no credentials, connection
     /// refused) is in the `source()` chain. Join the chain so an operator reads
    @@ -2174,6 +2404,81 @@ pub(crate) fn describe_error_chain(error: &(dyn std::error::Error + 'static)) ->
     mod tests {
         use super::*;
     
    +    /// A transport failure through `validate` keeps its class and names the
    +    /// bucket first in its chain; any other failure is `Other` with the same
    +    /// prefix.
    +    #[test]
    +    fn a_validation_transport_failure_keeps_its_class() {
    +        let transport: SdkError =
    +            SdkError::construction_failure("no region");
    +        let wrapped = validation_failure("b", map_sdk_error("Op", transport));
    +        assert!(
    +            matches!(wrapped, StoreError::Transport { .. }),
    +            "{wrapped:?}"
    +        );
    +        let text = wrapped.to_string();
    +        assert!(
    +            text.starts_with("transport error: backup bucket b failed validation: "),
    +            "{text}"
    +        );
    +        assert!(text.contains("no region"), "{text}");
    +        let other = validation_failure("b", StoreError::Other("x".to_owned()));
    +        assert!(
    +            matches!(&other, StoreError::Other(m) if m == "backup bucket b failed validation: x"),
    +            "{other:?}"
    +        );
    +    }
    +
    +    /// The label spells the endpoint one way: the two spellings an operator
    +    /// may type for one endpoint (a trailing `/`, upper-case letters, the
    +    /// scheme's default port) give one label, and a credential, query or
    +    /// fragment never reaches it.
    +    #[test]
    +    fn the_store_root_label_names_bucket_prefix_and_endpoint() {
    +        assert_eq!(store_root_label("b", "", None), "s3://b/");
    +        assert_eq!(store_root_label("b", "extenddb/", None), "s3://b/extenddb/");
    +        assert_eq!(
    +            store_root_label("b", "p/", Some("http://127.0.0.1:9000")),
    +            "s3://b/p/?endpoint=http://127.0.0.1:9000"
    +        );
    +        assert_eq!(
    +            store_root_label("b", "p/", Some("https://u:S3cretPW@h:1/base?x=1#f")),
    +            "s3://b/p/?endpoint=https://h:1/base"
    +        );
    +        for spelling in [
    +            "http://127.0.0.1:9000",
    +            "http://127.0.0.1:9000/",
    +            "HTTP://127.0.0.1:9000/",
    +            " http://127.0.0.1:9000 ",
    +        ] {
    +            assert_eq!(
    +                store_root_label("b", "p/", Some(spelling)),
    +                "s3://b/p/?endpoint=http://127.0.0.1:9000",
    +                "{spelling:?}"
    +            );
    +        }
    +        for spelling in [
    +            "https://S3.Example.com",
    +            "https://s3.example.com/",
    +            "https://s3.example.com:443",
    +            "HTTPS://s3.example.com:443/",
    +        ] {
    +            assert_eq!(
    +                store_root_label("b", "", Some(spelling)),
    +                "s3://b/?endpoint=https://s3.example.com",
    +                "{spelling:?}"
    +            );
    +        }
    +        assert_eq!(
    +            store_root_label("b", "", Some("http://LocalHost:80/base/")),
    +            "s3://b/?endpoint=http://localhost/base/"
    +        );
    +        assert_eq!(
    +            store_root_label("b", "", Some("http://[::1]:9000")),
    +            "s3://b/?endpoint=http://[::1]:9000"
    +        );
    +    }
    +
         #[test]
         fn store_prefix_normalization() {
             assert_eq!(normalize_store_prefix("").unwrap(), "");
    @@ -2246,6 +2551,47 @@ mod tests {
             )
         }
     
    +    /// The two answers a conditional write classifies, from synthetic
    +    /// service errors, by status alone, by code alone, and by both; every
    +    /// other answer is `None` and takes `map_sdk_error`'s path.
    +    #[test]
    +    fn conditional_outcomes_from_a_reply_document() {
    +        for (status, code, expected) in [
    +            (412, Some("PreconditionFailed"), Some(Conditional::Present)),
    +            (412, None, Some(Conditional::Present)),
    +            (400, Some("PreconditionFailed"), Some(Conditional::Present)),
    +            (
    +                409,
    +                Some("ConditionalRequestConflict"),
    +                Some(Conditional::Conflict),
    +            ),
    +            (
    +                503,
    +                Some("ConditionalRequestConflict"),
    +                Some(Conditional::Conflict),
    +            ),
    +            // A bare 409 is some other conflict (a bucket operation's), not
    +            // a conditional one.
    +            (409, None, None),
    +            (409, Some("BucketAlreadyOwnedByYou"), None),
    +            (403, Some("AccessDenied"), None),
    +            (404, Some("NoSuchKey"), None),
    +            (500, Some("InternalError"), None),
    +        ] {
    +            assert_eq!(
    +                conditional_outcome(&service_error(status, code)),
    +                expected,
    +                "{status} {code:?}"
    +            );
    +        }
    +        let transport: SdkError =
    +            SdkError::construction_failure("no region");
    +        assert_eq!(conditional_outcome(&transport), None);
    +        assert_eq!(conditional_backoff(1).as_millis(), 100);
    +        assert_eq!(conditional_backoff(5).as_millis(), 1600);
    +        assert_eq!(conditional_backoff(9).as_millis(), 1600, "capped");
    +    }
    +
         /// Every cell of `map_sdk_error`'s table, from synthetic service errors,
         /// so the mapping is pinned without an endpoint.
         #[test]
    diff --git a/crates/backup-store/tests/conformance.rs b/crates/backup-store/tests/conformance.rs
    index 1511b6164..30adcdd53 100644
    --- a/crates/backup-store/tests/conformance.rs
    +++ b/crates/backup-store/tests/conformance.rs
    @@ -441,6 +441,146 @@ mod suite {
             );
             assert!(list_keys(store, "aborted").await.is_empty());
         }
    +
    +    /// `put_if_absent` on an empty key stores the body and reports it; on a
    +    /// key that holds an object (its own, or one `put` wrote) it reports
    +    /// `None` and leaves the first body in place, whatever body the second
    +    /// call carried, and the key's metadata is the first object's. The cell
    +    /// over one part runs the multipart path on S3 (a conditional
    +    /// `CompleteMultipartUpload`) and a plain link on the filesystem store;
    +    /// its second call carries a body over one part too, so on S3 a
    +    /// conditional complete meets the present key and has an upload to
    +    /// abort, and its third call a body of one part.
    +    pub async fn put_if_absent_creates_once(store: &dyn BackupStore) {
    +        let small = b"first body".to_vec();
    +        let large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() * 2 + 4096);
    +        let mut other_large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() * 2 + 2048);
    +        other_large.reverse();
    +        for (key, first, seconds) in [
    +            (
    +                "ident/small.json",
    +                small.clone(),
    +                vec![b"second body, longer".to_vec()],
    +            ),
    +            (
    +                "ident/large.bin",
    +                large.clone(),
    +                vec![other_large, small.clone()],
    +            ),
    +        ] {
    +            let stored = store
    +                .put_if_absent(key, chunked_body(first.clone()))
    +                .await
    +                .expect("first put_if_absent succeeds")
    +                .expect("an empty key is created");
    +            assert_eq!(stored.key, key);
    +            assert_eq!(stored.size, first.len() as u64);
    +            for second in seconds {
    +                let again = store
    +                    .put_if_absent(key, chunked_body(second.clone()))
    +                    .await
    +                    .expect("a later put_if_absent succeeds");
    +                assert!(again.is_none(), "{key}: a present key reports None");
    +                let read = read_all(store.get(key).await.unwrap()).await;
    +                assert_eq!(read, first, "{key}: the first body stays");
    +            }
    +            let meta = store.head(key).await.unwrap().expect("object exists");
    +            assert_eq!(meta.size, first.len() as u64);
    +            assert_eq!(list_keys(store, key).await, vec![key.to_owned()]);
    +        }
    +        // A key `put` wrote is present to `put_if_absent` too.
    +        store
    +            .put("ident/put-first", chunked_body(b"by put"))
    +            .await
    +            .unwrap();
    +        assert!(
    +            store
    +                .put_if_absent("ident/put-first", chunked_body(b"by put_if_absent"))
    +                .await
    +                .unwrap()
    +                .is_none()
    +        );
    +        assert_eq!(
    +            read_all(store.get("ident/put-first").await.unwrap()).await,
    +            b"by put"
    +        );
    +        // And a key `put_if_absent` wrote is replaced by `put`, as any object
    +        // is.
    +        store
    +            .put("ident/small.json", chunked_body(b"replaced"))
    +            .await
    +            .unwrap();
    +        assert_eq!(
    +            read_all(store.get("ident/small.json").await.unwrap()).await,
    +            b"replaced"
    +        );
    +        // A body that fails leaves nothing under the key and the key stays
    +        // free for the next call.
    +        let err = store
    +            .put_if_absent("ident/failing", failing_body(b"partial"))
    +            .await
    +            .expect_err("mid-body failure surfaces");
    +        assert!(matches!(&err, StoreError::Other(msg) if msg.contains("injected")));
    +        assert!(store.head("ident/failing").await.unwrap().is_none());
    +        assert!(
    +            store
    +                .put_if_absent("ident/failing", chunked_body(b"ok"))
    +                .await
    +                .unwrap()
    +                .is_some()
    +        );
    +    }
    +
    +    /// Concurrent `put_if_absent` calls on one key: exactly one stores its
    +    /// body, every other reads `None`, and the object under the key is the
    +    /// winner's body. The bodies are distinct so the winner is identifiable;
    +    /// the race is driven by starting every call before any is polled to
    +    /// completion (`join_all`), and repeated so a store whose calls happen to
    +    /// serialise once is still exercised across rounds.
    +    pub async fn concurrent_put_if_absent_leaves_the_first_body(store: &dyn BackupStore) {
    +        const CALLERS: usize = 8;
    +        const ROUNDS: usize = 3;
    +        for round in 0..ROUNDS {
    +            let key = format!("race/round-{round}/extenddb-store.json");
    +            let bodies: Vec> = (0..CALLERS)
    +                .map(|i| format!("{{\"round\":{round},\"caller\":{i}}}").into_bytes())
    +                .collect();
    +            let outcomes = futures::future::join_all(
    +                bodies
    +                    .iter()
    +                    .map(|body| store.put_if_absent(&key, chunked_body(body.clone()))),
    +            )
    +            .await;
    +            let winners: Vec = outcomes
    +                .iter()
    +                .enumerate()
    +                .filter_map(|(i, outcome)| {
    +                    outcome
    +                        .as_ref()
    +                        .expect("every call succeeds")
    +                        .as_ref()
    +                        .map(|_| i)
    +                })
    +                .collect();
    +            assert_eq!(
    +                winners.len(),
    +                1,
    +                "round {round}: exactly one caller stores its body, got {winners:?}"
    +            );
    +            let read = read_all(store.get(&key).await.unwrap()).await;
    +            assert_eq!(
    +                read, bodies[winners[0]],
    +                "round {round}: the object is the winner's body"
    +            );
    +            let meta = store.head(&key).await.unwrap().expect("object exists");
    +            assert_eq!(meta.size, bodies[winners[0]].len() as u64);
    +            assert_eq!(
    +                list_keys(store, &format!("race/round-{round}")).await,
    +                vec![key.clone()],
    +                "round {round}: one object under the prefix"
    +            );
    +        }
    +    }
     }
     
     #[cfg(unix)]
    @@ -557,6 +697,65 @@ mod filesystem {
             }
         }
     
    +    #[tokio::test]
    +    async fn put_if_absent_creates_once() {
    +        let (dir, store) = fs_store().await;
    +        suite::put_if_absent_creates_once(&store).await;
    +        // The link discipline: no temporary stays, whether the call stored
    +        // its object, found the key present, or failed mid-body, and the
    +        // object the winning call linked is private.
    +        assert!(temporaries_under(dir.path()).is_empty());
    +        assert_eq!(
    +            mode_of(&dir.path().join("ident/large.bin")) & 0o777,
    +            0o600,
    +            "a linked object keeps the temporary's mode"
    +        );
    +    }
    +
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn concurrent_put_if_absent_leaves_the_first_body() {
    +        let (dir, store) = fs_store().await;
    +        suite::concurrent_put_if_absent_leaves_the_first_body(&store).await;
    +        assert!(temporaries_under(dir.path()).is_empty());
    +    }
    +
    +    /// `put_if_absent` applies the destination rules `put` does: a symlink at
    +    /// the key is refused without being followed and its target untouched, a
    +    /// directory at the key is a key-space collision, and an object at an
    +    /// ancestor of the key makes the key unusable. None of the three
    +    /// consumes the body or leaves a temporary.
    +    #[tokio::test]
    +    async fn put_if_absent_refuses_what_put_refuses() {
    +        let (dir, store) = fs_store().await;
    +        let outside = private_tempdir();
    +        let target = outside.path().join("target");
    +        std::fs::write(&target, b"outside").unwrap();
    +        private_dir(&dir.path().join("a"));
    +        std::os::unix::fs::symlink(&target, dir.path().join("a/link")).unwrap();
    +        let err = store
    +            .put_if_absent("a/link", chunked_body(b"x"))
    +            .await
    +            .expect_err("a symlink at the key is refused");
    +        assert!(matches!(err, StoreError::PermissionDenied(_)), "{err}");
    +        assert_eq!(std::fs::read(&target).unwrap(), b"outside");
    +
    +        store
    +            .put("dir/under/obj", chunked_body(b"x"))
    +            .await
    +            .unwrap();
    +        let err = store
    +            .put_if_absent("dir/under", chunked_body(b"x"))
    +            .await
    +            .expect_err("a directory at the key is refused");
    +        assert!(matches!(err, StoreError::Io { .. }), "{err}");
    +        let err = store
    +            .put_if_absent("dir/under/obj/below", chunked_body(b"x"))
    +            .await
    +            .expect_err("an object at an ancestor is refused");
    +        assert!(matches!(err, StoreError::Io { .. }), "{err}");
    +        assert!(temporaries_under(dir.path()).is_empty());
    +    }
    +
         /// Every `.put-*` entry under `root`, at any depth.
         fn temporaries_under(root: &std::path::Path) -> Vec {
             let mut found = Vec::new();
    @@ -1547,6 +1746,14 @@ mod filesystem {
             assert!(FilesystemStore::open(&missing).await.is_err());
         }
     
    +    /// An empty root path is refused by name, not as a stat of "" (whose
    +    /// message would show an empty name between two spaces).
    +    #[tokio::test]
    +    async fn open_refuses_an_empty_root_path() {
    +        let err = FilesystemStore::open("").await.err().expect("refused");
    +        assert!(err.to_string().contains("root path is empty"), "{err}");
    +    }
    +
         #[tokio::test]
         async fn symlinked_directory_escape_is_refused() {
             let outside = tempfile::tempdir().expect("outside dir");
    @@ -2553,6 +2760,41 @@ mod s3 {
             suite::list_order_is_key_bytes(&store).await;
         }
     
    +    /// Both cells of the conditional write: a single `PutObject` and a
    +    /// `CompleteMultipartUpload`, each with `If-None-Match: *`. A present key
    +    /// answers 412 to either; the multipart cell's upload is aborted on that
    +    /// answer, which the listing of open uploads below checks.
    +    #[tokio::test]
    +    async fn put_if_absent_creates_once() {
    +        let Some(endpoint) = endpoint_or_skip() else {
    +            return;
    +        };
    +        let prefix = unique_prefix();
    +        let config = test_config(endpoint, test_bucket(), prefix.clone());
    +        let store = S3Store::open(&config).await.expect("S3Store::open");
    +        ensure_bucket(&config).await;
    +        suite::put_if_absent_creates_once(&store).await;
    +        let open = raw_client(&config)
    +            .await
    +            .list_multipart_uploads()
    +            .bucket(&config.bucket)
    +            .prefix(&prefix)
    +            .send()
    +            .await
    +            .expect("list_multipart_uploads");
    +        assert!(
    +            open.uploads().is_empty(),
    +            "a conditional complete that found the key present aborts its upload: {:?}",
    +            open.uploads()
    +        );
    +    }
    +
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn concurrent_put_if_absent_leaves_the_first_body() {
    +        let Some(store) = store().await else { return };
    +        suite::concurrent_put_if_absent_leaves_the_first_body(&store).await;
    +    }
    +
         #[tokio::test]
         async fn delete_prefix_removes_and_counts() {
             let Some(store) = store().await else { return };
    @@ -2859,6 +3101,11 @@ mod s3 {
                 .validate()
                 .await
                 .expect_err("nothing listens on port 9");
    +        assert!(
    +            matches!(err, StoreError::Transport { .. }),
    +            "a transport failure keeps its class through validate, so a server can tell it \
    +             from a configuration error: {err:?}"
    +        );
             let text = err.to_string();
             assert!(text.contains("unused"), "names the bucket: {text}");
             assert!(
    @@ -2867,6 +3114,284 @@ mod s3 {
             );
         }
     
    +    /// The 409 `ConditionalRequestConflict` loop of `put_if_absent`, driven
    +    /// by a server of the test's own that answers from a script (moto never
    +    /// answers 409): a `PutObject` answered 409 twice is re-sent and the
    +    /// third answer stands; a `CompleteMultipartUpload` answered 409 and
    +    /// then 412 is "present" and its upload is aborted; a `PutObject`
    +    /// answered 409 nine times is the error the ninth answer is, after
    +    /// eight retries. Runs in a child with the suite's own credentials, as
    +    /// every request needs them. No endpoint is needed.
    +    #[tokio::test]
    +    async fn a_conditional_request_conflict_is_retried_from_a_reply_script() {
    +        const NAME: &str = "s3::a_conditional_request_conflict_is_retried_from_a_reply_script";
    +        if !is_child(NAME) {
    +            run_with_test_credentials(NAME);
    +            return;
    +        }
    +        let server = ScriptedS3::start().await;
    +        let config = test_config(server.endpoint(), "scripted".to_owned(), String::new());
    +        let store = S3Store::open(&config).await.expect("S3Store::open");
    +
    +        // Two conflicts, then created.
    +        server.conflicts_before_success(2);
    +        let meta = store
    +            .put_if_absent("k/one-part", chunked_body(b"body"))
    +            .await
    +            .expect("the third answer is 200")
    +            .expect("created");
    +        assert_eq!(meta.size, 4);
    +        let puts = server.requests("PUT /scripted/k/one-part?x-id=PutObject");
    +        assert_eq!(puts, 3, "one send and two re-sends");
    +
    +        // The multipart cell: the complete answers 409 once, then 412.
    +        server.conflicts_then_present(1);
    +        let large = patterned_bytes(usize::try_from(PART_SIZE).unwrap() + 16);
    +        let again = store
    +            .put_if_absent("k/two-parts", chunked_body(large))
    +            .await
    +            .expect("412 is an answer, not an error");
    +        assert!(again.is_none(), "a present key reports None");
    +        assert_eq!(
    +            server.requests("POST /scripted/k/two-parts?uploadId="),
    +            2,
    +            "the complete was re-sent once"
    +        );
    +        assert_eq!(
    +            server.requests("DELETE /scripted/k/two-parts?x-id=AbortMultipartUpload"),
    +            1,
    +            "the upload the present key refused was aborted"
    +        );
    +
    +        // Conflicts that never end: eight retries, then the error.
    +        server.conflicts_before_success(u32::MAX);
    +        let started = std::time::Instant::now();
    +        let err = store
    +            .put_if_absent("k/never", chunked_body(b"body"))
    +            .await
    +            .expect_err("the ninth 409 is reported");
    +        let text = err.to_string();
    +        assert!(
    +            text.contains("ConditionalRequestConflict"),
    +            "names the conflict: {text}"
    +        );
    +        assert_eq!(server.requests("PUT /scripted/k/never"), 9);
    +        let waited = started.elapsed();
    +        assert!(
    +            waited >= std::time::Duration::from_millis(7_000)
    +                && waited < std::time::Duration::from_secs(20),
    +            "the back-offs sum to about 7.9 s, measured {waited:?}"
    +        );
    +    }
    +
    +    /// An HTTP/1.1 server that plays S3 from a script: every request's
    +    /// method, path and query are recorded; a `PutObject` or
    +    /// `CompleteMultipartUpload` is answered 409 `ConditionalRequestConflict`
    +    /// a scripted number of times, then 200, or 412 `PreconditionFailed`;
    +    /// `CreateMultipartUpload`, `UploadPart` and `AbortMultipartUpload` are
    +    /// answered as S3 answers them.
    +    struct ScriptedS3 {
    +        port: u16,
    +        state: std::sync::Arc>,
    +    }
    +
    +    #[derive(Default)]
    +    struct ScriptState {
    +        /// Conflicts left to answer before the terminal answer.
    +        conflicts: u32,
    +        /// Whether the terminal answer is 412 rather than 200.
    +        then_present: bool,
    +        /// `" "` of every request, in order.
    +        requests: Vec,
    +    }
    +
    +    impl ScriptedS3 {
    +        async fn start() -> Self {
    +            let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
    +                .await
    +                .expect("bind");
    +            let port = listener.local_addr().unwrap().port();
    +            let state = std::sync::Arc::new(std::sync::Mutex::new(ScriptState::default()));
    +            let shared = std::sync::Arc::clone(&state);
    +            tokio::spawn(async move {
    +                loop {
    +                    let Ok((socket, _)) = listener.accept().await else {
    +                        return;
    +                    };
    +                    tokio::spawn(Self::serve(socket, std::sync::Arc::clone(&shared)));
    +                }
    +            });
    +            Self { port, state }
    +        }
    +
    +        fn endpoint(&self) -> String {
    +            format!("http://127.0.0.1:{}", self.port)
    +        }
    +
    +        fn conflicts_before_success(&self, n: u32) {
    +            let mut state = self.state.lock().unwrap();
    +            state.conflicts = n;
    +            state.then_present = false;
    +        }
    +
    +        fn conflicts_then_present(&self, n: u32) {
    +            let mut state = self.state.lock().unwrap();
    +            state.conflicts = n;
    +            state.then_present = true;
    +        }
    +
    +        /// How many recorded requests start with `prefix`.
    +        fn requests(&self, prefix: &str) -> usize {
    +            self.state
    +                .lock()
    +                .unwrap()
    +                .requests
    +                .iter()
    +                .filter(|r| r.starts_with(prefix))
    +                .count()
    +        }
    +
    +        async fn serve(
    +            mut socket: tokio::net::TcpStream,
    +            state: std::sync::Arc>,
    +        ) {
    +            use tokio::io::{AsyncReadExt, AsyncWriteExt};
    +            let mut buffer = Vec::new();
    +            loop {
    +                // Read one request: the head, then the body it announces.
    +                let head_end = loop {
    +                    if let Some(at) = buffer.windows(4).position(|w| w == b"\r\n\r\n") {
    +                        break at + 4;
    +                    }
    +                    let mut chunk = [0u8; 8192];
    +                    match socket.read(&mut chunk).await {
    +                        Ok(0) | Err(_) => return,
    +                        Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                    }
    +                };
    +                let head = String::from_utf8_lossy(&buffer[..head_end]).into_owned();
    +                let mut lines = head.split("\r\n");
    +                let request_line = lines.next().unwrap_or_default().to_owned();
    +                let mut content_length = 0usize;
    +                let mut chunked = false;
    +                for line in lines {
    +                    let Some((name, value)) = line.split_once(':') else {
    +                        continue;
    +                    };
    +                    let value = value.trim();
    +                    if name.eq_ignore_ascii_case("content-length") {
    +                        content_length = value.parse().unwrap_or(0);
    +                    } else if name.eq_ignore_ascii_case("transfer-encoding")
    +                        && value.eq_ignore_ascii_case("chunked")
    +                    {
    +                        chunked = true;
    +                    }
    +                }
    +                buffer.drain(..head_end);
    +                if chunked {
    +                    // Read until the terminating chunk.
    +                    loop {
    +                        if buffer.windows(5).any(|w| w == b"0\r\n\r\n") {
    +                            let at = buffer.windows(5).position(|w| w == b"0\r\n\r\n").unwrap();
    +                            buffer.drain(..at + 5);
    +                            break;
    +                        }
    +                        let mut chunk = [0u8; 65536];
    +                        match socket.read(&mut chunk).await {
    +                            Ok(0) | Err(_) => return,
    +                            Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                        }
    +                    }
    +                } else {
    +                    while buffer.len() < content_length {
    +                        let mut chunk = [0u8; 65536];
    +                        match socket.read(&mut chunk).await {
    +                            Ok(0) | Err(_) => return,
    +                            Ok(n) => buffer.extend_from_slice(&chunk[..n]),
    +                        }
    +                    }
    +                    buffer.drain(..content_length);
    +                }
    +
    +                let mut parts = request_line.split(' ');
    +                let method = parts.next().unwrap_or_default().to_owned();
    +                let target = parts.next().unwrap_or_default().to_owned();
    +                let (status, body) = {
    +                    let mut state = state.lock().unwrap();
    +                    state.requests.push(format!("{method} {target}"));
    +                    Self::answer(&mut state, &method, &target)
    +                };
    +                let reason = match status {
    +                    200 => "OK",
    +                    204 => "No Content",
    +                    409 => "Conflict",
    +                    412 => "Precondition Failed",
    +                    _ => "Error",
    +                };
    +                let response = format!(
    +                    "HTTP/1.1 {status} {reason}\r\nContent-Type: application/xml\r\n\
    +                     Content-Length: {}\r\nETag: \"etag-1\"\r\nConnection: keep-alive\r\n\r\n{body}",
    +                    body.len()
    +                );
    +                if socket.write_all(response.as_bytes()).await.is_err() {
    +                    return;
    +                }
    +            }
    +        }
    +
    +        /// The scripted answer to one request.
    +        fn answer(state: &mut ScriptState, method: &str, target: &str) -> (u16, String) {
    +            let conflict = || {
    +                (
    +                    409,
    +                    "ConditionalRequestConflictConditional request \
    +                     cannot succeed due to a conflicting operation against this \
    +                     resource."
    +                        .to_owned(),
    +                )
    +            };
    +            let present = || {
    +                (
    +                    412,
    +                    "PreconditionFailedAt least one of the \
    +                     pre-conditions you specified did not hold"
    +                        .to_owned(),
    +                )
    +            };
    +            let conditional = |state: &mut ScriptState, success: String| {
    +                if state.conflicts > 0 {
    +                    state.conflicts = state.conflicts.saturating_sub(1);
    +                    conflict()
    +                } else if state.then_present {
    +                    present()
    +                } else {
    +                    (200, success)
    +                }
    +            };
    +            match method {
    +                "PUT" if target.contains("partNumber=") => (200, String::new()),
    +                "PUT" => conditional(state, String::new()),
    +                "POST" if target.ends_with("?uploads") => (
    +                    200,
    +                    "scriptedk\
    +                     upload-1"
    +                        .to_owned(),
    +                ),
    +                "POST" if target.contains("uploadId=") => conditional(
    +                    state,
    +                    "scriptedk\
    +                     \"etag-1\""
    +                        .to_owned(),
    +                ),
    +                "DELETE" => (204, String::new()),
    +                _ => (
    +                    404,
    +                    "NoSuchKeyno".to_owned(),
    +                ),
    +            }
    +        }
    +    }
    +
         /// A put whose future is dropped mid-upload (a cancelled job) aborts its
         /// multipart upload from the drop guard, and `delete_prefix` aborts any
         /// upload a crashed process left, so no parts accrue storage.
    @@ -5172,6 +5697,16 @@ mod s3 {
             };
             println!("request URI: {uri}");
             assert!(uri.starts_with(expected), "{uri} does not go to {expected}");
    +        // The root label names the endpoint the request went to, in its one
    +        // spelling, whichever source supplied it.
    +        let endpoint = expected.trim_end_matches('/');
    +        let label = store.store_root();
    +        println!("store root: {label}");
    +        assert_eq!(
    +            label,
    +            format!("s3://env-bucket/?endpoint={endpoint}"),
    +            "the label names the effective endpoint"
    +        );
         }
     
         /// Rewrites every request's URI, after the endpoint is resolved and
    diff --git a/crates/config/Cargo.toml b/crates/config/Cargo.toml
    index 1c2c59ebf..f9f086124 100644
    --- a/crates/config/Cargo.toml
    +++ b/crates/config/Cargo.toml
    @@ -15,8 +15,12 @@ dev-mode = []
     [dependencies]
     extenddb-core = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     serde = { workspace = true }
     toml = { workspace = true }
     config = { workspace = true }
     anyhow = { workspace = true }
     tracing = { workspace = true }
    +
    +[dev-dependencies]
    +tempfile = "3"
    diff --git a/crates/config/src/backup.rs b/crates/config/src/backup.rs
    index 92339fa4f..83e3a9c83 100644
    --- a/crates/config/src/backup.rs
    +++ b/crates/config/src/backup.rs
    @@ -1,7 +1,8 @@
     // Copyright 2026 ExtendDB contributors
     // SPDX-License-Identifier: Apache-2.0
     
    -//! The `[backup]` section: the knobs backup and restore jobs read.
    +//! The `[backup]` section: the knobs backup and restore jobs read, and the
    +//! store they write to.
     //!
     //! ```toml
     //! [backup]
    @@ -12,6 +13,11 @@
     //! reconcile_interval_secs = 60       # reconciler cycle
     //! max_concurrent_jobs = 4            # backup and restore jobs running at once on this instance
     //! # spool_path = "/var/lib/extenddb/spool"  # restore download area for an S3 store
    +//!
    +//! [backup.store]
    +//! kind = "filesystem"                # "filesystem" | "s3" | "disabled"
    +//! [backup.store.filesystem]
    +//! path = "/var/lib/extenddb/backups"
     //! ```
     //!
     //! `stale_job_after_secs` must be at least three heartbeat periods: a job
    @@ -33,9 +39,56 @@
     //! at load, and the PostgreSQL pool rule makes anything over a few hundred
     //! meaningless anyway.
     //!
    -//! The store itself (`[backup.store]`) is configured separately and is not
    -//! part of this struct.
    +//! The store. `[backup.store]` is the backup store crate's
    +//! [`BackupStoreConfig`], nested under its own key so the one definition of
    +//! the shape stays in that crate and `deny_unknown_fields` holds on both
    +//! tables (`serde(flatten)` would drop it). Its keys, defaults and `open`'s
    +//! refusals are that crate's `config` module doc. What this module adds is
    +//! the per-backend rule for an absent section and the spool
    +//! ([`BackupConfig::resolve_store`]):
    +//!
    +//! - On a backend whose database is one file on this host (file-backed
    +//!   SQLite), a missing `[backup.store]` is `kind = "filesystem"` with
    +//!   `path = "backups/"` beside the database file. That defaulted root is
    +//!   created `0700` if missing, and a failure of its validation is a logged
    +//!   warning that leaves the store unconfigured, not a startup error, so an
    +//!   upgrade never stops a server that never configured backups; an
    +//!   explicitly configured root that fails validation is a startup error.
    +//! - On PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database
    +//!   (no file, so no default root), a missing section leaves the store
    +//!   unconfigured and startup logs that.
    +//! - `kind = "disabled"` leaves the store unconfigured on purpose, on any
    +//!   backend.
    +//! - Without a configured store, CreateBackup writes the in-catalog format
    +//!   and logs a deprecation warning naming `[backup.store]` on each call;
    +//!   backups of either format stay readable, listable, restorable and
    +//!   deletable.
    +//!
    +//! The spool. Restores from an S3 store download to `spool_path`, created
    +//! `0700` with files `0600`. On file-backed SQLite it defaults to
    +//! `backup-spool/` beside the database file; on every other backend an S3
    +//! store needs `spool_path` and startup refuses without it (a filesystem
    +//! store needs no spool: files are verified in place). A `spool_path` that
    +//! contains or is contained in a filesystem store root is refused (the store
    +//! would list and delete it, and a sync would read its first component as an
    +//! account id), and so is one that is or contains the database file's
    +//! directory. A filesystem store root that is or contains the database
    +//! file's directory is refused too: the database and its sidecars would sit
    +//! among the backups. An empty root is compared with nothing (it names no
    +//! directory) and left for `open` to refuse by name. The spool's `.lock`
    +//! another live instance holds is refused at startup by the server, which
    +//! owns the lock file.
    +//!
    +//! A relative `[backup.store.filesystem] path` or `spool_path` is made
    +//! absolute against the working directory at config load
    +//! ([`BackupConfig::make_paths_absolute`]), where `storage.path` is: `serve`
    +//! daemonizes after loading, and the child's working directory is `/`.
     
    +use std::path::{Path, PathBuf};
    +
    +pub use extenddb_backup_store::{
    +    BackupStoreConfig, BackupStoreKind, FilesystemStoreConfig, S3StoreConfig,
    +};
     use extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL;
     use serde::Deserialize;
     
    @@ -50,6 +103,13 @@ pub const MIN_STALE_HEARTBEATS: u64 = 3;
     /// The largest `max_concurrent_jobs`.
     pub const MAX_CONCURRENT_JOBS: usize = 1024;
     
    +/// The directory name of the defaulted filesystem store root beside a
    +/// database file.
    +pub const DEFAULT_ROOT_DIR_NAME: &str = "backups";
    +
    +/// The directory name of the defaulted restore spool beside a database file.
    +pub const DEFAULT_SPOOL_DIR_NAME: &str = "backup-spool";
    +
     #[derive(Debug, Clone, Deserialize, PartialEq, Eq)]
     #[serde(deny_unknown_fields)]
     pub struct BackupConfig {
    @@ -76,10 +136,14 @@ pub struct BackupConfig {
         #[serde(default = "default_max_concurrent_jobs")]
         pub max_concurrent_jobs: usize,
         /// Restore download area for an S3 store. Defaults beside the database
    -    /// file on SQLite; required on the other backends once a store is
    -    /// configured.
    +    /// file on SQLite; required on the other backends with an S3 store.
         #[serde(default)]
         pub spool_path: Option,
    +    /// `[backup.store]`: the store this instance writes backups to. `None`
    +    /// when the section is absent, which [`BackupConfig::resolve_store`]
    +    /// turns into the per-backend default.
    +    #[serde(default)]
    +    pub store: Option,
     }
     
     impl Default for BackupConfig {
    @@ -92,10 +156,45 @@ impl Default for BackupConfig {
                 reconcile_interval_secs: default_reconcile_interval_secs(),
                 max_concurrent_jobs: default_max_concurrent_jobs(),
                 spool_path: None,
    +            store: None,
             }
         }
     }
     
    +/// Why no store is configured.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +pub enum Unconfigured {
    +    /// `[backup.store]` is absent and this backend has no default root (a
    +    /// server database, or an in-memory SQLite database).
    +    Absent,
    +    /// `kind = "disabled"`.
    +    Disabled,
    +}
    +
    +/// The store `[backup.store]` and the backend together select.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum EffectiveStore {
    +    /// No store: CreateBackup writes the in-catalog format.
    +    Unconfigured(Unconfigured),
    +    /// A store to open and validate at startup.
    +    Configured {
    +        config: BackupStoreConfig,
    +        /// The section was absent and the root is the default beside the
    +        /// database file: the server creates it if missing and treats a
    +        /// validation failure as a warning that leaves the store
    +        /// unconfigured.
    +        defaulted: bool,
    +    },
    +}
    +
    +/// What [`BackupConfig::resolve_store`] decided.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct ResolvedStore {
    +    pub store: EffectiveStore,
    +    /// The restore spool, when the configured store is S3.
    +    pub spool_path: Option,
    +}
    +
     impl BackupConfig {
         /// The period the jobs heartbeat at: the larger of
         /// `heartbeat_interval_secs` and [`JOB_HEARTBEAT_INTERVAL`], which is
    @@ -106,6 +205,36 @@ impl BackupConfig {
                 .max(JOB_HEARTBEAT_INTERVAL.as_secs())
         }
     
    +    /// Make `[backup.store.filesystem] path` and `spool_path` absolute
    +    /// against `cwd` (`~` expanded first), so a relative value written in
    +    /// the config file names the directory it does from the shell that
    +    /// started the server, in the daemonized child too. An absolute value
    +    /// is left as it is, and so is an empty one, which `open` refuses as it
    +    /// did (it would otherwise name the working directory as the root).
    +    pub fn make_paths_absolute(&mut self, cwd: &Path) {
    +        if let Some(fs) = self.store.as_mut().and_then(|s| s.filesystem.as_mut())
    +            && !fs.path.is_absolute()
    +            && !fs.path.as_os_str().is_empty()
    +        {
    +            let expanded = PathBuf::from(crate::expand_tilde(&fs.path.to_string_lossy()));
    +            fs.path = if expanded.is_absolute() {
    +                expanded
    +            } else {
    +                cwd.join(expanded)
    +            };
    +        }
    +        if let Some(spool) = self.spool_path.as_mut() {
    +            let expanded = PathBuf::from(crate::expand_tilde(spool));
    +            *spool = if expanded.is_absolute() {
    +                expanded
    +            } else {
    +                cwd.join(expanded)
    +            }
    +            .to_string_lossy()
    +            .into_owned();
    +        }
    +    }
    +
         /// The cross-field rules, checked at load.
         ///
         /// # Errors
    @@ -148,8 +277,197 @@ impl BackupConfig {
                     self.max_concurrent_jobs
                 ));
             }
    +        if let Some(spool) = &self.spool_path
    +            && spool.trim().is_empty()
    +        {
    +            return Err("backup.spool_path must not be empty".to_owned());
    +        }
             Ok(())
         }
    +
    +    /// The store and spool this configuration selects on a backend whose
    +    /// database is `database_file` (`Some` for file-backed SQLite, `None`
    +    /// for a server database or an in-memory one), by the rules in the
    +    /// module doc.
    +    ///
    +    /// # Errors
    +    ///
    +    /// A message naming the key: an S3 store without `spool_path` on a
    +    /// backend with no default spool; a `spool_path` inside or containing a
    +    /// filesystem store root, or inside or containing the database file's
    +    /// directory; a filesystem root that is or contains the database file's
    +    /// directory.
    +    pub fn resolve_store(&self, database_file: Option<&Path>) -> Result {
    +        let database_dir = database_file.and_then(Path::parent).map(Path::to_path_buf);
    +        let store = match &self.store {
    +            None => match &database_dir {
    +                Some(dir) => EffectiveStore::Configured {
    +                    config: BackupStoreConfig {
    +                        kind: BackupStoreKind::Filesystem,
    +                        filesystem: Some(FilesystemStoreConfig {
    +                            path: dir.join(DEFAULT_ROOT_DIR_NAME),
    +                        }),
    +                        s3: None,
    +                    },
    +                    defaulted: true,
    +                },
    +                None => EffectiveStore::Unconfigured(Unconfigured::Absent),
    +            },
    +            Some(config) if config.kind == BackupStoreKind::Disabled => {
    +                EffectiveStore::Unconfigured(Unconfigured::Disabled)
    +            }
    +            Some(config) => EffectiveStore::Configured {
    +                config: config.clone(),
    +                defaulted: false,
    +            },
    +        };
    +
    +        let explicit_spool = self
    +            .spool_path
    +            .as_deref()
    +            .map(|p| PathBuf::from(crate::expand_tilde(p)));
    +        let filesystem_root = match &store {
    +            EffectiveStore::Configured { config, .. }
    +                if config.kind == BackupStoreKind::Filesystem =>
    +            {
    +                // An empty `path` names no directory: it is compared with
    +                // nothing here (compared as written it would be the working
    +                // directory, and refused by name as the database directory
    +                // when the server starts there), and `open` refuses it.
    +                config
    +                    .filesystem
    +                    .as_ref()
    +                    .map(|f| f.path.clone())
    +                    .filter(|path| !path.as_os_str().is_empty())
    +            }
    +            _ => None,
    +        };
    +        let is_s3 = matches!(
    +            &store,
    +            EffectiveStore::Configured { config, .. } if config.kind == BackupStoreKind::S3
    +        );
    +
    +        // The spool rules apply to every explicit `spool_path`, so a
    +        // configuration that breaks them is refused before the store it
    +        // will one day be used with is selected. Each message spells both
    +        // paths as they were compared (resolved through existing symlinks).
    +        if let Some(spool) = &explicit_spool {
    +            if let Some(root) = &filesystem_root
    +                && paths_overlap(spool, root)
    +            {
    +                return Err(format!(
    +                    "backup.spool_path {} is inside or contains the filesystem backup store \
    +                     root {}; the store would list and delete the spool. Put the spool \
    +                     outside the root",
    +                    comparable_path(spool).display(),
    +                    comparable_path(root).display()
    +                ));
    +            }
    +            if let Some(dir) = &database_dir
    +                && path_is_or_contains(spool, dir)
    +            {
    +                return Err(format!(
    +                    "backup.spool_path {} is or contains the database directory {}; put the \
    +                     spool in a directory of its own",
    +                    comparable_path(spool).display(),
    +                    comparable_path(dir).display()
    +                ));
    +            }
    +        }
    +        // A filesystem root that is the database directory or an ancestor of
    +        // it would hold the database file and its sidecars among the
    +        // backups: a listing would read the directory's entries as account
    +        // ids and a deletion of the root's whole prefix would walk them. The
    +        // default root is a child of the directory and passes.
    +        if let (Some(root), Some(dir)) = (&filesystem_root, &database_dir)
    +            && path_is_or_contains(root, dir)
    +        {
    +            return Err(format!(
    +                "backup.store.filesystem.path {} is or contains the database directory {}; \
    +                 put the backup root in a directory of its own (the default is \
    +                 {DEFAULT_ROOT_DIR_NAME}/ beside the database file)",
    +                comparable_path(root).display(),
    +                comparable_path(dir).display()
    +            ));
    +        }
    +
    +        let spool_path = if is_s3 {
    +            match (explicit_spool, &database_dir) {
    +                (Some(spool), _) => Some(spool),
    +                (None, Some(dir)) => Some(dir.join(DEFAULT_SPOOL_DIR_NAME)),
    +                (None, None) => {
    +                    return Err(
    +                        "backup.spool_path is required with an S3 backup store on this \
    +                         backend: a restore downloads its files there before loading them, \
    +                         and only a file-backed SQLite database has a default beside its file"
    +                            .to_owned(),
    +                    );
    +                }
    +            }
    +        } else {
    +            None
    +        };
    +        Ok(ResolvedStore { store, spool_path })
    +    }
    +}
    +
    +/// Whether one path is the other or an ancestor of it, after each is made
    +/// absolute and lexically normalized (`.` and `..` resolved without
    +/// consulting the filesystem, since neither directory need exist yet) and
    +/// canonicalized when it exists, so a symlinked spelling of the same
    +/// directory compares equal.
    +#[must_use]
    +pub fn paths_overlap(a: &Path, b: &Path) -> bool {
    +    let a = comparable_path(a);
    +    let b = comparable_path(b);
    +    a.starts_with(&b) || b.starts_with(&a)
    +}
    +
    +/// Whether `a` is `b` or an ancestor of it, compared as [`paths_overlap`]
    +/// compares.
    +#[must_use]
    +pub fn path_is_or_contains(a: &Path, b: &Path) -> bool {
    +    comparable_path(b).starts_with(comparable_path(a))
    +}
    +
    +fn comparable_path(path: &Path) -> PathBuf {
    +    let absolute = if path.is_absolute() {
    +        path.to_path_buf()
    +    } else {
    +        std::env::current_dir()
    +            .map(|cwd| cwd.join(path))
    +            .unwrap_or_else(|_| path.to_path_buf())
    +    };
    +    let mut normalized = PathBuf::new();
    +    for component in absolute.components() {
    +        match component {
    +            std::path::Component::CurDir => {}
    +            std::path::Component::ParentDir => {
    +                normalized.pop();
    +            }
    +            other => normalized.push(other.as_os_str()),
    +        }
    +    }
    +    // The deepest existing ancestor decides the canonical spelling; the
    +    // rest is appended as written.
    +    let mut existing = normalized.clone();
    +    let mut tail = Vec::new();
    +    loop {
    +        if let Ok(canonical) = std::fs::canonicalize(&existing) {
    +            let mut out = canonical;
    +            for part in tail.iter().rev() {
    +                out.push(part);
    +            }
    +            return out;
    +        }
    +        match (existing.file_name(), existing.parent()) {
    +            (Some(name), Some(parent)) => {
    +                tail.push(name.to_os_string());
    +                existing = parent.to_path_buf();
    +            }
    +            _ => return normalized,
    +        }
    +    }
     }
     
     fn default_data_file_target_bytes() -> u64 {
    @@ -185,15 +503,97 @@ mod tests {
             assert_eq!(c.reconcile_interval_secs, 60);
             assert_eq!(c.max_concurrent_jobs, 4);
             assert_eq!(c.spool_path, None);
    +        assert_eq!(c.store, None);
             c.validate().unwrap();
             let parsed: BackupConfig = toml::from_str("").unwrap();
             assert_eq!(parsed, c);
         }
     
    +    /// A relative root or spool is made absolute against the directory
    +    /// given (the shell's at load); an absolute one, and `~`, resolve as
    +    /// written.
         #[test]
    -    fn an_unknown_key_is_refused() {
    +    fn relative_store_and_spool_paths_are_made_absolute_at_load() {
    +        let mut c: BackupConfig = toml::from_str(
    +            "spool_path = \"rel-spool\"\n[store]\nkind = \"filesystem\"\n\
    +             [store.filesystem]\npath = \"rel-backups\"\n",
    +        )
    +        .unwrap();
    +        c.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            c.store.as_ref().unwrap().filesystem.as_ref().unwrap().path,
    +            PathBuf::from("/srv/extenddb/rel-backups")
    +        );
    +        assert_eq!(c.spool_path.as_deref(), Some("/srv/extenddb/rel-spool"));
    +
    +        let mut absolute: BackupConfig = toml::from_str(
    +            "spool_path = \"~/spool\"\n[store]\nkind = \"filesystem\"\n\
    +             [store.filesystem]\npath = \"/var/lib/extenddb/backups\"\n",
    +        )
    +        .unwrap();
    +        absolute.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            absolute
    +                .store
    +                .as_ref()
    +                .unwrap()
    +                .filesystem
    +                .as_ref()
    +                .unwrap()
    +                .path,
    +            PathBuf::from("/var/lib/extenddb/backups")
    +        );
    +        let home = std::env::var("HOME").unwrap();
    +        assert_eq!(
    +            absolute.spool_path.as_deref(),
    +            Some(format!("{home}/spool").as_str())
    +        );
    +        // An S3 store has no path to resolve; `None` stays `None`.
    +        let mut none = BackupConfig::default();
    +        none.make_paths_absolute(Path::new("/srv"));
    +        assert_eq!(none, BackupConfig::default());
    +        // An empty root stays empty, for `open` to refuse; it must not
    +        // become the working directory.
    +        let mut empty: BackupConfig =
    +            toml::from_str("[store]\nkind = \"filesystem\"\n[store.filesystem]\npath = \"\"\n")
    +                .unwrap();
    +        empty.make_paths_absolute(Path::new("/srv/extenddb"));
    +        assert_eq!(
    +            empty
    +                .store
    +                .as_ref()
    +                .unwrap()
    +                .filesystem
    +                .as_ref()
    +                .unwrap()
    +                .path,
    +            PathBuf::from("")
    +        );
    +    }
    +
    +    #[test]
    +    fn an_unknown_key_is_refused_at_every_level() {
             let e = toml::from_str::("store_kind = \"s3\"\n").unwrap_err();
             assert!(e.to_string().contains("store_kind"), "{e}");
    +        let e = toml::from_str::("[store]\nkid = \"s3\"\n").unwrap_err();
    +        assert!(e.to_string().contains("kid"), "{e}");
    +        let e = toml::from_str::(
    +            "[store]\nkind = \"s3\"\n[store.s3]\nbucket = \"b\"\nbukcet = \"b\"\n",
    +        )
    +        .unwrap_err();
    +        assert!(e.to_string().contains("bukcet"), "{e}");
    +    }
    +
    +    #[test]
    +    fn the_store_section_is_the_crates_shape() {
    +        let c: BackupConfig =
    +            toml::from_str("[store]\nkind = \"s3\"\n[store.s3]\nbucket = \"b\"\nprefix = \"p/\"\n")
    +                .unwrap();
    +        let store = c.store.unwrap();
    +        assert_eq!(store.kind, BackupStoreKind::S3);
    +        assert_eq!(store.s3.unwrap().bucket, "b");
    +        let c: BackupConfig = toml::from_str("[store]\nkind = \"disabled\"\n").unwrap();
    +        assert_eq!(c.store.unwrap().kind, BackupStoreKind::Disabled);
         }
     
         #[test]
    @@ -313,5 +713,267 @@ mod tests {
                     .unwrap_err()
                     .contains("reconcile_interval_secs")
             );
    +        let empty_spool = BackupConfig {
    +            spool_path: Some("  ".to_owned()),
    +            ..BackupConfig::default()
    +        };
    +        assert!(empty_spool.validate().unwrap_err().contains("spool_path"));
    +    }
    +
    +    fn s3_store() -> BackupStoreConfig {
    +        toml::from_str("kind = \"s3\"\n[s3]\nbucket = \"b\"\n").unwrap()
    +    }
    +
    +    fn fs_store(path: &str) -> BackupStoreConfig {
    +        toml::from_str(&format!(
    +            "kind = \"filesystem\"\n[filesystem]\npath = {path:?}\n"
    +        ))
    +        .unwrap()
    +    }
    +
    +    /// An absent section on a file-backed database is the filesystem store
    +    /// beside the file, marked defaulted; on a server or in-memory database
    +    /// it is unconfigured; `disabled` is unconfigured on both.
    +    #[test]
    +    fn an_absent_section_defaults_per_backend() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolved = BackupConfig::default().resolve_store(Some(db)).unwrap();
    +        assert_eq!(
    +            resolved.store,
    +            EffectiveStore::Configured {
    +                config: fs_store("/var/lib/extenddb/backups"),
    +                defaulted: true,
    +            }
    +        );
    +        assert_eq!(
    +            resolved.spool_path, None,
    +            "a filesystem store needs no spool"
    +        );
    +
    +        let resolved = BackupConfig::default().resolve_store(None).unwrap();
    +        assert_eq!(
    +            resolved.store,
    +            EffectiveStore::Unconfigured(Unconfigured::Absent)
    +        );
    +        assert_eq!(resolved.spool_path, None);
    +
    +        let disabled = BackupConfig {
    +            store: Some(toml::from_str("kind = \"disabled\"").unwrap()),
    +            ..BackupConfig::default()
    +        };
    +        for database in [Some(db), None] {
    +            assert_eq!(
    +                disabled.resolve_store(database).unwrap().store,
    +                EffectiveStore::Unconfigured(Unconfigured::Disabled)
    +            );
    +        }
    +
    +        let explicit = BackupConfig {
    +            store: Some(fs_store("/srv/backups")),
    +            ..BackupConfig::default()
    +        };
    +        assert_eq!(
    +            explicit.resolve_store(Some(db)).unwrap().store,
    +            EffectiveStore::Configured {
    +                config: fs_store("/srv/backups"),
    +                defaulted: false,
    +            }
    +        );
    +    }
    +
    +    /// An S3 store's spool: explicit on any backend, defaulted beside a
    +    /// database file, required otherwise with the key named.
    +    #[test]
    +    fn the_spool_is_required_with_s3_except_beside_a_database_file() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let s3 = BackupConfig {
    +            store: Some(s3_store()),
    +            ..BackupConfig::default()
    +        };
    +        assert_eq!(
    +            s3.resolve_store(Some(db)).unwrap().spool_path,
    +            Some(PathBuf::from("/var/lib/extenddb/backup-spool"))
    +        );
    +        let e = s3.resolve_store(None).unwrap_err();
    +        assert!(e.contains("backup.spool_path is required"), "{e}");
    +
    +        let with_spool = BackupConfig {
    +            spool_path: Some("/srv/spool".to_owned()),
    +            ..s3.clone()
    +        };
    +        for database in [Some(db), None] {
    +            assert_eq!(
    +                with_spool.resolve_store(database).unwrap().spool_path,
    +                Some(PathBuf::from("/srv/spool"))
    +            );
    +        }
    +    }
    +
    +    /// Every cell of the spool placement rule: inside the root, containing
    +    /// the root, equal to the root; the database directory itself, an
    +    /// ancestor of it; and the accepted shapes (a sibling of the root, a
    +    /// directory under the database directory, a path elsewhere), each
    +    /// against a filesystem store and against an S3 store.
    +    #[test]
    +    fn a_spool_inside_or_containing_the_root_or_the_database_directory_is_refused() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolve = |store: BackupStoreConfig, spool: &str| {
    +            BackupConfig {
    +                store: Some(store),
    +                spool_path: Some(spool.to_owned()),
    +                ..BackupConfig::default()
    +            }
    +            .resolve_store(Some(db))
    +        };
    +        for spool in [
    +            "/srv/backups/spool",
    +            "/srv",
    +            "/srv/backups",
    +            "/srv/backups/../backups",
    +        ] {
    +            let e = resolve(fs_store("/srv/backups"), spool).unwrap_err();
    +            assert!(e.contains("filesystem backup store root"), "{spool}: {e}");
    +        }
    +        for spool in ["/var/lib/extenddb", "/var/lib", "/var/lib/extenddb/./"] {
    +            let e = resolve(fs_store("/srv/backups"), spool).unwrap_err();
    +            assert!(e.contains("database directory"), "{spool}: {e}");
    +            let e = resolve(s3_store(), spool).unwrap_err();
    +            assert!(e.contains("database directory"), "{spool}: {e}");
    +        }
    +        for spool in [
    +            "/srv/backup-spool",
    +            "/var/lib/extenddb/backup-spool",
    +            "/tmp/extenddb-spool",
    +        ] {
    +            resolve(fs_store("/srv/backups"), spool).unwrap_or_else(|e| panic!("{spool}: {e}"));
    +            let resolved = resolve(s3_store(), spool).unwrap_or_else(|e| panic!("{spool}: {e}"));
    +            assert_eq!(resolved.spool_path, Some(PathBuf::from(spool)));
    +        }
    +        // The defaulted root beside the database file is a root too.
    +        let e = BackupConfig {
    +            spool_path: Some("/var/lib/extenddb/backups/spool".to_owned()),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(db))
    +        .unwrap_err();
    +        assert!(e.contains("filesystem backup store root"), "{e}");
    +    }
    +
    +    #[test]
    +    fn path_overlap_follows_an_existing_symlink_and_normalizes_dots() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let real = dir.path().join("real");
    +        std::fs::create_dir(&real).unwrap();
    +        let link = dir.path().join("link");
    +        std::os::unix::fs::symlink(&real, &link).unwrap();
    +        assert!(paths_overlap(&link.join("spool"), &real));
    +        assert!(paths_overlap(&real, &link));
    +        assert!(paths_overlap(
    +            &dir.path().join("real/x/../y"),
    +            &dir.path().join("real/y/z")
    +        ));
    +        assert!(!paths_overlap(
    +            &dir.path().join("real-2"),
    +            &dir.path().join("real")
    +        ));
    +    }
    +
    +    /// A filesystem root that is the database directory (`"."` with the
    +    /// database file in the working directory, or the directory spelled
    +    /// out) or an ancestor of it is refused naming the key and both
    +    /// directories; a child of the directory (the default root among them),
    +    /// a sibling and a root elsewhere are accepted, and the rule does not
    +    /// apply to an S3 store or to a server database (no directory).
    +    #[test]
    +    fn a_root_that_is_or_contains_the_database_directory_is_refused() {
    +        let db = Path::new("/var/lib/extenddb/data.sqlite");
    +        let resolve = |path: &str| {
    +            BackupConfig {
    +                store: Some(fs_store(path)),
    +                ..BackupConfig::default()
    +            }
    +            .resolve_store(Some(db))
    +        };
    +        for root in ["/var/lib/extenddb", "/var/lib/extenddb/./", "/var/lib", "/"] {
    +            let e = resolve(root).unwrap_err();
    +            assert!(e.contains("backup.store.filesystem.path"), "{root}: {e}");
    +            assert!(
    +                e.contains("database directory /var/lib/extenddb"),
    +                "{root}: {e}"
    +            );
    +        }
    +        for root in [
    +            "/var/lib/extenddb/backups",
    +            "/var/lib/extenddb-backups",
    +            "/srv/backups",
    +        ] {
    +            resolve(root).unwrap_or_else(|e| panic!("{root}: {e}"));
    +        }
    +        BackupConfig::default()
    +            .resolve_store(Some(db))
    +            .expect("the default root is a child of the directory");
    +        BackupConfig {
    +            store: Some(fs_store("/var/lib/extenddb")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(None)
    +        .expect("a server database has no directory to protect");
    +    }
    +
    +    /// An empty root is compared with nothing: a server started from the
    +    /// database's directory (`cd /var/lib/extenddb && extenddb serve`, the
    +    /// layout `init` leaves) with `path = ""` is not refused as a root that
    +    /// is the database directory, a path the operator never wrote; the
    +    /// configuration resolves with the empty path, which `open` refuses by
    +    /// name. The spool rule likewise compares nothing with it.
    +    #[test]
    +    fn an_empty_root_is_left_for_open_to_refuse() {
    +        let cwd = std::env::current_dir().unwrap();
    +        let db = cwd.join("data.sqlite");
    +        let resolved = BackupConfig {
    +            store: Some(fs_store("")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&db))
    +        .expect("an empty root is not the working directory");
    +        assert!(matches!(
    +            resolved.store,
    +            EffectiveStore::Configured { config, defaulted: false }
    +                if config.filesystem.as_ref().unwrap().path.as_os_str().is_empty()
    +        ));
    +        BackupConfig {
    +            spool_path: Some("/srv/spool".to_owned()),
    +            store: Some(fs_store("")),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&db))
    +        .expect("a spool is not inside an empty root");
    +    }
    +
    +    /// A refusal spells both paths as they were compared, so a spool given
    +    /// through a symlink and a database directory given by its real path do
    +    /// not read as two different places.
    +    #[test]
    +    fn a_placement_refusal_spells_both_paths_the_same_way() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let real = dir.path().join("real");
    +        std::fs::create_dir(&real).unwrap();
    +        let link = dir.path().join("link");
    +        std::os::unix::fs::symlink(&real, &link).unwrap();
    +        let e = BackupConfig {
    +            spool_path: Some(link.display().to_string()),
    +            store: Some(toml::from_str("kind = \"disabled\"").unwrap()),
    +            ..BackupConfig::default()
    +        }
    +        .resolve_store(Some(&real.join("data.sqlite")))
    +        .unwrap_err();
    +        let real = real.canonicalize().unwrap().display().to_string();
    +        assert!(
    +            e.contains(&format!(
    +                "backup.spool_path {real} is or contains the database directory {real}"
    +            )),
    +            "{e}"
    +        );
    +        assert!(!e.contains("link"), "{e}");
         }
     }
    diff --git a/crates/config/src/lib.rs b/crates/config/src/lib.rs
    index 5b1cb603a..ccdb5833d 100644
    --- a/crates/config/src/lib.rs
    +++ b/crates/config/src/lib.rs
    @@ -179,6 +179,13 @@ impl StorageConfig {
             self.config.startup_warnings()
         }
     
    +    /// The local file the database lives in, when the backend's database is
    +    /// one file on this host ([`extenddb_storage::config::StorageConfig::database_file`]).
    +    #[must_use]
    +    pub fn database_file(&self) -> Option {
    +        self.config.database_file()
    +    }
    +
         /// Get a reference to the underlying trait object for factory calls.
         pub fn as_trait(&self) -> &(dyn extenddb_storage::config::StorageConfig + 'static) {
             &*self.config
    @@ -423,9 +430,13 @@ pub fn load(config_path: &str) -> anyhow::Result {
         validated(config.try_deserialize()?)
     }
     
    -/// The cross-field checks a deserialized config still has to pass.
    -fn validated(config: AppConfig) -> anyhow::Result {
    +/// The cross-field checks a deserialized config still has to pass, and the
    +/// paths made absolute while the working directory is still the shell's.
    +fn validated(mut config: AppConfig) -> anyhow::Result {
         config.backup.validate().map_err(|e| anyhow::anyhow!(e))?;
    +    if let Ok(cwd) = std::env::current_dir() {
    +        config.backup.make_paths_absolute(&cwd);
    +    }
         Ok(config)
     }
     
    diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml
    index 9f58bea6e..d8e20bc15 100755
    --- a/crates/engine/Cargo.toml
    +++ b/crates/engine/Cargo.toml
    @@ -11,6 +11,7 @@ license.workspace = true
     extenddb-auth = { workspace = true }
     extenddb-core = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     serde = { workspace = true }
     serde_json = { workspace = true }
     tracing = { workspace = true }
    @@ -22,3 +23,7 @@ sha2 = "0.10"
     hex = "0.4"
     tokio = { workspace = true }
     futures = { workspace = true }
    +
    +[dev-dependencies]
    +# `start_paused` for the tests of timers that must not depend on the wall clock.
    +tokio = { workspace = true, features = ["test-util"] }
    diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs
    index 0475e9271..56445e86b 100755
    --- a/crates/engine/src/backup.rs
    +++ b/crates/engine/src/backup.rs
    @@ -35,7 +35,12 @@ use extenddb_storage::{
     use futures::TryStreamExt;
     use serde_json::{Value, json};
     
    -pub use runtime::BackupRuntime;
    +pub use maintenance::HeldClaim;
    +pub use runtime::{
    +    BackupRuntime, ConfiguredStore, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, StoreHealth,
    +    StoreHealthReport, check_configured_store, read_identity_bounded,
    +};
    +pub use store_identity::{STORE_IDENTITY_KEY, StoreIdentityFile, StoreProblem};
     
     use crate::{OperationContext, serialize_output};
     
    @@ -436,8 +441,9 @@ pub(crate) async fn handle_create_backup(
     /// `probe` is the `DescribeTable` of the table, which the handler ran.
     ///
     /// This is the in-catalog create, the path while no `[backup.store]` is
    -/// configured; each call logs that the format is deprecated.
    -async fn create_backup(
    +/// configured; each call logs that the format is deprecated. Last before the
    +/// permit, the maintenance claim ([`refuse_while_maintenance_held`]).
    +async fn create_backup(
         storage: &S,
         runtime: &BackupRuntime,
         account_id: &str,
    @@ -458,18 +464,78 @@ async fn create_backup(
         };
         check_backup_arn_fits(region)?;
         refuse_backup_of_vector_table(&desc)?;
    +    // A create that writes to the configured store needs it healthy, and is
    +    // refused before a slot is taken when it is not, as every operation that
    +    // touches the store is; the in-catalog create touches no store, so the
    +    // store's health does not gate it (a backend bound to the in-catalog
    +    // create, or a server with an unhealthy store, creates as a server
    +    // without a store does).
    +    let store_backed = runtime.configured_store().is_some()
    +        && !runtime.legacy_create_only()
    +        && STORE_BACKED_CREATE;
    +    if store_backed {
    +        runtime.require_healthy_store()?;
    +    }
    +    refuse_while_maintenance_held(storage, runtime).await?;
         let permit = runtime.take_permit().await?;
    -    tracing::warn!(
    -        "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog backup \
    -         format, which a later release stops creating; configure [backup.store] to write \
    -         backups to a backup store"
    -    );
    +    match runtime.configured_store() {
    +        None => tracing::warn!(
    +            "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog \
    +             backup format, which a later release stops creating; configure [backup.store] to \
    +             write backups to a backup store"
    +        ),
    +        Some(store) => tracing::info!(
    +            "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog \
    +             backup format; the configured {} backup store serves restores and deletes of the \
    +             backups it holds, and a later release's CreateBackup writes to it",
    +            store.kind
    +        ),
    +    }
         storage
             .create(account_id, table_name, backup_name, permit)
             .await
             .map_err(|e| create_backup_storage_err_to_dynamo(e, table_name))
     }
     
    +/// The jobs' side of the catalog-level maintenance claim: a backup or
    +/// restore job is refused with `LimitExceededException` while a maintenance
    +/// command (`extenddb backup purge`) holds the claim, read with
    +/// `stale_job_after_secs` so a dead command's claim expires. The check runs
    +/// after every other refusal and before the permit, which on this head is
    +/// before the job's row exists: the in-catalog create and restore commit
    +/// their row inside the backend and detach, so the engine has no point
    +/// between the row and the copy to check at. A claim taken between this
    +/// check and the row's insert is therefore not seen by the job; the command
    +/// create that slips through writes no object of the store the command
    +/// deletes from, and a restore that slips through is a RUNNING record the
    +/// command's wait lists, so the window costs nothing in this release. The
    +/// store-backed job of a later release owns its row from the
    +/// engine and checks after the row exists, as the design's 5.1 step 2 says.
    +async fn refuse_while_maintenance_held(
    +    storage: &S,
    +    runtime: &BackupRuntime,
    +) -> Result<(), DynamoDbError> {
    +    let held = storage
    +        .check_maintenance_claim(runtime.stale_job_after())
    +        .await
    +        .map_err(storage_err_to_dynamo)?;
    +    if held {
    +        return Err(DynamoDbError::LimitExceededException(
    +            "A backup maintenance command holds this catalog's maintenance claim; backup and \
    +             restore jobs are refused until it finishes"
    +                .to_owned(),
    +        ));
    +    }
    +    Ok(())
    +}
    +
    +/// Whether CreateBackup writes to the configured store. No store-backed
    +/// create exists on this head: every create writes the in-catalog format,
    +/// whatever the store configuration, and the release that adds the
    +/// store-backed job turns this on for a server with a store configured and
    +/// not bound to the in-catalog create.
    +const STORE_BACKED_CREATE: bool = false;
    +
     /// The one backup ARN rule CreateBackup applies on this path: the server's
     /// own region (`server.region` outside `[a-z0-9-]` forms no backup ARN
     /// whatever the name, so the message names the setting, not the table). The
    @@ -1080,16 +1146,26 @@ async fn delete_backup(
     /// `BackupNotFoundException`; a CREATING row is `BackupInUseException`; a
     /// backup a CREATING restore target still reads is `BackupInUseException`
     /// naming that table ([`BackupCatalog::backup_in_use`]); a row of a store
    -/// other than the configured one is a `ValidationException` naming no root;
    +/// other than the configured one is a `ValidationException` naming no root,
    +/// and a row of the configured store while that store is unhealthy or its
    +/// first check pending is a `ValidationException` naming the store's kind;
     /// then the in-use check runs again with the AVAILABLE-to-DELETING write in
     /// one transaction under the row's lock
     /// ([`BackupCatalog::start_backup_delete`]), so a restore's registration and
     /// this order wholly one before the other: a registration that won leaves
     /// this `BackupInUseException` naming its target. A DELETING row (a retried
    -/// DeleteBackup) resumes the deletion. The request returns the description
    -/// with `BackupStatus` DELETED; the store's objects are deleted by the
    -/// duty a configured store brings, and the row stays DELETING until a
    -/// listing of its prefix returns nothing.
    +/// DeleteBackup) resumes the deletion. Then the deletion itself
    +/// ([`finish_backup_delete`], through the store's bounded calls,
    +/// [`ConfiguredStore::calls`]): the manifest first, so the prefix is no
    +/// longer a complete backup to any reader, then the rest of the prefix,
    +/// then the row to DELETED once a listing of the prefix returns nothing. The
    +/// request answers the description with `BackupStatus` DELETED once the
    +/// DELETING write has committed, whatever the deletion does: every reader
    +/// already reports the backup deleted from that write on, and the service
    +/// has no counterpart for a failed object deletion. If a step fails (the
    +/// store not answering, a listing that still shows an object) the row stays
    +/// DELETING, the failure is logged, and the reconciler finishes the deletion
    +/// at its next cycle.
     ///
     /// The description is built before the DELETING write: a row whose
     /// `source_table_details_json` this build cannot read is answered as an
    @@ -1118,6 +1194,7 @@ async fn delete_catalog_backup(
         if !runtime.is_configured_store(&row.store) {
             return Err(backup_in_another_store(&arn_text));
         }
    +    let store = runtime.require_healthy_store()?;
         let mut desc = row.description().map_err(storage_err_to_dynamo)?;
         match storage
             .start_backup_delete(&key)
    @@ -1142,10 +1219,65 @@ async fn delete_catalog_backup(
                 ))));
             }
         }
    +    if let Err(e) = finish_backup_delete(storage, &store.calls("deletion"), arn).await {
    +        tracing::error!(
    +            "DeleteBackup {arn_text}: the row is DELETING but its objects could not be deleted; \
    +             the reconciler retries: {e}"
    +        );
    +    }
         desc.backup_details.backup_status = BackupStatus::Deleted.as_str().to_owned();
         Ok(desc)
     }
     
    +/// The deletion of a DELETING row's objects and the row's move to DELETED:
    +/// `extenddb-manifest.json` first (a `delete_prefix` on its exact key, so the
    +/// prefix is no longer a complete backup to any reader), then the rest of the
    +/// prefix, then, once a listing of the prefix returns nothing,
    +/// `set_backup_deleted`. Shared by DeleteBackup and the reconciler's duty for
    +/// a DELETING row; either may run it on a row the other started, and a run
    +/// that finds the row no longer DELETING (`false` from the catalog) changes
    +/// nothing more. Returns whether this run moved the row.
    +///
    +/// # Errors
    +///
    +/// A store failure, a listing that still shows objects (a put that
    +/// overlapped the deletion, which the next run removes), or a catalog write
    +/// failure; the row stays DELETING for the next run.
    +pub(crate) async fn finish_backup_delete(
    +    storage: &S,
    +    store: &dyn extenddb_backup_store::BackupStore,
    +    arn: &BackupArn,
    +) -> Result {
    +    let prefix = arn.store_prefix();
    +    let manifest_key = format!(
    +        "{prefix}{}",
    +        extenddb_core::backup_format::EXTENDDB_MANIFEST_FILE
    +    );
    +    store
    +        .delete_prefix(&manifest_key)
    +        .await
    +        .map_err(|e| format!("deleting the manifest: {e}"))?;
    +    store
    +        .delete_prefix(&prefix)
    +        .await
    +        .map_err(|e| format!("deleting the prefix: {e}"))?;
    +    let remaining = store
    +        .list(&prefix)
    +        .try_next()
    +        .await
    +        .map_err(|e| format!("listing the prefix after the deletion: {e}"))?;
    +    if let Some(object) = remaining {
    +        return Err(format!(
    +            "the prefix still lists {} after the deletion",
    +            object.key
    +        ));
    +    }
    +    storage
    +        .set_backup_deleted(&BackupKey::from_arn(arn))
    +        .await
    +        .map_err(|e| format!("marking the row DELETED: {e}"))
    +}
    +
     /// Overrides refused outright. `VectorIndexOverride` is not in the botocore
     /// 1.33.13 model; newer models and SDKs send it. The index overrides and
     /// `OnDemandThroughputOverride` are applied as the service applies them; see
    @@ -1332,17 +1464,21 @@ where
     /// answers `TableAlreadyExistsException`); then, for a row, the rest of its
     /// checks ([`check_catalog_row_for_restore`]: a backup in use is
     /// `BackupInUseException`, a row of another store a `ValidationException`,
    -/// and an AVAILABLE row of the configured store the `ValidationException`
    -/// naming the store, since this release restores in-catalog backups only);
    +/// a row of the configured store while it is unhealthy one naming the
    +/// store's kind, and an AVAILABLE row of the healthy store the
    +/// `ValidationException` naming the store, since this release restores
    +/// in-catalog backups only);
     /// (6) refuse what no
     /// backend restores faithfully (multi-part key, vector indexes); (7) check
     /// the throughput override against the backup's billing mode, build the
     /// target's `CreateTableInput`, and apply the index and on-demand throughput
     /// overrides to it, refusing what the service refuses ([`overrides`]);
    -/// (8) validate that input as CreateTable does; (9) take the job permit,
    -/// last among the checks (`LimitExceededException` when none is freed within
    -/// the runtime's wait), and ask the backend to create and register the
    -/// target; (10) map its errors, a taken name being
    +/// (8) validate that input as CreateTable does; (9) refuse while a
    +/// maintenance command holds the catalog's claim
    +/// ([`refuse_while_maintenance_held`]), then take the job permit, last among
    +/// the checks (`LimitExceededException` for either; the permit when none is
    +/// freed within the runtime's wait), and ask the backend to create and
    +/// register the target; (10) map its errors, a taken name being
     /// `TableAlreadyExistsException`, a name being restored into
     /// `TableInUseException` (the backend checks the name again, fenced, under
     /// the registration), a backup being restored from `BackupInUseException`.
    @@ -1408,14 +1544,15 @@ async fn restore_table_from_backup Result<(), Dyna
     /// a CREATING restore target still reads is `BackupInUseException` naming
     /// that table in the log ([`BackupCatalog::backup_in_use`]); a row of a
     /// store other than the configured one is a `ValidationException` naming no
    -/// root. An AVAILABLE row of the configured store is a backup in a store,
    -/// whose restore this release does not run.
    +/// root, and a row of the configured store while that store is unhealthy or
    +/// its first check pending is a `ValidationException` naming the store's
    +/// kind ([`BackupRuntime::require_healthy_store`]). An AVAILABLE row of a
    +/// healthy configured store is a backup in a store, whose restore this
    +/// release does not run.
     async fn check_catalog_row_for_restore(
         storage: &S,
         runtime: &BackupRuntime,
    @@ -1466,6 +1606,7 @@ async fn check_catalog_row_for_restore(
         if !runtime.is_configured_store(&row.store) {
             return Err(backup_in_another_store(&arn_text));
         }
    +    runtime.require_healthy_store()?;
         Err(DynamoDbError::ValidationException(format!(
             "Backup {arn_text} is in a backup store; this release restores in-catalog backups only"
         )))
    @@ -1641,10 +1782,12 @@ fn storage_err_to_dynamo(e: StorageError) -> DynamoDbError {
         }
     }
     
    +pub mod maintenance;
     mod overrides;
     mod pitr;
     pub mod reconciler;
     mod runtime;
    +pub mod store_identity;
     #[cfg(test)]
     mod tests;
     
    diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs
    new file mode 100644
    index 000000000..dd95800da
    --- /dev/null
    +++ b/crates/engine/src/backup/maintenance.rs
    @@ -0,0 +1,648 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! The holder's side of the catalog-level maintenance claim, and the steps a
    +//! maintenance command (`extenddb backup purge`) runs under it.
    +//!
    +//! [`HeldClaim`] is the handle the command keeps from the take to the
    +//! release, with the heartbeat running on a timer for the handle's whole
    +//! life, so every reader of the claim (`check_maintenance_claim` on every
    +//! backend measures the heartbeat's age, PostgreSQL the lock first) sees it
    +//! held through a wait and a deletion phase of any length. A heartbeat that
    +//! finds the claim lost (another holder took it over after this one looked
    +//! dead), or fails, is recorded for the command to stop at before its next
    +//! destructive step. Dropping the handle without `release` stops the timer
    +//! (the claim then expires by the stale rule, as a dead holder's does);
    +//! `release` stops it, waits for a heartbeat in flight, then releases.
    +//!
    +//! [`wait_for_quiet`] is the wait for the jobs the claim excludes from
    +//! starting to finish, [`purge_row`] the deletion of one row's objects and
    +//! the row, and [`purge_rows`] the loop over the rows that stops at a lost
    +//! claim or a signal, each written against the catalog's trait so the
    +//! engine's tests run them against a double; the command prints what they
    +//! return. An
    +//! [`Interrupt`] carries a signal the command received to both, so an
    +//! operator's Ctrl-C ends the wait at once and the deletion loop before its
    +//! next row, never inside one (the row's own steps are the idempotent
    +//! sequence a second deleter finishes).
    +
    +use std::sync::{Arc, Mutex, PoisonError};
    +use std::time::Duration;
    +
    +use extenddb_backup_store::BackupStore;
    +use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE};
    +use extenddb_storage::backup_catalog::{
    +    BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, StaleClock, StoreIdentity,
    +    is_backup_not_found,
    +};
    +use extenddb_storage::{BackupCatalog, MaintenanceClaim};
    +use futures::TryStreamExt;
    +use tokio::sync::watch;
    +use tokio::time::Instant;
    +
    +/// A maintenance claim with its heartbeat on a timer.
    +pub struct HeldClaim {
    +    claim: Arc,
    +    lost: Arc>>,
    +    timer: tokio::task::JoinHandle<()>,
    +}
    +
    +impl HeldClaim {
    +    /// Hold `claim`, heartbeating it every `interval` from now until
    +    /// [`HeldClaim::release`] or the drop (the caller passes the jobs'
    +    /// heartbeat period, `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL`,
    +    /// which every stale window the config accepts is at least three of).
    +    #[must_use]
    +    pub fn start(claim: Box, interval: Duration) -> Self {
    +        let claim: Arc = Arc::from(claim);
    +        let lost = Arc::new(Mutex::new(None));
    +        let timer = tokio::spawn({
    +            let claim = Arc::clone(&claim);
    +            let lost = Arc::clone(&lost);
    +            async move {
    +                let mut ticks = tokio::time::interval(interval);
    +                ticks.tick().await;
    +                loop {
    +                    ticks.tick().await;
    +                    let reason = match claim.heartbeat().await {
    +                        Ok(Heartbeat::Confirmed) => continue,
    +                        Ok(Heartbeat::Lost) => {
    +                            "the maintenance claim was lost (another holder took it over)"
    +                                .to_owned()
    +                        }
    +                        Err(e) => format!("the maintenance claim's heartbeat failed: {e}"),
    +                    };
    +                    *lost.lock().unwrap_or_else(PoisonError::into_inner) = Some(reason);
    +                    return;
    +                }
    +            }
    +        });
    +        Self { claim, lost, timer }
    +    }
    +
    +    /// The claim's fencing token.
    +    #[must_use]
    +    pub fn generation(&self) -> u64 {
    +        self.claim.generation()
    +    }
    +
    +    /// Why the claim is no longer held, once a heartbeat said so; `None`
    +    /// while every heartbeat has confirmed it.
    +    #[must_use]
    +    pub fn lost(&self) -> Option {
    +        self.lost
    +            .lock()
    +            .unwrap_or_else(PoisonError::into_inner)
    +            .clone()
    +    }
    +
    +    /// Stop the heartbeat, wait for one in flight to end, and release the
    +    /// claim, under [`RELEASE_TIMEOUT`]: a catalog that does not answer the
    +    /// release leaves the claim to expire by the stale rule, and the command
    +    /// ends.
    +    ///
    +    /// # Errors
    +    ///
    +    /// The backend's, when the release could not be written, or the bound;
    +    /// the claim then expires by the stale rule.
    +    pub async fn release(mut self) -> Result<(), extenddb_storage::error::StorageError> {
    +        self.timer.abort();
    +        // The abort lands at the task's next await; a heartbeat already sent
    +        // finishes first, so the release never overlaps it on the claim's
    +        // connection. A cancelled task is the expected outcome.
    +        let _ = (&mut self.timer).await;
    +        tokio::time::timeout(RELEASE_TIMEOUT, self.claim.release())
    +            .await
    +            .unwrap_or_else(|_| {
    +                Err(extenddb_storage::error::StorageError::Internal(format!(
    +                    "the release did not finish within {} s",
    +                    RELEASE_TIMEOUT.as_secs()
    +                )))
    +            })
    +    }
    +}
    +
    +/// How long [`HeldClaim::release`] waits for the catalog: past it the claim
    +/// is left to the stale rule, as a dead holder's is, so a stalled database
    +/// does not hold a command that received a signal.
    +pub const RELEASE_TIMEOUT: Duration = Duration::from_secs(10);
    +
    +impl Drop for HeldClaim {
    +    fn drop(&mut self) {
    +        self.timer.abort();
    +    }
    +}
    +
    +/// A signal the command received (`SIGINT` or `SIGTERM`), for the wait and
    +/// the deletion loop to stop at: the wait at once, the loop before its next
    +/// row. The sender side is the command's signal listener; a test's is the
    +/// test.
    +#[derive(Clone)]
    +pub struct Interrupt {
    +    rx: watch::Receiver>,
    +}
    +
    +impl Interrupt {
    +    /// A pair: the sender a signal listener sets with the signal's name, and
    +    /// the handle the wait and the loop read.
    +    #[must_use]
    +    pub fn new() -> (watch::Sender>, Self) {
    +        let (tx, rx) = watch::channel(None);
    +        (tx, Self { rx })
    +    }
    +
    +    /// The signal's name once one was received.
    +    #[must_use]
    +    pub fn signalled(&self) -> Option<&'static str> {
    +        *self.rx.borrow()
    +    }
    +
    +    /// Resolves with the signal's name once one is received; pending until
    +    /// then (forever, when the sender is gone and none was).
    +    pub async fn wait(&mut self) -> &'static str {
    +        loop {
    +            if let Some(name) = *self.rx.borrow_and_update() {
    +                return name;
    +            }
    +            if self.rx.changed().await.is_err() {
    +                return std::future::pending().await;
    +            }
    +        }
    +    }
    +}
    +
    +/// How [`wait_for_quiet`] ended.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum WaitOutcome {
    +    /// No job the claim excludes is running: the deletion may start.
    +    Quiet,
    +    /// The limit passed with these jobs still running (the operator's text,
    +    /// one per job).
    +    Busy(Vec),
    +    /// A heartbeat found the claim lost or failed; the reason.
    +    Lost(String),
    +    /// The rows could not be read; the backend's error.
    +    CatalogError(String),
    +    /// The command received this signal.
    +    Interrupted(&'static str),
    +}
    +
    +/// The durations of a [`wait_for_quiet`].
    +#[derive(Debug, Clone, Copy)]
    +pub struct Wait {
    +    /// The window a live owner is told apart by: a backup row the backend
    +    /// does not list as older than this on the heartbeat clock has a live
    +    /// owner; zero where every owner is known dead (SQLite under the serve
    +    /// lock).
    +    pub owner_window: Duration,
    +    /// How long to wait before giving up with the jobs still running.
    +    pub limit: Duration,
    +    /// How often the rows are re-read.
    +    pub poll: Duration,
    +}
    +
    +/// Wait until no CREATING or FAILED backup row of `identity` has a live
    +/// owner and no restore job is RUNNING, re-reading every `wait.poll`, for
    +/// at most `wait.limit`; the claim lost, a catalog read failing, or a
    +/// signal ends it early. The jobs found running are reported through
    +/// `announce` the first time they are seen.
    +pub async fn wait_for_quiet(
    +    storage: &S,
    +    identity: &StoreIdentity,
    +    claim: &HeldClaim,
    +    wait: Wait,
    +    interrupt: &mut Interrupt,
    +    announce: impl Fn(&[String]),
    +) -> WaitOutcome {
    +    let started = Instant::now();
    +    let mut announced = false;
    +    loop {
    +        if let Some(reason) = claim.lost() {
    +            return WaitOutcome::Lost(reason);
    +        }
    +        if let Some(signal) = interrupt.signalled() {
    +            return WaitOutcome::Interrupted(signal);
    +        }
    +        let remaining = match busy_jobs(storage, identity, wait.owner_window).await {
    +            Ok(remaining) => remaining,
    +            Err(e) => return WaitOutcome::CatalogError(e),
    +        };
    +        if remaining.is_empty() {
    +            return WaitOutcome::Quiet;
    +        }
    +        if started.elapsed() >= wait.limit {
    +            return WaitOutcome::Busy(remaining);
    +        }
    +        if !announced {
    +            announce(&remaining);
    +            announced = true;
    +        }
    +        tokio::select! {
    +            () = tokio::time::sleep(wait.poll) => {}
    +            signal = interrupt.wait() => return WaitOutcome::Interrupted(signal),
    +        }
    +    }
    +}
    +
    +/// The jobs purge waits on, as text for the operator. Every age is the
    +/// backend's comparison on the catalog's clock: a CREATING or FAILED backup
    +/// row has a live owner while `list_backup_rows_older_than` on the
    +/// heartbeat clock does not list it under the window (a FAILED row whose
    +/// heartbeat is young may still be written to by a job that has not seen
    +/// its claim lost; its prefix waits for the window as the reconciler's
    +/// deletion of it does), and a RUNNING restore record is listed by status
    +/// (`list_running_restore_jobs`), with no age.
    +async fn busy_jobs(
    +    storage: &S,
    +    identity: &StoreIdentity,
    +    owner_window: Duration,
    +) -> Result, String> {
    +    let mut busy = Vec::new();
    +    for status in [BackupStatus::Creating, BackupStatus::Failed] {
    +        let rows: Vec = storage
    +            .list_backup_rows(Some(status), identity)
    +            .try_collect()
    +            .await
    +            .map_err(|e| format!("cannot list the catalog's backup rows: {e}"))?;
    +        let dead: Vec = storage
    +            .list_backup_rows_older_than(status, identity, StaleClock::Heartbeat, owner_window)
    +            .try_collect()
    +            .await
    +            .map_err(|e| format!("cannot list the catalog's backup rows by age: {e}"))?;
    +        for row in rows {
    +            if dead.iter().any(|d| d.backup_arn == row.backup_arn) {
    +                continue;
    +            }
    +            busy.push(format!(
    +                "{} {} has a live owner (heartbeat within stale_job_after_secs)",
    +                if status == BackupStatus::Creating {
    +                    "CreateBackup"
    +                } else {
    +                    "the FAILED backup"
    +                },
    +                row.backup_arn
    +            ));
    +        }
    +    }
    +    // Every RUNNING restore job, by status alone: a record stamped in the
    +    // listing's own instant, or ahead of the catalog's clock, is a running
    +    // job too, which an age-based listing with a zero window would miss. The
    +    // FAILED records whose target is still CREATING are the reconciler's to
    +    // finish, not jobs: nothing runs under one, and purge deletes no table,
    +    // so they are not waited on.
    +    let running = storage
    +        .list_running_restore_jobs()
    +        .await
    +        .map_err(|e| format!("cannot list restore jobs: {e}"))?;
    +    for job in running {
    +        if job.status != JobStatus::Running {
    +            continue;
    +        }
    +        busy.push(format!(
    +            "a restore into {} (account {}) is RUNNING",
    +            job.target.table_name, job.target.account_id
    +        ));
    +    }
    +    Ok(busy)
    +}
    +
    +/// Delete one row's objects and the row: the manifest first, then the
    +/// prefix, then the row through the status machine to a tombstone and out.
    +/// The row is `row` as the command read it; the status another deleter may
    +/// have moved it to since is read from `start_backup_delete`'s answer, and
    +/// each state the row can be in when the command reaches it ends with no
    +/// row: AVAILABLE is moved to DELETING here; DELETING (another deleter is
    +/// mid-way) and DELETED (another finished) are finished here, the prefix
    +/// deletion being the same idempotent sequence on both sides; a row already
    +/// gone counts as deleted. A CREATING row's owner is dead (the wait saw no
    +/// live one): the row goes first, through the dead-owner removal on the
    +/// catalog's clock, and the prefix only once the row is gone, so an owner
    +/// that heartbeated after the wait leaves the row and its objects together
    +/// rather than a row over a deleted prefix. A prefix left by a failure
    +/// between the two steps holds no manifest (a CREATING backup has none
    +/// yet), so nothing adopts it.
    +///
    +/// # Errors
    +///
    +/// The reason the row was left, for the operator: a FAILED or CREATING
    +/// status another writer moved an AVAILABLE row to (neither follows
    +/// AVAILABLE; the row is left for a look), a restore reading the backup, a
    +/// CREATING row whose owner heartbeated after the wait, a store or catalog
    +/// call failing, or a prefix that still lists an object.
    +pub async fn purge_row(
    +    storage: &S,
    +    store: &dyn BackupStore,
    +    row: &BackupRow,
    +    stale_after: Duration,
    +) -> Result<(), String> {
    +    let arn = BackupArn::parse(&row.backup_arn)
    +        .map_err(|e| format!("the row's ARN does not parse: {e}"))?;
    +    let key = row.key();
    +    if row.status == BackupStatus::Creating
    +        && !storage
    +            .remove_dead_creating_backup_row(&key, stale_after)
    +            .await
    +            .map_err(|e| format!("cannot remove the CREATING row: {e}"))?
    +    {
    +        return Err(
    +            "the CREATING row could not be removed (its owner heartbeated after the wait, or its \
    +             status changed); its objects are left with it"
    +                .to_owned(),
    +        );
    +    }
    +    // The status to finish from: the row's as read, or as another deleter
    +    // left it.
    +    let mut status = row.status;
    +    if status == BackupStatus::Available {
    +        match storage.start_backup_delete(&key).await {
    +            Ok(DeleteStart::Deleting | DeleteStart::Status(BackupStatus::Deleting)) => {
    +                status = BackupStatus::Deleting;
    +            }
    +            Ok(DeleteStart::Status(BackupStatus::Deleted)) => status = BackupStatus::Deleted,
    +            Ok(DeleteStart::InUse(table)) => {
    +                return Err(format!("a restore into {table} is reading it"));
    +            }
    +            Ok(DeleteStart::Status(other)) => {
    +                return Err(format!("its status changed to {other}"));
    +            }
    +            Err(e) if is_backup_not_found(&e) => return Ok(()),
    +            Err(e) => return Err(format!("cannot mark the row DELETING: {e}")),
    +        }
    +    }
    +    let prefix = arn.store_prefix();
    +    store
    +        .delete_prefix(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}"))
    +        .await
    +        .map_err(|e| format!("cannot delete the manifest: {e}"))?;
    +    store
    +        .delete_prefix(&prefix)
    +        .await
    +        .map_err(|e| format!("cannot delete the prefix: {e}"))?;
    +    if let Some(left) = store
    +        .list(&prefix)
    +        .try_next()
    +        .await
    +        .map_err(|e| format!("cannot list the prefix after the deletion: {e}"))?
    +    {
    +        return Err(format!(
    +            "the prefix still lists {} after the deletion",
    +            left.key
    +        ));
    +    }
    +    if status == BackupStatus::Creating {
    +        return Ok(());
    +    }
    +    if status == BackupStatus::Deleting {
    +        storage
    +            .set_backup_deleted(&key)
    +            .await
    +            .map_err(|e| format!("cannot mark the row DELETED: {e}"))?;
    +    }
    +    if storage
    +        .remove_backup_row(&key)
    +        .await
    +        .map_err(|e| format!("cannot remove the row: {e}"))?
    +    {
    +        return Ok(());
    +    }
    +    // Not a tombstone: another deleter removed it, or a writer moved it.
    +    match storage
    +        .read_backup_row(&arn)
    +        .await
    +        .map_err(|e| format!("cannot re-read the row: {e}"))?
    +    {
    +        None => Ok(()),
    +        Some(current) => Err(format!(
    +            "the row could not be removed (it is {} now, not a tombstone)",
    +            current.status
    +        )),
    +    }
    +}
    +
    +/// What [`purge_rows`] did.
    +#[derive(Debug, Default, Clone, PartialEq, Eq)]
    +pub struct PurgeOutcome {
    +    /// The rows deleted, in order.
    +    pub deleted: Vec,
    +    /// The rows left, each with the reason (`: `).
    +    pub skipped: Vec,
    +    /// Why the loop stopped before its next row, if it did: the claim lost
    +    /// or its heartbeat failed, or the signal received.
    +    pub stopped: Option,
    +}
    +
    +/// Why [`purge_rows`] stopped before its next row.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum Stop {
    +    /// A heartbeat found the claim lost or failed; the reason.
    +    Lost(String),
    +    /// The command received this signal.
    +    Interrupted(&'static str),
    +}
    +
    +/// The deletion phase: [`purge_row`] on each of `rows` in order, stopping
    +/// before the next row (never inside one) once a heartbeat has found the
    +/// claim lost or failed, or a signal has been received. A row that could not
    +/// be deleted is skipped with its reason and the loop goes on; `deleted` is
    +/// called with each row deleted, for the command's audit line and output.
    +pub async fn purge_rows(
    +    storage: &S,
    +    store: &dyn BackupStore,
    +    rows: &[BackupRow],
    +    stale_after: Duration,
    +    claim: &HeldClaim,
    +    interrupt: &Interrupt,
    +    mut deleted: impl FnMut(&BackupRow),
    +) -> PurgeOutcome {
    +    let mut outcome = PurgeOutcome::default();
    +    for row in rows {
    +        if let Some(reason) = claim.lost() {
    +            outcome.stopped = Some(Stop::Lost(reason));
    +            break;
    +        }
    +        if let Some(signal) = interrupt.signalled() {
    +            outcome.stopped = Some(Stop::Interrupted(signal));
    +            break;
    +        }
    +        match purge_row(storage, store, row, stale_after).await {
    +            Ok(()) => {
    +                deleted(row);
    +                outcome.deleted.push(row.backup_arn.clone());
    +            }
    +            Err(e) => outcome.skipped.push(format!("{}: {e}", row.backup_arn)),
    +        }
    +    }
    +    outcome
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use extenddb_storage::error::StorageError;
    +    use futures::future::BoxFuture;
    +    use tokio::time::Instant;
    +
    +    use super::*;
    +
    +    /// A claim whose heartbeats are timestamps on tokio's clock, with the age
    +    /// rule every backend's `check_maintenance_claim` applies.
    +    #[derive(Default)]
    +    struct FakeClaim {
    +        beats: Mutex>,
    +        lost: Mutex,
    +        released: Mutex,
    +    }
    +
    +    impl FakeClaim {
    +        /// `check_maintenance_claim(stale)`: held while the last heartbeat is
    +        /// within `stale`.
    +        fn held(&self, stale: Duration) -> bool {
    +            self.beats
    +                .lock()
    +                .unwrap()
    +                .last()
    +                .is_some_and(|at| at.elapsed() < stale)
    +        }
    +        fn beats(&self) -> usize {
    +            self.beats.lock().unwrap().len()
    +        }
    +    }
    +
    +    impl MaintenanceClaim for FakeClaim {
    +        fn generation(&self) -> u64 {
    +            7
    +        }
    +        fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +            let lost = *self.lost.lock().unwrap();
    +            if !lost {
    +                self.beats.lock().unwrap().push(Instant::now());
    +            }
    +            Box::pin(async move {
    +                Ok(if lost {
    +                    Heartbeat::Lost
    +                } else {
    +                    Heartbeat::Confirmed
    +                })
    +            })
    +        }
    +        fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +            *self.released.lock().unwrap() = true;
    +            Box::pin(async { Ok(()) })
    +        }
    +    }
    +
    +    /// The claim object handed to `HeldClaim`, forwarding to the test's own.
    +    struct ForwardingClaim(Arc);
    +
    +    impl MaintenanceClaim for ForwardingClaim {
    +        fn generation(&self) -> u64 {
    +            self.0.generation()
    +        }
    +        fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +            self.0.heartbeat()
    +        }
    +        fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +            self.0.release()
    +        }
    +    }
    +
    +    fn held_over(fake: &Arc, interval: Duration) -> HeldClaim {
    +        fake.beats.lock().unwrap().push(Instant::now());
    +        HeldClaim::start(Box::new(ForwardingClaim(Arc::clone(fake))), interval)
    +    }
    +
    +    /// On a paused clock (the timer and the samples share it, so a stall of
    +    /// the test process cannot read as a missed heartbeat): a held claim
    +    /// stays "held" to the jobs' check through a run of thirty stale
    +    /// windows with no heartbeat from the holder's own loop, a lost
    +    /// heartbeat is reported, and the release stops the timer.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_heartbeat_timer_keeps_the_claim_held_through_a_long_run() {
    +        let fake = Arc::new(FakeClaim::default());
    +        let stale = Duration::from_secs(3);
    +        let held = held_over(&fake, Duration::from_secs(1));
    +        assert_eq!(held.generation(), 7);
    +        let started = Instant::now();
    +        let mut samples = 0;
    +        while started.elapsed() < Duration::from_secs(90) {
    +            assert!(
    +                fake.held(stale),
    +                "the claim read as free {:?} after the take",
    +                started.elapsed()
    +            );
    +            assert_eq!(held.lost(), None);
    +            samples += 1;
    +            tokio::time::sleep(Duration::from_millis(500)).await;
    +        }
    +        assert!(samples >= 150, "{samples}");
    +        assert!(
    +            fake.beats() >= 85,
    +            "the timer heartbeated: {}",
    +            fake.beats()
    +        );
    +        // Taken over: the next heartbeat finds the claim lost.
    +        *fake.lost.lock().unwrap() = true;
    +        tokio::time::sleep(Duration::from_millis(2_500)).await;
    +        assert!(
    +            held.lost().is_some_and(|reason| reason.contains("lost")),
    +            "{:?}",
    +            held.lost()
    +        );
    +        let beats = fake.beats();
    +        held.release().await.unwrap();
    +        assert!(*fake.released.lock().unwrap());
    +        tokio::time::sleep(Duration::from_secs(5)).await;
    +        assert_eq!(fake.beats(), beats, "no heartbeat after the release");
    +    }
    +
    +    /// A handle dropped without `release` stops its timer: the claim gets no
    +    /// further heartbeat and expires by the stale rule, as a dead holder's
    +    /// does, rather than being kept alive by a detached task.
    +    #[tokio::test(start_paused = true)]
    +    async fn a_dropped_handle_stops_the_heartbeat() {
    +        let fake = Arc::new(FakeClaim::default());
    +        let held = held_over(&fake, Duration::from_secs(1));
    +        tokio::time::sleep(Duration::from_millis(3_500)).await;
    +        let before = fake.beats();
    +        assert!(before >= 3, "{before}");
    +        drop(held);
    +        tokio::time::sleep(Duration::from_secs(10)).await;
    +        assert_eq!(fake.beats(), before, "a dropped handle heartbeats no more");
    +        assert!(!*fake.released.lock().unwrap(), "a drop is not a release");
    +        assert!(
    +            !fake.held(Duration::from_secs(3)),
    +            "the claim expires by the stale rule"
    +        );
    +    }
    +
    +    /// A claim whose release never answers (a stalled database).
    +    struct StalledRelease;
    +
    +    impl MaintenanceClaim for StalledRelease {
    +        fn generation(&self) -> u64 {
    +            1
    +        }
    +        fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +            Box::pin(async { Ok(Heartbeat::Confirmed) })
    +        }
    +        fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +            Box::pin(std::future::pending())
    +        }
    +    }
    +
    +    /// `release` is bounded: a catalog that does not answer it ends the
    +    /// release at [`RELEASE_TIMEOUT`] with an error naming the bound, so a
    +    /// command that received a signal exits and leaves the claim to the
    +    /// stale rule, rather than holding on a stalled database.
    +    #[tokio::test(start_paused = true)]
    +    async fn a_release_the_catalog_does_not_answer_ends_at_its_bound() {
    +        let held = HeldClaim::start(Box::new(StalledRelease), Duration::from_secs(1));
    +        let started = Instant::now();
    +        let err = held.release().await.unwrap_err();
    +        assert_eq!(started.elapsed(), RELEASE_TIMEOUT);
    +        assert!(
    +            err.to_string()
    +                .contains("the release did not finish within 10 s"),
    +            "{err}"
    +        );
    +    }
    +}
    diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs
    index bf9904ca9..10c0068e6 100644
    --- a/crates/engine/src/backup/reconciler.rs
    +++ b/crates/engine/src/backup/reconciler.rs
    @@ -2,14 +2,26 @@
     // SPDX-License-Identifier: Apache-2.0
     
     //! The reconciler: the periodic worker that resolves catalog rows whose
    -//! owner is dead.
    +//! owner is dead, and keeps the configured store's health current.
     //!
     //! One cycle runs every `interval`. It first checks the catalog-level
     //! maintenance claim; while a maintenance command holds it the reconciler
    -//! runs only the duties that resolve the states the command waits on, which
    -//! today are every duty it has, so the check is what decides when the other
    -//! duties are added. The duties, over the records `list_dead_restore_jobs`
    -//! returns:
    +//! runs only the duties below, which resolve the states the command waits
    +//! on or finishes itself (the dead-owner duties, the store's check and the
    +//! DELETING rows); the duties a later release adds that delete or move a
    +//! row the command does not wait on (a FAILED row's prefix, tombstone
    +//! retention) go in the branch the flag skips.
    +//!
    +//! The claim is read through `check_maintenance_claim(stale_job_after)`,
    +//! except in a cycle a maintenance command runs itself while it holds the
    +//! claim (`backup purge` on SQLite, with a zero dead-owner window): that
    +//! caller's state says so ([`ReconcilerState::for_maintenance_holder`]) and
    +//! the cycle takes the held branch without reading the claim. Read through
    +//! the clock, a zero window would see the command's own claim held only
    +//! when its heartbeat and the check land in the same instant, so the
    +//! branch would be decided by scheduler timing.
    +//!
    +//! The duties, over the records `list_dead_restore_jobs` returns:
     //!
     //! - A RUNNING restore job record whose `heartbeat_at` is older than
     //!   `stale_job_after` (on the catalog's clock) has a dead owner. One of kind
    @@ -34,6 +46,28 @@
     //!   `delete`, which takes its items with it, unless the backend removed it
     //!   in the call that reported it (`AbandonedLegacyBackup::removed`), when
     //!   the reconciler only logs it.
    +//! - With a store configured: the store's check runs again
    +//!   ([`super::check_configured_store`]: `validate` while it has not passed,
    +//!   then the identity check and manifest probe) and sets the store's
    +//!   health, so a store whose checks run after the listener is up, or that
    +//!   failed at startup or later, is healthy once a check passes, and one
    +//!   that fails is refused from the next cycle on. The check runs after the
    +//!   duties above and under the store's bound
    +//!   ([`super::ConfiguredStore::check_timeout`]: 5 s for a filesystem root,
    +//!   30 s for a bucket), so a store that has stopped answering costs each
    +//!   cycle that bound and nothing more: the dead-owner duties run every
    +//!   `interval` plus the bound, the check is recorded as a transport
    +//!   failure, and a filesystem check leaves one blocking thread parked on
    +//!   the mount until it answers, after which no check starts until that
    +//!   call returns (the runtime module's in-flight rule), so a mount that
    +//!   stays stalled parks one thread and not one per cycle. The check is
    +//!   also selected against the shutdown token, so a cycle that is checking
    +//!   when the server stops returns at once and the drain finishes.
    +//!   While the store is healthy, each DELETING row of the configured store
    +//!   (a DeleteBackup whose deletion did not finish) is finished
    +//!   (`finish_backup_delete` in the engine's backup module): the manifest, then the prefix, then
    +//!   the row to DELETED. While it is unhealthy the reconciler acts on no row
    +//!   of the store.
     //!
     //! A cycle that fails logs and waits for the next; each cycle records the
     //! worker metrics the other workers record (`WorkerLastSuccess` and
    @@ -44,10 +78,16 @@ use std::collections::HashSet;
     use std::sync::Arc;
     use std::time::Duration;
     
    +use extenddb_core::backup_format::BackupArn;
     use extenddb_core::metrics::{MetricsCollector, QuerySource};
    -use extenddb_storage::backup_catalog::{JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey};
    +use extenddb_storage::backup_catalog::{
    +    BackupStatus, JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey,
    +};
     use extenddb_storage::error::StorageError;
     use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader, sleep_or_shutdown};
    +use futures::TryStreamExt;
    +
    +use super::BackupRuntime;
     
     /// The two durations a reconciler reads from `[backup]`.
     #[derive(Debug, Clone, Copy, PartialEq, Eq)]
    @@ -82,25 +122,65 @@ pub struct CycleReport {
         pub manifest_restores_logged: Vec,
         /// In-catalog CREATING backup rows removed, by ARN.
         pub legacy_backups_removed: Vec,
    +    /// The configured store's health after this cycle's check; `None` with
    +    /// no store configured.
    +    pub store_healthy: Option,
    +    /// DELETING rows of the configured store moved to DELETED, by ARN.
    +    pub deletions_finished: Vec,
     }
     
     /// What a reconciler carries from one cycle to the next: the manifest
     /// records it has already logged, so a record it cannot act on is reported
    -/// once rather than every cycle. The caller of [`run`] owns it and hands the
    +/// once rather than every cycle (the caller of [`run`] owns it and hands the
     /// same state to every worker it starts, so the set outlives a worker
    -/// restart (`serve` restarts its workers each time it takes the serving
    -/// registration again).
    +/// restart: `serve` restarts its workers each time it takes the serving
    +/// registration again), the worker's shutdown token, which the
    +/// store's check is selected against, and, for a maintenance command's own
    +/// cycle, that its caller holds the maintenance claim.
     #[derive(Debug, Default)]
     pub struct ReconcilerState {
         logged_manifest_restores: HashSet,
    +    shutdown: Option,
    +    /// `Some(true)` when the caller holds the maintenance claim: the cycle
    +    /// takes the held branch and does not read the claim. `None` reads it
    +    /// through `check_maintenance_claim`.
    +    maintenance_held: Option,
    +}
    +
    +impl ReconcilerState {
    +    /// A state whose cycles stop waiting on the store's check when `shutdown`
    +    /// fires (the health is then left as the last check left it). A default
    +    /// state, which commands and tests use, waits the check's bound.
    +    #[must_use]
    +    pub fn with_shutdown(shutdown: CancellationToken) -> Self {
    +        Self {
    +            shutdown: Some(shutdown),
    +            ..Self::default()
    +        }
    +    }
    +
    +    /// The state of a cycle that a maintenance command runs while it holds
    +    /// the maintenance claim: the cycle takes the held branch without
    +    /// reading the claim, whatever window it is given.
    +    #[must_use]
    +    pub fn for_maintenance_holder() -> Self {
    +        Self {
    +            maintenance_held: Some(true),
    +            ..Self::default()
    +        }
    +    }
     }
     
     /// Run cycles every `settings.interval` until `shutdown` fires. One cycle
     /// runs at once at startup, so a restart resolves what the stopped process
    -/// left without waiting an interval. `state` is locked for each cycle.
    +/// left without waiting an interval. `backend` names the storage backend in
    +/// an identity file this worker has to rewrite. `state` is locked for each
    +/// cycle.
     pub async fn run(
         storage: Arc,
    +    runtime: Arc,
         settings: ReconcilerSettings,
    +    backend: String,
         metrics: Arc,
         state: Arc>,
         shutdown: CancellationToken,
    @@ -111,7 +191,17 @@ pub async fn run(
             let started = std::time::Instant::now();
             let outcome = {
                 let mut state = state.lock().await;
    -            cycle(&*storage, settings.stale_job_after, &mut state).await
    +            // The state outlives this worker (it is `serve`'s); the token is
    +            // this worker's, so it is set here, each cycle.
    +            state.shutdown = Some(shutdown.clone());
    +            cycle(
    +                &*storage,
    +                &runtime,
    +                settings.stale_job_after,
    +                &backend,
    +                &mut state,
    +            )
    +            .await
             };
             match outcome {
                 Ok(report) => {
    @@ -121,12 +211,14 @@ pub async fn run(
                     if !report.legacy_restores_removed.is_empty()
                         || !report.failed_targets_removed.is_empty()
                         || !report.legacy_backups_removed.is_empty()
    +                    || !report.deletions_finished.is_empty()
                     {
                         tracing::info!(
                             "backup reconciler: removed {} abandoned restore target(s) and {} \
    -                         abandoned in-catalog backup(s)",
    +                         abandoned in-catalog backup(s), finished {} deletion(s)",
                             report.legacy_restores_removed.len() + report.failed_targets_removed.len(),
    -                        report.legacy_backups_removed.len()
    +                        report.legacy_backups_removed.len(),
    +                        report.deletions_finished.len()
                         );
                     }
                 }
    @@ -141,7 +233,10 @@ pub async fn run(
         }
     }
     
    -/// One cycle: the maintenance check, then the duties.
    +/// One cycle: the maintenance check (or the caller's word that it holds
    +/// the claim, from `state`), then the duties. `stale_job_after` is the
    +/// dead-owner window; it is also the claim's stale window when the claim is
    +/// read.
     ///
     /// # Errors
     ///
    @@ -149,20 +244,26 @@ pub async fn run(
     /// logged and the cycle goes on to the next row.
     pub async fn cycle(
         storage: &S,
    +    runtime: &BackupRuntime,
         stale_job_after: Duration,
    +    backend: &str,
         state: &mut ReconcilerState,
     ) -> Result
     where
         S: BackupCatalog + LegacyBackupReader + ?Sized,
     {
    +    let maintenance_held = match state.maintenance_held {
    +        Some(held) => held,
    +        None => storage.check_maintenance_claim(stale_job_after).await?,
    +    };
         let mut report = CycleReport {
    -        maintenance_held: storage.check_maintenance_claim().await?,
    +        maintenance_held,
             ..CycleReport::default()
         };
         if report.maintenance_held {
             tracing::info!(
                 "backup reconciler: a maintenance claim is held; running only the duties that \
    -             resolve rows whose owner is dead"
    +             resolve rows whose owner is dead, the store's check and the DELETING rows"
             );
         }
     
    @@ -246,9 +347,118 @@ where
                 ),
             }
         }
    +
    +    if let Some(store) = runtime.configured_store() {
    +        let check = super::check_configured_store(store, storage, backend);
    +        let outcome = match &state.shutdown {
    +            Some(shutdown) => tokio::select! {
    +                outcome = check => Some(outcome),
    +                () = shutdown.cancelled() => None,
    +            },
    +            None => Some(check.await),
    +        };
    +        match outcome {
    +            Some(outcome) => {
    +                // The defaulted SQLite root's rule, applied to the first
    +                // check when the startup checks did not finish and left the
    +                // store checking: a failure with no rows for the root is the
    +                // warning the startup check would have given, and the store
    +                // is left unconfigured rather than unhealthy.
    +                if store.is_defaulted()
    +                    && store.health() == super::StoreHealth::Checking
    +                    && let Err(problem) = &outcome
    +                    && !matches!(problem, super::StoreProblem::Catalog(_))
    +                    && storage
    +                        .read_store_ids(store.kind, &store.root)
    +                        .await
    +                        .is_ok_and(|ids| ids.is_empty())
    +                {
    +                    tracing::warn!(
    +                        "backup store: the default root {} failed its first check ({problem}); \
    +                         the catalog has no backup rows for it, so the store is unconfigured and \
    +                         CreateBackup writes in-catalog backups. Fix the root (owned by this \
    +                         user, mode 0700, under directories others cannot write to, its \
    +                         extenddb-store.json naming this catalog), configure [backup.store] to \
    +                         choose another, or set kind = \"disabled\" to silence this",
    +                        store.root
    +                    );
    +                    store.set_aside();
    +                    return Ok(report);
    +                }
    +                store.record_check(outcome);
    +            }
    +            None => {
    +                tracing::info!(
    +                    "backup store ({}) check was stopped by shutdown; its health is left as the \
    +                     last check left it",
    +                    store.kind
    +                );
    +                report.store_healthy = Some(store.is_healthy());
    +                return Ok(report);
    +            }
    +        }
    +        report.store_healthy = Some(store.is_healthy());
    +        if let Some(identity) = store.identity().filter(|_| store.is_healthy()) {
    +            report.deletions_finished = finish_deleting_rows(storage, store, &identity).await?;
    +        }
    +    }
    +
    +    if !report.maintenance_held {
    +        // The duties a maintenance command does not wait on and must not
    +        // race: a later release's deletion of a FAILED row's prefix and its
    +        // removal of tombstones past their retention run here.
    +    }
         Ok(report)
     }
     
    +/// The duty for DELETING rows of the configured store: finish each deletion.
    +/// Returns the ARNs moved to DELETED; a row whose deletion fails is logged
    +/// and left for the next cycle.
    +async fn finish_deleting_rows(
    +    storage: &S,
    +    store: &super::ConfiguredStore,
    +    identity: &extenddb_storage::backup_catalog::StoreIdentity,
    +) -> Result, StorageError> {
    +    let rows: Vec<_> = storage
    +        .list_backup_rows(Some(BackupStatus::Deleting), identity)
    +        .try_collect()
    +        .await?;
    +    let mut finished = Vec::new();
    +    for row in rows {
    +        let arn = match BackupArn::parse(&row.backup_arn) {
    +            Ok(arn) => arn,
    +            Err(e) => {
    +                tracing::error!(
    +                    "DELETING row {} carries an ARN this build cannot parse; left as it is: {e}",
    +                    row.backup_arn
    +                );
    +                continue;
    +            }
    +        };
    +        match super::finish_backup_delete(storage, &store.calls("deletion"), &arn).await {
    +            Ok(true) => {
    +                tracing::info!(
    +                    "finished the deletion of backup {} (account {})",
    +                    row.backup_arn,
    +                    row.account_id
    +                );
    +                finished.push(row.backup_arn);
    +            }
    +            Ok(false) => tracing::info!(
    +                "backup {} was no longer DELETING when its objects were gone; left to its owner",
    +                row.backup_arn
    +            ),
    +            Err(e) => tracing::error!(
    +                "could not finish the deletion of backup {} (account {}); the next cycle \
    +                 retries: {e}",
    +                row.backup_arn,
    +                row.account_id
    +            ),
    +        }
    +    }
    +    Ok(finished)
    +}
    +
     /// Mark a dead legacy restore FAILED and remove its target. The FAILED write
     /// is conditioned on the `owner_generation` and `heartbeat_at` the listing
     /// read (design 4(e)), so a heartbeat that landed after the listing, which
    diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs
    index 7a2794f80..e57ada84f 100644
    --- a/crates/engine/src/backup/runtime.rs
    +++ b/crates/engine/src/backup/runtime.rs
    @@ -3,16 +3,62 @@
     
     //! The per-process state of the backup operations: the job slots every
     //! backup and restore job holds one of, and the store this instance is
    -//! configured to write.
    +//! configured to write, with its health.
    +//!
    +//! The store's health is a state the checks set ([`super::store_identity`]
    +//! runs them; [`check_configured_store`] is the one the reconciler calls each
    +//! cycle): `Checking` until the first check of a store whose checks run after
    +//! the listener is up (an S3 store; the server binds and serves first, so an
    +//! endpoint that stalls never holds the data plane), `Healthy`, or
    +//! `Unhealthy` with the problem. A transition between the three is logged
    +//! once; while checking or unhealthy, the operations that touch the store
    +//! refuse with a `ValidationException` that names the store's kind and not
    +//! its root ([`BackupRuntime::require_healthy_store`]), `/health` reports the
    +//! `backup_store` component `checking` or `unhealthy`
    +//! ([`BackupRuntime::store_health`]), and the reconciler acts on no row of
    +//! the store. A check that cannot read the catalog changes nothing: the
    +//! condition is the catalog's, not the store's, and the next cycle reads
    +//! again.
    +//!
    +//! A store call that does not finish within the store's bound is abandoned,
    +//! and the check with it; the catalog's reads run beside the calls and not
    +//! under the bound, so a slow catalog never marks the store unhealthy. An
    +//! S3 store's stalled request ends with the dropped future, before the SDK
    +//! retries it, and the next cycle sends a fresh one. A filesystem store's
    +//! stalled call is a blocking thread parked in a filesystem call until the
    +//! mount answers, which dropping the future cannot end, so a filesystem
    +//! root's calls run as tasks of their own that the abandoned call does not
    +//! take with it, and the store remembers that an abandoned call is in flight
    +//! ([`ConfiguredStore::check_call_in_flight`]): no call starts while one is
    +//! (the next [`check_configured_store`] records the same `unreachable` the
    +//! abandoned one did), so a mount that stays stalled parks one thread, not
    +//! one per cycle. A filesystem root's calls also run one at a time: a call
    +//! that finds another running (a DeleteBackup beside a check, two
    +//! DeleteBackups together) waits for it under its own bound, so calls that
    +//! start before the first reaches its bound park no thread of their own
    +//! and answer at the bound with the same transport failure. The deletions
    +//! DeleteBackup and the reconciler run against the store go through the
    +//! same adapter ([`BoundedCalls`]), under the same bound and rules. The
    +//! thread costs nothing at exit: the binary ends its runtime with a bounded
    +//! shutdown that detaches it.
     
    -use std::sync::Arc;
    +use std::sync::atomic::{AtomicBool, Ordering};
    +use std::sync::{Arc, RwLock};
     use std::time::Duration;
     
    +use extenddb_backup_store::{BackupStore, ByteStream, ObjectMeta, StoreError};
     use extenddb_core::error::DynamoDbError;
    -use extenddb_storage::JobPermit;
    -use extenddb_storage::backup_catalog::{JobBound, StoreIdentity, limit_exceeded_message};
    +use extenddb_storage::backup_catalog::{
    +    JobBound, StoreIdentity, StoreKind, limit_exceeded_message,
    +};
    +use extenddb_storage::{BackupCatalog, JobPermit};
    +use futures::StreamExt;
    +use futures::future::BoxFuture;
    +use futures::stream::BoxStream;
     use tokio::sync::Semaphore;
     
    +use super::store_identity::{self, StoreProblem};
    +
     /// The configuration key of the job bound the runtime enforces, as the
     /// refusal names it.
     const MAX_CONCURRENT_JOBS_KEY: &str = "[backup] max_concurrent_jobs";
    @@ -25,9 +71,28 @@ const MAX_CONCURRENT_JOBS_KEY: &str = "[backup] max_concurrent_jobs";
     /// over the bound is served in turn rather than refused.
     pub const PERMIT_WAIT: Duration = Duration::from_secs(30);
     
    +/// How long a filesystem store's check may run, at startup and in a
    +/// reconciler cycle, before the store is treated as not answering: a local
    +/// disk answers in milliseconds, and a network mount that takes longer is
    +/// one that has stopped answering.
    +pub const LOCAL_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
    +
    +/// How long an S3 store's check may run in a reconciler cycle. The check
    +/// sends a few small requests (`validate`'s listing, the identity file's
    +/// `GetObject`, the manifest's `HeadObject`), each answered in well under a
    +/// second by a store that answers at all; the bound sits under the SDK's
    +/// per-attempt timeout ([`extenddb_backup_store::S3_ATTEMPT_TIMEOUT`], 60 s)
    +/// so a timed-out check is dropped before the SDK starts a retry of its
    +/// stalled request, and no retry outlives the check. A stalled endpoint thus
    +/// costs a cycle this bound, not the SDK's whole retry budget.
    +pub const S3_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
    +const _: () = assert!(
    +    S3_CHECK_TIMEOUT.as_secs() < extenddb_backup_store::S3_ATTEMPT_TIMEOUT.as_secs(),
    +    "the S3 check bound must fire before the SDK's attempt timeout"
    +);
    +
     /// One per server process, shared by every request through the operation
     /// context.
    -#[derive(Debug)]
     pub struct BackupRuntime {
         slots: Arc,
         max_concurrent_jobs: usize,
    @@ -37,7 +102,682 @@ pub struct BackupRuntime {
         /// The store this instance writes new backups to and deletes from;
         /// `None` while no store is configured, when every catalog row is a row
         /// of another store.
    -    configured_store: Option,
    +    configured_store: Option,
    +    /// Whether CreateBackup takes the in-catalog path whatever the store
    +    /// configuration (a backend that cannot pin a table snapshot); such a
    +    /// create is not refused by an unhealthy store.
    +    legacy_create_only: bool,
    +    /// `[backup] stale_job_after_secs`: the window a job's claim check reads
    +    /// the maintenance claim with.
    +    stale_job_after: std::time::Duration,
    +}
    +
    +impl std::fmt::Debug for BackupRuntime {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.debug_struct("BackupRuntime")
    +            .field("max_concurrent_jobs", &self.max_concurrent_jobs)
    +            .field(
    +                "configured_store",
    +                &self
    +                    .configured_store
    +                    .as_ref()
    +                    .map(ConfiguredStore::identity),
    +            )
    +            .field("legacy_create_only", &self.legacy_create_only)
    +            .finish_non_exhaustive()
    +    }
    +}
    +
    +/// The store this instance is configured with: its kind and root (the
    +/// configuration's), its id once the identity check has read it, the handle,
    +/// the spool for restores from it, and its health.
    +///
    +/// The id is `None` until a check passes: a store whose startup check failed
    +/// (a transport failure, say) has an unknown id, and a row of its kind and
    +/// root is still a row of this store for the refusals, so those rows answer
    +/// "the store is unhealthy" rather than "another store".
    +pub struct ConfiguredStore {
    +    /// `filesystem` or `s3`.
    +    pub kind: StoreKind,
    +    /// The root label rows record (`store_root`).
    +    pub root: String,
    +    /// The opened store.
    +    pub store: Arc,
    +    /// The restore spool for an S3 store: the directory and the
    +    /// `:` label a restore job record names.
    +    pub spool: Option,
    +    store_id: RwLock>,
    +    health: RwLock,
    +    /// Whether the store's `validate` has passed; the checks run it first
    +    /// until it has.
    +    validated: AtomicBool,
    +    /// Whether a call on a filesystem root abandoned at its bound is still
    +    /// running (module doc): set when the call is abandoned, cleared by the
    +    /// call's own task when the mount answers. Never set on an S3 store.
    +    check_call_in_flight: Arc,
    +    /// The one slot a filesystem root's calls run in (module doc): a call
    +    /// waits for it under its bound, held by the call's task until the
    +    /// store answers, so callers that start before the first call reaches
    +    /// its bound park no thread of their own. Unused on an S3 store.
    +    call_slot: Arc,
    +    /// Whether the store is the defaulted SQLite root beside the database
    +    /// file, whose first failed check with no rows for it is a warning that
    +    /// leaves the store unconfigured ([`ConfiguredStore::set_aside`]) rather
    +    /// than an unhealthy configured store.
    +    defaulted: bool,
    +    /// Set once a defaulted root's first check failed with no rows for it:
    +    /// the runtime then answers as a server with no store configured.
    +    set_aside: AtomicBool,
    +}
    +
    +/// The restore spool of an S3 store.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct Spool {
    +    pub path: std::path::PathBuf,
    +    /// `:`, the host whose spool holds a restore's
    +    /// downloaded files.
    +    pub spool_host: String,
    +}
    +
    +/// The store's health as the last check left it.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum StoreHealth {
    +    /// No check has finished yet: the store's checks run after the listener
    +    /// is up and the first has not returned.
    +    Checking,
    +    Healthy,
    +    Unhealthy(StoreProblem),
    +}
    +
    +impl StoreHealth {
    +    /// The one or two words the API refusal and `/health` name: the
    +    /// problem's category ([`StoreProblem::category`]), or `check pending`.
    +    #[must_use]
    +    pub fn category(&self) -> Option<&'static str> {
    +        match self {
    +            Self::Healthy => None,
    +            Self::Checking => Some("check pending"),
    +            Self::Unhealthy(problem) => Some(problem.category()),
    +        }
    +    }
    +}
    +
    +/// What `/health` reports for the `backup_store` component.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub struct StoreHealthReport {
    +    /// `filesystem` or `s3`.
    +    pub kind: &'static str,
    +    /// `healthy`, `unhealthy` or `checking`.
    +    pub status: &'static str,
    +    /// The problem's category and nothing else: `/health` is unauthenticated,
    +    /// and the ids, ARNs and catalog id the operator's log line carries stay
    +    /// out of it. `None` when healthy or checking.
    +    pub reason: Option<&'static str>,
    +}
    +
    +impl ConfiguredStore {
    +    /// A store of `kind` at `root` whose startup check (`validate`, then
    +    /// `establish`) ended in `outcome`: the identity it settled (healthy), or
    +    /// the problem that makes the store unhealthy from the start (a transport
    +    /// failure, or a failed identity check or probe: section 7's rule that a
    +    /// store problem never stops the data plane). A problem leaves the id
    +    /// unknown until a later check passes. Whether `validate` passed is read
    +    /// from the outcome: a store whose startup failed as a transport failure,
    +    /// a denial or a failed validation (the three problems `validate` can end
    +    /// in; [`StoreProblem::from`]) runs `validate` again at its next check,
    +    /// so a probe write that failed is not reported healthy by a later check
    +    /// that only re-read the identity file.
    +    #[must_use]
    +    pub fn new(
    +        kind: StoreKind,
    +        root: String,
    +        store: Arc,
    +        spool: Option,
    +        outcome: Result,
    +    ) -> Self {
    +        let validated = !matches!(
    +            outcome,
    +            Err(StoreProblem::Transport(_) | StoreProblem::Denied(_) | StoreProblem::Validate(_))
    +        );
    +        let (store_id, health) = match outcome {
    +            Ok(identity) => (Some(identity.store_id), StoreHealth::Healthy),
    +            Err(problem) => (None, StoreHealth::Unhealthy(problem)),
    +        };
    +        Self {
    +            kind,
    +            root,
    +            store,
    +            spool,
    +            store_id: RwLock::new(store_id),
    +            health: RwLock::new(health),
    +            validated: AtomicBool::new(validated),
    +            check_call_in_flight: Arc::new(AtomicBool::new(false)),
    +            call_slot: Arc::new(Semaphore::new(1)),
    +            defaulted: false,
    +            set_aside: AtomicBool::new(false),
    +        }
    +    }
    +
    +    /// A store of `kind` at `root` that `open` accepted and whose checks
    +    /// (`validate`, then the identity check) have not run: the first
    +    /// [`check_configured_store`] runs both and records the result. Until
    +    /// then the store is [`StoreHealth::Checking`].
    +    #[must_use]
    +    pub fn checking(
    +        kind: StoreKind,
    +        root: String,
    +        store: Arc,
    +        spool: Option,
    +    ) -> Self {
    +        Self {
    +            kind,
    +            root,
    +            store,
    +            spool,
    +            store_id: RwLock::new(None),
    +            health: RwLock::new(StoreHealth::Checking),
    +            validated: AtomicBool::new(false),
    +            check_call_in_flight: Arc::new(AtomicBool::new(false)),
    +            call_slot: Arc::new(Semaphore::new(1)),
    +            defaulted: false,
    +            set_aside: AtomicBool::new(false),
    +        }
    +    }
    +
    +    /// The same store marked as the defaulted SQLite root (or not): the
    +    /// first check's failure with no rows for the root then sets it aside
    +    /// ([`ConfiguredStore::set_aside`]), as the startup check's would have.
    +    #[must_use]
    +    pub fn defaulted(mut self, defaulted: bool) -> Self {
    +        self.defaulted = defaulted;
    +        self
    +    }
    +
    +    /// Whether this is the defaulted SQLite root.
    +    #[must_use]
    +    pub fn is_defaulted(&self) -> bool {
    +        self.defaulted
    +    }
    +
    +    /// Leave the store unconfigured from now on: the runtime answers as a
    +    /// server with no store (`configured_store` is `None`, `/health` has no
    +    /// component, CreateBackup logs the no-store warning). The reconciler
    +    /// applies this to a defaulted root whose first check failed while the
    +    /// catalog has no rows for it, as the server's startup applies it to a
    +    /// check that failed before the listener.
    +    pub fn set_aside(&self) {
    +        self.set_aside.store(true, Ordering::Release);
    +    }
    +
    +    /// Whether the store was set aside.
    +    #[must_use]
    +    pub fn is_set_aside(&self) -> bool {
    +        self.set_aside.load(Ordering::Acquire)
    +    }
    +
    +    /// The store's identity, once a check has read its id.
    +    #[must_use]
    +    pub fn identity(&self) -> Option {
    +        let id = self
    +            .store_id
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .clone()?;
    +        Some(StoreIdentity {
    +            store_kind: self.kind,
    +            store_root: self.root.clone(),
    +            store_id: id,
    +        })
    +    }
    +
    +    /// Whether a row's store is this one: the kind and the root, and the id
    +    /// when this store's is known.
    +    #[must_use]
    +    pub fn is_store_of(&self, row_store: &StoreIdentity) -> bool {
    +        if row_store.store_kind != self.kind || row_store.store_root != self.root {
    +            return false;
    +        }
    +        match self
    +            .store_id
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .as_deref()
    +        {
    +            Some(id) => id == row_store.store_id,
    +            None => true,
    +        }
    +    }
    +
    +    /// The health as the last check left it.
    +    #[must_use]
    +    pub fn health(&self) -> StoreHealth {
    +        self.health
    +            .read()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner)
    +            .clone()
    +    }
    +
    +    /// Whether the last check passed.
    +    #[must_use]
    +    pub fn is_healthy(&self) -> bool {
    +        matches!(self.health(), StoreHealth::Healthy)
    +    }
    +
    +    /// Record the outcome of a check: the identity it settled (the id is
    +    /// kept), or the problem. A transition is logged once: to unhealthy at
    +    /// `error` with the problem's detail, to healthy at `info`; a check
    +    /// whose outcome is in the last one's category logs nothing, so a store
    +    /// that stays down costs one line however its error text varies between
    +    /// cycles. A check that could not read the catalog
    +    /// ([`StoreProblem::Catalog`]) leaves the health as it was and logs the
    +    /// read failure: the condition is not the store's. Returns whether this
    +    /// check changed the health's category (and so logged a transition).
    +    pub fn record_check(&self, outcome: Result) -> bool {
    +        let next = match outcome {
    +            Ok(identity) => {
    +                *self
    +                    .store_id
    +                    .write()
    +                    .unwrap_or_else(std::sync::PoisonError::into_inner) = Some(identity.store_id);
    +                StoreHealth::Healthy
    +            }
    +            Err(StoreProblem::Catalog(e)) => {
    +                tracing::warn!(
    +                    "backup store ({}) check could not read the catalog ({e}); the store's \
    +                     health is left as it was and the next cycle reads again",
    +                    self.kind
    +                );
    +                return false;
    +            }
    +            Err(problem) => StoreHealth::Unhealthy(problem),
    +        };
    +        let mut health = self
    +            .health
    +            .write()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner);
    +        if health.category() == next.category() {
    +            *health = next;
    +            return false;
    +        }
    +        match (&*health, &next) {
    +            (StoreHealth::Checking, StoreHealth::Healthy) => tracing::info!(
    +                "backup store ({}) {} passed its check (store id {}); it serves restores and \
    +                 deletes of the backups it holds",
    +                self.kind,
    +                self.root,
    +                self.identity().map(|i| i.store_id).unwrap_or_default()
    +            ),
    +            (_, StoreHealth::Healthy) => {
    +                tracing::info!("backup store ({}) is healthy again", self.kind);
    +            }
    +            (_, StoreHealth::Unhealthy(problem)) => tracing::error!(
    +                "backup store ({}) {} is unhealthy: {problem}; restore and delete of a backup \
    +                 in the store are refused until the check passes; the check runs again each \
    +                 reconciler cycle",
    +                self.kind,
    +                self.root
    +            ),
    +            (_, StoreHealth::Checking) => {}
    +        }
    +        *health = next;
    +        true
    +    }
    +
    +    /// Whether the store's `validate` has passed.
    +    #[must_use]
    +    pub fn is_validated(&self) -> bool {
    +        self.validated.load(Ordering::Acquire)
    +    }
    +
    +    /// How long one check of this store may run before it is abandoned
    +    /// ([`LOCAL_CHECK_TIMEOUT`] for a filesystem root, [`S3_CHECK_TIMEOUT`]
    +    /// for a bucket).
    +    #[must_use]
    +    pub fn check_timeout(&self) -> std::time::Duration {
    +        match self.kind {
    +            StoreKind::Filesystem => LOCAL_CHECK_TIMEOUT,
    +            StoreKind::S3 => S3_CHECK_TIMEOUT,
    +        }
    +    }
    +
    +    /// Whether a call abandoned at its bound has not returned (module doc).
    +    #[must_use]
    +    pub fn check_call_in_flight(&self) -> bool {
    +        self.check_call_in_flight.load(Ordering::Acquire)
    +    }
    +
    +    /// The store as a check, a command's identity read or a deletion calls
    +    /// it: every store call under [`ConfiguredStore::check_timeout`] and the
    +    /// in-flight rule (module doc), and nothing else under the bound, so a
    +    /// slow catalog read beside the calls does not count against it. `what`
    +    /// names the caller in the transport failure a call that does not finish
    +    /// is recorded as ("the check did not finish within 5 s").
    +    pub(crate) fn calls(&self, what: &'static str) -> BoundedCalls<'_> {
    +        BoundedCalls {
    +            store: &self.store,
    +            in_flight: &self.check_call_in_flight,
    +            slot: &self.call_slot,
    +            bound: self.check_timeout(),
    +            what,
    +            // A filesystem root's calls are blocking threads, which dropping
    +            // the future cannot end; an S3 store's are the SDK's requests,
    +            // which it does end, before the SDK retries them.
    +            park: self.kind == StoreKind::Filesystem,
    +            timed_out: AtomicBool::new(false),
    +        }
    +    }
    +}
    +
    +/// The check of the configured store a reconciler cycle, and a command that
    +/// needs the store's identity, run and record through
    +/// [`ConfiguredStore::record_check`]: `validate` first while it has not
    +/// passed (its failure is the problem [`StoreProblem::from`] makes of the
    +/// store's error, a transport failure [`StoreProblem::Transport`] with the
    +/// cause chain, a denial [`StoreProblem::Denied`], anything else
    +/// [`StoreProblem::Validate`]; each is retried next cycle), then the
    +/// identity check, which is [`store_identity::recheck`] for a store whose
    +/// id is known and [`store_identity::establish`] for one whose checks have
    +/// not passed yet (it settles the id when it passes). `backend` names the
    +/// storage backend in an identity file the check has to write.
    +///
    +/// The store's calls run under [`ConfiguredStore::check_timeout`]
    +/// ([`BoundedCalls`]): a store that does not answer a call within it (a
    +/// stalled endpoint, a mount that has stopped answering) ends the check as
    +/// [`StoreProblem::Transport`] naming the bound, and the caller's cycle goes
    +/// on. The catalog's reads run beside the calls and not under the bound, so
    +/// a slow catalog costs the check time and never marks the store unhealthy.
    +/// An S3 store's abandoned request is dropped with the call's future. A
    +/// filesystem root's calls run as tasks of their own, which the abandoned
    +/// call does not take with it: the call's synchronous work runs on the
    +/// blocking pool, where the thread stays parked until the mount answers,
    +/// and the task is what clears the store's in-flight mark when it does.
    +/// While that mark is set no call starts: the store is recorded
    +/// `unreachable` again, with the reason, and nothing new is parked.
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy
    +/// ([`store_identity`]'s module doc).
    +pub async fn check_configured_store(
    +    store: &ConfiguredStore,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    if store.check_call_in_flight() {
    +        tracing::info!(
    +            "backup store ({}) check was not started: a call of the check abandoned at an \
    +             earlier cycle has not returned (the store is still not answering); the next cycle \
    +             checks again",
    +            store.kind
    +        );
    +        return Err(StoreProblem::Transport(
    +            "a call of an earlier check has not returned".to_owned(),
    +        ));
    +    }
    +    let via = store.calls("check");
    +    let outcome = check_through(store, &via, catalog, backend).await;
    +    if via.timed_out() {
    +        tracing::info!(
    +            "backup store ({}) check did not finish within {} s and was abandoned; the \
    +             cycle's other duties ran, and the next cycle checks again",
    +            store.kind,
    +            via.bound.as_secs()
    +        );
    +        return Err(via.did_not_finish_problem());
    +    }
    +    outcome
    +}
    +
    +/// [`check_configured_store`]'s steps, calling the store through `via`.
    +async fn check_through(
    +    store: &ConfiguredStore,
    +    via: &dyn BackupStore,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    if !store.is_validated() {
    +        match via.validate().await {
    +            Ok(()) => store.validated.store(true, Ordering::Release),
    +            Err(e) => return Err(StoreProblem::from(e)),
    +        }
    +    }
    +    match store.identity() {
    +        Some(identity) => store_identity::recheck(via, &identity, catalog, backend)
    +            .await
    +            .map(|()| identity),
    +        None => store_identity::establish(via, store.kind, &store.root, catalog, backend).await,
    +    }
    +}
    +
    +/// A command's read of the store's identity ([`store_identity::read_identity`],
    +/// which writes nothing), with the store's calls under
    +/// [`ConfiguredStore::check_timeout`] as the server's check runs them: a call
    +/// that does not finish within the bound is [`StoreProblem::Transport`]
    +/// naming it, where the store's own retries (the SDK's three attempts of
    +/// 60 s on an endpoint that accepts and never answers) or a stalled mount
    +/// would hold the command for minutes or for good.
    +///
    +/// # Errors
    +///
    +/// The problem that would make the store unhealthy
    +/// ([`store_identity`]'s module doc), or the bound.
    +pub async fn read_identity_bounded(
    +    store: &ConfiguredStore,
    +    catalog: &C,
    +) -> Result, StoreProblem> {
    +    let via = store.calls("identity read");
    +    let read = store_identity::read_identity(&via, store.kind, &store.root, catalog).await;
    +    if via.timed_out() {
    +        return Err(via.did_not_finish_problem());
    +    }
    +    read
    +}
    +
    +/// The configured store as the checks, a command's identity read and the
    +/// deletions call it (module doc): each call runs under `bound`, and a call
    +/// that does not finish is a transport failure naming `what` and the bound.
    +/// On a filesystem root (`park`) each call runs as a task of its own, owning
    +/// the store handle, so a call abandoned at its bound leaves the task
    +/// running until the mount answers, marked in flight until then; a call that
    +/// finds the mark set (a call abandoned earlier has not returned) is refused
    +/// at once as a transport failure, so no second thread is parked beside the
    +/// first. The root's calls run one at a time (`slot`): a call that finds
    +/// another running waits for it under the bound and, once it holds the slot,
    +/// has the bound for itself, so two callers whose calls start before the
    +/// first reaches its bound park one thread between them, the second
    +/// answering at its wait's bound with nothing parked when the first never
    +/// returns, and finishing when the first does. On an S3
    +/// store the call is the SDK's request, which the dropped future ends, so
    +/// nothing is marked and nothing waits. The check calls `validate`, `get`,
    +/// `head` and `put_if_absent`, the deletions `delete_prefix` and `list`;
    +/// `put` is forwarded for the trait's sake and bounded the same way.
    +pub(crate) struct BoundedCalls<'a> {
    +    store: &'a Arc,
    +    in_flight: &'a Arc,
    +    slot: &'a Arc,
    +    bound: std::time::Duration,
    +    what: &'static str,
    +    park: bool,
    +    timed_out: AtomicBool,
    +}
    +
    +/// The call task's end, by any exit: clears the in-flight mark under the
    +/// call's own lock, so a caller that abandons the call at its bound marks it
    +/// only while the task still runs (a task that ended between the timeout's
    +/// last poll and the caller's mark would otherwise leave the mark set with
    +/// nothing left to clear it, and every later check refused until a restart).
    +struct ClearOnDrop {
    +    in_flight: Arc,
    +    running: Arc>,
    +}
    +
    +impl Drop for ClearOnDrop {
    +    fn drop(&mut self) {
    +        let mut running = self
    +            .running
    +            .lock()
    +            .unwrap_or_else(std::sync::PoisonError::into_inner);
    +        *running = false;
    +        self.in_flight.store(false, Ordering::Release);
    +    }
    +}
    +
    +impl BoundedCalls<'_> {
    +    /// Whether a call of this adapter was abandoned at the bound.
    +    pub(crate) fn timed_out(&self) -> bool {
    +        self.timed_out.load(Ordering::Acquire)
    +    }
    +
    +    /// The text of a call that did not finish within the bound.
    +    fn did_not_finish_text(&self) -> String {
    +        format!(
    +            "the {} did not finish within {} s",
    +            self.what,
    +            self.bound.as_secs()
    +        )
    +    }
    +
    +    /// The transport failure a call that did not finish within the bound is.
    +    fn did_not_finish(&self) -> StoreError {
    +        StoreError::Transport {
    +            source: self.did_not_finish_text().into(),
    +        }
    +    }
    +
    +    /// The problem the check or read is recorded as once a call was
    +    /// abandoned: the transport failure naming the bound and nothing else
    +    /// (the store error's own prefix left out, so the operator's line and
    +    /// the recorded reason read as the bound).
    +    pub(crate) fn did_not_finish_problem(&self) -> StoreProblem {
    +        StoreProblem::Transport(self.did_not_finish_text())
    +    }
    +
    +    /// Run `call` on the store under the bound.
    +    fn run(&self, call: F) -> BoxFuture<'_, Result>
    +    where
    +        T: Send + 'static,
    +        F: FnOnce(Arc) -> BoxFuture<'static, Result>
    +            + Send
    +            + 'static,
    +    {
    +        if self.in_flight.load(Ordering::Acquire) {
    +            return Box::pin(std::future::ready(Err(StoreError::Transport {
    +                source: "a call of an earlier check has not returned".into(),
    +            })));
    +        }
    +        let store = Arc::clone(self.store);
    +        let bound = self.bound;
    +        if !self.park {
    +            return Box::pin(async move {
    +                match tokio::time::timeout(bound, call(store)).await {
    +                    Ok(outcome) => outcome,
    +                    Err(_) => {
    +                        self.timed_out.store(true, Ordering::Release);
    +                        Err(self.did_not_finish())
    +                    }
    +                }
    +            });
    +        }
    +        // The task owns the slot and the mark's clearing, so a call
    +        // abandoned below is marked, and holds the slot, until the mount
    +        // answers it. A caller that finds the slot taken waits for it under
    +        // the bound, and the call's own bound starts once it holds the slot
    +        // (a 3 s call behind a 3 s call finishes, where one bound over both
    +        // would abandon it); a wait that ends at the bound marks nothing,
    +        // since the call holding the slot marks itself when it is abandoned.
    +        let slot = Arc::clone(self.slot);
    +        let in_flight = Arc::clone(self.in_flight);
    +        Box::pin(async move {
    +            let Ok(permit) = tokio::time::timeout(bound, slot.acquire_owned()).await else {
    +                self.timed_out.store(true, Ordering::Release);
    +                return Err(self.did_not_finish());
    +            };
    +            let permit = permit.expect("the store's call slot is never closed");
    +            let running = Arc::new(std::sync::Mutex::new(true));
    +            let clear = ClearOnDrop {
    +                in_flight,
    +                running: Arc::clone(&running),
    +            };
    +            let task = tokio::spawn(async move {
    +                let _permit = permit;
    +                let _clear = clear;
    +                call(store).await
    +            });
    +            match tokio::time::timeout(bound, task).await {
    +                Ok(joined) => {
    +                    joined.map_err(|e| StoreError::Other(format!("store call task failed: {e}")))?
    +                }
    +                Err(_) => {
    +                    // Under the task's lock: a task that has already ended
    +                    // (its guard cleared the mark) is not marked again.
    +                    let running = running
    +                        .lock()
    +                        .unwrap_or_else(std::sync::PoisonError::into_inner);
    +                    if *running {
    +                        self.in_flight.store(true, Ordering::Release);
    +                    }
    +                    drop(running);
    +                    self.timed_out.store(true, Ordering::Release);
    +                    Err(self.did_not_finish())
    +                }
    +            }
    +        })
    +    }
    +}
    +
    +impl BackupStore for BoundedCalls<'_> {
    +    fn put<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.put(&key, body).await }))
    +    }
    +
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: ByteStream,
    +    ) -> BoxFuture<'a, Result, StoreError>> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.put_if_absent(&key, body).await }))
    +    }
    +
    +    fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.get(&key).await }))
    +    }
    +
    +    fn head<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result, StoreError>> {
    +        let key = key.to_owned();
    +        self.run(move |store| Box::pin(async move { store.head(&key).await }))
    +    }
    +
    +    fn list<'a>(&'a self, prefix: &'a str) -> BoxStream<'a, Result> {
    +        let prefix = prefix.to_owned();
    +        let listed = self.run(move |store| {
    +            Box::pin(async move {
    +                use futures::TryStreamExt as _;
    +                store.list(&prefix).try_collect::>().await
    +            })
    +        });
    +        Box::pin(
    +            futures::stream::once(listed).flat_map(|listed| match listed {
    +                Ok(metas) => futures::stream::iter(metas.into_iter().map(Ok)).boxed(),
    +                Err(e) => futures::stream::once(async move { Err(e) }).boxed(),
    +            }),
    +        )
    +    }
    +
    +    fn delete_prefix<'a>(&'a self, prefix: &'a str) -> BoxFuture<'a, Result> {
    +        let prefix = prefix.to_owned();
    +        self.run(move |store| Box::pin(async move { store.delete_prefix(&prefix).await }))
    +    }
    +
    +    fn validate(&self) -> BoxFuture<'_, Result<(), StoreError>> {
    +        self.run(|store| Box::pin(async move { store.validate().await }))
    +    }
     }
     
     impl BackupRuntime {
    @@ -49,12 +789,29 @@ impl BackupRuntime {
                 max_concurrent_jobs,
                 permit_wait: PERMIT_WAIT,
                 configured_store: None,
    +            legacy_create_only: false,
    +            stale_job_after: std::time::Duration::from_secs(300),
             }
         }
     
    +    /// The same runtime with `[backup] stale_job_after_secs` set, the window
    +    /// the jobs' maintenance-claim check reads with (the default is the
    +    /// configuration's default, 300 s).
    +    #[must_use]
    +    pub fn with_stale_job_after(mut self, stale_job_after: std::time::Duration) -> Self {
    +        self.stale_job_after = stale_job_after;
    +        self
    +    }
    +
    +    /// `[backup] stale_job_after_secs`.
    +    #[must_use]
    +    pub fn stale_job_after(&self) -> std::time::Duration {
    +        self.stale_job_after
    +    }
    +
         /// The same runtime with `store` configured.
         #[must_use]
    -    pub fn with_store(mut self, store: StoreIdentity) -> Self {
    +    pub fn with_store(mut self, store: ConfiguredStore) -> Self {
             self.configured_store = Some(store);
             self
         }
    @@ -67,6 +824,14 @@ impl BackupRuntime {
             self
         }
     
    +    /// The same runtime with CreateBackup bound to the in-catalog path
    +    /// whatever the store configuration.
    +    #[must_use]
    +    pub fn with_legacy_create_only(mut self, legacy_create_only: bool) -> Self {
    +        self.legacy_create_only = legacy_create_only;
    +        self
    +    }
    +
         /// The slots jobs take permits from.
         #[must_use]
         pub fn slots(&self) -> &Arc {
    @@ -79,6 +844,54 @@ impl BackupRuntime {
             self.slots.available_permits()
         }
     
    +    /// The configured store, if any; `None` too for a defaulted root set
    +    /// aside after its first check ([`ConfiguredStore::set_aside`]).
    +    #[must_use]
    +    pub fn configured_store(&self) -> Option<&ConfiguredStore> {
    +        self.configured_store
    +            .as_ref()
    +            .filter(|store| !store.is_set_aside())
    +    }
    +
    +    /// The configured store, taken out of a runtime a command built only to
    +    /// reach it.
    +    #[must_use]
    +    pub fn into_configured_store(self) -> Option {
    +        self.configured_store
    +    }
    +
    +    /// Whether CreateBackup takes the in-catalog path whatever the store
    +    /// configuration.
    +    #[must_use]
    +    pub fn legacy_create_only(&self) -> bool {
    +        self.legacy_create_only
    +    }
    +
    +    /// What `/health` reports for the `backup_store` component; `None` when
    +    /// no store is configured.
    +    #[must_use]
    +    pub fn store_health(&self) -> Option {
    +        let store = self.configured_store()?;
    +        let kind = store.kind.as_str();
    +        Some(match store.health() {
    +            StoreHealth::Healthy => StoreHealthReport {
    +                kind,
    +                status: "healthy",
    +                reason: None,
    +            },
    +            StoreHealth::Checking => StoreHealthReport {
    +                kind,
    +                status: "checking",
    +                reason: None,
    +            },
    +            StoreHealth::Unhealthy(problem) => StoreHealthReport {
    +                kind,
    +                status: "unhealthy",
    +                reason: Some(problem.category()),
    +            },
    +        })
    +    }
    +
         /// Take a job slot, waiting up to `permit_wait` for one, or
         /// `LimitExceededException` when none was freed within the bound. The
         /// request holds nothing while it waits: the permit is the last of its
    @@ -99,6 +912,33 @@ impl BackupRuntime {
         /// Whether `store` is the one this instance is configured with. With no
         /// store configured, no row's store is.
         pub(crate) fn is_configured_store(&self, store: &StoreIdentity) -> bool {
    -        self.configured_store.as_ref() == Some(store)
    +        self.configured_store()
    +            .is_some_and(|configured| configured.is_store_of(store))
    +    }
    +
    +    /// The configured store, when its last check passed; otherwise (no
    +    /// check has finished, or the last one failed) the `ValidationException`
    +    /// the operations that touch the store answer, naming the store's kind
    +    /// and the problem's category and not its root (no API error names a
    +    /// root). Every caller has checked that a store is configured
    +    /// (`is_configured_store`, or the store-backed create's own condition).
    +    pub(crate) fn require_healthy_store(&self) -> Result<&ConfiguredStore, DynamoDbError> {
    +        let store = self
    +            .configured_store()
    +            .expect("every caller checks that a store is configured first");
    +        match store.health() {
    +            StoreHealth::Healthy => Ok(store),
    +            StoreHealth::Checking => Err(DynamoDbError::ValidationException(format!(
    +                "The {} backup store configured on this server has not passed a check yet (its \
    +                 first check is pending); the operation is refused until it does",
    +                store.kind
    +            ))),
    +            StoreHealth::Unhealthy(problem) => Err(DynamoDbError::ValidationException(format!(
    +                "The {} backup store configured on this server is unhealthy ({}); the operation \
    +                 is refused until the store's check passes",
    +                store.kind,
    +                problem.category()
    +            ))),
    +        }
         }
     }
    diff --git a/crates/engine/src/backup/store_identity.rs b/crates/engine/src/backup/store_identity.rs
    new file mode 100644
    index 000000000..b9f846eca
    --- /dev/null
    +++ b/crates/engine/src/backup/store_identity.rs
    @@ -0,0 +1,664 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! The store identity file and the checks that decide a configured store's
    +//! health.
    +//!
    +//! A store's root carries `extenddb-store.json` ([`STORE_IDENTITY_KEY`]),
    +//! holding a random store id and the `catalog_id` of the catalog that owns
    +//! the root; every backup row records the store id in `store_id`. Server
    +//! startup ([`establish`]) and every reconciler cycle ([`recheck`]) read the
    +//! file and check it against the catalog. The store is unhealthy when:
    +//!
    +//! - the file names another catalog (a root another deployment owns, or a
    +//!   copy of one; an operator adopts it on purpose, never a server by
    +//!   accident);
    +//! - its store id differs from the `store_id` of the catalog's rows for the
    +//!   configured kind and root (the root was replaced under the rows);
    +//! - the catalog has rows for the root and the root has no file (the root
    +//!   was emptied or re-pointed);
    +//! - those rows carry more than one `store_id`;
    +//! - the manifest probe fails: the newest AVAILABLE row's
    +//!   `extenddb-manifest.json` is not under the root. Instances that share a
    +//!   catalog must share one store root; host-local roots at one path give
    +//!   each instance different bytes under one `store_root`, which the ids
    +//!   alone cannot see, so the probe reads the root independently of the
    +//!   host;
    +//! - the store cannot be reached or read, or the file cannot be parsed.
    +//!
    +//! On a root with no file and a catalog with no rows for it, the server
    +//! creates the file with `put_if_absent`, then re-reads it and uses the id
    +//! it holds. Of two instances of one catalog starting against one empty root,
    +//! one creates the file and the other's create finds it present, and both
    +//! use the winner's id. An S3-compatible server that ignores `If-None-Match`
    +//! makes the create a plain put, last writer wins: the re-read catches a race
    +//! lost before it (the file holds another id than this instance wrote) and
    +//! marks the store unhealthy naming both ids; a race lost after it is caught
    +//! by the next check, which finds rows carrying an id the file does not.
    +//!
    +//! An unhealthy store is not a startup refusal: the caller logs the problem,
    +//! reports it on `/health`, refuses the operations that touch the store, and
    +//! runs the check again each cycle until it passes.
    +
    +use extenddb_backup_store::{BackupStore, StoreError, byte_stream_from};
    +use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE};
    +use extenddb_storage::BackupCatalog;
    +use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind};
    +use futures::TryStreamExt;
    +use serde::{Deserialize, Serialize};
    +
    +/// The key of the identity file, at the root, under no `/` prefix.
    +pub const STORE_IDENTITY_KEY: &str = "extenddb-store.json";
    +
    +/// The one version of the file's shape this build writes and reads.
    +pub const STORE_IDENTITY_FORMAT_VERSION: u32 = 1;
    +
    +/// The most bytes a check reads of `extenddb-store.json`. The file this
    +/// build writes is under 300 bytes; an object larger than this under the
    +/// key is not an identity file, and is reported as unreadable without being
    +/// read into memory on every cycle.
    +pub const STORE_IDENTITY_MAX_BYTES: usize = 64 * 1024;
    +
    +/// The contents of `extenddb-store.json`.
    +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
    +pub struct StoreIdentityFile {
    +    pub format_version: u32,
    +    /// The random id every row of this store carries.
    +    pub store_id: String,
    +    /// The catalog that owns the root.
    +    pub catalog_id: String,
    +    /// The storage backend of the owning catalog, for the operator reading
    +    /// the file; nothing is checked against it.
    +    pub backend: String,
    +    /// When the file was created, epoch milliseconds on the creating host.
    +    pub created_at_ms: u64,
    +}
    +
    +impl StoreIdentityFile {
    +    /// A new identity for `catalog_id`.
    +    fn new(catalog_id: &str, backend: &str) -> Self {
    +        Self {
    +            format_version: STORE_IDENTITY_FORMAT_VERSION,
    +            store_id: uuid::Uuid::new_v4().to_string(),
    +            catalog_id: catalog_id.to_owned(),
    +            backend: backend.to_owned(),
    +            created_at_ms: std::time::SystemTime::now()
    +                .duration_since(std::time::UNIX_EPOCH)
    +                .map_or(0, |d| u64::try_from(d.as_millis()).unwrap_or(u64::MAX)),
    +        }
    +    }
    +
    +    /// Parse the file's bytes.
    +    ///
    +    /// # Errors
    +    ///
    +    /// [`StoreProblem::FileUnreadable`] for bytes that are not the documented
    +    /// shape, or carry a version this build does not read.
    +    pub fn parse(bytes: &[u8]) -> Result {
    +        let file: Self = serde_json::from_slice(bytes)
    +            .map_err(|e| StoreProblem::FileUnreadable(format!("not a store identity file: {e}")))?;
    +        if file.format_version != STORE_IDENTITY_FORMAT_VERSION {
    +            return Err(StoreProblem::FileUnreadable(format!(
    +                "format_version {} is not {STORE_IDENTITY_FORMAT_VERSION}",
    +                file.format_version
    +            )));
    +        }
    +        if file.store_id.is_empty() || file.catalog_id.is_empty() {
    +            return Err(StoreProblem::FileUnreadable(
    +                "store_id and catalog_id must be non-empty".to_owned(),
    +            ));
    +        }
    +        Ok(file)
    +    }
    +
    +    /// The file's bytes.
    +    #[must_use]
    +    pub fn to_bytes(&self) -> Vec {
    +        let mut bytes = serde_json::to_vec_pretty(self).expect("a plain struct serializes");
    +        bytes.push(b'\n');
    +        bytes
    +    }
    +}
    +
    +/// Why a store is unhealthy. `Display` is the operator's line (ids, the
    +/// catalog id, the store's own error text, never a credential);
    +/// [`StoreProblem::public_reason`] is the part of it that names no root, and
    +/// [`StoreProblem::category`] alone is what `/health` reports.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum StoreProblem {
    +    /// The store could not be reached, read or written; the store's error.
    +    Transport(String),
    +    /// The store refused an operation: a root or a directory under it that
    +    /// another user can reach, a denied grant; the store's error.
    +    Denied(String),
    +    /// The store's `validate`, or a request of the identity check, failed for
    +    /// a reason that is neither a transport failure nor a denial (a bucket
    +    /// that does not exist, an I/O error on a filesystem root); the store's
    +    /// error. The configuration errors `validate` reports before the
    +    /// listener stop startup instead. One category whether `validate` or a
    +    /// later check's request found the condition, so a bucket deleted under
    +    /// a running server reads as it does at startup.
    +    Validate(String),
    +    /// `extenddb-store.json` is there but is not a readable identity file.
    +    FileUnreadable(String),
    +    /// The file names a catalog other than this one.
    +    ForeignCatalog {
    +        file_catalog_id: String,
    +        catalog_id: String,
    +    },
    +    /// The file's id is not the id the catalog's rows for the root carry.
    +    IdMismatch {
    +        file_id: String,
    +        row_ids: Vec,
    +    },
    +    /// The catalog's rows for the root carry more than one id.
    +    MultipleRowIds(Vec),
    +    /// The catalog has rows for the root and the root has no file.
    +    FileMissingWithRows(Vec),
    +    /// This instance's create reported success and the re-read holds another
    +    /// id: the store did not honour the conditional put.
    +    CreateRaceLost { ours: String, file: String },
    +    /// The newest AVAILABLE row's manifest is not under the root.
    +    ManifestMissing { backup_arn: String },
    +    /// The catalog could not be read.
    +    Catalog(String),
    +}
    +
    +impl StoreProblem {
    +    /// The problem's category, one or two words, for an API refusal.
    +    #[must_use]
    +    pub fn category(&self) -> &'static str {
    +        match self {
    +            Self::Transport(_) => "unreachable",
    +            Self::Denied(_) => "permission denied",
    +            Self::Validate(_) => "failed validation",
    +            Self::FileUnreadable(_) => "identity file unreadable",
    +            Self::ForeignCatalog { .. } => "owned by another catalog",
    +            Self::IdMismatch { .. } | Self::MultipleRowIds(_) | Self::CreateRaceLost { .. } => {
    +                "identity mismatch"
    +            }
    +            Self::FileMissingWithRows(_) => "identity file missing",
    +            Self::ManifestMissing { .. } => "manifest probe failed",
    +            Self::Catalog(_) => "catalog unreadable",
    +        }
    +    }
    +
    +    /// The problem with the ids involved and never a root (a filesystem path
    +    /// or a bucket) or the store's own error text, which may carry one: the
    +    /// operator's line for the identity problems. Not the `/health` text,
    +    /// which is [`StoreProblem::category`].
    +    #[must_use]
    +    pub fn public_reason(&self) -> String {
    +        match self {
    +            Self::Transport(_) => "the store could not be reached".to_owned(),
    +            Self::Denied(_) => "the store refused an operation (permission denied)".to_owned(),
    +            Self::Validate(_) => "the store failed its validation".to_owned(),
    +            Self::FileUnreadable(_) => format!("{STORE_IDENTITY_KEY} could not be read"),
    +            Self::ForeignCatalog {
    +                file_catalog_id,
    +                catalog_id,
    +            } => format!(
    +                "{STORE_IDENTITY_KEY} names catalog {file_catalog_id}; this catalog is \
    +                 {catalog_id}"
    +            ),
    +            Self::IdMismatch { file_id, row_ids } => format!(
    +                "{STORE_IDENTITY_KEY} holds store id {file_id}; the catalog's rows for the root \
    +                 carry {}",
    +                row_ids.join(", ")
    +            ),
    +            Self::MultipleRowIds(ids) => format!(
    +                "the catalog's rows for the root carry more than one store id: {}",
    +                ids.join(", ")
    +            ),
    +            Self::FileMissingWithRows(ids) => format!(
    +                "the root has no {STORE_IDENTITY_KEY} but the catalog has rows for it (store id \
    +                 {})",
    +                ids.join(", ")
    +            ),
    +            Self::CreateRaceLost { ours, file } => format!(
    +                "this server created {STORE_IDENTITY_KEY} with store id {ours} but the file \
    +                 holds {file}; the store did not honour the conditional put"
    +            ),
    +            Self::ManifestMissing { backup_arn } => {
    +                format!("the manifest of {backup_arn} is not under the root")
    +            }
    +            Self::Catalog(_) => "the catalog could not be read".to_owned(),
    +        }
    +    }
    +}
    +
    +impl std::fmt::Display for StoreProblem {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        match self {
    +            Self::Transport(e) => write!(f, "the store could not be reached: {e}"),
    +            Self::Denied(e) => write!(f, "the store refused an operation: {e}"),
    +            Self::Validate(e) => write!(f, "the store failed its validation: {e}"),
    +            Self::FileUnreadable(e) => write!(f, "{STORE_IDENTITY_KEY} could not be read: {e}"),
    +            Self::Catalog(e) => write!(f, "the catalog could not be read: {e}"),
    +            other => f.write_str(&other.public_reason()),
    +        }
    +    }
    +}
    +
    +/// The problem a store's error is: a transport failure keeps its class with
    +/// the store error's whole cause chain (the SDK's top link is a bare
    +/// category such as "dispatch failure"; the cause is a link below it), a
    +/// denial is [`StoreProblem::Denied`], and every other refusal
    +/// ([`StoreProblem::Validate`]'s doc) is the same category `validate` gives
    +/// it.
    +impl From for StoreProblem {
    +    fn from(e: StoreError) -> Self {
    +        match e {
    +            StoreError::PermissionDenied(m) => Self::Denied(m),
    +            transport @ StoreError::Transport { .. } => Self::Transport(transport.to_string()),
    +            other => Self::Validate(other.to_string()),
    +        }
    +    }
    +}
    +
    +/// Read the identity file, or `None` when the root has none. At most
    +/// [`STORE_IDENTITY_MAX_BYTES`] are read; a larger object is
    +/// [`StoreProblem::FileUnreadable`].
    +async fn read_file(store: &dyn BackupStore) -> Result, StoreProblem> {
    +    let mut body = match store.get(STORE_IDENTITY_KEY).await {
    +        Ok(body) => body,
    +        Err(StoreError::NotFound) => return Ok(None),
    +        Err(e) => return Err(e.into()),
    +    };
    +    let mut bytes = Vec::new();
    +    while let Some(chunk) = body.try_next().await? {
    +        if bytes.len() + chunk.len() > STORE_IDENTITY_MAX_BYTES {
    +            return Err(StoreProblem::FileUnreadable(format!(
    +                "larger than {STORE_IDENTITY_MAX_BYTES} bytes"
    +            )));
    +        }
    +        bytes.extend_from_slice(&chunk);
    +    }
    +    StoreIdentityFile::parse(&bytes).map(Some)
    +}
    +
    +/// The distinct ids the catalog's rows for the root carry, at most one.
    +async fn row_ids(
    +    catalog: &C,
    +    kind: StoreKind,
    +    root: &str,
    +) -> Result, StoreProblem> {
    +    let ids = catalog
    +        .read_store_ids(kind, root)
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    if ids.len() > 1 {
    +        return Err(StoreProblem::MultipleRowIds(ids));
    +    }
    +    Ok(ids)
    +}
    +
    +/// The startup check: `validate` has passed; read the file, create it when
    +/// the root is empty of both file and rows, compare it with the rows and
    +/// with `catalog_id`, then probe the newest manifest. Returns the store's
    +/// identity, the rows' and the file's.
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy (module doc).
    +pub async fn establish(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, kind, root).await?;
    +    let file = match read_file(store).await? {
    +        Some(file) => file,
    +        None if !row_ids.is_empty() => {
    +            return Err(StoreProblem::FileMissingWithRows(row_ids));
    +        }
    +        None => {
    +            let ours = StoreIdentityFile::new(&catalog_id, backend);
    +            let created = store
    +                .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes()))
    +                .await?
    +                .is_some();
    +            // The key holds nothing the store reads as an object and refuses
    +            // the create (a FIFO or a socket at the path, say): nothing was
    +            // created, and nothing can be read.
    +            let file = read_file(store).await?.ok_or_else(|| {
    +                StoreProblem::Transport(format!(
    +                    "{STORE_IDENTITY_KEY} is neither readable nor creatable"
    +                ))
    +            })?;
    +            if created && file.store_id != ours.store_id {
    +                return Err(StoreProblem::CreateRaceLost {
    +                    ours: ours.store_id,
    +                    file: file.store_id,
    +                });
    +            }
    +            if created {
    +                tracing::info!(
    +                    "backup store ({kind}) root was empty; created {STORE_IDENTITY_KEY} with \
    +                     store id {}",
    +                    file.store_id
    +                );
    +            } else {
    +                tracing::info!(
    +                    "backup store ({kind}) root already carried {STORE_IDENTITY_KEY} (store id \
    +                     {}), created by another instance of this catalog",
    +                    file.store_id
    +                );
    +            }
    +            file
    +        }
    +    };
    +    let identity = compare(&file, &catalog_id, row_ids, kind, root)?;
    +    manifest_probe(store, &identity, catalog).await?;
    +    Ok(identity)
    +}
    +
    +/// A read of the store's identity that writes nothing: the file, compared
    +/// with the catalog as [`establish`] compares it, without the create of a
    +/// missing file and without the manifest probe. `Ok(None)` for a root with
    +/// no file and no rows (a fresh root). For a command that reads beside a
    +/// server (`backup list`), which must leave the root as it found it.
    +///
    +/// # Errors
    +///
    +/// The problem that would make the store unhealthy (module doc).
    +pub async fn read_identity(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +) -> Result, StoreProblem> {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, kind, root).await?;
    +    match read_file(store).await? {
    +        Some(file) => compare(&file, &catalog_id, row_ids, kind, root).map(Some),
    +        None if !row_ids.is_empty() => Err(StoreProblem::FileMissingWithRows(row_ids)),
    +        None => Ok(None),
    +    }
    +}
    +
    +/// The file against the catalog: it must name this catalog, and the rows
    +/// for the root, when there are any, must carry its id.
    +fn compare(
    +    file: &StoreIdentityFile,
    +    catalog_id: &str,
    +    row_ids: Vec,
    +    kind: StoreKind,
    +    root: &str,
    +) -> Result {
    +    if file.catalog_id != catalog_id {
    +        return Err(StoreProblem::ForeignCatalog {
    +            file_catalog_id: file.catalog_id.clone(),
    +            catalog_id: catalog_id.to_owned(),
    +        });
    +    }
    +    if !row_ids.is_empty() && row_ids[0] != file.store_id {
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id.clone(),
    +            row_ids,
    +        });
    +    }
    +    Ok(StoreIdentity {
    +        store_kind: kind,
    +        store_root: root.to_owned(),
    +        store_id: file.store_id.clone(),
    +    })
    +}
    +
    +/// The check every reconciler cycle runs on a store whose identity startup
    +/// settled: the file is still there, names this catalog and holds the
    +/// identity's id; the rows for the root carry no other id; the newest
    +/// manifest is under the root. A root that lost its file while the catalog
    +/// has no rows for it gets the file back, written with the identity's id
    +/// (`put_if_absent`, so a file another instance wrote first wins and is then
    +/// compared like any other).
    +///
    +/// # Errors
    +///
    +/// The problem that makes the store unhealthy (module doc).
    +pub async fn recheck(
    +    store: &dyn BackupStore,
    +    identity: &StoreIdentity,
    +    catalog: &C,
    +    backend: &str,
    +) -> Result<(), StoreProblem> {
    +    let catalog_id = catalog
    +        .catalog_id()
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?;
    +    let row_ids = row_ids(catalog, identity.store_kind, &identity.store_root).await?;
    +    let file = match read_file(store).await? {
    +        Some(file) => file,
    +        None if !row_ids.is_empty() => {
    +            return Err(StoreProblem::FileMissingWithRows(row_ids));
    +        }
    +        None => {
    +            let ours = StoreIdentityFile {
    +                store_id: identity.store_id.clone(),
    +                ..StoreIdentityFile::new(&catalog_id, backend)
    +            };
    +            store
    +                .put_if_absent(STORE_IDENTITY_KEY, byte_stream_from(ours.to_bytes()))
    +                .await?;
    +            tracing::warn!(
    +                "backup store ({}) root had lost {STORE_IDENTITY_KEY}; rewrote it with store id \
    +                 {}",
    +                identity.store_kind,
    +                identity.store_id
    +            );
    +            read_file(store).await?.ok_or_else(|| {
    +                StoreProblem::Transport(format!(
    +                    "{STORE_IDENTITY_KEY} is neither readable nor creatable"
    +                ))
    +            })?
    +        }
    +    };
    +    if file.catalog_id != catalog_id {
    +        return Err(StoreProblem::ForeignCatalog {
    +            file_catalog_id: file.catalog_id,
    +            catalog_id,
    +        });
    +    }
    +    if file.store_id != identity.store_id {
    +        // The file changed under a running server: the root was replaced.
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id,
    +            row_ids: vec![identity.store_id.clone()],
    +        });
    +    }
    +    if let Some(other) = row_ids.iter().find(|id| **id != identity.store_id) {
    +        return Err(StoreProblem::IdMismatch {
    +            file_id: file.store_id,
    +            row_ids: vec![other.clone()],
    +        });
    +    }
    +    manifest_probe(store, identity, catalog).await
    +}
    +
    +/// `head` the manifest of the newest AVAILABLE row of the store; a root with
    +/// no AVAILABLE row passes (there is nothing to probe).
    +///
    +/// # Errors
    +///
    +/// [`StoreProblem::ManifestMissing`] when the manifest is absent,
    +/// [`StoreProblem::Transport`] when the store cannot be read,
    +/// [`StoreProblem::Catalog`] when the rows cannot be listed.
    +pub async fn manifest_probe(
    +    store: &dyn BackupStore,
    +    identity: &StoreIdentity,
    +    catalog: &C,
    +) -> Result<(), StoreProblem> {
    +    // One row, not the listing: the probe runs each reconciler cycle on
    +    // every instance and in each command's identity check.
    +    let Some(newest) = catalog
    +        .newest_available_backup_row(identity)
    +        .await
    +        .map_err(|e| StoreProblem::Catalog(e.to_string()))?
    +    else {
    +        return Ok(());
    +    };
    +    let arn = BackupArn::parse(&newest.backup_arn).map_err(|e| {
    +        StoreProblem::Catalog(format!("row {} carries a bad ARN: {e}", newest.backup_arn))
    +    })?;
    +    let key = format!("{}{EXTENDDB_MANIFEST_FILE}", arn.store_prefix());
    +    match store.head(&key).await? {
    +        Some(_) => Ok(()),
    +        None => Err(StoreProblem::ManifestMissing {
    +            backup_arn: newest.backup_arn.clone(),
    +        }),
    +    }
    +}
    +
    +#[cfg(test)]
    +mod tests {
    +    use super::*;
    +
    +    #[test]
    +    fn the_file_round_trips_and_refuses_other_shapes() {
    +        let file = StoreIdentityFile::new("cat-1", "postgres");
    +        assert_eq!(file.format_version, 1);
    +        assert_eq!(file.catalog_id, "cat-1");
    +        assert_eq!(file.backend, "postgres");
    +        assert!(!file.store_id.is_empty());
    +        let parsed = StoreIdentityFile::parse(&file.to_bytes()).unwrap();
    +        assert_eq!(parsed, file);
    +
    +        let v2 = serde_json::json!({
    +            "format_version": 2, "store_id": "s", "catalog_id": "c", "backend": "b",
    +            "created_at_ms": 0
    +        });
    +        let err = StoreIdentityFile::parse(v2.to_string().as_bytes()).unwrap_err();
    +        assert!(matches!(&err, StoreProblem::FileUnreadable(m) if m.contains("format_version 2")));
    +        let empty = serde_json::json!({
    +            "format_version": 1, "store_id": "", "catalog_id": "c", "backend": "b",
    +            "created_at_ms": 0
    +        });
    +        assert!(matches!(
    +            StoreIdentityFile::parse(empty.to_string().as_bytes()),
    +            Err(StoreProblem::FileUnreadable(_))
    +        ));
    +        assert!(matches!(
    +            StoreIdentityFile::parse(b"not json"),
    +            Err(StoreProblem::FileUnreadable(_))
    +        ));
    +    }
    +
    +    /// `public_reason` names ids and the catalog, never a root or the store's
    +    /// own error text (which may carry a path or a bucket); `Display` carries
    +    /// that text, and `/health` carries the category alone.
    +    #[test]
    +    fn the_public_reason_names_no_root() {
    +        let transport = StoreProblem::Transport("connect /srv/backups: refused".to_owned());
    +        assert!(!transport.public_reason().contains("/srv"));
    +        assert!(transport.to_string().contains("/srv/backups"));
    +        assert_eq!(transport.category(), "unreachable");
    +        let foreign = StoreProblem::ForeignCatalog {
    +            file_catalog_id: "other".to_owned(),
    +            catalog_id: "mine".to_owned(),
    +        };
    +        assert!(foreign.public_reason().contains("other"));
    +        assert!(foreign.public_reason().contains("mine"));
    +        assert_eq!(foreign.category(), "owned by another catalog");
    +        let probe = StoreProblem::ManifestMissing {
    +            backup_arn: "arn:x".to_owned(),
    +        };
    +        assert!(probe.public_reason().contains("arn:x"));
    +        assert_eq!(probe.category(), "manifest probe failed");
    +        let unreadable = StoreProblem::FileUnreadable("at /srv/x".to_owned());
    +        assert!(!unreadable.public_reason().contains("/srv"));
    +        let denied: StoreProblem =
    +            StoreError::PermissionDenied("/srv/backups/a has mode 0755".to_owned()).into();
    +        assert_eq!(denied.category(), "permission denied");
    +        assert!(!denied.public_reason().contains("/srv"));
    +        assert!(denied.to_string().contains("/srv/backups/a"));
    +        // A transport failure keeps its whole cause chain: the SDK's top
    +        // link names only the class, and the cause is the link below it.
    +        let chain: StoreProblem = StoreError::Transport {
    +            source: Box::new(TwoLinks),
    +        }
    +        .into();
    +        assert_eq!(chain.category(), "unreachable");
    +        assert!(chain.to_string().contains("dispatch failure"), "{chain}");
    +        assert!(chain.to_string().contains("connection refused"), "{chain}");
    +        // Any other store refusal is the category `validate` gives it, so a
    +        // bucket that disappears under a running server reads as a bucket
    +        // that was missing at startup.
    +        let missing: StoreProblem = StoreError::Other("NoSuchBucket".to_owned()).into();
    +        assert_eq!(missing.category(), "failed validation");
    +        assert!(missing.to_string().contains("NoSuchBucket"));
    +    }
    +
    +    /// A two-link error chain: "dispatch failure" over "connection refused".
    +    #[derive(Debug)]
    +    struct TwoLinks;
    +
    +    impl std::fmt::Display for TwoLinks {
    +        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +            f.write_str("dispatch failure")
    +        }
    +    }
    +
    +    impl std::error::Error for TwoLinks {
    +        fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +            Some(&CONNECTION_REFUSED)
    +        }
    +    }
    +
    +    static CONNECTION_REFUSED: ConnectionRefused = ConnectionRefused;
    +
    +    #[derive(Debug)]
    +    struct ConnectionRefused;
    +
    +    impl std::fmt::Display for ConnectionRefused {
    +        fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +            f.write_str("connection refused")
    +        }
    +    }
    +
    +    impl std::error::Error for ConnectionRefused {}
    +
    +    #[test]
    +    fn compare_applies_the_three_rules() {
    +        let file = StoreIdentityFile {
    +            format_version: 1,
    +            store_id: "s-1".to_owned(),
    +            catalog_id: "c-1".to_owned(),
    +            backend: "sqlite".to_owned(),
    +            created_at_ms: 0,
    +        };
    +        let ok = compare(&file, "c-1", vec![], StoreKind::Filesystem, "/r").unwrap();
    +        assert_eq!(ok.store_id, "s-1");
    +        assert_eq!(ok.store_root, "/r");
    +        compare(
    +            &file,
    +            "c-1",
    +            vec!["s-1".to_owned()],
    +            StoreKind::Filesystem,
    +            "/r",
    +        )
    +        .unwrap();
    +        assert!(matches!(
    +            compare(&file, "c-2", vec![], StoreKind::Filesystem, "/r"),
    +            Err(StoreProblem::ForeignCatalog { .. })
    +        ));
    +        assert!(matches!(
    +            compare(
    +                &file,
    +                "c-1",
    +                vec!["s-9".to_owned()],
    +                StoreKind::S3,
    +                "s3://b/"
    +            ),
    +            Err(StoreProblem::IdMismatch { .. })
    +        ));
    +    }
    +}
    diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs
    index d306f8fd4..f3c72b532 100644
    --- a/crates/engine/src/backup/tests.rs
    +++ b/crates/engine/src/backup/tests.rs
    @@ -5,7 +5,7 @@
     //! reader that records what the engine asked of it.
     
     use std::collections::HashMap;
    -use std::sync::Mutex;
    +use std::sync::{Arc, Mutex};
     use std::time::Duration;
     
     use extenddb_core::backup_format::BackupArn;
    @@ -22,7 +22,7 @@ use extenddb_core::types::{
     };
     use extenddb_storage::backup_catalog::{
         AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, JobClaim,
    -    RestoreJobRow, RestoreTargetKey, StoreIdentity, StoreKind, TimeBounds,
    +    RestoreJobRow, RestoreTargetKey, StaleClock, StoreIdentity, StoreKind, TimeBounds,
     };
     use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition};
     use extenddb_storage::error::StorageError;
    @@ -30,6 +30,7 @@ use extenddb_storage::{
         BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, RestoreRequest,
         TableEngine,
     };
    +use futures::TryStreamExt;
     use futures::stream::BoxStream;
     use serde_json::{Value, json};
     
    @@ -91,11 +92,20 @@ struct Fake {
         abandoned_backups: Mutex>,
         deleted: Mutex>,
         maintenance_held: Mutex,
    -    /// How many times `check_maintenance_claim` ran: one per reconciler
    -    /// cycle, so a test can see that a cycle ran.
    -    maintenance_checks: Mutex,
    +    /// The windows `check_maintenance_claim` was called with: one entry per
    +    /// reconciler cycle, so a test can see that a cycle ran.
    +    maintenance_checks: Mutex>,
         /// When set, `check_maintenance_claim` fails with it (every call).
         maintenance_error: Mutex>,
    +    /// When set, the age of a live maintenance claim's last heartbeat, and
    +    /// `check_maintenance_claim(stale_after)` answers held while the age is
    +    /// under `stale_after`, the rule the backends apply; `maintenance_held`
    +    /// answers otherwise.
    +    maintenance_beat_age: Mutex>,
    +    /// How far the catalog's clock runs ahead of this process's (negative:
    +    /// behind), in seconds; `list_backup_rows_older_than` measures ages on
    +    /// the catalog's clock.
    +    catalog_clock_offset_secs: Mutex,
         /// The `TimeBounds` `list_account_backup_rows` received.
         bounds_seen: Mutex>,
         /// The account's tables, by name, as `describe_table` reports them; a
    @@ -103,6 +113,22 @@ struct Fake {
         tables: Mutex>,
         /// The names `describe_table` was asked for.
         table_reads: Mutex>,
    +    /// What `catalog_id` answers; `catalog-1` unless a test changes it.
    +    catalog_id: Mutex>,
    +    /// When set, `catalog_id` fails as a catalog that cannot be read.
    +    catalog_id_error: Mutex,
    +    /// Rows `set_backup_deleted` moved to DELETED, by ARN.
    +    set_deleted: Mutex>,
    +    /// Rows `remove_backup_row` removed, by ARN.
    +    removed_rows: Mutex>,
    +    /// When set, `list_backup_rows` fails with it.
    +    list_error: Mutex>,
    +    /// The stores `newest_available_backup_row` was asked for.
    +    newest_reads: Mutex>,
    +    /// RUNNING restore records with a live heartbeat (a fresh one, say),
    +    /// which `list_running_restore_jobs` lists and `list_dead_restore_jobs`
    +    /// does not.
    +    running_jobs: Mutex>,
     }
     
     impl Fake {
    @@ -139,6 +165,328 @@ fn store() -> StoreIdentity {
         }
     }
     
    +/// An in-memory `BackupStore` for the engine's tests: objects by key, a
    +/// switch that makes every call fail as a transport error, a switch that
    +/// makes every call wait forever (a store that has stopped answering), and
    +/// the keys each operation was called with.
    +#[derive(Default)]
    +struct MemStore {
    +    objects: Mutex>>,
    +    /// When set, every operation fails with this text as the cause under a
    +    /// "dispatch failure" link, the shape the SDK's transport errors have.
    +    down: Mutex>,
    +    /// When set, every operation's future never completes; `stalls` counts
    +    /// the calls that entered the stall. A stalled call returns once the
    +    /// store is un-stalled, as a parked filesystem call returns when its
    +    /// mount answers.
    +    stalled: std::sync::atomic::AtomicBool,
    +    stalls: std::sync::atomic::AtomicUsize,
    +    /// When set, every operation takes this long before it answers: a
    +    /// healthy root whose calls are slow.
    +    delay: Mutex>,
    +    deletes: Mutex>,
    +    puts: Mutex>,
    +    /// When set, the bytes `put_if_absent` leaves under the key right after
    +    /// it stored its own: a server that ignores `If-None-Match` and a second
    +    /// writer that won the plain-put race.
    +    overwrite_after_conditional_put: Mutex>>,
    +    /// When set, `put_if_absent` finds the key taken (`None`) while `get`
    +    /// and `head` find no object: something that is not an object (a FIFO,
    +    /// a socket) sits at the path.
    +    key_taken_by_a_non_object: std::sync::atomic::AtomicBool,
    +    /// When set, `validate` fails with this text as a store condition (the
    +    /// probe write failing with an I/O error) while every other call
    +    /// answers.
    +    validate_error: Mutex>,
    +    /// When set, a `delete_prefix` of a prefix (not of a manifest key) is
    +    /// followed by a put of this key under the prefix: a writer whose put
    +    /// overlapped the deletion.
    +    late_put_after_delete: Mutex>,
    +}
    +
    +/// A transport error with the SDK's shape: a top link naming only the class
    +/// and the cause one link below it.
    +#[derive(Debug)]
    +struct DispatchFailure(ConnectError);
    +
    +#[derive(Debug)]
    +struct ConnectError(String);
    +
    +impl std::fmt::Display for DispatchFailure {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str("dispatch failure")
    +    }
    +}
    +
    +impl std::error::Error for DispatchFailure {
    +    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
    +        Some(&self.0)
    +    }
    +}
    +
    +impl std::fmt::Display for ConnectError {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.write_str(&self.0)
    +    }
    +}
    +
    +impl std::error::Error for ConnectError {}
    +
    +impl MemStore {
    +    /// A store that is down fails every call as the S3 store fails one it
    +    /// cannot send: a transport error whose cause is a link below the top.
    +    fn check(&self) -> Result<(), extenddb_backup_store::StoreError> {
    +        match self.down.lock().unwrap().as_ref() {
    +            Some(text) => Err(extenddb_backup_store::StoreError::Transport {
    +                source: Box::new(DispatchFailure(ConnectError(text.clone()))),
    +            }),
    +            None => Ok(()),
    +        }
    +    }
    +    /// Wait while the store is stalled, and the delay if one is set;
    +    /// otherwise return at once.
    +    async fn stall(&self) {
    +        let delay = *self.delay.lock().unwrap();
    +        if let Some(delay) = delay {
    +            tokio::time::sleep(delay).await;
    +        }
    +        if self.stalled.load(std::sync::atomic::Ordering::Acquire) {
    +            self.stalls
    +                .fetch_add(1, std::sync::atomic::Ordering::AcqRel);
    +            while self.stalled.load(std::sync::atomic::Ordering::Acquire) {
    +                tokio::time::sleep(Duration::from_millis(10)).await;
    +            }
    +        }
    +    }
    +    fn set_down(&self, down: bool) {
    +        *self.down.lock().unwrap() = down.then(|| "connection refused".to_owned());
    +    }
    +    fn set_stalled(&self, stalled: bool) {
    +        self.stalled
    +            .store(stalled, std::sync::atomic::Ordering::Release);
    +    }
    +    fn set_delay(&self, delay: Option) {
    +        *self.delay.lock().unwrap() = delay;
    +    }
    +    fn insert(&self, key: &str, body: &[u8]) {
    +        self.objects
    +            .lock()
    +            .unwrap()
    +            .insert(key.to_owned(), body.to_vec());
    +    }
    +    fn keys(&self) -> Vec {
    +        self.objects.lock().unwrap().keys().cloned().collect()
    +    }
    +    fn matches(key: &str, prefix: &str) -> bool {
    +        let prefix = prefix.trim_end_matches('/');
    +        prefix.is_empty() || key == prefix || key.starts_with(&format!("{prefix}/"))
    +    }
    +}
    +
    +async fn collect_body(
    +    body: extenddb_backup_store::ByteStream,
    +) -> Result, extenddb_backup_store::StoreError> {
    +    body.try_fold(Vec::new(), |mut acc, chunk| async move {
    +        acc.extend_from_slice(&chunk);
    +        Ok(acc)
    +    })
    +    .await
    +}
    +
    +impl extenddb_backup_store::BackupStore for MemStore {
    +    fn put<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: extenddb_backup_store::ByteStream,
    +    ) -> BoxedFuture<'a, Result>
    +    {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = collect_body(body).await?;
    +            self.puts.lock().unwrap().push(key.to_owned());
    +            let size = bytes.len() as u64;
    +            self.insert(key, &bytes);
    +            Ok(extenddb_backup_store::ObjectMeta {
    +                key: key.to_owned(),
    +                size,
    +                last_modified: None,
    +                etag: None,
    +            })
    +        })
    +    }
    +
    +    fn put_if_absent<'a>(
    +        &'a self,
    +        key: &'a str,
    +        body: extenddb_backup_store::ByteStream,
    +    ) -> BoxedFuture<
    +        'a,
    +        Result, extenddb_backup_store::StoreError>,
    +    > {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = collect_body(body).await?;
    +            if self
    +                .key_taken_by_a_non_object
    +                .load(std::sync::atomic::Ordering::Acquire)
    +            {
    +                return Ok(None);
    +            }
    +            let mut objects = self.objects.lock().unwrap();
    +            if objects.contains_key(key) {
    +                return Ok(None);
    +            }
    +            self.puts.lock().unwrap().push(key.to_owned());
    +            let size = bytes.len() as u64;
    +            objects.insert(key.to_owned(), bytes);
    +            if let Some(other) = self.overwrite_after_conditional_put.lock().unwrap().clone() {
    +                objects.insert(key.to_owned(), other);
    +            }
    +            Ok(Some(extenddb_backup_store::ObjectMeta {
    +                key: key.to_owned(),
    +                size,
    +                last_modified: None,
    +                etag: None,
    +            }))
    +        })
    +    }
    +
    +    fn get<'a>(
    +        &'a self,
    +        key: &'a str,
    +    ) -> BoxedFuture<'a, Result>
    +    {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            let bytes = self
    +                .objects
    +                .lock()
    +                .unwrap()
    +                .get(key)
    +                .cloned()
    +                .ok_or(extenddb_backup_store::StoreError::NotFound)?;
    +            Ok(extenddb_backup_store::byte_stream_from(bytes))
    +        })
    +    }
    +
    +    fn head<'a>(
    +        &'a self,
    +        key: &'a str,
    +    ) -> BoxedFuture<
    +        'a,
    +        Result, extenddb_backup_store::StoreError>,
    +    > {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            Ok(self.objects.lock().unwrap().get(key).map(|bytes| {
    +                extenddb_backup_store::ObjectMeta {
    +                    key: key.to_owned(),
    +                    size: bytes.len() as u64,
    +                    last_modified: None,
    +                    etag: None,
    +                }
    +            }))
    +        })
    +    }
    +
    +    fn list<'a>(
    +        &'a self,
    +        prefix: &'a str,
    +    ) -> BoxStream<'a, Result>
    +    {
    +        if let Err(e) = self.check() {
    +            return Box::pin(futures::stream::once(async move { Err(e) }));
    +        }
    +        let metas: Vec<_> = self
    +            .objects
    +            .lock()
    +            .unwrap()
    +            .iter()
    +            .filter(|(k, _)| Self::matches(k, prefix))
    +            .map(|(k, v)| {
    +                Ok(extenddb_backup_store::ObjectMeta {
    +                    key: k.clone(),
    +                    size: v.len() as u64,
    +                    last_modified: None,
    +                    etag: None,
    +                })
    +            })
    +            .collect();
    +        Box::pin(futures::stream::iter(metas))
    +    }
    +
    +    fn delete_prefix<'a>(
    +        &'a self,
    +        prefix: &'a str,
    +    ) -> BoxedFuture<'a, Result> {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            self.deletes.lock().unwrap().push(prefix.to_owned());
    +            let mut objects = self.objects.lock().unwrap();
    +            let before = objects.len();
    +            objects.retain(|k, _| !Self::matches(k, prefix));
    +            let deleted = (before - objects.len()) as u64;
    +            if let Some(late) = self.late_put_after_delete.lock().unwrap().as_deref()
    +                && !prefix.ends_with(".json")
    +            {
    +                objects.insert(format!("{prefix}{late}"), b"late".to_vec());
    +            }
    +            Ok(deleted)
    +        })
    +    }
    +
    +    fn validate(&self) -> BoxedFuture<'_, Result<(), extenddb_backup_store::StoreError>> {
    +        Box::pin(async move {
    +            self.stall().await;
    +            self.check()?;
    +            match self.validate_error.lock().unwrap().clone() {
    +                Some(text) => Err(extenddb_backup_store::StoreError::Other(text)),
    +                None => Ok(()),
    +            }
    +        })
    +    }
    +}
    +
    +/// A runtime with `store` configured as `identity`, healthy.
    +fn runtime_on(mem: &Arc, identity: StoreIdentity) -> BackupRuntime {
    +    runtime().with_store(ConfiguredStore::new(
    +        identity.store_kind,
    +        identity.store_root.clone(),
    +        Arc::clone(mem) as Arc,
    +        None,
    +        Ok(identity),
    +    ))
    +}
    +
    +/// A runtime whose configured store (`store()`'s kind and root) failed its
    +/// startup check with `problem`: the id is unknown.
    +fn runtime_unhealthy(mem: &Arc, problem: StoreProblem) -> BackupRuntime {
    +    runtime().with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(mem) as Arc,
    +        None,
    +        Err(problem),
    +    ))
    +}
    +
    +/// The identity file of `store()` for catalog `catalog_id`, as bytes.
    +fn identity_file(store_id: &str, catalog_id: &str) -> Vec {
    +    StoreIdentityFile {
    +        format_version: 1,
    +        store_id: store_id.to_owned(),
    +        catalog_id: catalog_id.to_owned(),
    +        backend: "test".to_owned(),
    +        created_at_ms: 0,
    +    }
    +    .to_bytes()
    +}
    +
     /// A catalog row for `arn`, with the manifest fixture's source table (the
     /// `Music` table of account 123456789012).
     fn row(arn: &str, status: BackupStatus) -> BackupRow {
    @@ -168,7 +516,16 @@ fn row(arn: &str, status: BackupStatus) -> BackupRow {
     
     impl BackupCatalog for Fake {
         fn catalog_id(&self) -> BoxedFuture<'_, Result> {
    -        Box::pin(async { Ok("catalog-1".to_owned()) })
    +        if *self.catalog_id_error.lock().unwrap() {
    +            return Box::pin(async { Err(StorageError::Internal("pool exhausted".to_owned())) });
    +        }
    +        let id = self
    +            .catalog_id
    +            .lock()
    +            .unwrap()
    +            .clone()
    +            .unwrap_or_else(|| "catalog-1".to_owned());
    +        Box::pin(async move { Ok(id) })
         }
     
         fn claim_backup<'a>(
    @@ -178,12 +535,58 @@ impl BackupCatalog for Fake {
             unused()
         }
     
    +    /// The backend's listing: rows of the store (kind, root and id), of the
    +    /// status when given, in (`created_at`, `backup_arn`) order.
         fn list_backup_rows<'a>(
             &'a self,
    -        _status: Option,
    -        _store: &'a StoreIdentity,
    +        status: Option,
    +        store: &'a StoreIdentity,
         ) -> BoxStream<'a, Result> {
    -        Box::pin(futures::stream::empty())
    +        if let Some(e) = self.list_error.lock().unwrap().clone() {
    +            return Box::pin(futures::stream::once(async move {
    +                Err(StorageError::Internal(e))
    +            }));
    +        }
    +        let mut rows: Vec = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|r| r.store == *store)
    +            .filter(|r| status.is_none_or(|s| r.status == s))
    +            .cloned()
    +            .collect();
    +        rows.sort_by(|a, b| {
    +            a.created_at
    +                .total_cmp(&b.created_at)
    +                .then_with(|| a.backup_arn.cmp(&b.backup_arn))
    +        });
    +        Box::pin(futures::stream::iter(rows.into_iter().map(Ok)))
    +    }
    +
    +    /// The last AVAILABLE row of the store in the listing's order, read as
    +    /// one row; `list_error` fails it as it fails the listing.
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxedFuture<'a, Result, StorageError>> {
    +        self.newest_reads.lock().unwrap().push(store.clone());
    +        if let Some(e) = self.list_error.lock().unwrap().clone() {
    +            return Box::pin(async move { Err(StorageError::Internal(e)) });
    +        }
    +        let newest = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|r| r.store == *store && r.status == BackupStatus::Available)
    +            .max_by(|a, b| {
    +                a.created_at
    +                    .total_cmp(&b.created_at)
    +                    .then_with(|| a.backup_arn.cmp(&b.backup_arn))
    +            })
    +            .cloned();
    +        Box::pin(async move { Ok(newest) })
         }
     
         fn list_account_backup_rows<'a>(
    @@ -235,12 +638,17 @@ impl BackupCatalog for Fake {
                 .unwrap()
                 .get(&key.backup_arn)
                 .cloned();
    -        let status = self
    -            .rows
    -            .lock()
    -            .unwrap()
    -            .get(&key.backup_arn)
    -            .map(|r| r.status);
    +        let mut rows = self.rows.lock().unwrap();
    +        let status = rows.get(&key.backup_arn).map(|r| r.status);
    +        // The natural answer moves an AVAILABLE row to DELETING, as the
    +        // backends do; a preset answer leaves the row as the test set it.
    +        if answer.is_none()
    +            && status == Some(BackupStatus::Available)
    +            && let Some(row) = rows.get_mut(&key.backup_arn)
    +        {
    +            row.status = BackupStatus::Deleting;
    +        }
    +        drop(rows);
             Box::pin(async move {
                 match (answer, status) {
                     (Some(answer), _) => Ok(answer),
    @@ -251,18 +659,105 @@ impl BackupCatalog for Fake {
             })
         }
     
    +    /// DELETING to DELETED, conditional, as the backends do.
         fn set_backup_deleted<'a>(
             &'a self,
    -        _key: &'a BackupKey,
    +        key: &'a BackupKey,
         ) -> BoxedFuture<'a, Result> {
    -        unused()
    +        let moved = match self.rows.lock().unwrap().get_mut(&key.backup_arn) {
    +            Some(row) if row.status == BackupStatus::Deleting => {
    +                row.status = BackupStatus::Deleted;
    +                true
    +            }
    +            _ => false,
    +        };
    +        if moved {
    +            self.set_deleted
    +                .lock()
    +                .unwrap()
    +                .push(key.backup_arn.clone());
    +        }
    +        Box::pin(async move { Ok(moved) })
         }
     
    +    /// Remove a DELETED or FAILED tombstone, as the backends do.
         fn remove_backup_row<'a>(
             &'a self,
    -        _key: &'a BackupKey,
    +        key: &'a BackupKey,
         ) -> BoxedFuture<'a, Result> {
    -        unused()
    +        let mut rows = self.rows.lock().unwrap();
    +        let removed = match rows.get(&key.backup_arn) {
    +            Some(row) if matches!(row.status, BackupStatus::Deleted | BackupStatus::Failed) => {
    +                rows.remove(&key.backup_arn);
    +                true
    +            }
    +            _ => false,
    +        };
    +        if removed {
    +            self.removed_rows
    +                .lock()
    +                .unwrap()
    +                .push(key.backup_arn.clone());
    +        }
    +        Box::pin(async move { Ok(removed) })
    +    }
    +
    +    /// Remove a CREATING row whose heartbeat is older than `stale_after`
    +    /// (on this process's clock, which stands for the catalog's here).
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxedFuture<'a, Result> {
    +        let now = std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_secs_f64();
    +        let mut rows = self.rows.lock().unwrap();
    +        let removed = match rows.get(&key.backup_arn) {
    +            Some(row)
    +                if row.status == BackupStatus::Creating
    +                    && row.heartbeat_at < now - stale_after.as_secs_f64() =>
    +            {
    +                rows.remove(&key.backup_arn);
    +                true
    +            }
    +            _ => false,
    +        };
    +        Box::pin(async move { Ok(removed) })
    +    }
    +
    +    /// The backends' listing on the named clock, the catalog's being this
    +    /// process's moved by `catalog_clock_offset_secs`: rows of the store
    +    /// with `status` whose `heartbeat_at`, or `completed_at` when set, is
    +    /// older than `age`.
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result> {
    +        let cutoff = seconds_since_epoch() + *self.catalog_clock_offset_secs.lock().unwrap()
    +            - age.as_secs_f64();
    +        let mut rows: Vec = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|r| r.store == *store && r.status == status)
    +            .filter(|r| match clock {
    +                StaleClock::Heartbeat => r.heartbeat_at < cutoff,
    +                StaleClock::Completed => r.completed_at.is_some_and(|c| c < cutoff),
    +            })
    +            .cloned()
    +            .collect();
    +        rows.sort_by(|a, b| {
    +            a.created_at
    +                .total_cmp(&b.created_at)
    +                .then_with(|| a.backup_arn.cmp(&b.backup_arn))
    +        });
    +        Box::pin(futures::stream::iter(rows.into_iter().map(Ok)))
         }
     
         fn backup_in_use<'a>(
    @@ -290,6 +785,23 @@ impl BackupCatalog for Fake {
             Box::pin(async move { Ok(jobs) })
         }
     
    +    /// The RUNNING records of `dead_jobs` and every record of
    +    /// `running_jobs`, by status, whatever their heartbeat.
    +    fn list_running_restore_jobs(
    +        &self,
    +    ) -> BoxedFuture<'_, Result, StorageError>> {
    +        let mut jobs: Vec = self
    +            .dead_jobs
    +            .lock()
    +            .unwrap()
    +            .iter()
    +            .filter(|j| j.status == extenddb_storage::backup_catalog::JobStatus::Running)
    +            .cloned()
    +            .collect();
    +        jobs.extend(self.running_jobs.lock().unwrap().iter().cloned());
    +        Box::pin(async move { Ok(jobs) })
    +    }
    +
         /// Matches as a backend does: only a record still RUNNING under the
         /// listed `owner_generation` and `heartbeat_at` (the record's current
         /// values are those of its `dead_jobs` entry, moved by
    @@ -320,9 +832,41 @@ impl BackupCatalog for Fake {
             Box::pin(async move { Ok(answer) })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxedFuture<'_, Result> {
    -        *self.maintenance_checks.lock().unwrap() += 1;
    -        let held = *self.maintenance_held.lock().unwrap();
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxedFuture<'a, Result, StorageError>> {
    +        let mut ids: Vec = self
    +            .rows
    +            .lock()
    +            .unwrap()
    +            .values()
    +            .filter(|row| row.store.store_kind == store_kind && row.store.store_root == store_root)
    +            .map(|row| row.store.store_id.clone())
    +            .collect();
    +        ids.sort();
    +        ids.dedup();
    +        Box::pin(async move { Ok(ids) })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        _holder: &'a str,
    +        _stale_after: Duration,
    +    ) -> BoxedFuture<'a, Result, StorageError>> {
    +        unused()
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxedFuture<'_, Result> {
    +        self.maintenance_checks.lock().unwrap().push(stale_after);
    +        let held = match *self.maintenance_beat_age.lock().unwrap() {
    +            Some(age) => age < stale_after,
    +            None => *self.maintenance_held.lock().unwrap(),
    +        };
             let error = self.maintenance_error.lock().unwrap().clone();
             Box::pin(async move {
                 match error {
    @@ -670,13 +1214,22 @@ fn restore_body(extra: Value) -> Value {
     }
     
     async fn restore(fake: &Fake, body: Value) -> Result {
    +    restore_on(fake, &runtime(), &body).await
    +}
    +
    +/// `restore` on `runtime` rather than the tests' default one.
    +async fn restore_on(
    +    fake: &Fake,
    +    runtime: &BackupRuntime,
    +    body: &Value,
    +) -> Result {
         restore_table_from_backup(
             fake,
    -        &runtime(),
    +        runtime,
             ACCOUNT,
             REGION,
             &LimitsConfig::default(),
    -        &body,
    +        body,
         )
         .await
         .map(|(_, v)| v)
    @@ -3701,7 +4254,7 @@ fn fake_with_row(status: BackupStatus) -> Fake {
     }
     
     fn runtime_with_store() -> BackupRuntime {
    -    runtime().with_store(store())
    +    runtime_on(&Arc::new(MemStore::default()), store())
     }
     
     /// DescribeBackup answers from the catalog row when one exists, derived from
    @@ -4320,7 +4873,13 @@ async fn restore_over_a_row_refuses_before_the_permit() {
             err
         }
         let configured = BackupRuntime::new(1)
    -        .with_store(store())
    +        .with_store(ConfiguredStore::new(
    +            store().store_kind,
    +            store().store_root,
    +            Arc::new(MemStore::default()) as Arc,
    +            None,
    +            Ok(store()),
    +        ))
             .with_permit_wait(Duration::ZERO);
         let _held_configured = JobPermit::try_acquire(configured.slots()).unwrap();
         let none = BackupRuntime::new(1).with_permit_wait(Duration::ZERO);
    @@ -4372,10 +4931,16 @@ async fn restore_over_a_row_refuses_before_the_permit() {
         // A configured store that is not the row's: the row is of another store,
         // not of "the" store because one is configured.
         let other = BackupRuntime::new(1)
    -        .with_store(StoreIdentity {
    -            store_id: "store-2".to_owned(),
    -            ..store()
    -        })
    +        .with_store(ConfiguredStore::new(
    +            store().store_kind,
    +            store().store_root,
    +            Arc::new(MemStore::default()) as Arc,
    +            None,
    +            Ok(StoreIdentity {
    +                store_id: "store-2".to_owned(),
    +                ..store()
    +            }),
    +        ))
             .with_permit_wait(Duration::ZERO);
         let _held_other = JobPermit::try_acquire(other.slots()).unwrap();
         let msg = validation_message(check(fake_with_row(BackupStatus::Available), &other).await);
    @@ -4873,74 +5438,540 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting
         assert_eq!(fake.row_reads.lock().unwrap().len(), 1, "no further read");
     }
     
    -// --- the reconciler -------------------------------------------------------------
    -
    -fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow {
    -    RestoreJobRow {
    -        target: RestoreTargetKey {
    -            account_id: ACCOUNT.to_owned(),
    -            table_name: name.to_owned(),
    -            table_id: format!("tid-{name}"),
    -        },
    -        source_backup_arn: arn(ACCOUNT),
    -        kind,
    -        status: extenddb_storage::backup_catalog::JobStatus::Running,
    -        owner_generation: 0,
    -        heartbeat_at: 0.0,
    -    }
    -}
    -
    -/// A legacy restore whose owner is dead is marked FAILED and its target
    -/// removed; a manifest restore is left for its resumer; an abandoned
    -/// in-catalog CREATING backup is removed through the reader's delete.
    +/// While a maintenance command holds the catalog's claim, CreateBackup and
    +/// RestoreTableFromBackup are refused with LimitExceeded naming a maintenance command
    +/// (generically: every maintenance command holds the same claim),
    +/// after every other check (a missing table is still TableNotFound) and
    +/// before the permit (no slot is taken, and the backend is not asked); once
    +/// the claim is gone both run.
     #[tokio::test]
    -async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() {
    -    use extenddb_storage::backup_catalog::RestoreKind;
    -    let fake = Fake::default();
    -    *fake.target_still_creating.lock().unwrap() = true;
    -    *fake.record_still_running.lock().unwrap() = true;
    -    fake.dead_jobs.lock().unwrap().extend([
    -        dead_job(RestoreKind::Legacy, "LegacyTarget"),
    -        dead_job(RestoreKind::Manifest, "ManifestTarget"),
    -    ]);
    -    fake.abandoned_backups
    -        .lock()
    -        .unwrap()
    -        .push(AbandonedLegacyBackup {
    -            account_id: ACCOUNT.to_owned(),
    -            backup_arn: arn(ACCOUNT),
    -            removed: false,
    -        });
    -    fake.descriptions
    -        .lock()
    -        .unwrap()
    -        .insert(arn(ACCOUNT), backup_description("CREATING", "PROVISIONED"));
    -
    -    let report = reconciler::cycle(
    +async fn jobs_are_refused_while_a_maintenance_claim_is_held() {
    +    let fake = Fake::with_source(source(Some(definition("PROVISIONED"))));
    +    *fake.maintenance_held.lock().unwrap() = true;
    +    let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7));
    +    let err = create_backup(
             &fake,
    -        Duration::from_secs(300),
    -        &mut reconciler::ReconcilerState::default(),
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
         )
         .await
    -    .unwrap();
    -    assert!(!report.maintenance_held);
    -    assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    -    assert!(report.failed_targets_removed.is_empty());
    -    assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]);
    -    assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]);
    -    assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]);
    -
    -    let failed = fake.failed_jobs.lock().unwrap().clone();
    -    assert_eq!(failed.len(), 1);
    -    assert_eq!(failed[0].0.table_name, "LegacyTarget");
    -    assert_eq!(failed[0].1, reconciler::OWNER_DEAD);
    -    assert_eq!(
    -        fake.removed_targets.lock().unwrap().as_slice(),
    -        [dead_job(RestoreKind::Legacy, "LegacyTarget").target]
    +    .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m)
    +            if m.contains("A backup maintenance command holds") && !m.contains("purge")),
    +        "{err:?}"
         );
    -    assert_eq!(fake.deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    -}
    -
    +    assert!(fake.creates.lock().unwrap().is_empty());
    +    assert_eq!(runtime.free_slots(), 1, "no slot was taken");
    +    let err = create_backup(
    +        &fake,
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Err(StorageError::TableNotFound("Music".to_owned())),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(err, DynamoDbError::TableNotFoundException(_)),
    +        "the table check comes first: {err:?}"
    +    );
    +
    +    let err = restore_on(&fake, &runtime, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m)
    +            if m.contains("A backup maintenance command holds") && !m.contains("purge")),
    +        "{err:?}"
    +    );
    +    assert!(fake.restores().is_empty(), "the backend was not asked");
    +    assert_eq!(runtime.free_slots(), 1);
    +    assert_eq!(
    +        *fake.maintenance_checks.lock().unwrap(),
    +        vec![Duration::from_secs(7); 2],
    +        "the claim is read with stale_job_after_secs"
    +    );
    +
    +    *fake.maintenance_held.lock().unwrap() = false;
    +    create_backup(
    +        &fake,
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap();
    +    fake.permits.lock().unwrap().clear();
    +    restore_on(&fake, &runtime, &restore_body(json!({})))
    +        .await
    +        .unwrap();
    +    assert_eq!(fake.creates.lock().unwrap().len(), 1);
    +    assert_eq!(fake.restores().len(), 1);
    +
    +    // Before the permit: with every slot held and the claim held, the
    +    // refusal names the maintenance claim, not the slot limit, so the check cannot
    +    // have run after `take_permit`.
    +    *fake.maintenance_held.lock().unwrap() = true;
    +    let full = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7));
    +    let held_slot = full.take_permit().await.unwrap();
    +    assert_eq!(full.free_slots(), 0, "every slot is held");
    +    let err = create_backup(
    +        &fake,
    +        &full,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m)
    +            if m.contains("A backup maintenance command holds") && !m.contains("purge")),
    +        "the claim is checked before the permit: {err:?}"
    +    );
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::LimitExceededException(m)
    +            if m.contains("A backup maintenance command holds") && !m.contains("purge")),
    +        "the claim is checked before the permit: {err:?}"
    +    );
    +    drop(held_slot);
    +}
    +
    +// --- purge: the wait and the row deletion (`maintenance`) ----------------------
    +
    +/// A maintenance claim handle for `HeldClaim`: heartbeats confirm until
    +/// `lost` is set.
    +#[derive(Default)]
    +struct FakeHolder {
    +    lost: Mutex,
    +    released: Mutex,
    +}
    +
    +impl extenddb_storage::MaintenanceClaim for FakeHolder {
    +    fn generation(&self) -> u64 {
    +        11
    +    }
    +    fn heartbeat(
    +        &self,
    +    ) -> BoxedFuture<'_, Result> {
    +        let lost = *self.lost.lock().unwrap();
    +        Box::pin(async move {
    +            Ok(if lost {
    +                extenddb_storage::backup_catalog::Heartbeat::Lost
    +            } else {
    +                extenddb_storage::backup_catalog::Heartbeat::Confirmed
    +            })
    +        })
    +    }
    +    fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> {
    +        *self.released.lock().unwrap() = true;
    +        Box::pin(async { Ok(()) })
    +    }
    +}
    +
    +/// A forwarding handle, so the test keeps its own reference to the holder.
    +struct Forward(Arc);
    +
    +impl extenddb_storage::MaintenanceClaim for Forward {
    +    fn generation(&self) -> u64 {
    +        self.0.generation()
    +    }
    +    fn heartbeat(
    +        &self,
    +    ) -> BoxedFuture<'_, Result> {
    +        self.0.heartbeat()
    +    }
    +    fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> {
    +        self.0.release()
    +    }
    +}
    +
    +fn held(holder: &Arc) -> maintenance::HeldClaim {
    +    maintenance::HeldClaim::start(
    +        Box::new(Forward(Arc::clone(holder))),
    +        Duration::from_secs(1),
    +    )
    +}
    +
    +fn seconds_since_epoch() -> f64 {
    +    std::time::SystemTime::now()
    +        .duration_since(std::time::UNIX_EPOCH)
    +        .unwrap()
    +        .as_secs_f64()
    +}
    +
    +/// A row of `store()` under ARN `arn` in `status`, with a manifest and a
    +/// data file under its prefix in `mem`.
    +fn seed(fake: &Fake, mem: &MemStore, arn: &str, status: BackupStatus) {
    +    let prefix = BackupArn::parse(arn).unwrap().store_prefix();
    +    mem.insert(&format!("{prefix}extenddb-manifest.json"), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"x");
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arn.to_owned(), row(arn, status));
    +}
    +
    +fn arn_n(n: u32) -> String {
    +    format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/0179197643143{n}-0000000{n}")
    +}
    +
    +/// The states a row can be in when purge reaches it after reading it
    +/// AVAILABLE, a DeleteBackup having run meanwhile: still AVAILABLE; moved to
    +/// DELETING (the other deleter mid-way); moved to DELETED (the other
    +/// finished); gone (finished and its tombstone removed). Each ends with no
    +/// row and no object, counted as deleted; a status that cannot follow
    +/// AVAILABLE (FAILED) is the one case left for a look. A tombstone read
    +/// and removed meanwhile (a second purge) counts as deleted too.
    +#[tokio::test]
    +async fn purge_finishes_a_row_another_deleter_moved_or_removed_meanwhile() {
    +    let fake = Fake::default();
    +    let mem = MemStore::default();
    +    let arns: Vec = (1..=6).map(arn_n).collect();
    +    for arn in &arns[..5] {
    +        seed(&fake, &mem, arn, BackupStatus::Available);
    +    }
    +    // A DELETED tombstone the command read, whose prefix is already empty.
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arns[5].clone(), row(&arns[5], BackupStatus::Deleted));
    +    // The command's read under the claim.
    +    let read: Vec = fake
    +        .list_backup_rows(None, &store())
    +        .try_collect()
    +        .await
    +        .unwrap();
    +    assert_eq!(read.len(), 6);
    +    // Between the read and the rows' turns, DeleteBackup moves rows 2 and 3
    +    // and finishes 4 (its tombstone removed too); a writer moves 5 to
    +    // FAILED, which no DeleteBackup does; the tombstone 6 is removed by a
    +    // second purge.
    +    {
    +        let mut rows = fake.rows.lock().unwrap();
    +        rows.get_mut(&arns[1]).unwrap().status = BackupStatus::Deleting;
    +        rows.get_mut(&arns[2]).unwrap().status = BackupStatus::Deleted;
    +        rows.remove(&arns[3]);
    +        rows.get_mut(&arns[4]).unwrap().status = BackupStatus::Failed;
    +        rows.remove(&arns[5]);
    +    }
    +    // The deleter that finished rows 3 and 4 deleted their objects.
    +    for arn in [&arns[2], &arns[3]] {
    +        let prefix = BackupArn::parse(arn).unwrap().store_prefix();
    +        extenddb_backup_store::BackupStore::delete_prefix(&mem, &prefix)
    +            .await
    +            .unwrap();
    +    }
    +
    +    let mut outcomes = Vec::new();
    +    for row in &read {
    +        outcomes.push(maintenance::purge_row(&fake, &mem, row, Duration::from_secs(300)).await);
    +    }
    +    assert_eq!(
    +        outcomes[0],
    +        Ok(()),
    +        "still AVAILABLE: moved and deleted here"
    +    );
    +    assert_eq!(
    +        outcomes[1],
    +        Ok(()),
    +        "moved to DELETING by another deleter: finished here"
    +    );
    +    assert_eq!(
    +        outcomes[2],
    +        Ok(()),
    +        "moved to DELETED by another deleter: the tombstone removed here, counted as deleted"
    +    );
    +    assert_eq!(outcomes[3], Ok(()), "gone: counted as deleted");
    +    assert_eq!(
    +        outcomes[4],
    +        Err("its status changed to FAILED".to_owned()),
    +        "a status that cannot follow AVAILABLE is left"
    +    );
    +    assert_eq!(
    +        outcomes[5],
    +        Ok(()),
    +        "a tombstone removed meanwhile: counted as deleted"
    +    );
    +    let left: Vec = fake.rows.lock().unwrap().keys().cloned().collect();
    +    assert_eq!(left, vec![arns[4].clone()], "no row of the five remains");
    +    let failed_prefix = BackupArn::parse(&arns[4]).unwrap().store_prefix();
    +    assert_eq!(
    +        mem.keys(),
    +        vec![
    +            format!("{failed_prefix}data/00001.json.gz"),
    +            format!("{failed_prefix}extenddb-manifest.json"),
    +        ],
    +        "only the left row's objects remain"
    +    );
    +    // The row moved to DELETED was never moved by purge (DELETING to
    +    // DELETED is the other deleter's write); the one left DELETING and the
    +    // one still AVAILABLE were finished by purge; the three tombstones were
    +    // removed here.
    +    assert_eq!(
    +        *fake.set_deleted.lock().unwrap(),
    +        vec![arns[0].clone(), arns[1].clone()]
    +    );
    +    assert_eq!(
    +        *fake.removed_rows.lock().unwrap(),
    +        vec![arns[0].clone(), arns[1].clone(), arns[2].clone()]
    +    );
    +}
    +
    +/// A CREATING row with a dead owner goes through the dead-owner removal and
    +/// then its prefix; one whose heartbeat is young is left with its objects
    +/// under the live window, and removed under the zero window SQLite uses.
    +#[tokio::test]
    +async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() {
    +    let fake = Fake::default();
    +    let mem = MemStore::default();
    +    let dead = arn_n(1);
    +    let live = arn_n(2);
    +    seed(&fake, &mem, &dead, BackupStatus::Creating);
    +    seed(&fake, &mem, &live, BackupStatus::Creating);
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .get_mut(&live)
    +        .unwrap()
    +        .heartbeat_at = seconds_since_epoch();
    +    let window = Duration::from_secs(300);
    +    let dead_row = fake.rows.lock().unwrap()[&dead].clone();
    +    let live_row = fake.rows.lock().unwrap()[&live].clone();
    +    assert_eq!(
    +        maintenance::purge_row(&fake, &mem, &dead_row, window).await,
    +        Ok(())
    +    );
    +    let err = maintenance::purge_row(&fake, &mem, &live_row, window)
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        err.contains("the CREATING row could not be removed"),
    +        "{err}"
    +    );
    +    let left: Vec = fake.rows.lock().unwrap().keys().cloned().collect();
    +    assert_eq!(left, vec![live.clone()]);
    +    assert_eq!(mem.keys().len(), 2, "the live row's objects stay");
    +    // A heartbeat written a moment ago is older than a zero window once a
    +    // moment has passed.
    +    tokio::time::sleep(Duration::from_millis(20)).await;
    +    assert_eq!(
    +        maintenance::purge_row(&fake, &mem, &live_row, Duration::ZERO).await,
    +        Ok(())
    +    );
    +    assert!(fake.rows.lock().unwrap().is_empty());
    +    assert!(mem.keys().is_empty());
    +}
    +
    +/// The wait's five outcomes are told apart: quiet, also with a FAILED
    +/// restore record whose target is still CREATING (the reconciler's duty,
    +/// not a running job) and a FAILED backup row whose owner is dead; busy at
    +/// the limit, with the jobs named (a CREATING row and a FAILED row with a
    +/// live heartbeat, a RUNNING restore) and announced once; a signal, during
    +/// a sleep and before the first read; the catalog unreadable; the claim
    +/// lost.
    +#[tokio::test(start_paused = true)]
    +async fn the_wait_reports_each_outcome_as_itself() {
    +    use extenddb_storage::backup_catalog::{JobStatus, RestoreKind};
    +    use maintenance::{Interrupt, Wait, WaitOutcome, wait_for_quiet};
    +    let fake = Fake::default();
    +    let holder = Arc::new(FakeHolder::default());
    +    let claim = held(&holder);
    +    let limit = Duration::from_secs(20);
    +    let wait = Wait {
    +        owner_window: Duration::from_secs(10),
    +        limit,
    +        poll: Duration::from_secs(2),
    +    };
    +    let announced: Mutex>> = Mutex::new(Vec::new());
    +    let announce = |jobs: &[String]| announced.lock().unwrap().push(jobs.to_vec());
    +
    +    let (_tx, mut quiet) = Interrupt::new();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Quiet
    +    );
    +    // A FAILED record whose target is still CREATING is listed by the
    +    // zero-window dead-job listing with the RUNNING ones; it is the
    +    // reconciler's to finish and no job runs under it. A FAILED backup row
    +    // whose heartbeat is older than the window has a dead owner.
    +    let mut failed_record = dead_job(RestoreKind::Legacy, "FailedTarget");
    +    failed_record.status = JobStatus::Failed;
    +    fake.dead_jobs.lock().unwrap().push(failed_record);
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .insert(arn_n(3), row(&arn_n(3), BackupStatus::Failed));
    +    let started = tokio::time::Instant::now();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Quiet,
    +        "a FAILED restore record and a FAILED row with a dead owner are not waited on"
    +    );
    +    assert_eq!(started.elapsed(), Duration::ZERO, "quiet at the first read");
    +    assert!(announced.lock().unwrap().is_empty());
    +
    +    // Busy: a CREATING row and a FAILED row whose heartbeats are young, and
    +    // a RUNNING restore.
    +    for (n, status) in [(1, BackupStatus::Creating), (2, BackupStatus::Failed)] {
    +        fake.rows.lock().unwrap().insert(arn_n(n), {
    +            let mut r = row(&arn_n(n), status);
    +            r.heartbeat_at = seconds_since_epoch();
    +            r
    +        });
    +    }
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "Target"));
    +    let started = tokio::time::Instant::now();
    +    let busy = wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await;
    +    let expected = vec![
    +        format!(
    +            "CreateBackup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            arn_n(1)
    +        ),
    +        format!(
    +            "the FAILED backup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            arn_n(2)
    +        ),
    +        format!("a restore into Target (account {ACCOUNT}) is RUNNING"),
    +    ];
    +    assert_eq!(busy, WaitOutcome::Busy(expected.clone()));
    +    assert!(started.elapsed() >= limit, "{:?}", started.elapsed());
    +    assert_eq!(*announced.lock().unwrap(), vec![expected]);
    +
    +    // A signal during a sleep ends the wait before the limit.
    +    let (tx, mut interrupted) = Interrupt::new();
    +    tokio::spawn(async move {
    +        tokio::time::sleep(Duration::from_secs(3)).await;
    +        let _ = tx.send(Some("SIGTERM"));
    +    });
    +    let started = tokio::time::Instant::now();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut interrupted, announce).await,
    +        WaitOutcome::Interrupted("SIGTERM")
    +    );
    +    assert!(
    +        started.elapsed() < Duration::from_millis(3_500),
    +        "the signal ends the sleep, not the next poll: {:?}",
    +        started.elapsed()
    +    );
    +    // One already received ends it before the first read.
    +    assert_eq!(interrupted.signalled(), Some("SIGTERM"));
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut interrupted, announce).await,
    +        WaitOutcome::Interrupted("SIGTERM")
    +    );
    +
    +    // The catalog unreadable.
    +    *fake.list_error.lock().unwrap() = Some("pool exhausted".to_owned());
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::CatalogError(
    +            "cannot list the catalog's backup rows: Internal error: pool exhausted".to_owned()
    +        )
    +    );
    +    *fake.list_error.lock().unwrap() = None;
    +
    +    // The claim lost: the next heartbeat records it, and the wait stops.
    +    *holder.lost.lock().unwrap() = true;
    +    tokio::time::sleep(Duration::from_millis(1_500)).await;
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, announce).await,
    +        WaitOutcome::Lost(
    +            "the maintenance claim was lost (another holder took it over)".to_owned()
    +        )
    +    );
    +    claim.release().await.unwrap();
    +    assert!(*holder.released.lock().unwrap());
    +}
    +
    +// --- the reconciler -------------------------------------------------------------
    +
    +fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow {
    +    RestoreJobRow {
    +        target: RestoreTargetKey {
    +            account_id: ACCOUNT.to_owned(),
    +            table_name: name.to_owned(),
    +            table_id: format!("tid-{name}"),
    +        },
    +        source_backup_arn: arn(ACCOUNT),
    +        kind,
    +        status: extenddb_storage::backup_catalog::JobStatus::Running,
    +        owner_generation: 0,
    +        heartbeat_at: 0.0,
    +    }
    +}
    +
    +/// A legacy restore whose owner is dead is marked FAILED and its target
    +/// removed; a manifest restore is left for its resumer; an abandoned
    +/// in-catalog CREATING backup is removed through the reader's delete.
    +#[tokio::test]
    +async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let fake = Fake::default();
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    *fake.record_still_running.lock().unwrap() = true;
    +    fake.dead_jobs.lock().unwrap().extend([
    +        dead_job(RestoreKind::Legacy, "LegacyTarget"),
    +        dead_job(RestoreKind::Manifest, "ManifestTarget"),
    +    ]);
    +    fake.abandoned_backups
    +        .lock()
    +        .unwrap()
    +        .push(AbandonedLegacyBackup {
    +            account_id: ACCOUNT.to_owned(),
    +            backup_arn: arn(ACCOUNT),
    +            removed: false,
    +        });
    +    fake.descriptions
    +        .lock()
    +        .unwrap()
    +        .insert(arn(ACCOUNT), backup_description("CREATING", "PROVISIONED"));
    +
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(!report.maintenance_held);
    +    assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    +    assert!(report.failed_targets_removed.is_empty());
    +    assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]);
    +    assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]);
    +    assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]);
    +
    +    let failed = fake.failed_jobs.lock().unwrap().clone();
    +    assert_eq!(failed.len(), 1);
    +    assert_eq!(failed[0].0.table_name, "LegacyTarget");
    +    assert_eq!(failed[0].1, reconciler::OWNER_DEAD);
    +    assert_eq!(
    +        fake.removed_targets.lock().unwrap().as_slice(),
    +        [dead_job(RestoreKind::Legacy, "LegacyTarget").target]
    +    );
    +    assert_eq!(fake.deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    +}
    +
     /// A record that is no longer RUNNING when the reconciler reaches it belongs
     /// to whoever moved it: the target is not removed.
     #[tokio::test]
    @@ -4955,7 +5986,9 @@ async fn the_reconciler_leaves_a_target_whose_record_moved() {
             .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -4987,7 +6020,9 @@ async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_fai
             .extend([dead.clone(), failed.clone()]);
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -5012,11 +6047,16 @@ async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_fai
     }
     
     /// The maintenance claim is checked first and reported; the duties that
    -/// resolve dead owners still run while it is held.
    +/// resolve dead owners and the DELETING rows of the configured store still
    +/// run while it is held (the command waits on the first and finishes the
    +/// second itself, so neither races it).
     #[tokio::test]
     async fn the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead_owners() {
         use extenddb_storage::backup_catalog::RestoreKind;
    -    let fake = Fake::default();
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
         *fake.maintenance_held.lock().unwrap() = true;
         *fake.target_still_creating.lock().unwrap() = true;
         *fake.record_still_running.lock().unwrap() = true;
    @@ -5026,17 +6066,140 @@ async fn the_reconciler_reports_a_held_maintenance_claim_and_still_resolves_dead
             .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
         let report = reconciler::cycle(
             &fake,
    +        &runtime_on(&mem, store()),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
         .unwrap();
         assert!(report.maintenance_held);
         assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    -}
    -
    -/// The worker runs a cycle at once (observed through the maintenance check,
    -/// the cycle's first read), records the worker metrics the other workers
    +    assert_eq!(
    +        report.deletions_finished,
    +        [arn(ACCOUNT)],
    +        "a DELETING row is finished while the claim is held"
    +    );
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +}
    +
    +/// The wait tells a live owner from a dead one on the catalog's clock, the
    +/// clock the owner's heartbeat was written on, and never on this host's:
    +/// with the catalog's clock 400 s behind this host's, a CREATING row whose
    +/// heartbeat is the catalog's now has a live owner although this host's
    +/// clock reads it 400 s old, past the 300 s window; with the catalog's clock
    +/// 400 s ahead, a row stamped with this host's now is 400 s old to the
    +/// catalog and its owner dead.
    +#[tokio::test(start_paused = true)]
    +async fn the_wait_reads_a_live_owner_on_the_catalogs_clock() {
    +    use maintenance::{Interrupt, Wait, WaitOutcome, wait_for_quiet};
    +    let fake = Fake::default();
    +    let holder = Arc::new(FakeHolder::default());
    +    let claim = held(&holder);
    +    let wait = Wait {
    +        owner_window: Duration::from_secs(300),
    +        limit: Duration::from_secs(4),
    +        poll: Duration::from_secs(2),
    +    };
    +    let (_tx, mut quiet) = Interrupt::new();
    +
    +    *fake.catalog_clock_offset_secs.lock().unwrap() = -400.0;
    +    fake.rows.lock().unwrap().insert(arn_n(1), {
    +        let mut r = row(&arn_n(1), BackupStatus::Creating);
    +        r.heartbeat_at = seconds_since_epoch() - 400.0;
    +        r
    +    });
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, |_| {}).await,
    +        WaitOutcome::Busy(vec![format!(
    +            "CreateBackup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            arn_n(1)
    +        )]),
    +        "an owner whose heartbeat is young on the catalog's clock is live"
    +    );
    +
    +    *fake.catalog_clock_offset_secs.lock().unwrap() = 400.0;
    +    fake.rows
    +        .lock()
    +        .unwrap()
    +        .get_mut(&arn_n(1))
    +        .unwrap()
    +        .heartbeat_at = seconds_since_epoch();
    +    assert_eq!(
    +        wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, |_| {}).await,
    +        WaitOutcome::Quiet,
    +        "an owner whose heartbeat is old on the catalog's clock is dead"
    +    );
    +}
    +
    +/// A cycle run by the holder of the maintenance claim with a zero dead-owner
    +/// window (`backup purge` on SQLite) takes the held branch without reading
    +/// the claim: the command's claim was heartbeated a millisecond ago, which
    +/// a zero window reads as stale, so a cycle that read it would take the
    +/// branch for a free claim. The same cycle without the holder's state reads
    +/// the claim through the window and finds it free; that is the read the
    +/// holder's state replaces.
    +#[tokio::test]
    +async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_window() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    *fake.maintenance_beat_age.lock().unwrap() = Some(Duration::from_millis(1));
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    *fake.record_still_running.lock().unwrap() = true;
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime_on(&mem, store()),
    +        Duration::ZERO,
    +        "test",
    +        &mut reconciler::ReconcilerState::for_maintenance_holder(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(
    +        report.maintenance_held,
    +        "the holder's cycle took the branch for a free claim"
    +    );
    +    assert!(
    +        fake.maintenance_checks.lock().unwrap().is_empty(),
    +        "the holder's cycle read the claim: {:?}",
    +        fake.maintenance_checks.lock().unwrap()
    +    );
    +    // The duties the command waits on still ran under the zero window.
    +    assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]);
    +    assert_eq!(report.deletions_finished, [arn(ACCOUNT)]);
    +
    +    // Without the holder's state the zero window reads the same claim as
    +    // free.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime_on(&mem, store()),
    +        Duration::ZERO,
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(!report.maintenance_held);
    +    assert_eq!(
    +        *fake.maintenance_checks.lock().unwrap(),
    +        [Duration::ZERO],
    +        "a default state reads the claim through the window it is given"
    +    );
    +}
    +
    +/// The worker runs a cycle at once (observed through the maintenance check,
    +/// the cycle's first read), records the worker metrics the other workers
     /// record, then waits an interval, and stops at the shutdown token.
     #[tokio::test]
     async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
    @@ -5047,10 +6210,12 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
         let shutdown = extenddb_storage::CancellationToken::new();
         let worker = tokio::spawn(reconciler::run(
             Arc::clone(&fake),
    +        Arc::new(runtime()),
             reconciler::ReconcilerSettings {
                 interval: Duration::from_secs(3600),
                 stale_job_after: Duration::from_secs(300),
             },
    +        "test".to_owned(),
             Arc::clone(&metrics),
             Arc::new(tokio::sync::Mutex::new(
                 reconciler::ReconcilerState::default(),
    @@ -5058,7 +6223,7 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
             shutdown.clone(),
         ));
         let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    -    while *fake.maintenance_checks.lock().unwrap() == 0 {
    +    while fake.maintenance_checks.lock().unwrap().is_empty() {
             assert!(
                 tokio::time::Instant::now() < deadline,
                 "the first cycle did not run"
    @@ -5066,7 +6231,7 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
             tokio::time::sleep(Duration::from_millis(5)).await;
         }
         assert_eq!(
    -        *fake.maintenance_checks.lock().unwrap(),
    +        fake.maintenance_checks.lock().unwrap().len(),
             1,
             "one cycle before the first interval"
         );
    @@ -5075,7 +6240,7 @@ async fn the_reconciler_worker_runs_at_startup_and_stops_on_shutdown() {
             .await
             .expect("the worker stops at shutdown")
             .unwrap();
    -    assert_eq!(*fake.maintenance_checks.lock().unwrap(), 1);
    +    assert_eq!(fake.maintenance_checks.lock().unwrap().len(), 1);
         let recorded: Vec = metrics
             .drain(Duration::ZERO)
             .into_iter()
    @@ -5101,10 +6266,12 @@ async fn a_failed_reconciler_cycle_records_a_worker_error() {
         let shutdown = extenddb_storage::CancellationToken::new();
         let worker = tokio::spawn(reconciler::run(
             Arc::clone(&fake),
    +        Arc::new(runtime()),
             reconciler::ReconcilerSettings {
                 interval: Duration::from_millis(5),
                 stale_job_after: Duration::from_secs(300),
             },
    +        "test".to_owned(),
             Arc::clone(&metrics),
             Arc::new(tokio::sync::Mutex::new(
                 reconciler::ReconcilerState::default(),
    @@ -5112,7 +6279,7 @@ async fn a_failed_reconciler_cycle_records_a_worker_error() {
             shutdown.clone(),
         ));
         let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    -    while *fake.maintenance_checks.lock().unwrap() < 2 {
    +    while fake.maintenance_checks.lock().unwrap().len() < 2 {
             assert!(
                 tokio::time::Instant::now() < deadline,
                 "the worker did not go on after a failed cycle"
    @@ -5153,7 +6320,9 @@ async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_fa
             .extend([failed.clone(), failed_manifest.clone()]);
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -5185,14 +6354,26 @@ async fn a_dead_manifest_record_is_logged_once_per_record() {
             .unwrap()
             .push(dead_job(RestoreKind::Manifest, "ManifestTarget"));
         let mut state = reconciler::ReconcilerState::default();
    -    let first = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let first = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(first.manifest_restores_seen, ["ManifestTarget"]);
         assert_eq!(first.manifest_restores_logged, ["ManifestTarget"]);
    -    let second = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let second = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(second.manifest_restores_seen, ["ManifestTarget"]);
         assert!(
             second.manifest_restores_logged.is_empty(),
    @@ -5202,9 +6383,15 @@ async fn a_dead_manifest_record_is_logged_once_per_record() {
             .lock()
             .unwrap()
             .push(dead_job(RestoreKind::Manifest, "Another"));
    -    let third = reconciler::cycle(&fake, Duration::from_secs(300), &mut state)
    -        .await
    -        .unwrap();
    +    let third = reconciler::cycle(
    +        &fake,
    +        &runtime(),
    +        Duration::from_secs(300),
    +        "test",
    +        &mut state,
    +    )
    +    .await
    +    .unwrap();
         assert_eq!(third.manifest_restores_logged, ["Another"]);
         assert!(
             fake.failed_jobs.lock().unwrap().is_empty()
    @@ -5235,7 +6422,9 @@ async fn the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listin
     
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -5258,7 +6447,9 @@ async fn the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listin
         // removed.
         let report = reconciler::cycle(
             &fake,
    +        &runtime(),
             Duration::from_secs(300),
    +        "test",
             &mut reconciler::ReconcilerState::default(),
         )
         .await
    @@ -5270,6 +6461,1960 @@ async fn the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listin
         );
     }
     
    +// --- the store identity file and the store's health ---------------------------
    +
    +const MANIFEST_KEY_SUFFIX: &str = "extenddb-manifest.json";
    +
    +/// The manifest key of `arn`'s prefix.
    +fn manifest_key(arn: &str) -> String {
    +    format!(
    +        "{}{MANIFEST_KEY_SUFFIX}",
    +        BackupArn::parse(arn).unwrap().store_prefix()
    +    )
    +}
    +
    +/// An ACTIVE table description, the probe CreateBackup's handler ran.
    +fn active_table(name: &str) -> TableDescription {
    +    TableDescription {
    +        table_name: name.to_owned(),
    +        table_status: TableStatus::Active,
    +        ..Default::default()
    +    }
    +}
    +
    +/// A second, newer backup of the `Music` table.
    +fn newer_arn(account: &str) -> String {
    +    format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/01791976999999-0badf00d")
    +}
    +
    +fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake {
    +    let fake = Fake::default();
    +    for (arn, status) in rows {
    +        fake.rows
    +            .lock()
    +            .unwrap()
    +            .insert(arn.clone(), row(arn, *status));
    +    }
    +    fake
    +}
    +
    +/// An empty root and a catalog with no rows for it: startup creates the file
    +/// with this catalog's id and a fresh store id, re-reads it and uses that id.
    +#[tokio::test]
    +async fn establish_creates_the_identity_file_on_an_empty_root() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(identity.store_kind, StoreKind::Filesystem);
    +    assert_eq!(identity.store_root, "/r");
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap();
    +    assert_eq!(file.store_id, identity.store_id);
    +    assert_eq!(file.catalog_id, "catalog-1");
    +    assert_eq!(file.backend, "test");
    +    assert_eq!(mem.puts.lock().unwrap().as_slice(), [STORE_IDENTITY_KEY]);
    +    // A second startup against the same root reads the file and uses its id.
    +    let again = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(again, identity);
    +    assert_eq!(
    +        mem.puts.lock().unwrap().len(),
    +        1,
    +        "the file is not rewritten"
    +    );
    +}
    +
    +/// Two instances of one catalog starting against one empty root: one creates
    +/// the file, the other finds it present, and both use the winner's id.
    +#[tokio::test]
    +async fn two_instances_on_one_empty_root_end_with_one_id() {
    +    let mem = Arc::new(MemStore::default());
    +    let fakes = [Fake::default(), Fake::default()];
    +    let (a, b) = tokio::join!(
    +        store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[0], "test"),
    +        store_identity::establish(&*mem, StoreKind::S3, "s3://b/", &fakes[1], "test"),
    +    );
    +    let (a, b) = (a.unwrap(), b.unwrap());
    +    assert_eq!(a, b);
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    assert_eq!(
    +        mem.puts.lock().unwrap().len(),
    +        1,
    +        "one create stored its file"
    +    );
    +}
    +
    +/// A root whose file names another catalog is unhealthy, and stays so until
    +/// the file is restored; the problem names both catalogs and no root.
    +#[tokio::test]
    +async fn a_file_naming_another_catalog_is_unhealthy_until_restored() {
    +    let mem = MemStore::default();
    +    mem.insert(
    +        STORE_IDENTITY_KEY,
    +        &identity_file("store-1", "another-catalog"),
    +    );
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::ForeignCatalog {
    +            file_catalog_id: "another-catalog".to_owned(),
    +            catalog_id: "catalog-1".to_owned(),
    +        }
    +    );
    +    assert_eq!(err.category(), "owned by another catalog");
    +    assert!(!err.public_reason().contains("/r"));
    +    // Restored: the check passes.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let identity = store_identity::recheck(&mem, &store(), &fake, "test").await;
    +    assert_eq!(identity, Ok(()));
    +}
    +
    +/// Something that is not an object at the identity file's key (a FIFO, a
    +/// socket): the create finds the key taken and the read finds no object, so
    +/// nothing was created and nothing can be read, and the problem says so
    +/// rather than claiming a file was created.
    +#[tokio::test]
    +async fn a_non_object_at_the_identity_key_is_neither_readable_nor_creatable() {
    +    let mem = MemStore::default();
    +    mem.key_taken_by_a_non_object
    +        .store(true, std::sync::atomic::Ordering::Release);
    +    let fake = Fake::default();
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::Transport("extenddb-store.json is neither readable nor creatable".to_owned())
    +    );
    +    assert!(mem.keys().is_empty(), "nothing was created");
    +}
    +
    +/// The rows for the root carry an id the file does not, or more than one id,
    +/// or the root has no file while rows exist: each is unhealthy.
    +#[tokio::test]
    +async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() {
    +    // The file holds another id than the rows.
    +    let mem = MemStore::default();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1"));
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::IdMismatch {
    +            file_id: "store-2".to_owned(),
    +            row_ids: vec!["store-1".to_owned()],
    +        }
    +    );
    +    assert_eq!(err.category(), "identity mismatch");
    +
    +    // No file, rows exist.
    +    let mem = MemStore::default();
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()])
    +    );
    +    assert!(mem.keys().is_empty(), "nothing is created over rows");
    +    // The file put back: healthy.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +
    +    // Rows carrying two ids.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let mut other = row(&newer_arn(ACCOUNT), BackupStatus::Available);
    +    other.store.store_id = "store-9".to_owned();
    +    fake.rows.lock().unwrap().insert(newer_arn(ACCOUNT), other);
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        store().store_root.as_str(),
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::MultipleRowIds(vec!["store-1".to_owned(), "store-9".to_owned()])
    +    );
    +}
    +
    +/// The manifest probe: the newest AVAILABLE row's manifest must be under the
    +/// root. A second instance with a host-local root at the same path and the
    +/// first's file copied to it has the file and none of the objects, which is
    +/// what the probe catches; a root with no AVAILABLE row is not probed.
    +#[tokio::test]
    +async fn the_manifest_probe_reads_the_newest_available_rows_manifest() {
    +    let mem = MemStore::default();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = fake_with_rows(&[
    +        (arn(ACCOUNT), BackupStatus::Available),
    +        (newer_arn(ACCOUNT), BackupStatus::Available),
    +        (
    +            music_arn(ACCOUNT, "01791976999999-0bad0bad"),
    +            BackupStatus::Creating,
    +        ),
    +    ]);
    +    // Only the older manifest is there: the newest is probed and missing.
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    let err = store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::ManifestMissing {
    +            backup_arn: newer_arn(ACCOUNT),
    +        }
    +    );
    +    assert_eq!(err.category(), "manifest probe failed");
    +    mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +    store_identity::establish(
    +        &mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +    // The CREATING row's manifest is not probed: it has none yet.
    +    // A root with rows but none AVAILABLE is not probed either.
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Creating)]);
    +    let empty = MemStore::default();
    +    empty.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    store_identity::establish(
    +        &empty,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap();
    +}
    +
    +/// A store that ignores the conditional put makes the create a plain put: a
    +/// race lost before the re-read leaves another id in the file, which the
    +/// re-read catches, naming both ids.
    +#[tokio::test]
    +async fn a_create_race_lost_on_a_store_ignoring_the_condition_is_caught_by_the_re_read() {
    +    let mem = MemStore::default();
    +    *mem.overwrite_after_conditional_put.lock().unwrap() =
    +        Some(identity_file("the-other-instance", "catalog-1"));
    +    let fake = Fake::default();
    +    let err = store_identity::establish(&mem, StoreKind::S3, "s3://b/", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    match err {
    +        StoreProblem::CreateRaceLost { ours, file } => {
    +            assert_eq!(file, "the-other-instance");
    +            assert_ne!(ours, file);
    +            let reason = StoreProblem::CreateRaceLost {
    +                ours: ours.clone(),
    +                file: file.clone(),
    +            }
    +            .public_reason();
    +            assert!(reason.contains(&ours) && reason.contains(&file), "{reason}");
    +        }
    +        other => panic!("expected CreateRaceLost, got {other:?}"),
    +    }
    +}
    +
    +/// An unreachable store is unhealthy with the store's error in the log line
    +/// and not in the `/health` text; the check that runs again (the
    +/// reconciler's) recovers it once the store answers.
    +#[tokio::test]
    +async fn an_unreachable_store_is_unhealthy_and_recovers_on_recheck() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_down(true);
    +    let fake = Fake::default();
    +    let err = store_identity::establish(
    +        &*mem,
    +        StoreKind::Filesystem,
    +        &store().store_root,
    +        &fake,
    +        "test",
    +    )
    +    .await
    +    .unwrap_err();
    +    // The operator's line carries the whole cause chain: the top link names
    +    // the class ("dispatch failure"), the cause is the link below it.
    +    assert!(
    +        matches!(&err, StoreProblem::Transport(m)
    +            if m.contains("dispatch failure") && m.contains("connection refused")),
    +        "{err:?}"
    +    );
    +    assert_eq!(err.public_reason(), "the store could not be reached");
    +    assert!(err.to_string().contains("connection refused"));
    +
    +    let runtime = runtime_unhealthy(&mem, err);
    +    assert_eq!(
    +        runtime.configured_store().unwrap().identity(),
    +        None,
    +        "the id is unknown until a check passes"
    +    );
    +    let report = runtime.store_health().unwrap();
    +    assert_eq!(report.kind, "filesystem");
    +    assert_eq!(report.status, "unhealthy");
    +    assert_eq!(report.reason, Some("unreachable"));
    +
    +    // Still down: the cycle leaves it unhealthy.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    // Up again: the cycle's recheck passes and the component recovers.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert_eq!(runtime.store_health().unwrap().status, "healthy");
    +    assert_eq!(runtime.store_health().unwrap().reason, None);
    +    assert_eq!(
    +        runtime.configured_store().unwrap().identity(),
    +        Some(store()),
    +        "the check that passed settled the id from the file"
    +    );
    +    // And back to unhealthy when it goes away again.
    +    mem.set_down(true);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +}
    +
    +/// A store that has stopped answering costs a cycle its check bound and
    +/// nothing more: the dead-owner duties ran before the check, the check is
    +/// abandoned at `check_timeout` (5 s for a filesystem root) and recorded as
    +/// a transport failure naming the bound, the cycle returns, and the next
    +/// cycle on a store that answers again recovers it.
    +#[tokio::test(start_paused = true)]
    +async fn a_stalled_store_check_is_abandoned_at_its_bound() {
    +    use extenddb_storage::backup_catalog::RestoreKind;
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_stalled(true);
    +    let runtime = runtime_on(&mem, store());
    +    let configured = runtime.configured_store().unwrap();
    +    assert_eq!(configured.check_timeout(), LOCAL_CHECK_TIMEOUT);
    +    let fake = Fake::default();
    +    *fake.record_still_running.lock().unwrap() = true;
    +    *fake.target_still_creating.lock().unwrap() = true;
    +    fake.dead_jobs
    +        .lock()
    +        .unwrap()
    +        .push(dead_job(RestoreKind::Legacy, "LegacyTarget"));
    +
    +    let started = tokio::time::Instant::now();
    +    // The test's own bound, far past the check's: a cycle that waits on the
    +    // store for longer than it fails here instead of hanging.
    +    let report = tokio::time::timeout(
    +        Duration::from_secs(120),
    +        reconciler::cycle(
    +            &fake,
    +            &runtime,
    +            Duration::from_secs(300),
    +            "test",
    +            &mut reconciler::ReconcilerState::default(),
    +        ),
    +    )
    +    .await
    +    .expect("the cycle returns within the check's bound")
    +    .unwrap();
    +    assert_eq!(
    +        started.elapsed(),
    +        LOCAL_CHECK_TIMEOUT,
    +        "the check's bound is the whole cost"
    +    );
    +    assert_eq!(
    +        report.legacy_restores_removed,
    +        ["LegacyTarget"],
    +        "the duties ran"
    +    );
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert_eq!(
    +        configured.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport(
    +            "the check did not finish within 5 s".to_owned()
    +        ))
    +    );
    +    assert_eq!(runtime.store_health().unwrap().reason, Some("unreachable"));
    +    assert_eq!(mem.stalls.load(std::sync::atomic::Ordering::Acquire), 1);
    +    // The abandoned check's call is still in the store (on a filesystem
    +    // root, a thread parked on the mount). A cycle during the stall starts
    +    // no second check: it returns at once, the store stays `unreachable`
    +    // with the reason, and the store sees no second call.
    +    assert!(configured.check_call_in_flight());
    +    let started = tokio::time::Instant::now();
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(
    +        started.elapsed(),
    +        Duration::ZERO,
    +        "a cycle that starts no check waits no bound"
    +    );
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert_eq!(
    +        configured.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport(
    +            "a call of an earlier check has not returned".to_owned()
    +        ))
    +    );
    +    assert_eq!(
    +        mem.stalls.load(std::sync::atomic::Ordering::Acquire),
    +        1,
    +        "no second call reached the store while the first was in flight"
    +    );
    +    // Checking stores wait the same bound: an S3 store's is 30 s, under the
    +    // SDK's attempt timeout so no retry outlives the check.
    +    let s3 = ConfiguredStore::checking(
    +        StoreKind::S3,
    +        "s3://b/".to_owned(),
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    );
    +    assert_eq!(s3.check_timeout(), S3_CHECK_TIMEOUT);
    +    assert_eq!(S3_CHECK_TIMEOUT, Duration::from_secs(30));
    +    assert!(S3_CHECK_TIMEOUT < extenddb_backup_store::S3_ATTEMPT_TIMEOUT);
    +    // An S3 store's abandoned request is dropped with the check (the SDK
    +    // would otherwise retry it past the bound), so nothing is in flight
    +    // afterwards and the next cycle sends a fresh request.
    +    let s3_runtime = BackupRuntime::new(4).with_store(s3);
    +    let s3 = s3_runtime.configured_store().unwrap();
    +    let calls_before = mem.stalls.load(std::sync::atomic::Ordering::Acquire);
    +    for _ in 0..2 {
    +        let started = tokio::time::Instant::now();
    +        reconciler::cycle(
    +            &fake,
    +            &s3_runtime,
    +            Duration::from_secs(300),
    +            "test",
    +            &mut reconciler::ReconcilerState::default(),
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(started.elapsed(), S3_CHECK_TIMEOUT);
    +        assert!(
    +            !s3.check_call_in_flight(),
    +            "an S3 request ends with the check"
    +        );
    +    }
    +    assert_eq!(
    +        mem.stalls.load(std::sync::atomic::Ordering::Acquire),
    +        calls_before + 2,
    +        "each cycle's check reached the endpoint"
    +    );
    +
    +    // The store answers again: the parked call returns (its task clears the
    +    // in-flight mark), and the next cycle's check passes.
    +    mem.set_stalled(false);
    +    tokio::time::sleep(Duration::from_millis(20)).await;
    +    assert!(
    +        !configured.check_call_in_flight(),
    +        "the parked call returned once the store answered"
    +    );
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert!(configured.is_healthy());
    +}
    +
    +/// A shutdown that fires while a cycle waits on the store's check ends the
    +/// cycle at once (the health is left as it was), so the worker drains
    +/// without waiting the check's bound.
    +#[tokio::test(start_paused = true)]
    +async fn a_shutdown_during_the_store_check_ends_the_cycle() {
    +    use extenddb_core::metrics::MetricsCollector;
    +    let mem = Arc::new(MemStore::default());
    +    mem.set_stalled(true);
    +    // An S3 store, whose bound (30 s) is the longer one.
    +    let runtime = Arc::new(runtime().with_store(ConfiguredStore::checking(
    +        StoreKind::S3,
    +        "s3://b/".to_owned(),
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    )));
    +    let fake = Arc::new(Fake::default());
    +    let shutdown = extenddb_storage::CancellationToken::new();
    +    let worker = tokio::spawn(reconciler::run(
    +        Arc::clone(&fake),
    +        Arc::clone(&runtime),
    +        reconciler::ReconcilerSettings {
    +            interval: Duration::from_secs(3600),
    +            stale_job_after: Duration::from_secs(300),
    +        },
    +        "test".to_owned(),
    +        Arc::new(MetricsCollector::new()),
    +        Arc::new(tokio::sync::Mutex::new(
    +            reconciler::ReconcilerState::default(),
    +        )),
    +        shutdown.clone(),
    +    ));
    +    let deadline = tokio::time::Instant::now() + Duration::from_secs(5);
    +    while mem.stalls.load(std::sync::atomic::Ordering::Acquire) == 0 {
    +        assert!(
    +            tokio::time::Instant::now() < deadline,
    +            "the first cycle's check reached the store"
    +        );
    +        tokio::time::sleep(Duration::from_millis(5)).await;
    +    }
    +    let cancelled = tokio::time::Instant::now();
    +    shutdown.cancel();
    +    tokio::time::timeout(Duration::from_secs(5), worker)
    +        .await
    +        .expect("the worker stops at shutdown without waiting the check's bound")
    +        .unwrap();
    +    assert!(
    +        cancelled.elapsed() < Duration::from_secs(1),
    +        "the drain took {:?}",
    +        cancelled.elapsed()
    +    );
    +    assert_eq!(
    +        runtime.configured_store().unwrap().health(),
    +        StoreHealth::Checking,
    +        "a check shutdown ended records nothing"
    +    );
    +}
    +
    +/// A command's identity read (`backup list`, the purge's dry run) runs under
    +/// the store's check bound as the server's check does: a store that does not
    +/// answer costs the command the bound (5 s for a filesystem root) and
    +/// reports it, where the unbounded read would wait out the store's retries
    +/// or, on a stalled mount, never return; a store that answers is read.
    +#[tokio::test(start_paused = true)]
    +async fn a_commands_identity_read_is_bounded_as_the_check_is() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = Fake::default();
    +    let configured = ConfiguredStore::checking(
    +        StoreKind::Filesystem,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    );
    +    let read = runtime::read_identity_bounded(&configured, &fake)
    +        .await
    +        .unwrap();
    +    assert_eq!(read.map(|i| i.store_id), Some("store-1".to_owned()));
    +
    +    mem.set_stalled(true);
    +    let started = tokio::time::Instant::now();
    +    // The test's own bound, far past the read's: an unbounded read fails
    +    // here instead of hanging.
    +    let problem = tokio::time::timeout(
    +        Duration::from_secs(120),
    +        runtime::read_identity_bounded(&configured, &fake),
    +    )
    +    .await
    +    .expect("the read returns at its bound")
    +    .unwrap_err();
    +    assert_eq!(started.elapsed(), LOCAL_CHECK_TIMEOUT);
    +    assert_eq!(
    +        problem,
    +        StoreProblem::Transport("the identity read did not finish within 5 s".to_owned())
    +    );
    +    assert_eq!(problem.category(), "unreachable");
    +    assert_eq!(mem.stalls.load(std::sync::atomic::Ordering::Acquire), 1);
    +    mem.set_stalled(false);
    +}
    +
    +/// `record_check` logs a transition once per category: it reports a change
    +/// of category, and not a new text within the same one, so a store that
    +/// stays down with a varying error costs one line.
    +#[test]
    +fn record_check_reports_a_transition_only_across_categories() {
    +    let mem = Arc::new(MemStore::default());
    +    let runtime = runtime_on(&mem, store());
    +    let store = runtime.configured_store().unwrap();
    +    assert!(store.record_check(Err(StoreProblem::Transport("refused".to_owned()))));
    +    assert!(
    +        !store.record_check(Err(StoreProblem::Transport("timed out".to_owned()))),
    +        "a new text in the same category is not a transition"
    +    );
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("timed out".to_owned())),
    +        "the latest text is kept"
    +    );
    +    assert!(store.record_check(Err(StoreProblem::Validate("no bucket".to_owned()))));
    +    assert!(store.record_check(Ok(self::store())));
    +    assert!(!store.record_check(Ok(self::store())));
    +    assert!(
    +        !store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned()))),
    +        "a catalog read failure changes nothing"
    +    );
    +}
    +
    +/// A recheck of a root that lost its file while the catalog has no rows for
    +/// it writes the file back with the same id.
    +#[tokio::test]
    +async fn a_recheck_rewrites_a_lost_file_when_no_rows_exist() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    store_identity::recheck(&mem, &store(), &fake, "test")
    +        .await
    +        .unwrap();
    +    let file = StoreIdentityFile::parse(&mem.objects.lock().unwrap()[STORE_IDENTITY_KEY]).unwrap();
    +    assert_eq!(file.store_id, "store-1");
    +    assert_eq!(file.catalog_id, "catalog-1");
    +    // A file that changed under the running server (the root replaced) is
    +    // an id mismatch.
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1"));
    +    let err = store_identity::recheck(&mem, &store(), &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert_eq!(
    +        err,
    +        StoreProblem::IdMismatch {
    +            file_id: "store-2".to_owned(),
    +            row_ids: vec!["store-1".to_owned()],
    +        }
    +    );
    +}
    +
    +/// No store configured: `/health` has no component and no operation is
    +/// refused for the store.
    +#[test]
    +fn no_configured_store_reports_no_component() {
    +    assert_eq!(runtime().store_health(), None);
    +    assert_eq!(
    +        runtime_with_store().store_health().unwrap().status,
    +        "healthy"
    +    );
    +}
    +
    +/// The `/health` reason is the problem's category and nothing else: the
    +/// endpoint is unauthenticated, and the ARN (account, table and backup id)
    +/// a manifest probe names, and the catalog and store ids the identity
    +/// problems name, go to the operator's log line only.
    +#[test]
    +fn the_health_reason_is_the_category_only() {
    +    let mem = Arc::new(MemStore::default());
    +    for (problem, category) in [
    +        (
    +            StoreProblem::ManifestMissing {
    +                backup_arn: arn("999988887777"),
    +            },
    +            "manifest probe failed",
    +        ),
    +        (
    +            StoreProblem::ForeignCatalog {
    +                file_catalog_id: "cat-other".to_owned(),
    +                catalog_id: "cat-mine".to_owned(),
    +            },
    +            "owned by another catalog",
    +        ),
    +        (
    +            StoreProblem::IdMismatch {
    +                file_id: "store-9".to_owned(),
    +                row_ids: vec!["store-1".to_owned()],
    +            },
    +            "identity mismatch",
    +        ),
    +        (
    +            StoreProblem::FileMissingWithRows(vec!["store-1".to_owned()]),
    +            "identity file missing",
    +        ),
    +        (
    +            StoreProblem::Transport("connect /var/lib/extenddb: refused".to_owned()),
    +            "unreachable",
    +        ),
    +    ] {
    +        let report = runtime_unhealthy(&mem, problem.clone())
    +            .store_health()
    +            .unwrap();
    +        assert_eq!(report.status, "unhealthy");
    +        assert_eq!(report.reason, Some(category), "{problem:?}");
    +        for secret in ["999988887777", "cat-", "store-", "/var/lib", "arn:"] {
    +            assert!(
    +                !report.reason.unwrap().contains(secret),
    +                "{problem:?}: the reason carries {secret:?}"
    +            );
    +        }
    +    }
    +}
    +
    +/// An identity file over the read bound is reported unreadable without
    +/// being read in full; one under it parses.
    +#[tokio::test]
    +async fn an_oversized_identity_file_is_unreadable() {
    +    let mem = MemStore::default();
    +    let fake = Fake::default();
    +    let mut big = identity_file("store-1", "catalog-1");
    +    big.resize(store_identity::STORE_IDENTITY_MAX_BYTES + 1, b' ');
    +    mem.insert(STORE_IDENTITY_KEY, &big);
    +    let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, StoreProblem::FileUnreadable(m) if m.contains("larger than 65536 bytes")),
    +        "{err:?}"
    +    );
    +    assert_eq!(err.category(), "identity file unreadable");
    +    // Trailing blanks up to the bound are JSON whitespace: the file parses.
    +    let mut padded = identity_file("store-1", "catalog-1");
    +    padded.resize(store_identity::STORE_IDENTITY_MAX_BYTES, b' ');
    +    mem.insert(STORE_IDENTITY_KEY, &padded);
    +    let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test")
    +        .await
    +        .unwrap();
    +    assert_eq!(identity.store_id, "store-1");
    +}
    +
    +/// A check that could not read the catalog leaves the store's health as the
    +/// last check left it, healthy or unhealthy, and keeps its id.
    +#[tokio::test]
    +async fn a_catalog_read_failure_leaves_the_health_unchanged() {
    +    let mem = Arc::new(MemStore::default());
    +    let healthy = runtime_on(&mem, store());
    +    let store = healthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned())));
    +    assert_eq!(store.health(), StoreHealth::Healthy);
    +    assert_eq!(store.identity(), Some(self::store()));
    +    let unhealthy = runtime_unhealthy(&mem, StoreProblem::Transport("refused".to_owned()));
    +    let store = unhealthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Catalog("pool exhausted".to_owned())));
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("refused".to_owned()))
    +    );
    +    // And through a cycle: a catalog that fails `catalog_id` leaves a
    +    // healthy store healthy.
    +    let fake = Fake::default();
    +    *fake.catalog_id_error.lock().unwrap() = true;
    +    let report = reconciler::cycle(
    +        &fake,
    +        &healthy,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    // A problem of another category replaces the last one; one of the same
    +    // category is kept as the latest text.
    +    let store = healthy.configured_store().unwrap();
    +    store.record_check(Err(StoreProblem::Transport("refused".to_owned())));
    +    store.record_check(Err(StoreProblem::Transport("timed out".to_owned())));
    +    assert_eq!(
    +        store.health(),
    +        StoreHealth::Unhealthy(StoreProblem::Transport("timed out".to_owned()))
    +    );
    +}
    +
    +/// A store whose checks run after the listener is up is `checking` until
    +/// the first check records: `/health` says so with no reason, the
    +/// operations that touch the store refuse naming the pending check, and
    +/// the check runs `validate` first (a store that cannot be reached fails
    +/// there, with its id still unknown), then the identity check, which
    +/// settles the id.
    +#[tokio::test]
    +async fn a_store_checked_after_startup_is_checking_until_the_first_check() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.set_down(true);
    +    let runtime = runtime().with_store(ConfiguredStore::checking(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    ));
    +    let configured = runtime.configured_store().unwrap();
    +    assert_eq!(configured.health(), StoreHealth::Checking);
    +    assert!(!configured.is_validated());
    +    let report = runtime.store_health().unwrap();
    +    assert_eq!((report.status, report.reason), ("checking", None));
    +    let err = runtime.require_healthy_store().err().unwrap();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("first check is pending")),
    +        "{err:?}"
    +    );
    +    // The restore check of a row of the store refuses the same way, before
    +    // the permit: with no slot free, the refusal is still the store's.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::checking(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +    ));
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("first check is pending")),
    +        "{err:?}"
    +    );
    +
    +    // The first check: `validate` fails on the unreachable store.
    +    let fake = Fake::default();
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert!(!configured.is_validated());
    +    assert_eq!(runtime.store_health().unwrap().reason, Some("unreachable"));
    +    assert_eq!(configured.identity(), None);
    +    // Reachable: `validate` passes once and is not run again, the identity
    +    // check settles the id, and the store is healthy.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert!(configured.is_validated());
    +    assert_eq!(configured.identity(), Some(store()));
    +    assert_eq!(runtime.store_health().unwrap().status, "healthy");
    +}
    +
    +/// The refusals while the configured store is unhealthy: DeleteBackup of a
    +/// row of the store, and the restore of such a row, each a
    +/// ValidationException naming the store's kind and the problem's category
    +/// and never the root; the legacy reader is not reached, and the restore's
    +/// refusal comes before the permit (with no slot free it is still the
    +/// store's, not LimitExceeded). CreateBackup is not refused: the in-catalog
    +/// create, the one this release runs, touches no store, so it runs with an
    +/// unhealthy store as it does with none (a server that never configured
    +/// backups keeps CreateBackup through a failed check of the default root),
    +/// and on a backend bound to the in-catalog create.
    +#[tokio::test]
    +async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_create() {
    +    let mem = Arc::new(MemStore::default());
    +    let unhealthy = || {
    +        runtime_unhealthy(
    +            &mem,
    +            StoreProblem::Transport("connect /var/lib/extenddb/backups: refused".to_owned()),
    +        )
    +    };
    +    let check = |e: DynamoDbError| {
    +        let DynamoDbError::ValidationException(m) = e else {
    +            panic!("expected ValidationException, got {e:?}");
    +        };
    +        assert!(m.contains("filesystem backup store"), "{m}");
    +        assert!(m.contains("unreachable"), "{m}");
    +        assert!(!m.contains("/var/lib"), "{m}");
    +        assert!(!m.contains("connect"), "{m}");
    +    };
    +
    +    // CreateBackup runs the in-catalog create, taking its slot.
    +    let fake = Fake::default();
    +    for runtime in [unhealthy(), unhealthy().with_legacy_create_only(true)] {
    +        let before = fake.creates.lock().unwrap().len();
    +        create_backup(
    +            &fake,
    +            &runtime,
    +            ACCOUNT,
    +            REGION,
    +            "Music",
    +            "nightly",
    +            Ok(active_table("Music")),
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(fake.creates.lock().unwrap().len(), before + 1);
    +        assert_eq!(runtime.free_slots(), 3, "the create holds its slot");
    +        fake.permits.lock().unwrap().clear();
    +    }
    +    // With no slot free the create is LimitExceeded, the store's health
    +    // notwithstanding.
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +        Err(StoreProblem::Transport("refused".to_owned())),
    +    ));
    +    let err = create_backup(
    +        &fake,
    +        &full,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap_err();
    +    assert!(
    +        matches!(err, DynamoDbError::LimitExceededException(_)),
    +        "{err:?}"
    +    );
    +
    +    // DeleteBackup of a row of the store.
    +    let fake = fake_with_row(BackupStatus::Available);
    +    let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap_err();
    +    check(err);
    +    assert!(
    +        fake.delete_starts_seen.lock().unwrap().is_empty(),
    +        "no DELETING write"
    +    );
    +
    +    // The restore of a row of the store, with no slot free: the refusal is
    +    // the store's, so it came before the permit.
    +    let full = BackupRuntime::new(0).with_store(ConfiguredStore::new(
    +        store().store_kind,
    +        store().store_root,
    +        Arc::clone(&mem) as Arc,
    +        None,
    +        Err(StoreProblem::Transport(
    +            "connect /var/lib/extenddb/backups: refused".to_owned(),
    +        )),
    +    ));
    +    let err = restore_on(&fake, &full, &restore_body(json!({})))
    +        .await
    +        .unwrap_err();
    +    check(err);
    +    assert!(fake.restores().is_empty(), "the backend was not asked");
    +
    +    // A row of another store is the other-store refusal, whatever the health
    +    // of the configured one.
    +    let mut other = row(&arn(ACCOUNT), BackupStatus::Available);
    +    other.store.store_root = "/elsewhere".to_owned();
    +    let fake = Fake::default();
    +    fake.rows.lock().unwrap().insert(arn(ACCOUNT), other);
    +    let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap_err();
    +    assert!(
    +        matches!(&err, DynamoDbError::ValidationException(m) if m.contains("other than")),
    +        "{err:?}"
    +    );
    +}
    +
    +/// With a healthy store configured, CreateBackup takes the in-catalog path
    +/// (this release has no store-backed create) and logs that the store is
    +/// configured; the slot is taken and the reader's create runs.
    +#[tokio::test]
    +async fn a_healthy_store_leaves_create_backup_on_the_in_catalog_path() {
    +    let fake = Fake::default();
    +    let runtime = runtime_with_store();
    +    create_backup(
    +        &fake,
    +        &runtime,
    +        ACCOUNT,
    +        REGION,
    +        "Music",
    +        "nightly",
    +        Ok(active_table("Music")),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(fake.creates.lock().unwrap().len(), 1);
    +}
    +
    +/// DeleteBackup of a row of the configured store deletes the manifest first
    +/// (a `delete_prefix` on its exact key), then the prefix, and moves the row
    +/// to DELETED once the prefix lists nothing; a retry on a DELETING row runs
    +/// the same deletion. The identity file and other backups' prefixes stay.
    +#[tokio::test]
    +async fn delete_backup_of_a_configured_store_row_deletes_manifest_prefix_then_marks_deleted() {
    +    let mem = Arc::new(MemStore::default());
    +    let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"gz");
    +    mem.insert(&format!("{prefix}manifest-files.json"), b"{}");
    +    mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +    let fake = fake_with_rows(&[
    +        (arn(ACCOUNT), BackupStatus::Deleting),
    +        (newer_arn(ACCOUNT), BackupStatus::Available),
    +    ]);
    +    let runtime = runtime_on(&mem, store());
    +    let desc = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap();
    +    assert_eq!(desc.backup_details.backup_status, "DELETED");
    +    assert_eq!(
    +        mem.deletes.lock().unwrap().as_slice(),
    +        [manifest_key(&arn(ACCOUNT)), prefix.clone()],
    +        "the manifest first, then the prefix"
    +    );
    +    assert_eq!(
    +        mem.keys(),
    +        [
    +            manifest_key(&newer_arn(ACCOUNT)),
    +            STORE_IDENTITY_KEY.to_owned()
    +        ],
    +        "the other backup and the identity file stay"
    +    );
    +    assert_eq!(fake.set_deleted.lock().unwrap().as_slice(), [arn(ACCOUNT)]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +}
    +
    +/// A store failure during the deletion leaves the row DELETING and still
    +/// answers the description with `BackupStatus` DELETED (the service has no
    +/// counterpart for a failed object deletion, and every reader reports the
    +/// backup deleted from the DELETING write on); the reconciler's next cycle
    +/// finishes it once the store answers, and leaves it alone while the store
    +/// is unhealthy.
    +#[tokio::test]
    +async fn a_failed_deletion_stays_deleting_and_the_reconciler_finishes_it() {
    +    let mem = Arc::new(MemStore::default());
    +    let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix();
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +    mem.insert(&format!("{prefix}data/00001.json.gz"), b"gz");
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let runtime = runtime_on(&mem, store());
    +
    +    mem.set_down(true);
    +    let out = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT)))
    +        .await
    +        .unwrap();
    +    assert_eq!(
    +        out.backup_details.backup_status, "DELETED",
    +        "the description answers DELETED once DELETING committed"
    +    );
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleting
    +    );
    +    assert_eq!(mem.keys().len(), 3, "nothing was deleted");
    +
    +    // The reconciler finds the store down: unhealthy, and no row touched.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(false));
    +    assert!(report.deletions_finished.is_empty());
    +    assert!(runtime.require_healthy_store().is_err());
    +
    +    // The store answers again: the cycle recovers the health and finishes
    +    // the deletion.
    +    mem.set_down(false);
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.store_healthy, Some(true));
    +    assert_eq!(report.deletions_finished, [arn(ACCOUNT)]);
    +    assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleted
    +    );
    +    // Nothing left to finish.
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert!(report.deletions_finished.is_empty());
    +}
    +
    +/// The reconciler finishes DELETING rows of the configured store only: a
    +/// DELETING row of another store is left alone, and so is one whose prefix
    +/// still lists an object after the deletion (a put that overlapped it).
    +#[tokio::test]
    +async fn the_reconciler_finishes_only_the_configured_stores_deleting_rows() {
    +    let mem = Arc::new(MemStore::default());
    +    mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let mut other = row(&newer_arn(ACCOUNT), BackupStatus::Deleting);
    +    other.store.store_id = "store-9".to_owned();
    +    other.store.store_root = "/elsewhere".to_owned();
    +    fake.rows.lock().unwrap().insert(newer_arn(ACCOUNT), other);
    +    let runtime = runtime_on(&mem, store());
    +    let report = reconciler::cycle(
    +        &fake,
    +        &runtime,
    +        Duration::from_secs(300),
    +        "test",
    +        &mut reconciler::ReconcilerState::default(),
    +    )
    +    .await
    +    .unwrap();
    +    assert_eq!(report.deletions_finished, [arn(ACCOUNT)]);
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&newer_arn(ACCOUNT)].status,
    +        BackupStatus::Deleting,
    +        "another store's row is not touched"
    +    );
    +    assert!(
    +        mem.deletes
    +            .lock()
    +            .unwrap()
    +            .iter()
    +            .all(|p| !p.contains("0badf00d")),
    +        "nothing of the other store's prefix was deleted"
    +    );
    +}
    +
    +/// `finish_backup_delete` reports a prefix that still lists an object after
    +/// the deletion and leaves the row DELETING for the next run.
    +#[tokio::test]
    +async fn a_prefix_that_still_lists_an_object_is_left_deleting() {
    +    struct Refilling(MemStore);
    +    impl extenddb_backup_store::BackupStore for Refilling {
    +        fn put<'a>(
    +            &'a self,
    +            key: &'a str,
    +            body: extenddb_backup_store::ByteStream,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result,
    +        > {
    +            self.0.put(key, body)
    +        }
    +        fn put_if_absent<'a>(
    +            &'a self,
    +            key: &'a str,
    +            body: extenddb_backup_store::ByteStream,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result, extenddb_backup_store::StoreError>,
    +        > {
    +            self.0.put_if_absent(key, body)
    +        }
    +        fn get<'a>(
    +            &'a self,
    +            key: &'a str,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result,
    +        > {
    +            self.0.get(key)
    +        }
    +        fn head<'a>(
    +            &'a self,
    +            key: &'a str,
    +        ) -> BoxedFuture<
    +            'a,
    +            Result, extenddb_backup_store::StoreError>,
    +        > {
    +            self.0.head(key)
    +        }
    +        fn list<'a>(
    +            &'a self,
    +            prefix: &'a str,
    +        ) -> BoxStream<
    +            'a,
    +            Result,
    +        > {
    +            self.0.list(prefix)
    +        }
    +        /// The deletion runs, and a late put lands under the prefix after
    +        /// it.
    +        fn delete_prefix<'a>(
    +            &'a self,
    +            prefix: &'a str,
    +        ) -> BoxedFuture<'a, Result> {
    +            Box::pin(async move {
    +                let n = self.0.delete_prefix(prefix).await?;
    +                if !prefix.ends_with(MANIFEST_KEY_SUFFIX) {
    +                    self.0
    +                        .insert(&format!("{prefix}data/late.json.gz"), b"late");
    +                }
    +                Ok(n)
    +            })
    +        }
    +        fn validate(&self) -> BoxedFuture<'_, Result<(), extenddb_backup_store::StoreError>> {
    +            self.0.validate()
    +        }
    +    }
    +    let store_handle = Refilling(MemStore::default());
    +    let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +    let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap();
    +    let err = finish_backup_delete(&fake, &store_handle, &parsed)
    +        .await
    +        .unwrap_err();
    +    assert!(err.contains("still lists"), "{err}");
    +    assert_eq!(
    +        fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +        BackupStatus::Deleting
    +    );
    +}
    +
    +// --- the second review round: the store's health, the purge's steps, the orders --
    +
    +mod review_round_2 {
    +    use super::*;
    +    use extenddb_storage::backup_catalog::{JobStatus, RestoreKind};
    +    use maintenance::{
    +        Interrupt, PurgeOutcome, Stop, Wait, WaitOutcome, purge_rows, wait_for_quiet,
    +    };
    +
    +    async fn cycle_on(fake: &Fake, runtime: &BackupRuntime) -> reconciler::CycleReport {
    +        reconciler::cycle(
    +            fake,
    +            runtime,
    +            Duration::from_secs(300),
    +            "test",
    +            &mut reconciler::ReconcilerState::default(),
    +        )
    +        .await
    +        .unwrap()
    +    }
    +
    +    /// A store whose startup `validate` failed (an I/O error on the probe
    +    /// write) is built unvalidated, so the next check runs `validate` again
    +    /// and the store stays unhealthy while the probe fails, where a store
    +    /// built validated would re-read the identity file alone and report
    +    /// itself healthy after one cycle. Once the probe passes the store is
    +    /// healthy.
    +    #[tokio::test]
    +    async fn a_store_whose_startup_validate_failed_is_rechecked_by_validate() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        *mem.validate_error.lock().unwrap() =
    +            Some("write probe: Input/output error (os error 5)".to_owned());
    +        let fake = Fake::default();
    +        let runtime = runtime_unhealthy(
    +            &mem,
    +            StoreProblem::Validate("write probe: Input/output error (os error 5)".to_owned()),
    +        );
    +        let configured = runtime.configured_store().unwrap();
    +        assert!(
    +            !configured.is_validated(),
    +            "a failed validate leaves the store unvalidated"
    +        );
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(false));
    +        assert_eq!(
    +            configured.health(),
    +            StoreHealth::Unhealthy(StoreProblem::Validate(
    +                "write probe: Input/output error (os error 5)".to_owned()
    +            )),
    +            "validate ran again and failed again"
    +        );
    +        assert!(!configured.is_validated());
    +        assert_eq!(runtime.store_health().unwrap().status, "unhealthy");
    +
    +        *mem.validate_error.lock().unwrap() = None;
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(true));
    +        assert!(configured.is_validated());
    +        assert_eq!(
    +            configured.identity().map(|i| i.store_id),
    +            Some("store-1".to_owned())
    +        );
    +
    +        // A startup check that failed after `validate` passed (an identity
    +        // problem) is built validated: the probe is not repeated.
    +        let identity_problem = runtime_unhealthy(
    +            &mem,
    +            StoreProblem::ForeignCatalog {
    +                file_catalog_id: "other".to_owned(),
    +                catalog_id: "catalog-1".to_owned(),
    +            },
    +        );
    +        assert!(identity_problem.configured_store().unwrap().is_validated());
    +        for problem in [
    +            StoreProblem::Transport("refused".to_owned()),
    +            StoreProblem::Denied("mode 0755".to_owned()),
    +        ] {
    +            assert!(
    +                !runtime_unhealthy(&mem, problem)
    +                    .configured_store()
    +                    .unwrap()
    +                    .is_validated(),
    +                "the problems validate can end in leave the store unvalidated"
    +            );
    +        }
    +    }
    +
    +    /// `purge_row` deletes the manifest before the rest of the prefix (so
    +    /// the prefix is no longer a complete backup to any reader while the
    +    /// rest goes), lists the prefix after the deletion and leaves a row whose
    +    /// prefix still holds an object (a put that overlapped the deletion),
    +    /// and does not continue past `DeleteStart::InUse`: a restore reading
    +    /// the backup keeps its source and the row stays AVAILABLE.
    +    #[tokio::test]
    +    async fn purge_row_deletes_manifest_first_lists_after_and_stops_at_in_use() {
    +        // The order: the manifest key, then the prefix.
    +        let fake = Fake::default();
    +        let mem = MemStore::default();
    +        let arn = arn_n(1);
    +        seed(&fake, &mem, &arn, BackupStatus::Available);
    +        let prefix = BackupArn::parse(&arn).unwrap().store_prefix();
    +        let row = fake.rows.lock().unwrap()[&arn].clone();
    +        maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300))
    +            .await
    +            .unwrap();
    +        assert_eq!(
    +            *mem.deletes.lock().unwrap(),
    +            vec![format!("{prefix}{MANIFEST_KEY_SUFFIX}"), prefix.clone()],
    +            "the manifest first, then the prefix"
    +        );
    +        assert!(mem.keys().is_empty());
    +        assert!(fake.rows.lock().unwrap().is_empty());
    +
    +        // An object left under the prefix: the row is left, after the
    +        // DELETING write, for the operator and the reconciler.
    +        let fake = Fake::default();
    +        let mem = MemStore::default();
    +        *mem.late_put_after_delete.lock().unwrap() = Some("data/late.json.gz".to_owned());
    +        seed(&fake, &mem, &arn, BackupStatus::Available);
    +        let row = fake.rows.lock().unwrap()[&arn].clone();
    +        let err = maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300))
    +            .await
    +            .unwrap_err();
    +        assert_eq!(
    +            err,
    +            format!("the prefix still lists {prefix}data/late.json.gz after the deletion")
    +        );
    +        assert_eq!(
    +            fake.rows.lock().unwrap()[&arn].status,
    +            BackupStatus::Deleting,
    +            "the row is left DELETING, not removed, over objects"
    +        );
    +        assert!(fake.set_deleted.lock().unwrap().is_empty());
    +        assert!(fake.removed_rows.lock().unwrap().is_empty());
    +
    +        // In use: a restore reads the backup, so nothing is deleted.
    +        let fake = Fake::default();
    +        let mem = MemStore::default();
    +        seed(&fake, &mem, &arn, BackupStatus::Available);
    +        fake.delete_starts
    +            .lock()
    +            .unwrap()
    +            .insert(arn.clone(), DeleteStart::InUse("Restored".to_owned()));
    +        let row = fake.rows.lock().unwrap()[&arn].clone();
    +        let err = maintenance::purge_row(&fake, &mem, &row, Duration::from_secs(300))
    +            .await
    +            .unwrap_err();
    +        assert_eq!(err, "a restore into Restored is reading it");
    +        assert!(mem.deletes.lock().unwrap().is_empty(), "no delete ran");
    +        assert_eq!(mem.keys().len(), 2, "the objects stay");
    +        assert_eq!(
    +            fake.rows.lock().unwrap()[&arn].status,
    +            BackupStatus::Available
    +        );
    +    }
    +
    +    /// The reconciler's DELETING duty runs only while the store is healthy:
    +    /// with a foreign `extenddb-store.json` under the root (another
    +    /// catalog's), and a store that answers deletes, no delete runs and the
    +    /// row stays DELETING, so the reconciler never deletes under a root
    +    /// another catalog owns.
    +    #[tokio::test]
    +    async fn an_unhealthy_store_whose_deletes_would_answer_has_no_row_touched() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(
    +            STORE_IDENTITY_KEY,
    +            &identity_file("store-1", "other-catalog"),
    +        );
    +        mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +        let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]);
    +        let runtime = runtime_on(&mem, store());
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(false));
    +        assert_eq!(
    +            runtime.configured_store().unwrap().health(),
    +            StoreHealth::Unhealthy(StoreProblem::ForeignCatalog {
    +                file_catalog_id: "other-catalog".to_owned(),
    +                catalog_id: "catalog-1".to_owned(),
    +            })
    +        );
    +        assert!(
    +            report.deletions_finished.is_empty() && mem.deletes.lock().unwrap().is_empty(),
    +            "no delete ran under a root another catalog owns"
    +        );
    +        assert_eq!(
    +            fake.rows.lock().unwrap()[&arn(ACCOUNT)].status,
    +            BackupStatus::Deleting
    +        );
    +        assert_eq!(mem.keys().len(), 2, "the manifest and the file stay");
    +    }
    +
    +    /// The maintenance refusal comes after every other check: with the claim
    +    /// held, a create under a `server.region` `check_backup_arn_fits`
    +    /// refuses and a restore whose throughput override does not fit the
    +    /// backup's billing mode (`validate_override_for_source`) each answer
    +    /// their own ValidationException, not LimitExceededException.
    +    #[tokio::test]
    +    async fn the_maintenance_refusal_comes_after_the_requests_own_checks() {
    +        let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST"))));
    +        *fake.maintenance_held.lock().unwrap() = true;
    +        let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7));
    +
    +        let err = create_backup(
    +            &fake,
    +            &runtime,
    +            ACCOUNT,
    +            "US_EAST_1",
    +            "Music",
    +            "nightly",
    +            Ok(active_table("Music")),
    +        )
    +        .await
    +        .unwrap_err();
    +        assert!(
    +            matches!(&err, DynamoDbError::ValidationException(m) if m.contains("server.region")),
    +            "the region rule answers before the claim: {err:?}"
    +        );
    +
    +        let body = restore_body(json!({
    +            "BillingModeOverride": "PAY_PER_REQUEST",
    +            "ProvisionedThroughputOverride": {
    +                "ReadCapacityUnits": 5, "WriteCapacityUnits": 5
    +            },
    +        }));
    +        let err = restore_on(&fake, &runtime, &body).await.unwrap_err();
    +        assert!(
    +            matches!(&err, DynamoDbError::ValidationException(m)
    +                if m.contains("ProvisionedThroughputOverride can only be specified")),
    +            "the override rule answers before the claim: {err:?}"
    +        );
    +        assert!(
    +            fake.maintenance_checks.lock().unwrap().is_empty(),
    +            "the claim was not read for a request refused on its own terms"
    +        );
    +
    +        // The same requests with their own faults removed are refused on the
    +        // claim, so the claim is the next check.
    +        let err = create_backup(
    +            &fake,
    +            &runtime,
    +            ACCOUNT,
    +            REGION,
    +            "Music",
    +            "nightly",
    +            Ok(active_table("Music")),
    +        )
    +        .await
    +        .unwrap_err();
    +        assert!(matches!(err, DynamoDbError::LimitExceededException(_)));
    +        let err = restore_on(&fake, &runtime, &restore_body(json!({})))
    +            .await
    +            .unwrap_err();
    +        assert!(matches!(err, DynamoDbError::LimitExceededException(_)));
    +        assert_eq!(fake.maintenance_checks.lock().unwrap().len(), 2);
    +    }
    +
    +    /// `recheck` on a store startup settled: rows that arrive carrying
    +    /// another id are an identity mismatch; a file deleted while rows exist
    +    /// is `identity file missing` and is not rewritten; a file replaced with
    +    /// another catalog's is `owned by another catalog`. Each runs through
    +    /// the reconciler, as a running server runs it.
    +    #[tokio::test]
    +    async fn a_recheck_catches_rows_with_another_id_a_lost_file_and_a_foreign_file() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        let fake = Fake::default();
    +        let runtime = runtime_on(&mem, store());
    +        let configured = runtime.configured_store().unwrap();
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(true));
    +
    +        // Rows of the root with another id arrive after startup.
    +        let mut foreign_row = row(&arn(ACCOUNT), BackupStatus::Available);
    +        foreign_row.store.store_id = "store-2".to_owned();
    +        mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +        fake.rows.lock().unwrap().insert(arn(ACCOUNT), foreign_row);
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(false));
    +        assert_eq!(
    +            configured.health(),
    +            StoreHealth::Unhealthy(StoreProblem::IdMismatch {
    +                file_id: "store-1".to_owned(),
    +                row_ids: vec!["store-2".to_owned()],
    +            })
    +        );
    +        fake.rows.lock().unwrap().clear();
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(true));
    +
    +        // The file is deleted while rows exist: not rewritten.
    +        fake.rows
    +            .lock()
    +            .unwrap()
    +            .insert(arn(ACCOUNT), row(&arn(ACCOUNT), BackupStatus::Available));
    +        let puts_before = mem.puts.lock().unwrap().len();
    +        mem.objects.lock().unwrap().remove(STORE_IDENTITY_KEY);
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(false));
    +        assert_eq!(
    +            configured.health(),
    +            StoreHealth::Unhealthy(StoreProblem::FileMissingWithRows(vec![
    +                "store-1".to_owned()
    +            ]))
    +        );
    +        assert_eq!(
    +            mem.puts.lock().unwrap().len(),
    +            puts_before,
    +            "a root that lost its file while rows exist gets no new one"
    +        );
    +        assert!(!mem.keys().contains(&STORE_IDENTITY_KEY.to_owned()));
    +        assert_eq!(
    +            runtime.store_health().unwrap().reason,
    +            Some("identity file missing")
    +        );
    +
    +        // The file is replaced with another catalog's.
    +        mem.insert(
    +            STORE_IDENTITY_KEY,
    +            &identity_file("store-1", "other-catalog"),
    +        );
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(false));
    +        assert_eq!(
    +            configured.health(),
    +            StoreHealth::Unhealthy(StoreProblem::ForeignCatalog {
    +                file_catalog_id: "other-catalog".to_owned(),
    +                catalog_id: "catalog-1".to_owned(),
    +            })
    +        );
    +        // Restored, the store is healthy again.
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(true));
    +    }
    +
    +    /// The wait reads RUNNING restores by status, with no age: a record
    +    /// stamped this instant (one the dead-job listing never returns, its
    +    /// heartbeat being within any window) holds the wait, and the wait
    +    /// reports it.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_wait_sees_a_running_restore_with_a_fresh_heartbeat() {
    +        let fake = Fake::default();
    +        let holder = Arc::new(FakeHolder::default());
    +        let claim = held(&holder);
    +        let mut fresh = dead_job(RestoreKind::Legacy, "Fresh");
    +        fresh.heartbeat_at = seconds_since_epoch();
    +        fresh.status = JobStatus::Running;
    +        fake.running_jobs.lock().unwrap().push(fresh);
    +        let wait = Wait {
    +            owner_window: Duration::from_secs(10),
    +            limit: Duration::from_secs(4),
    +            poll: Duration::from_secs(2),
    +        };
    +        let (_tx, mut quiet) = Interrupt::new();
    +        let outcome = wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, |_| {}).await;
    +        assert_eq!(
    +            outcome,
    +            WaitOutcome::Busy(vec![format!(
    +                "a restore into Fresh (account {ACCOUNT}) is RUNNING"
    +            )])
    +        );
    +        fake.running_jobs.lock().unwrap().clear();
    +        assert_eq!(
    +            wait_for_quiet(&fake, &store(), &claim, wait, &mut quiet, |_| {}).await,
    +            WaitOutcome::Quiet
    +        );
    +        claim.release().await.unwrap();
    +    }
    +
    +    /// The deletion loop stops before its next row, never inside one: once a
    +    /// heartbeat has found the claim lost, and once a signal has been
    +    /// received (here sent as the first row's deletion is reported, so the
    +    /// loop stops before the second). A row left for a reason is skipped
    +    /// and the loop goes on.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_deletion_loop_stops_at_a_lost_claim_and_at_a_signal() {
    +        let fake = Fake::default();
    +        let mem = MemStore::default();
    +        let arns: Vec = (1..=3).map(arn_n).collect();
    +        for arn in &arns {
    +            seed(&fake, &mem, arn, BackupStatus::Available);
    +        }
    +        let rows: Vec = fake
    +            .list_backup_rows(None, &store())
    +            .try_collect()
    +            .await
    +            .unwrap();
    +        assert_eq!(rows.len(), 3);
    +
    +        // The claim lost before the loop: no row is deleted.
    +        let holder = Arc::new(FakeHolder::default());
    +        let claim = held(&holder);
    +        *holder.lost.lock().unwrap() = true;
    +        tokio::time::sleep(Duration::from_millis(1_500)).await;
    +        assert!(claim.lost().is_some());
    +        let (_tx, interrupt) = Interrupt::new();
    +        let mut reported = Vec::new();
    +        let outcome = purge_rows(
    +            &fake,
    +            &mem,
    +            &rows,
    +            Duration::from_secs(300),
    +            &claim,
    +            &interrupt,
    +            |row| reported.push(row.backup_arn.clone()),
    +        )
    +        .await;
    +        assert_eq!(
    +            outcome,
    +            PurgeOutcome {
    +                deleted: vec![],
    +                skipped: vec![],
    +                stopped: Some(Stop::Lost(
    +                    "the maintenance claim was lost (another holder took it over)".to_owned()
    +                )),
    +            }
    +        );
    +        assert!(reported.is_empty());
    +        assert_eq!(fake.rows.lock().unwrap().len(), 3, "no row was touched");
    +        assert_eq!(mem.keys().len(), 6, "no object was deleted");
    +        claim.release().await.unwrap();
    +
    +        // A signal received as the first row is reported stops the loop
    +        // before the second; the third is untouched.
    +        let holder = Arc::new(FakeHolder::default());
    +        let claim = held(&holder);
    +        let (tx, interrupt) = Interrupt::new();
    +        let mut reported = Vec::new();
    +        let outcome = purge_rows(
    +            &fake,
    +            &mem,
    +            &rows,
    +            Duration::from_secs(300),
    +            &claim,
    +            &interrupt,
    +            |row| {
    +                reported.push(row.backup_arn.clone());
    +                let _ = tx.send(Some("SIGTERM"));
    +            },
    +        )
    +        .await;
    +        assert_eq!(
    +            outcome,
    +            PurgeOutcome {
    +                deleted: vec![arns[0].clone()],
    +                skipped: vec![],
    +                stopped: Some(Stop::Interrupted("SIGTERM")),
    +            }
    +        );
    +        assert_eq!(reported, vec![arns[0].clone()]);
    +        let mut left: Vec = fake.rows.lock().unwrap().keys().cloned().collect();
    +        left.sort();
    +        assert_eq!(left, vec![arns[1].clone(), arns[2].clone()]);
    +        assert_eq!(mem.keys().len(), 4);
    +
    +        // Nothing stops it: the rest goes, a row left for a reason skipped.
    +        fake.delete_starts
    +            .lock()
    +            .unwrap()
    +            .insert(arns[1].clone(), DeleteStart::InUse("Restored".to_owned()));
    +        let (_tx, interrupt) = Interrupt::new();
    +        let rest: Vec = rows[1..].to_vec();
    +        let outcome = purge_rows(
    +            &fake,
    +            &mem,
    +            &rest,
    +            Duration::from_secs(300),
    +            &claim,
    +            &interrupt,
    +            |_| {},
    +        )
    +        .await;
    +        assert_eq!(
    +            outcome,
    +            PurgeOutcome {
    +                deleted: vec![arns[2].clone()],
    +                skipped: vec![format!(
    +                    "{}: a restore into Restored is reading it",
    +                    arns[1]
    +                )],
    +                stopped: None,
    +            }
    +        );
    +        claim.release().await.unwrap();
    +    }
    +
    +    /// A defaulted root whose startup checks did not finish starts checking
    +    /// with the rule kept: a first check that fails while the catalog has no
    +    /// rows for the root sets the store aside (no component, no store), as
    +    /// the startup failure would have; with rows for it the same failure is
    +    /// an unhealthy configured store.
    +    #[tokio::test]
    +    async fn a_defaulted_root_set_aside_after_its_first_failed_check() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(
    +            STORE_IDENTITY_KEY,
    +            &identity_file("store-1", "other-catalog"),
    +        );
    +        let fake = Fake::default();
    +        let checking = || {
    +            ConfiguredStore::checking(
    +                store().store_kind,
    +                store().store_root,
    +                Arc::clone(&mem) as Arc,
    +                None,
    +            )
    +            .defaulted(true)
    +        };
    +        let runtime = BackupRuntime::new(4).with_store(checking());
    +        assert!(runtime.configured_store().is_some());
    +        assert_eq!(runtime.store_health().unwrap().status, "checking");
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(
    +            report.store_healthy, None,
    +            "the store is set aside, as none configured"
    +        );
    +        assert!(runtime.configured_store().is_none());
    +        assert!(runtime.store_health().is_none());
    +        assert!(!runtime.is_configured_store(&store()));
    +        // The next cycle runs no check: the store is unconfigured for good.
    +        let puts = mem.puts.lock().unwrap().len();
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, None);
    +        assert_eq!(mem.puts.lock().unwrap().len(), puts);
    +
    +        // With rows for the root, the same failure is an unhealthy store.
    +        fake.rows
    +            .lock()
    +            .unwrap()
    +            .insert(arn(ACCOUNT), row(&arn(ACCOUNT), BackupStatus::Available));
    +        let runtime = BackupRuntime::new(4).with_store(checking());
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(false));
    +        assert_eq!(
    +            runtime.store_health().unwrap().reason,
    +            Some("owned by another catalog")
    +        );
    +        // A root that is not the defaulted one is never set aside.
    +        let explicit = BackupRuntime::new(4).with_store(checking().defaulted(false));
    +        fake.rows.lock().unwrap().clear();
    +        assert_eq!(cycle_on(&fake, &explicit).await.store_healthy, Some(false));
    +        assert!(explicit.configured_store().is_some());
    +    }
    +
    +    /// The deletions run under the store's bound and the in-flight rule:
    +    /// DeleteBackup on a stalled store answers DELETED at the bound with the
    +    /// row left DELETING, two DeleteBackups together park one thread between
    +    /// them (the second waits for the first's slot and answers at its bound
    +    /// with nothing parked), and the reconciler's deletion of those rows
    +    /// returns at the bound too, with no second call parked beside the first.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_deletions_run_under_the_stores_bound() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +        mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +        let fake = fake_with_rows(&[
    +            (arn(ACCOUNT), BackupStatus::Available),
    +            (newer_arn(ACCOUNT), BackupStatus::Available),
    +        ]);
    +        let runtime = runtime_on(&mem, store());
    +        let configured = runtime.configured_store().unwrap();
    +        mem.set_stalled(true);
    +        let started = tokio::time::Instant::now();
    +        let (first, second) = tokio::time::timeout(
    +            Duration::from_secs(120),
    +            futures::future::join(
    +                delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))),
    +                delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))),
    +            ),
    +        )
    +        .await
    +        .expect("both deletes return at the store's bound");
    +        assert_eq!(started.elapsed(), LOCAL_CHECK_TIMEOUT);
    +        assert_eq!(first.unwrap().backup_details.backup_status, "DELETED");
    +        assert_eq!(second.unwrap().backup_details.backup_status, "DELETED");
    +        for row in [arn(ACCOUNT), newer_arn(ACCOUNT)] {
    +            assert_eq!(
    +                fake.rows.lock().unwrap()[&row].status,
    +                BackupStatus::Deleting,
    +                "the row is left for the reconciler"
    +            );
    +        }
    +        assert!(
    +            configured.check_call_in_flight(),
    +            "the abandoned call is parked"
    +        );
    +        assert_eq!(
    +            mem.stalls.load(std::sync::atomic::Ordering::Acquire),
    +            1,
    +            "the second delete's call waited for the first's slot and parked nothing"
    +        );
    +
    +        // The reconciler's cycle: the check is refused at once (a call is
    +        // parked), so no deletion runs and nothing new is parked.
    +        let started = tokio::time::Instant::now();
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(started.elapsed(), Duration::ZERO);
    +        assert_eq!(report.store_healthy, Some(false));
    +        assert!(report.deletions_finished.is_empty());
    +        assert_eq!(mem.stalls.load(std::sync::atomic::Ordering::Acquire), 1);
    +
    +        // The mount answers: the parked call returns, the next cycle checks
    +        // and finishes the deletions.
    +        mem.set_stalled(false);
    +        tokio::time::sleep(Duration::from_millis(50)).await;
    +        assert!(!configured.check_call_in_flight());
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(true));
    +        let mut finished = report.deletions_finished.clone();
    +        finished.sort();
    +        let mut expected = [arn(ACCOUNT), newer_arn(ACCOUNT)];
    +        expected.sort();
    +        assert_eq!(finished, expected);
    +        assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]);
    +    }
    +
    +    /// A call's bound starts once it holds the root's slot: on a healthy
    +    /// root whose calls take 3 s under a 5 s bound, two DeleteBackups
    +    /// together both finish (the second waits 3 s for the slot and then has
    +    /// the bound for its own 3 s call), both rows reach DELETED, nothing is
    +    /// marked in flight and the next check finds the store healthy. With one
    +    /// bound over the wait and the call the second was abandoned at 5 s, its
    +    /// row left DELETING, and the next check marked the store unreachable.
    +    #[tokio::test(start_paused = true)]
    +    async fn a_calls_bound_starts_when_it_holds_the_slot() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +        mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +        let fake = fake_with_rows(&[
    +            (arn(ACCOUNT), BackupStatus::Available),
    +            (newer_arn(ACCOUNT), BackupStatus::Available),
    +        ]);
    +        let runtime = runtime_on(&mem, store());
    +        let configured = runtime.configured_store().unwrap();
    +        mem.set_delay(Some(Duration::from_secs(3)));
    +        let started = tokio::time::Instant::now();
    +        let (first, second) = futures::future::join(
    +            delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))),
    +            delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))),
    +        )
    +        .await;
    +        assert_eq!(first.unwrap().backup_details.backup_status, "DELETED");
    +        assert_eq!(second.unwrap().backup_details.backup_status, "DELETED");
    +        assert!(
    +            started.elapsed() > LOCAL_CHECK_TIMEOUT,
    +            "the second call ran after the first's 3 s: {:?}",
    +            started.elapsed()
    +        );
    +        for row in [arn(ACCOUNT), newer_arn(ACCOUNT)] {
    +            assert_eq!(
    +                fake.rows.lock().unwrap()[&row].status,
    +                BackupStatus::Deleted,
    +                "both deletions finished"
    +            );
    +        }
    +        assert!(!configured.check_call_in_flight(), "nothing abandoned");
    +        mem.set_delay(None);
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(true));
    +    }
    +
    +    /// The manifest probe reads one row (`newest_available_backup_row`) and
    +    /// not the listing: the catalog double records the read, and the probe
    +    /// heads the newest AVAILABLE row's manifest.
    +    #[tokio::test]
    +    async fn the_manifest_probe_reads_one_row() {
    +        let mem = Arc::new(MemStore::default());
    +        mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1"));
    +        mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}");
    +        let fake = fake_with_rows(&[
    +            (arn(ACCOUNT), BackupStatus::Available),
    +            (newer_arn(ACCOUNT), BackupStatus::Available),
    +        ]);
    +        let runtime = runtime_on(&mem, store());
    +        let report = cycle_on(&fake, &runtime).await;
    +        assert_eq!(report.store_healthy, Some(false));
    +        assert_eq!(
    +            runtime.configured_store().unwrap().health(),
    +            StoreHealth::Unhealthy(StoreProblem::ManifestMissing {
    +                backup_arn: newer_arn(ACCOUNT),
    +            }),
    +            "the newest row's manifest is the one probed"
    +        );
    +        assert_eq!(fake.newest_reads.lock().unwrap().as_slice(), [store()]);
    +        mem.insert(&manifest_key(&newer_arn(ACCOUNT)), b"{}");
    +        assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(true));
    +    }
    +}
    +
     /// A restore of a backup the catalog index has a row for checks the target
     /// name once the row is known to exist and AVAILABLE, before the backup's
     /// in-use rule and before the store checks, the service's order (measured on
    @@ -5283,7 +8428,9 @@ async fn the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listin
     /// before the name is read.
     #[tokio::test]
     async fn a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_its_in_use_rule() {
    -    let runtime = runtime().with_store(store());
    +    // The row's store is the configured, healthy one, so a free name reaches
    +    // the store's refusal after the in-use rule.
    +    let runtime = runtime_on(&Arc::new(MemStore::default()), store());
         let fake = fake_with_row(BackupStatus::Available);
         fake.in_use
             .lock()
    diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs
    index 187f4f59f..3f8828d0b 100755
    --- a/crates/engine/src/lib.rs
    +++ b/crates/engine/src/lib.rs
    @@ -8,7 +8,11 @@
     //! `DynamoDB`-format error responses.
     
     mod backup;
    -pub use backup::{BackupRuntime, reconciler};
    +pub use backup::{
    +    BackupRuntime, ConfiguredStore, HeldClaim, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT,
    +    STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem,
    +    check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity,
    +};
     mod batch_get_item;
     mod batch_write_item;
     pub mod capacity_helpers;
    diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml
    index 716cf07a7..e674e5c58 100755
    --- a/crates/server/Cargo.toml
    +++ b/crates/server/Cargo.toml
    @@ -16,6 +16,7 @@ extenddb-core = { workspace = true }
     extenddb-config = { workspace = true }
     extenddb-engine = { workspace = true }
     extenddb-storage = { workspace = true }
    +extenddb-backup-store = { workspace = true }
     extenddb-auth = { workspace = true }
     futures = { workspace = true }
     axum = { workspace = true }
    @@ -51,3 +52,5 @@ libc = { workspace = true }
     
     [dev-dependencies]
     extenddb-cache = { workspace = true, features = ["test-util"] }
    +tempfile = "3"
    +tokio = { workspace = true, features = ["test-util"] }
    diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs
    new file mode 100644
    index 000000000..14dc6949a
    --- /dev/null
    +++ b/crates/server/src/backup_store.rs
    @@ -0,0 +1,1468 @@
    +// Copyright 2026 ExtendDB contributors
    +// SPDX-License-Identifier: Apache-2.0
    +
    +//! Startup of the backup store: the `[backup.store]` resolution, `open`,
    +//! the spool and its lock, and, for a filesystem store, `validate` and the
    +//! identity check; the split between a configuration error, which stops
    +//! startup, and a store condition or identity failure, which starts the
    +//! server with the store unhealthy; and the rule that nothing here waits on
    +//! the network, so a store endpoint that stalls never holds the data plane.
    +//!
    +//! Configuration errors, each named with the path or bucket in the store's
    +//! own message, every one decided without a request to the store: `open`
    +//! refusing the configuration (a missing section, an empty or S3 Express
    +//! bucket name, a part size, prefix or key-length problem, an insecure
    +//! endpoint, a filesystem root that does not exist or fails the owner-only
    +//! rule), a filesystem store's `validate` refusing the root for a reason of
    +//! the same kind (a root another user can reach, a root replaced by a
    +//! symlink), a spool that breaks its placement rule, that another live
    +//! instance holds or that exists with a mode other than owner-only, and an
    +//! S3 store without a spool on a backend with no default. A filesystem
    +//! root's `validate` also writes a probe file, which is a request to the
    +//! store: an I/O failure there or in resolving the root (a full or read-only
    +//! disk, a stale network handle, a root that vanished after `open`) is a
    +//! store condition and degrades the store, as the same failure one cycle
    +//! later does, rather than stopping startup.
    +//!
    +//! The one exception to the configuration errors is the defaulted SQLite
    +//! root (`backups/` beside the database file when `[backup.store]` is
    +//! absent) while the catalog has no backup rows for it: a failure of its
    +//! creation, `open`, `validate` or identity check is a logged warning that
    +//! leaves the store unconfigured, so an upgrade, a fresh deployment or a
    +//! catalog re-created by `destroy` and `init` under the old root never stops
    +//! a server that never configured backups. A defaulted root whose startup
    +//! checks did not finish within their bound starts `checking` with the rule
    +//! kept ([`ConfiguredStore::defaulted`]): the reconciler applies it to the
    +//! first check's outcome, so a failure then with no rows is the same warning
    +//! and leaves the store unconfigured. Once the catalog has rows for
    +//! that root ([`extenddb_storage::BackupCatalog::read_store_ids`]), the root
    +//! has been used and its backups must not vanish behind a warning: the
    +//! defaulted root is then handled as an explicitly configured one, a
    +//! refusal of `open` or `validate` a startup error, and an identity failure
    +//! an unhealthy store (reported on `/health`, its rows' restores and deletes
    +//! refused naming the kind and the category, rechecked each cycle). A
    +//! command that found no store learns why from [`NotConfigured`].
    +//!
    +//! A filesystem store's `validate` and identity check run here, before the
    +//! listener, bounded by [`LOCAL_CHECK_TIMEOUT`]: a root on a local disk
    +//! answers at once, and a mount that stops answering after the root is
    +//! opened leaves the server starting with the store `checking`; a root that
    +//! does not answer at startup holds the server before the listener, as the
    +//! database directory would (the banner's resolution of the root, the
    +//! default root's creation and `open`'s own resolution run unbounded, before
    +//! the checks). The socket is bound before the banner, so a startup held
    +//! that way leaves the port accepting connections it never answers until
    +//! the mount answers or the process is killed. A check abandoned at the
    +//! bound leaves its filesystem call parked on a blocking thread until the
    +//! mount answers (the engine's runtime module says how the later checks
    +//! avoid parking another); the thread costs nothing at exit, where the
    +//! binary ends its runtime with a bounded shutdown. An S3 store's
    +//! `validate` and identity check are requests to the endpoint, and run after
    +//! the listener is up, in the reconciler's first cycle
    +//! ([`extenddb_engine::check_configured_store`]); the store is `checking`
    +//! until that cycle records its result, and the operations that touch the
    +//! store refuse meanwhile as they do while it is unhealthy. A `validate`
    +//! refusal that is not a transport failure (a missing or denied bucket) then
    +//! makes the store unhealthy rather than stopping startup.
    +//!
    +//! Transport failures (the store unreachable: no region, no credentials, a
    +//! refused connection) and failures of the identity check or the manifest
    +//! probe (a denial the store reports while a check runs, say a directory
    +//! under the root that another user can reach, included) leave the server
    +//! running with the store unhealthy: logged once, reported on `/health`,
    +//! the operations that touch the store refused, and the check run again
    +//! each reconciler cycle until it passes.
    +
    +use std::path::{Path, PathBuf};
    +use std::sync::Arc;
    +
    +use extenddb_backup_store::{BackupStore, BackupStoreKind, StoreError};
    +use extenddb_config::backup::{BackupConfig, EffectiveStore, Unconfigured};
    +pub use extenddb_engine::LOCAL_CHECK_TIMEOUT;
    +use extenddb_engine::{BackupRuntime, ConfiguredStore, Spool, StoreProblem, store_identity};
    +use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind};
    +use extenddb_storage::{BackupCatalog, StorageEngine};
    +
    +/// The lock on the spool directory, held for the process's life so a second
    +/// instance on this host refuses the same `spool_path`.
    +pub struct SpoolLock {
    +    _file: std::fs::File,
    +    path: PathBuf,
    +}
    +
    +impl std::fmt::Debug for SpoolLock {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        f.debug_struct("SpoolLock")
    +            .field("path", &self.path)
    +            .finish()
    +    }
    +}
    +
    +/// The name of the lock file inside the spool.
    +pub const SPOOL_LOCK_FILE: &str = ".lock";
    +
    +/// Who is configuring the store, which decides the spool and the checks.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq)]
    +pub enum Caller {
    +    /// A server: create the spool and hold its lock for the process's life;
    +    /// a filesystem store's checks run here (module doc).
    +    Server,
    +    /// A command that writes (`backup purge`): the spool rule is checked,
    +    /// the directory neither created nor locked; the checks run as a
    +    /// server's do.
    +    Command,
    +    /// A command that reads beside a server (`backup list`, the purge's dry
    +    /// run): as `Command`, and no check of the store runs, so a root is left
    +    /// as it is found (neither the default directory nor an identity file is
    +    /// created; a default root that does not exist yet leaves the store
    +    /// unconfigured, logged); the store is returned `checking` and the
    +    /// command reads what it needs itself.
    +    ReadOnlyCommand,
    +}
    +
    +/// Why [`configure_described`] left the store unconfigured; `Display` is the
    +/// sentence a command prints for it.
    +#[derive(Debug, Clone, PartialEq, Eq)]
    +pub enum NotConfigured {
    +    /// `[backup.store]` is absent and this backend has no default root.
    +    Absent,
    +    /// `kind = "disabled"`.
    +    Disabled,
    +    /// A read-only command found that the default root does not exist yet:
    +    /// no server has run with this database file.
    +    DefaultRootMissing(PathBuf),
    +    /// The defaulted root could not be created, opened, validated or checked
    +    /// while the catalog has no rows for it (module doc): the warning path.
    +    DefaultRootUnusable {
    +        /// The root, as the configuration or `open` spelled it.
    +        root: String,
    +        /// The failure, as the warning logged it.
    +        problem: String,
    +    },
    +}
    +
    +impl std::fmt::Display for NotConfigured {
    +    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    +        match self {
    +            Self::Absent => f.write_str(
    +                "no backup store is configured: [backup.store] is absent and this backend has \
    +                 no default root, so there are no store-backed backups to list or purge",
    +            ),
    +            Self::Disabled => f.write_str(
    +                "no backup store is configured: [backup.store] is disabled, so there are no \
    +                 store-backed backups to list or purge",
    +            ),
    +            Self::DefaultRootMissing(root) => write!(
    +                f,
    +                "the default root {} does not exist yet: no server has run with this database \
    +                 file, so there are no store-backed backups to list or purge",
    +                root.display()
    +            ),
    +            Self::DefaultRootUnusable { root, problem } => write!(
    +                f,
    +                "the default root {root} failed its check ({problem}) and the catalog has no \
    +                 rows for it, so there are no store-backed backups to list or purge; the \
    +                 warning above says how to fix the root"
    +            ),
    +        }
    +    }
    +}
    +
    +/// What [`configure_described`] built.
    +pub struct StoreSetup {
    +    /// The job slots and, when one is configured, the store with its health.
    +    pub runtime: BackupRuntime,
    +    /// The spool's lock, held by a server for the process's life.
    +    pub spool_lock: Option,
    +    /// Why the runtime has no store, when it has none.
    +    pub not_configured: Option,
    +}
    +
    +impl StoreSetup {
    +    fn without_store(runtime: BackupRuntime, why: NotConfigured) -> Self {
    +        Self {
    +            runtime,
    +            spool_lock: None,
    +            not_configured: Some(why),
    +        }
    +    }
    +
    +    fn with_store(
    +        runtime: BackupRuntime,
    +        store: ConfiguredStore,
    +        spool_lock: Option,
    +    ) -> Self {
    +        Self {
    +            runtime: runtime.with_store(store),
    +            spool_lock,
    +            not_configured: None,
    +        }
    +    }
    +}
    +
    +/// Build the backup runtime for this process: the job slots and, when the
    +/// configuration and the backend select one, the store with its health
    +/// ([`configure_described`] without the reason for an absent store).
    +///
    +/// # Errors
    +///
    +/// A configuration error (module doc), with the key, path or bucket named.
    +pub async fn configure(
    +    backup: &BackupConfig,
    +    database_file: Option<&Path>,
    +    backend: &str,
    +    storage: &Arc,
    +    caller: Caller,
    +) -> anyhow::Result<(BackupRuntime, Option)> {
    +    let setup = configure_described(backup, database_file, backend, &**storage, caller).await?;
    +    Ok((setup.runtime, setup.spool_lock))
    +}
    +
    +/// Build the backup runtime for this process: the job slots and, when the
    +/// configuration and the backend select one, the store with its health;
    +/// otherwise why there is none. `catalog` is the catalog the identity check
    +/// reads and the rows rule of the module doc consults.
    +///
    +/// # Errors
    +///
    +/// A configuration error (module doc), with the key, path or bucket named.
    +pub async fn configure_described(
    +    backup: &BackupConfig,
    +    database_file: Option<&Path>,
    +    backend: &str,
    +    catalog: &C,
    +    caller: Caller,
    +) -> anyhow::Result {
    +    let runtime = BackupRuntime::new(backup.max_concurrent_jobs)
    +        .with_legacy_create_only(backend == "cassandra")
    +        .with_stale_job_after(std::time::Duration::from_secs(backup.stale_job_after_secs));
    +    let resolved = backup
    +        .resolve_store(database_file)
    +        .map_err(|e| anyhow::anyhow!("[backup] configuration: {e}"))?;
    +    let (config, defaulted) = match resolved.store {
    +        EffectiveStore::Unconfigured(Unconfigured::Absent) => {
    +            tracing::info!(
    +                "backup store: none configured ([backup.store] is absent and this backend has \
    +                 no default root); CreateBackup writes in-catalog backups, with a deprecation \
    +                 warning on each call"
    +            );
    +            return Ok(StoreSetup::without_store(runtime, NotConfigured::Absent));
    +        }
    +        EffectiveStore::Unconfigured(Unconfigured::Disabled) => {
    +            tracing::info!(
    +                "backup store: disabled ([backup.store] kind = \"disabled\"); CreateBackup \
    +                 writes in-catalog backups, with a deprecation warning on each call"
    +            );
    +            return Ok(StoreSetup::without_store(runtime, NotConfigured::Disabled));
    +        }
    +        EffectiveStore::Configured { config, defaulted } => (config, defaulted),
    +    };
    +
    +    // The defaulted root beside the database file is created if missing,
    +    // private to this user; an explicitly configured root must exist. A
    +    // read-only command creates nothing: a defaulted root no server has
    +    // created yet is reported as such, and the command has nothing to read.
    +    if defaulted
    +        && caller == Caller::ReadOnlyCommand
    +        && let Some(fs) = &config.filesystem
    +        && !fs.path.exists()
    +    {
    +        tracing::info!(
    +            "backup store: the default root {} does not exist yet (no server has run with this \
    +             database file); there are no store-backed backups to list",
    +            fs.path.display()
    +        );
    +        return Ok(StoreSetup::without_store(
    +            runtime,
    +            NotConfigured::DefaultRootMissing(fs.path.clone()),
    +        ));
    +    }
    +    if defaulted
    +        && let Some(fs) = &config.filesystem
    +        && let Err(e) = create_private_dir_if_missing(&fs.path)
    +    {
    +        // A root that could not be created does not exist, so no row can
    +        // name it: the warning path, whatever the catalog holds.
    +        let root = fs.path.display().to_string();
    +        let problem = format!("it could not be created: {e}");
    +        warn_default_root_unusable(&root, &problem);
    +        return Ok(StoreSetup::without_store(
    +            runtime,
    +            NotConfigured::DefaultRootUnusable { root, problem },
    +        ));
    +    }
    +
    +    let opened = match extenddb_backup_store::open_described(&config).await {
    +        Ok(opened) => opened,
    +        Err(e) => {
    +            // The label rows record is the root's canonical path, which the
    +            // refused root may still have (a parent others can write to).
    +            let label = config
    +                .filesystem
    +                .as_ref()
    +                .and_then(|fs| std::fs::canonicalize(&fs.path).ok())
    +                .map(|p| p.display().to_string());
    +            if defaulted
    +                && let Some(root) = label.as_deref()
    +                && !default_root_has_rows(catalog, root).await
    +            {
    +                let problem = format!("it is not usable: {e}");
    +                warn_default_root_unusable(root, &problem);
    +                return Ok(StoreSetup::without_store(
    +                    runtime,
    +                    NotConfigured::DefaultRootUnusable {
    +                        root: root.to_owned(),
    +                        problem,
    +                    },
    +                ));
    +            }
    +            anyhow::bail!(
    +                "[backup.store] ({}) cannot be opened: {e}{}",
    +                describe_target(&config),
    +                rows_note(defaulted)
    +            );
    +        }
    +    };
    +    let kind = match opened.kind {
    +        BackupStoreKind::Filesystem => StoreKind::Filesystem,
    +        BackupStoreKind::S3 => StoreKind::S3,
    +        BackupStoreKind::Disabled => unreachable!("open_described never returns Disabled"),
    +    };
    +
    +    // The spool and its lock: configuration errors on every backend.
    +    let (spool, spool_lock) = match (resolved.spool_path, caller) {
    +        (Some(path), Caller::Server) => {
    +            let (spool, lock) = take_spool(&path)?;
    +            (Some(spool), Some(lock))
    +        }
    +        (Some(_), Caller::Command | Caller::ReadOnlyCommand) | (None, _) => (None, None),
    +    };
    +    let spool_line = spool
    +        .as_ref()
    +        .map(|s| format!(", spool {}", s.path.display()))
    +        .unwrap_or_default();
    +
    +    // An S3 store's checks are requests to the endpoint: they run once the
    +    // listener is up, in the reconciler's first cycle. A read-only command
    +    // runs no check at all.
    +    if caller == Caller::ReadOnlyCommand {
    +        let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool);
    +        return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +    }
    +    if kind == StoreKind::S3 {
    +        if caller == Caller::Server {
    +            tracing::info!(
    +                "backup store: {kind} {}{spool_line}; its validation and identity check run \
    +                 after the listener is up, and CreateBackup writes in-catalog backups in this \
    +                 release",
    +                opened.root
    +            );
    +        }
    +        let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool);
    +        return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +    }
    +
    +    // A filesystem store: `validate` and the identity check read the root
    +    // here (`run_local_checks`). A `validate` refusal is a configuration
    +    // error; a store condition or the identity check's failure degrades the
    +    // store; on the defaulted root without rows either is the warning.
    +    let outcome = match run_local_checks(&*opened.store, kind, &opened.root, catalog, backend).await
    +    {
    +        LocalChecks::Done(outcome) => outcome,
    +        LocalChecks::Refused(e) => {
    +            if defaulted && !default_root_has_rows(catalog, &opened.root).await {
    +                let problem = format!("it failed its validation: {e}");
    +                warn_default_root_unusable(&opened.root, &problem);
    +                return Ok(StoreSetup::without_store(
    +                    runtime,
    +                    NotConfigured::DefaultRootUnusable {
    +                        root: opened.root,
    +                        problem,
    +                    },
    +                ));
    +            }
    +            anyhow::bail!(
    +                "[backup.store] ({}) failed its startup validation: {e}{}",
    +                describe_target(&config),
    +                rows_note(defaulted)
    +            );
    +        }
    +        LocalChecks::TimedOut => {
    +            tracing::warn!(
    +                "backup store: {kind} {} did not answer its startup checks within {} s (a mount \
    +                 that has stopped answering?); the server starts with the store checking, and \
    +                 the reconciler's first cycle runs the checks again{}",
    +                opened.root,
    +                LOCAL_CHECK_TIMEOUT.as_secs(),
    +                if defaulted {
    +                    " (the default root beside the database file: a first check that fails \
    +                     while the catalog has no backup rows for it leaves the store unconfigured, \
    +                     as a startup failure would)"
    +                } else {
    +                    ""
    +                }
    +            );
    +            // `defaulted` travels with the store, so the reconciler applies
    +            // the rows rule to the first check's outcome.
    +            let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool)
    +                .defaulted(defaulted);
    +            return Ok(StoreSetup::with_store(runtime, configured, spool_lock));
    +        }
    +    };
    +    match &outcome {
    +        Ok(identity) => tracing::info!(
    +            "backup store: {kind} {} (store id {}){spool_line}; CreateBackup writes in-catalog \
    +             backups in this release, and the store serves restores and deletes of the backups \
    +             it holds",
    +            opened.root,
    +            identity.store_id
    +        ),
    +        Err(problem) if defaulted && !default_root_has_rows(catalog, &opened.root).await => {
    +            let problem = problem.to_string();
    +            warn_default_root_unusable(&opened.root, &problem);
    +            return Ok(StoreSetup::without_store(
    +                runtime,
    +                NotConfigured::DefaultRootUnusable {
    +                    root: opened.root,
    +                    problem,
    +                },
    +            ));
    +        }
    +        Err(problem) => tracing::error!(
    +            "backup store: {kind} {} is unhealthy: {problem}; the server starts, restore and \
    +             delete of a backup in the store are refused, /health reports the backup_store \
    +             component unhealthy, and the check runs again each reconciler cycle{}",
    +            opened.root,
    +            rows_note(defaulted)
    +        ),
    +    }
    +    let configured = ConfiguredStore::new(kind, opened.root, opened.store, spool, outcome);
    +    Ok(StoreSetup::with_store(runtime, configured, spool_lock))
    +}
    +
    +/// What a filesystem store's startup checks decided.
    +enum LocalChecks {
    +    /// `validate` passed and the identity check ran, or `validate` failed on
    +    /// a store condition (module doc): the identity, or the problem that
    +    /// makes the store unhealthy.
    +    Done(Result),
    +    /// `validate` refused the root for a configuration reason.
    +    Refused(StoreError),
    +    /// Nothing answered within [`LOCAL_CHECK_TIMEOUT`].
    +    TimedOut,
    +}
    +
    +/// A filesystem store's startup checks: `validate`, then
    +/// [`store_identity::establish`], bounded by [`LOCAL_CHECK_TIMEOUT`] so a
    +/// root on a mount that stops answering after `open` does not hold the
    +/// listener. The store's synchronous filesystem calls run on the blocking
    +/// pool, so the bound fires whichever call stalls; a timed-out check leaves
    +/// that thread parked until the mount answers (module doc).
    +async fn run_local_checks(
    +    store: &dyn BackupStore,
    +    kind: StoreKind,
    +    root: &str,
    +    catalog: &C,
    +    backend: &str,
    +) -> LocalChecks {
    +    let checks = async {
    +        match store.validate().await {
    +            Ok(()) => LocalChecks::Done(
    +                store_identity::establish(store, kind, root, catalog, backend).await,
    +            ),
    +            // A store condition, not a configuration error: the probe write
    +            // or the root's resolution failed (module doc).
    +            Err(e @ (StoreError::Transport { .. } | StoreError::Io { .. })) => {
    +                LocalChecks::Done(Err(StoreProblem::from(e)))
    +            }
    +            Err(e) => LocalChecks::Refused(e),
    +        }
    +    };
    +    tokio::time::timeout(LOCAL_CHECK_TIMEOUT, checks)
    +        .await
    +        .unwrap_or(LocalChecks::TimedOut)
    +}
    +
    +/// Whether the catalog has backup rows for the defaulted root at `root`
    +/// (the label rows record), which ends the root's leniency (module doc). A
    +/// catalog that cannot be read is answered as having rows, so a read
    +/// failure never makes a used root's backups vanish behind a warning.
    +async fn default_root_has_rows(catalog: &C, root: &str) -> bool {
    +    match catalog.read_store_ids(StoreKind::Filesystem, root).await {
    +        Ok(ids) => !ids.is_empty(),
    +        Err(e) => {
    +            tracing::warn!(
    +                "backup store: could not read the catalog's rows for the default root {root} \
    +                 ({e}); the root is handled as a configured one"
    +            );
    +            true
    +        }
    +    }
    +}
    +
    +/// The clause a defaulted root's startup error or unhealthy line carries,
    +/// saying why it was not the warning.
    +fn rows_note(defaulted: bool) -> &'static str {
    +    if defaulted {
    +        " (the default root beside the database file; the catalog has backup rows for it, so \
    +         it is handled as a configured root: fix it, or configure [backup.store])"
    +    } else {
    +        ""
    +    }
    +}
    +
    +/// The warning for a defaulted root whose creation, `open`, `validate` or
    +/// identity check failed while the catalog has no rows for it: the store is
    +/// left unconfigured, as the module doc says.
    +fn warn_default_root_unusable(root: &str, problem: &str) {
    +    tracing::warn!(
    +        "backup store: the default root {root} failed its check ({problem}); the catalog has no \
    +         backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog \
    +         backups. Fix the root (it must be owned by this user with mode 0700, under directories \
    +         owned by root or this user that others cannot write to, and its extenddb-store.json, \
    +         if any, must name this catalog: a catalog re-created by `destroy` and `init` needs the \
    +         old file removed or a new root), configure [backup.store] to choose another, or set \
    +         kind = \"disabled\" to silence this"
    +    );
    +}
    +
    +/// The store line of the console banner: what `[backup.store]` and the
    +/// backend select, before the store is opened. The configuration's own
    +/// refusals are not applied here; `configure` reports them. A filesystem
    +/// root is shown in the spelling the rows record and the startup log line
    +/// prints (the canonical directory, `banner_target`), with the configured
    +/// spelling beside it only where the two differ.
    +#[must_use]
    +pub fn banner_line(backup: &BackupConfig, database_file: Option<&Path>) -> String {
    +    let resolved = match backup.resolve_store(database_file) {
    +        Ok(resolved) => resolved,
    +        Err(e) => return format!("backup store: configuration error ({e})"),
    +    };
    +    let spool = resolved
    +        .spool_path
    +        .as_ref()
    +        .map(|p| format!(", spool {}", p.display()))
    +        .unwrap_or_default();
    +    match resolved.store {
    +        EffectiveStore::Unconfigured(Unconfigured::Absent) => {
    +            "backup store: none ([backup.store] is absent); CreateBackup writes in-catalog \
    +             backups"
    +                .to_owned()
    +        }
    +        EffectiveStore::Unconfigured(Unconfigured::Disabled) => {
    +            "backup store: disabled; CreateBackup writes in-catalog backups".to_owned()
    +        }
    +        EffectiveStore::Configured { config, defaulted } => format!(
    +            "backup store: {}{}{spool}",
    +            banner_target(&config),
    +            if defaulted {
    +                " (the default beside the database file)"
    +            } else {
    +                ""
    +            }
    +        ),
    +    }
    +}
    +
    +/// The root a configuration names, for a startup error: the bucket of an S3
    +/// store, the path of a filesystem one as written. Neither is a secret, and
    +/// an operator reading the error wants to know which store it is about.
    +fn describe_target(config: &extenddb_backup_store::BackupStoreConfig) -> String {
    +    match config.kind {
    +        BackupStoreKind::S3 => config.s3.as_ref().map_or_else(
    +            || "s3".to_owned(),
    +            |s3| format!("s3 bucket {:?}", s3.bucket),
    +        ),
    +        BackupStoreKind::Filesystem => config.filesystem.as_ref().map_or_else(
    +            || "filesystem".to_owned(),
    +            |fs| format!("filesystem path {}", shown_path(&fs.path)),
    +        ),
    +        BackupStoreKind::Disabled => "disabled".to_owned(),
    +    }
    +}
    +
    +/// The banner's spelling of a configuration's root: as [`describe_target`],
    +/// except that a filesystem root is the canonical directory the store opens
    +/// (the spelling its rows record, `backup list` prints and the startup log
    +/// line names), followed by the configured spelling when that differs (a
    +/// root reached through a symlink, or written relative). A root not created
    +/// yet (the default root before the server's first start) is its canonical
    +/// parent joined with its name; one whose parent does not resolve either is
    +/// shown as configured.
    +fn banner_target(config: &extenddb_backup_store::BackupStoreConfig) -> String {
    +    let Some(fs) = config
    +        .filesystem
    +        .as_ref()
    +        .filter(|_| config.kind == BackupStoreKind::Filesystem)
    +    else {
    +        return describe_target(config);
    +    };
    +    let canonical = std::fs::canonicalize(&fs.path).ok().or_else(|| {
    +        let name = fs.path.file_name()?;
    +        let parent = fs.path.parent().filter(|p| !p.as_os_str().is_empty())?;
    +        Some(std::fs::canonicalize(parent).ok()?.join(name))
    +    });
    +    match canonical {
    +        Some(canonical) if canonical == fs.path => {
    +            format!("filesystem {}", shown_path(&canonical))
    +        }
    +        Some(canonical) => format!(
    +            "filesystem {} (configured as {})",
    +            shown_path(&canonical),
    +            shown_path(&fs.path)
    +        ),
    +        None => format!("filesystem {}", shown_path(&fs.path)),
    +    }
    +}
    +
    +/// A path for an operator's line: quoted when it is empty or begins or ends
    +/// with whitespace, so the line shows where it ends.
    +fn shown_path(path: &Path) -> String {
    +    let path = path.to_string_lossy();
    +    if path.is_empty() {
    +        "\"\" (empty)".to_owned()
    +    } else if path.trim() != path {
    +        format!("{path:?}")
    +    } else {
    +        path.into_owned()
    +    }
    +}
    +
    +/// Create `path` with mode `0700` unless it exists.
    +fn create_private_dir_if_missing(path: &Path) -> std::io::Result<()> {
    +    if path.exists() {
    +        return Ok(());
    +    }
    +    let mut builder = std::fs::DirBuilder::new();
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::DirBuilderExt;
    +        builder.mode(0o700);
    +    }
    +    builder.recursive(true).create(path)
    +}
    +
    +/// Create the spool directory (`0700`) if missing and take its `.lock`
    +/// (`0600`, `flock`), refusing a spool another live instance holds. An
    +/// existing spool must meet the root's owner-only rule (owned by this user,
    +/// no group or other permission bits): it will hold downloaded backup
    +/// files, and a readable lock file would let any local user hold the spool.
    +fn take_spool(path: &Path) -> anyhow::Result<(Spool, SpoolLock)> {
    +    create_private_dir_if_missing(path).map_err(|e| {
    +        anyhow::anyhow!(
    +            "backup.spool_path {} cannot be created: {e}",
    +            path.display()
    +        )
    +    })?;
    +    let canonical = path
    +        .canonicalize()
    +        .map_err(|e| anyhow::anyhow!("backup.spool_path {} is not usable: {e}", path.display()))?;
    +    check_spool_mode(&canonical)?;
    +    let lock_path = canonical.join(SPOOL_LOCK_FILE);
    +    let mut options = std::fs::OpenOptions::new();
    +    options.read(true).write(true).create(true).truncate(false);
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::OpenOptionsExt;
    +        options.mode(0o600);
    +    }
    +    let file = options
    +        .open(&lock_path)
    +        .map_err(|e| anyhow::anyhow!("cannot open the spool lock {}: {e}", lock_path.display()))?;
    +    // `create` honours the umask and the file may predate this rule: flock
    +    // needs only a read handle, so a readable lock file would let any local
    +    // user hold the spool. Set the mode on every open.
    +    #[cfg(unix)]
    +    {
    +        use std::os::unix::fs::PermissionsExt;
    +        file.set_permissions(std::fs::Permissions::from_mode(0o600))
    +            .map_err(|e| anyhow::anyhow!("cannot set the mode of {}: {e}", lock_path.display()))?;
    +    }
    +    try_lock_exclusive(&file).map_err(|e| {
    +        if e.kind() == std::io::ErrorKind::WouldBlock {
    +            anyhow::anyhow!(
    +                "backup.spool_path {} is held by another running extenddb instance (lock {}); \
    +                 every instance on a host needs a spool of its own",
    +                canonical.display(),
    +                lock_path.display()
    +            )
    +        } else {
    +            anyhow::anyhow!("cannot lock {}: {e}", lock_path.display())
    +        }
    +    })?;
    +    let spool = Spool {
    +        spool_host: format!("{}:{}", hostname(), canonical.display()),
    +        path: canonical,
    +    };
    +    Ok((
    +        spool,
    +        SpoolLock {
    +            _file: file,
    +            path: lock_path,
    +        },
    +    ))
    +}
    +
    +/// The owner-only rule on the spool directory, the one the filesystem store
    +/// applies to its root (`extenddb_backup_store::check_private_directory`):
    +/// owned by this user with mode `0700`, under directories owned by root or
    +/// this user that others cannot write to.
    +fn check_spool_mode(spool: &Path) -> anyhow::Result<()> {
    +    if !spool.is_dir() {
    +        anyhow::bail!("backup.spool_path {} is not a directory", spool.display());
    +    }
    +    extenddb_backup_store::check_private_directory(spool, "spool").map_err(|e| {
    +        anyhow::anyhow!(
    +            "backup.spool_path {} is not private: {e}; the spool must have mode 0700 under \
    +             directories others cannot write to",
    +            spool.display()
    +        )
    +    })
    +}
    +
    +/// This host's name, for `spool_host` and a maintenance claim's holder;
    +/// `unknown-host` when the system does not say.
    +#[must_use]
    +pub fn hostname() -> String {
    +    #[cfg(unix)]
    +    {
    +        let mut buf = [0u8; 256];
    +        // SAFETY: `gethostname` writes at most `len` bytes into the buffer it
    +        // is given, which is valid for its whole length.
    +        let rc = unsafe { libc::gethostname(buf.as_mut_ptr().cast(), buf.len()) };
    +        if rc == 0 {
    +            let end = buf.iter().position(|b| *b == 0).unwrap_or(buf.len());
    +            if let Ok(name) = std::str::from_utf8(&buf[..end])
    +                && !name.is_empty()
    +            {
    +                return name.to_owned();
    +            }
    +        }
    +    }
    +    "unknown-host".to_owned()
    +}
    +
    +#[cfg(unix)]
    +fn try_lock_exclusive(file: &std::fs::File) -> std::io::Result<()> {
    +    use std::os::unix::io::AsRawFd;
    +    // SAFETY: `flock` takes a file descriptor and flags and touches no memory
    +    // the caller owns; the descriptor belongs to `file`, borrowed for the
    +    // call.
    +    let rc = unsafe { libc::flock(file.as_raw_fd(), libc::LOCK_EX | libc::LOCK_NB) };
    +    if rc == 0 {
    +        Ok(())
    +    } else {
    +        Err(std::io::Error::last_os_error())
    +    }
    +}
    +
    +#[cfg(not(unix))]
    +fn try_lock_exclusive(_file: &std::fs::File) -> std::io::Result<()> {
    +    tracing::warn!(
    +        "no file lock is taken on the spool on this platform: nothing prevents a second \
    +         extenddb instance from using the same spool_path"
    +    );
    +    Ok(())
    +}
    +
    +#[cfg(all(test, unix))]
    +mod tests {
    +    use super::*;
    +
    +    fn mode_of(path: &Path) -> u32 {
    +        use std::os::unix::fs::PermissionsExt;
    +        std::fs::metadata(path).unwrap().permissions().mode() & 0o777
    +    }
    +
    +    /// The spool is created private with a private lock file, a second take
    +    /// of the same spool is refused while the first lock lives, and a spool
    +    /// elsewhere is not.
    +    #[test]
    +    fn the_spool_lock_refuses_a_second_instance() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        let (first, lock) = take_spool(&spool).unwrap();
    +        assert_eq!(mode_of(&spool), 0o700);
    +        assert_eq!(mode_of(&spool.join(SPOOL_LOCK_FILE)), 0o600);
    +        assert!(
    +            first
    +                .spool_host
    +                .ends_with(&format!(":{}", spool.canonicalize().unwrap().display()))
    +        );
    +        assert!(!first.spool_host.starts_with(':'), "the host name is set");
    +        let err = take_spool(&spool).unwrap_err().to_string();
    +        assert!(
    +            err.contains("held by another running extenddb instance"),
    +            "{err}"
    +        );
    +        assert!(err.contains(&spool.display().to_string()), "{err}");
    +        take_spool(&dir.path().join("other")).unwrap();
    +        drop(lock);
    +        take_spool(&spool).expect("free once the holder is gone");
    +    }
    +
    +    /// An existing spool that group or others can reach, or that another
    +    /// user owns, is refused naming the path and the mode; one with mode
    +    /// 0700 is taken.
    +    #[test]
    +    fn an_existing_spool_must_be_owner_only() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        std::fs::create_dir(&spool).unwrap();
    +        for mode in [0o755u32, 0o750, 0o707, 0o770] {
    +            std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(mode)).unwrap();
    +            let err = take_spool(&spool).unwrap_err().to_string();
    +            assert!(
    +                err.contains(&format!("{mode:04o}")) && err.contains("mode 0700"),
    +                "{mode:04o}: {err}"
    +            );
    +            assert!(err.contains("not private"), "{err}");
    +            assert!(err.contains(&spool.display().to_string()), "{err}");
    +            assert!(
    +                !spool.join(SPOOL_LOCK_FILE).exists(),
    +                "no lock file is created"
    +            );
    +        }
    +        std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        take_spool(&spool).expect("an owner-only spool is taken");
    +        // A spool under a parent others can write to is refused as the root
    +        // would be: the parent could be swapped under it.
    +        let parent = dir.path().join("open");
    +        std::fs::create_dir(&parent).unwrap();
    +        std::fs::set_permissions(&parent, std::fs::Permissions::from_mode(0o777)).unwrap();
    +        let under = parent.join("spool");
    +        std::fs::create_dir(&under).unwrap();
    +        std::fs::set_permissions(&under, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let err = take_spool(&under).unwrap_err().to_string();
    +        assert!(
    +            err.contains("an ancestor of the spool") && err.contains("0777"),
    +            "{err}"
    +        );
    +    }
    +
    +    /// A lock file that predates the mode rule is narrowed to 0600 on open.
    +    #[test]
    +    fn a_permissive_lock_file_is_narrowed() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let spool = dir.path().join("spool");
    +        std::fs::create_dir(&spool).unwrap();
    +        std::fs::set_permissions(&spool, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let lock = spool.join(SPOOL_LOCK_FILE);
    +        std::fs::write(&lock, b"").unwrap();
    +        std::fs::set_permissions(&lock, std::fs::Permissions::from_mode(0o644)).unwrap();
    +        let _held = take_spool(&spool).unwrap();
    +        assert_eq!(mode_of(&lock), 0o600);
    +    }
    +
    +    // --- the startup checks against doubles of the catalog and the store ---
    +
    +    use extenddb_backup_store::{ByteStream, ObjectMeta};
    +    use extenddb_core::backup_format::BackupArn;
    +    use extenddb_storage::BoxedFuture;
    +    use extenddb_storage::backup_catalog::{
    +        BackupKey, BackupRow, BackupStatus, DeleteStart, JobClaim, MaintenanceClaim, RestoreJobRow,
    +        RestoreTargetKey, TimeBounds,
    +    };
    +    use extenddb_storage::error::StorageError;
    +    use futures::stream::BoxStream;
    +
    +    /// A catalog whose rows for every root carry `ids`, or whose rows cannot
    +    /// be read (`rows_unreadable`), with no backup row to list; every other
    +    /// method is an error, since the startup checks never reach it.
    +    #[derive(Default)]
    +    struct StubCatalog {
    +        ids: Vec,
    +        rows_unreadable: bool,
    +    }
    +
    +    fn unused() -> BoxedFuture<'static, Result> {
    +        Box::pin(async {
    +            Err(StorageError::Internal(
    +                "not reached by the startup checks".to_owned(),
    +            ))
    +        })
    +    }
    +
    +    impl BackupCatalog for StubCatalog {
    +        fn catalog_id(&self) -> BoxedFuture<'_, Result> {
    +            Box::pin(async { Ok("catalog-1".to_owned()) })
    +        }
    +        fn claim_backup<'a>(
    +            &'a self,
    +            _row: &'a BackupRow,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn list_backup_rows<'a>(
    +            &'a self,
    +            _status: Option,
    +            _store: &'a StoreIdentity,
    +        ) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn newest_available_backup_row<'a>(
    +            &'a self,
    +            _store: &'a StoreIdentity,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            Box::pin(async { Ok(None) })
    +        }
    +        fn list_running_restore_jobs(
    +            &self,
    +        ) -> BoxedFuture<
    +            '_,
    +            Result, StorageError>,
    +        > {
    +            unused()
    +        }
    +        fn list_account_backup_rows<'a>(
    +            &'a self,
    +            _account_id: &'a str,
    +            _table_name: Option<&'a str>,
    +            _bounds: &'a TimeBounds,
    +        ) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn read_backup_row<'a>(
    +            &'a self,
    +            _arn: &'a BackupArn,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn start_backup_delete<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn set_backup_deleted<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn remove_backup_row<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn remove_dead_creating_backup_row<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn list_backup_rows_older_than<'a>(
    +            &'a self,
    +            _status: BackupStatus,
    +            _store: &'a StoreIdentity,
    +            _clock: extenddb_storage::backup_catalog::StaleClock,
    +            _age: std::time::Duration,
    +        ) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn backup_in_use<'a>(
    +            &'a self,
    +            _key: &'a BackupKey,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn remove_restore_target<'a>(
    +            &'a self,
    +            _key: &'a RestoreTargetKey,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn list_dead_restore_jobs(
    +            &self,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'_, Result, StorageError>> {
    +            unused()
    +        }
    +        fn fail_restore_job<'a>(
    +            &'a self,
    +            _key: &'a RestoreTargetKey,
    +            _owner_generation: u64,
    +            _heartbeat_at: f64,
    +            _reason: &'a str,
    +        ) -> BoxedFuture<'a, Result> {
    +            unused()
    +        }
    +        fn read_store_ids<'a>(
    +            &'a self,
    +            _store_kind: StoreKind,
    +            _store_root: &'a str,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            if self.rows_unreadable {
    +                return Box::pin(async {
    +                    Err(StorageError::Internal(
    +                        "the catalog is not answering".to_owned(),
    +                    ))
    +                });
    +            }
    +            let ids = self.ids.clone();
    +            Box::pin(async move { Ok(ids) })
    +        }
    +        fn maintenance_claim<'a>(
    +            &'a self,
    +            _holder: &'a str,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'a, Result, StorageError>> {
    +            unused()
    +        }
    +        fn check_maintenance_claim(
    +            &self,
    +            _stale_after: std::time::Duration,
    +        ) -> BoxedFuture<'_, Result> {
    +            unused()
    +        }
    +    }
    +
    +    /// A store whose `validate` answers as told: never (a mount that has
    +    /// stopped answering), or with a given error; nothing else is reached.
    +    enum Validate {
    +        Never,
    +        Fails(fn() -> StoreError),
    +    }
    +
    +    struct FakeStore(Validate);
    +
    +    fn not_reached() -> futures::future::BoxFuture<'static, Result>
    +    {
    +        Box::pin(async { Err(StoreError::Other("not reached".to_owned())) })
    +    }
    +
    +    impl BackupStore for FakeStore {
    +        fn put<'a>(
    +            &'a self,
    +            _key: &'a str,
    +            _body: ByteStream,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn put_if_absent<'a>(
    +            &'a self,
    +            _key: &'a str,
    +            _body: ByteStream,
    +        ) -> futures::future::BoxFuture<'a, Result, StoreError>> {
    +            not_reached()
    +        }
    +        fn get<'a>(
    +            &'a self,
    +            _key: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn head<'a>(
    +            &'a self,
    +            _key: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result, StoreError>> {
    +            not_reached()
    +        }
    +        fn list<'a>(&'a self, _prefix: &'a str) -> BoxStream<'a, Result> {
    +            Box::pin(futures::stream::empty())
    +        }
    +        fn delete_prefix<'a>(
    +            &'a self,
    +            _prefix: &'a str,
    +        ) -> futures::future::BoxFuture<'a, Result> {
    +            not_reached()
    +        }
    +        fn validate(&self) -> futures::future::BoxFuture<'_, Result<(), StoreError>> {
    +            match &self.0 {
    +                Validate::Never => Box::pin(std::future::pending()),
    +                Validate::Fails(make) => Box::pin(std::future::ready(Err(make()))),
    +            }
    +        }
    +    }
    +
    +    /// A SQLite deployment's `[backup]` without a store, and the database
    +    /// file whose directory the default root is resolved against.
    +    fn defaulted(dir: &Path) -> (BackupConfig, PathBuf) {
    +        (BackupConfig::default(), dir.join("extenddb.sqlite"))
    +    }
    +
    +    /// The defaulted root's leniency ends with the catalog's first row for
    +    /// it. The root is created empty (no identity file) each time: with no
    +    /// rows for it, a fresh root gets its file and the store is healthy;
    +    /// with rows for it and no file, an explicit root would be unhealthy
    +    /// (`identity file missing`), and so is the defaulted one, configured
    +    /// and reported rather than left unconfigured behind a warning.
    +    #[tokio::test]
    +    async fn a_defaulted_root_with_rows_is_handled_as_a_configured_one() {
    +        let dir = tempfile::tempdir().unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let root = dir.path().join("backups");
    +
    +        // Rows for the root (seeded, as a server that used the root leaves
    +        // them), no identity file: an explicit root's `identity file
    +        // missing`, on `/health` and in the refusals.
    +        let with_rows = StubCatalog {
    +            ids: vec!["store-1".to_owned()],
    +            ..StubCatalog::default()
    +        };
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &with_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None, "the store is configured");
    +        let store = setup
    +            .runtime
    +            .configured_store()
    +            .expect("a defaulted root with rows is a configured store");
    +        assert_eq!(store.kind, StoreKind::Filesystem);
    +        assert_eq!(
    +            store.root,
    +            root.canonicalize().unwrap().display().to_string()
    +        );
    +        assert_eq!(
    +            store.health().category(),
    +            Some("identity file missing"),
    +            "{:?}",
    +            store.health()
    +        );
    +        let report = setup.runtime.store_health().unwrap();
    +        assert_eq!(
    +            (report.status, report.reason),
    +            ("unhealthy", Some("identity file missing"))
    +        );
    +        assert!(
    +            !root.join(store_identity::STORE_IDENTITY_KEY).exists(),
    +            "no file is created over rows that name another id"
    +        );
    +
    +        // No rows: the fresh-root case, the file created, the store healthy.
    +        let without_rows = StubCatalog::default();
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &without_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None);
    +        assert!(setup.runtime.configured_store().unwrap().is_healthy());
    +        assert!(root.join(store_identity::STORE_IDENTITY_KEY).exists());
    +
    +        // No rows and a failed check (the file names another catalog, as a
    +        // catalog re-created by `destroy` and `init` leaves it): the
    +        // warning path, with the reason carried out for the commands.
    +        let foreign = store_identity::StoreIdentityFile::parse(
    +            br#"{"format_version":1,"store_id":"s-9","catalog_id":"other","backend":"sqlite","created_at_ms":0}"#,
    +        )
    +        .unwrap();
    +        std::fs::write(
    +            root.join(store_identity::STORE_IDENTITY_KEY),
    +            foreign.to_bytes(),
    +        )
    +        .unwrap();
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &without_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert!(setup.runtime.configured_store().is_none());
    +        let why = setup.not_configured.expect("the reason is carried out");
    +        assert!(
    +            matches!(&why, NotConfigured::DefaultRootUnusable { root, problem }
    +                if root.contains("backups") && problem.contains("names catalog other")),
    +            "{why:?}"
    +        );
    +        assert!(why.to_string().contains("no rows for it"), "{why}");
    +        // The same file over rows for the root: a configured, unhealthy
    +        // store (`owned by another catalog`), not the warning.
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &with_rows,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(
    +            setup
    +                .runtime
    +                .configured_store()
    +                .unwrap()
    +                .health()
    +                .category(),
    +            Some("owned by another catalog")
    +        );
    +    }
    +
    +    /// A defaulted root `open` refuses (a parent others can write to) is the
    +    /// warning while the catalog has no rows for it, and a startup error
    +    /// naming the root once it has; an explicitly configured root is the
    +    /// startup error either way.
    +    #[tokio::test]
    +    async fn a_defaulted_root_open_refuses_is_a_warning_only_without_rows() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o777)).unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &StubCatalog::default(),
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        let why = setup.not_configured.expect("the warning path");
    +        assert!(
    +            matches!(&why, NotConfigured::DefaultRootUnusable { problem, .. }
    +                if problem.contains("writable by group or others")),
    +            "{why:?}"
    +        );
    +        let err = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &StubCatalog {
    +                ids: vec!["store-1".to_owned()],
    +                ..StubCatalog::default()
    +            },
    +            Caller::Server,
    +        )
    +        .await
    +        .err()
    +        .expect("with rows, the refusal stops startup")
    +        .to_string();
    +        assert!(err.contains("cannot be opened"), "{err}");
    +        assert!(err.contains("backup rows for it"), "{err}");
    +        assert!(err.contains(&dir.path().display().to_string()), "{err}");
    +    }
    +
    +    /// A catalog whose rows cannot be read is answered as having rows, so a
    +    /// defaulted root whose check then fails is a configured, unhealthy
    +    /// store and not the warning that leaves the store unconfigured: the
    +    /// root may hold backups the read would have shown. Here the root
    +    /// carries a foreign identity file; the read failure reaches `establish`
    +    /// first, so the problem is the catalog's (`catalog unreadable`), and
    +    /// the next cycle's check reads again.
    +    #[tokio::test]
    +    async fn an_unreadable_catalog_keeps_a_failing_default_root_configured() {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().unwrap();
    +        let (backup, database) = defaulted(dir.path());
    +        let root = dir.path().join("backups");
    +        std::fs::create_dir(&root).unwrap();
    +        std::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).unwrap();
    +        let foreign = store_identity::StoreIdentityFile::parse(
    +            br#"{"format_version":1,"store_id":"s-9","catalog_id":"other","backend":"sqlite","created_at_ms":0}"#,
    +        )
    +        .unwrap();
    +        std::fs::write(
    +            root.join(store_identity::STORE_IDENTITY_KEY),
    +            foreign.to_bytes(),
    +        )
    +        .unwrap();
    +        let unreadable = StubCatalog {
    +            rows_unreadable: true,
    +            ..StubCatalog::default()
    +        };
    +        let setup = configure_described(
    +            &backup,
    +            Some(&database),
    +            "sqlite",
    +            &unreadable,
    +            Caller::Server,
    +        )
    +        .await
    +        .unwrap();
    +        assert_eq!(setup.not_configured, None, "the store is configured");
    +        let store = setup
    +            .runtime
    +            .configured_store()
    +            .expect("a root whose rows cannot be read is a configured store");
    +        assert_eq!(
    +            store.health().category(),
    +            Some("catalog unreadable"),
    +            "{:?}",
    +            store.health()
    +        );
    +    }
    +
    +    /// The timed startup checks: a store that never answers leaves the
    +    /// server starting with the store `checking` after `LOCAL_CHECK_TIMEOUT`
    +    /// (not sooner, since a store that answers must get its outcome; not
    +    /// later, since the listener waits on this); `validate`'s I/O failure
    +    /// is a store condition that degrades (`failed validation`), its denial
    +    /// a configuration refusal.
    +    #[tokio::test(start_paused = true)]
    +    async fn the_startup_checks_are_bounded_and_split_store_conditions_from_refusals() {
    +        let catalog = StubCatalog::default();
    +        // The bound's value is part of the rule (a local disk answers in
    +        // milliseconds), so it is pinned here rather than read back.
    +        assert_eq!(LOCAL_CHECK_TIMEOUT, std::time::Duration::from_secs(5));
    +        let started = tokio::time::Instant::now();
    +        let stalled = FakeStore(Validate::Never);
    +        // The test's own guard, far past the bound: a check with no bound
    +        // fails here instead of hanging.
    +        let outcome = tokio::time::timeout(
    +            std::time::Duration::from_secs(120),
    +            run_local_checks(&stalled, StoreKind::Filesystem, "/r", &catalog, "test"),
    +        )
    +        .await
    +        .expect("the checks return at their bound");
    +        assert!(matches!(outcome, LocalChecks::TimedOut));
    +        assert_eq!(
    +            started.elapsed(),
    +            LOCAL_CHECK_TIMEOUT,
    +            "the bound is the whole wait, no more and no less"
    +        );
    +
    +        let io = FakeStore(Validate::Fails(|| StoreError::Io {
    +            context: "write probe /r/.put-validate.tmp".to_owned(),
    +            source: std::io::Error::from_raw_os_error(28),
    +        }));
    +        let outcome = run_local_checks(&io, StoreKind::Filesystem, "/r", &catalog, "test").await;
    +        let LocalChecks::Done(Err(problem)) = outcome else {
    +            panic!("an I/O failure of the probe degrades the store");
    +        };
    +        assert_eq!(problem.category(), "failed validation");
    +        assert!(problem.to_string().contains("write probe"), "{problem}");
    +
    +        let denied = FakeStore(Validate::Fails(|| {
    +            StoreError::PermissionDenied("/r has mode 0755".to_owned())
    +        }));
    +        let outcome =
    +            run_local_checks(&denied, StoreKind::Filesystem, "/r", &catalog, "test").await;
    +        assert!(matches!(
    +            outcome,
    +            LocalChecks::Refused(StoreError::PermissionDenied(_))
    +        ));
    +    }
    +
    +    /// `NotConfigured`'s sentences: each says why there are no store-backed
    +    /// backups, and the unusable root's names the root and the failure.
    +    #[test]
    +    fn not_configured_reads_as_the_commands_print_it() {
    +        assert!(
    +            NotConfigured::Absent
    +                .to_string()
    +                .contains("[backup.store] is absent")
    +        );
    +        assert!(NotConfigured::Disabled.to_string().contains("is disabled"));
    +        let missing = NotConfigured::DefaultRootMissing(PathBuf::from("/srv/db/backups"));
    +        assert!(
    +            missing
    +                .to_string()
    +                .contains("/srv/db/backups does not exist yet")
    +        );
    +        let unusable = NotConfigured::DefaultRootUnusable {
    +            root: "/srv/db/backups".to_owned(),
    +            problem: "extenddb-store.json names catalog other".to_owned(),
    +        }
    +        .to_string();
    +        assert!(
    +            unusable.contains("/srv/db/backups failed its check"),
    +            "{unusable}"
    +        );
    +        assert!(unusable.contains("names catalog other"), "{unusable}");
    +        assert!(unusable.contains("no rows for it"), "{unusable}");
    +    }
    +
    +    /// The banner quotes a root that is empty or begins or ends with
    +    /// whitespace so the line shows where the path ends (none of these
    +    /// resolves to another spelling on the test host).
    +    #[test]
    +    fn a_root_with_edge_whitespace_is_quoted_in_the_banner() {
    +        let line = |path: &str| {
    +            let backup = BackupConfig {
    +                store: Some(extenddb_backup_store::BackupStoreConfig {
    +                    kind: BackupStoreKind::Filesystem,
    +                    filesystem: Some(extenddb_backup_store::FilesystemStoreConfig {
    +                        path: PathBuf::from(path),
    +                    }),
    +                    s3: None,
    +                }),
    +                ..BackupConfig::default()
    +            };
    +            banner_line(&backup, None)
    +        };
    +        let missing = format!("/nonexistent-{}/backups", uuid::Uuid::new_v4().simple());
    +        assert!(
    +            line(&format!("{missing} ")).contains(&format!("filesystem \"{missing} \"")),
    +            "{}",
    +            line(&format!("{missing} "))
    +        );
    +        assert!(
    +            line(&format!(" {missing}")).contains(&format!("filesystem \" {missing}\"")),
    +            "{}",
    +            line(&format!(" {missing}"))
    +        );
    +        assert!(
    +            line(&missing).ends_with(&format!("filesystem {missing}")),
    +            "{}",
    +            line(&missing)
    +        );
    +        assert!(line("").contains("filesystem \"\" (empty)"), "{}", line(""));
    +    }
    +
    +    /// The banner spells a filesystem root as the store's rows and the
    +    /// startup log line do, the canonical directory, and names the
    +    /// configured spelling beside it only where the two differ: a root
    +    /// reached through a symlink, and a root not created yet under a
    +    /// symlinked parent (the default root before the first start).
    +    #[cfg(unix)]
    +    #[test]
    +    fn the_banner_shows_the_canonical_root_and_the_configured_spelling_where_they_differ() {
    +        let dir = std::env::temp_dir().join(format!(
    +            "extenddb-banner-root-{}",
    +            uuid::Uuid::new_v4().simple()
    +        ));
    +        let real = dir.join("real");
    +        std::fs::create_dir_all(real.join("backups")).unwrap();
    +        let link = dir.join("link");
    +        std::os::unix::fs::symlink(&real, &link).unwrap();
    +        let canonical_real = std::fs::canonicalize(&real).unwrap();
    +        let line = |path: &Path| {
    +            let backup = BackupConfig {
    +                store: Some(extenddb_backup_store::BackupStoreConfig {
    +                    kind: BackupStoreKind::Filesystem,
    +                    filesystem: Some(extenddb_backup_store::FilesystemStoreConfig {
    +                        path: path.to_path_buf(),
    +                    }),
    +                    s3: None,
    +                }),
    +                ..BackupConfig::default()
    +            };
    +            banner_line(&backup, None)
    +        };
    +        let canonical = canonical_real.join("backups");
    +        assert_eq!(
    +            line(&canonical),
    +            format!("backup store: filesystem {}", canonical.display()),
    +            "one spelling where the configured root is canonical"
    +        );
    +        for configured in [link.join("backups"), link.join("not-yet")] {
    +            let expected = canonical_real.join(configured.file_name().unwrap());
    +            assert_eq!(
    +                line(&configured),
    +                format!(
    +                    "backup store: filesystem {} (configured as {})",
    +                    expected.display(),
    +                    configured.display()
    +                ),
    +            );
    +        }
    +        let _ = std::fs::remove_dir_all(&dir);
    +    }
    +}
    diff --git a/crates/server/src/lib.rs b/crates/server/src/lib.rs
    index 73e1a1fe9..dd4cb480b 100755
    --- a/crates/server/src/lib.rs
    +++ b/crates/server/src/lib.rs
    @@ -11,6 +11,7 @@
     pub(crate) mod authorization;
     pub mod authz_cache;
     pub(crate) mod authz_resource;
    +pub mod backup_store;
     pub mod console;
     pub mod dev_credentials;
     mod handler;
    @@ -29,7 +30,8 @@ mod throttle_helpers;
     mod workers;
     
     pub use serve::{
    -    BuildInfo, CatalogRefusal, LogTarget, RegistrationLost, ServeParams, log_to_syslog_raw, serve,
    +    BuildInfo, CatalogRefusal, LogTarget, RegistrationLost, ServeParams, catalog_refusal,
    +    log_to_syslog_raw, serve,
     };
     
     use std::path::PathBuf;
    @@ -301,9 +303,39 @@ pub async fn start_server(
         Ok(())
     }
     
    -/// GET /health — REQ-OBS-006
    -async fn health_check() -> impl IntoResponse {
    -    (StatusCode::OK, axum::Json(json!({"status": "healthy"})))
    +/// GET /health — REQ-OBS-006.
    +///
    +/// Always 200 while the server answers: the data plane is up. `status` is
    +/// `healthy`, or `degraded` when a component is unhealthy, and `components`
    +/// carries each component the server has. Today the one component is
    +/// `backup_store`, present when a store is configured: `status` `healthy`,
    +/// `checking` (an S3 store's first check has not returned; the server's
    +/// `status` stays `healthy`) or `unhealthy`, `kind` (`filesystem` or `s3`)
    +/// and, when unhealthy, `reason`, the problem's category in one or two
    +/// words. The endpoint is unauthenticated, so the reason carries no id, ARN,
    +/// table name or root; the operator's log line does. A store problem never
    +/// takes the data plane out of rotation, which is why the HTTP status stays
    +/// 200.
    +async fn health_check(
    +    axum::extract::State(state): axum::extract::State>,
    +) -> impl IntoResponse {
    +    let mut components = serde_json::Map::new();
    +    let mut status = "healthy";
    +    if let Some(store) = state.backup_runtime.store_health() {
    +        let mut component = json!({
    +            "status": store.status,
    +            "kind": store.kind,
    +        });
    +        if let Some(reason) = store.reason {
    +            component["reason"] = serde_json::Value::String(reason.to_owned());
    +            status = "degraded";
    +        }
    +        components.insert("backup_store".to_owned(), component);
    +    }
    +    (
    +        StatusCode::OK,
    +        axum::Json(json!({ "status": status, "components": components })),
    +    )
     }
     
     async fn graceful_shutdown(pid_file: Option, admission: registration::Admission) {
    diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs
    index 64ad37527..dc4acc2f4 100644
    --- a/crates/server/src/serve.rs
    +++ b/crates/server/src/serve.rs
    @@ -176,6 +176,43 @@ impl std::fmt::Display for CatalogRefusal {
     
     impl std::error::Error for CatalogRefusal {}
     
    +/// The refusal for a catalog error the backend's factory returned, for
    +/// `serve` and for every command that opens the catalog through the factory
    +/// (`extenddb backup list` and `purge`): the message, with the command that
    +/// resolves the status, and the exit code `migrate` and `verify` use for it
    +/// (3 for a catalog behind this binary, 6 while a migration runs).
    +pub async fn catalog_refusal(
    +    error: &extenddb_storage::migrations::MigrationError,
    +    release: &'static str,
    +    app_config: &config::AppConfig,
    +    config_path: Option<&str>,
    +) -> CatalogRefusal {
    +    use extenddb_storage::migrations::{Target, advice, installed};
    +    // A server started without a config file (a dev server configured
    +    // by `EXTENDDB__` variables) advises `migrate` without one, which
    +    // reads the same variables. The backup clause names the data
    +    // store the catalog records.
    +    let target = Target::at(app_config.storage.connection_config());
    +    let named = match installed() {
    +        Some(migrations) => Target {
    +            data_store: migrations.data_store_name(&target).await,
    +            ..target
    +        },
    +        None => target,
    +    };
    +    let advice = advice::AdviceFor::new(
    +        release,
    +        config::redact_password(app_config.storage.connection_config()),
    +        installed(),
    +        &named,
    +        config_path,
    +    );
    +    CatalogRefusal {
    +        exit_code: error.exit_code(),
    +        message: advice::error_message(error, &advice.advice()),
    +    }
    +}
    +
     /// `serve` stopped because the backend's serving registration was lost and
     /// could not be taken again. It stopped through the shutdown path a signal
     /// takes: the listener drained, the workers stopped and the PID file was
    @@ -358,30 +395,13 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
         {
             Ok(components) => components,
             Err(extenddb_storage::server_components::BackendError::Catalog(error)) => {
    -            use extenddb_storage::migrations::{Target, advice, installed};
    -            // A server started without a config file (a dev server configured
    -            // by `EXTENDDB__` variables) advises `migrate` without one, which
    -            // reads the same variables. The backup clause names the data
    -            // store the catalog records.
    -            let target = Target::at(app_config.storage.connection_config());
    -            let named = match installed() {
    -                Some(migrations) => Target {
    -                    data_store: migrations.data_store_name(&target).await,
    -                    ..target
    -                },
    -                None => target,
    -            };
    -            let advice = advice::AdviceFor::new(
    +            return Err(catalog_refusal(
    +                &error,
                     build.version,
    -                config::redact_password(app_config.storage.connection_config()),
    -                installed(),
    -                &named,
    +                &app_config,
                     config_path.as_deref(),
    -            );
    -            return Err(CatalogRefusal {
    -                exit_code: error.exit_code(),
    -                message: advice::error_message(&error, &advice.advice()),
    -            }
    +            )
    +            .await
                 .into());
             }
             Err(other) => return Err(other.into()),
    @@ -641,9 +661,21 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
             initial_throttling,
         ));
     
    -    let backup_runtime = Arc::new(extenddb_engine::BackupRuntime::new(
    -        app_config.backup.max_concurrent_jobs,
    -    ));
    +    // The backup store: resolved per backend, opened, validated, its identity
    +    // checked. A configuration error stops startup here; a transport or
    +    // identity failure starts the server with the store unhealthy
    +    // (`backup_store` module doc). The spool lock lives until shutdown.
    +    let backend_name = app_config.storage.backend.clone();
    +    let (backup_runtime, _spool_lock) = crate::backup_store::configure(
    +        &app_config.backup,
    +        app_config.storage.database_file().as_deref(),
    +        &backend_name,
    +        &storage,
    +        crate::backup_store::Caller::Server,
    +    )
    +    .await?;
    +    let backup_runtime = Arc::new(backup_runtime);
    +    let backup_runtime_for_reconciler = Arc::clone(&backup_runtime);
         let state = AppState {
             storage: storage.clone(),
             backup_runtime,
    @@ -698,6 +730,7 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
             let config_log_level = app_config.logging.level.clone();
             let metrics = metrics.clone();
             let storage = storage.clone();
    +        let backup_runtime = backup_runtime_for_reconciler;
             Box::new(move |token: CancellationToken| {
                 let catalog_store = catalog_store.clone();
                 let reload_handle = reload_handle.clone();
    @@ -707,6 +740,8 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
                 let hooks = hooks.clone();
                 let storage = storage.clone();
                 let reconciler_state = reconciler_state.clone();
    +            let backup_runtime = Arc::clone(&backup_runtime);
    +            let backend_name = backend_name.clone();
                 futures::FutureExt::boxed(async move {
                     let mut handles = vec![
                         // D-22: Poll log_level from the settings table.
    @@ -746,7 +781,9 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> {
                         // them.
                         tokio::spawn(extenddb_engine::reconciler::run(
                             storage,
    +                        backup_runtime,
                             reconciler_settings,
    +                        backend_name,
                             metrics.clone(),
                             reconciler_state,
                             token.clone(),
    diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs
    index 0acf7db12..68bc77c50 100644
    --- a/crates/storage-cassandra/src/backup_catalog.rs
    +++ b/crates/storage-cassandra/src/backup_catalog.rs
    @@ -41,6 +41,21 @@
     //! server's `WriteTimeout` or `WriteFailure`, has an unknown outcome
     //! (`WriteOutcome::Unknown`: the Paxos round may still commit); every other
     //! error is `Err`.
    +//!
    +//! The maintenance claim is the one row of `backup_maintenance`, taken with
    +//! `INSERT ... IF NOT EXISTS`; when a row exists, one whose heartbeat is older
    +//! than the stale window on the coordinator's clock is taken over with
    +//! `IF owner_generation = `, so two commands racing for a dead holder's
    +//! row see one win. An insert or a takeover whose answer was lost is settled
    +//! by a re-read at `LOCAL_SERIAL`: a row carrying the call's own generation is
    +//! its claim. Heartbeats and the
    +//! release are lightweight transactions on the generation. The two reads of
    +//! the claim row that decide whether another process's claim is live
    +//! (`check_maintenance_claim`, the row before a takeover) run at
    +//! `LOCAL_SERIAL`, so a heartbeat an LWT just wrote is seen; the RUNNING
    +//! restore records `list_dead_restore_jobs` scans are a range read, where a
    +//! serial level is a local quorum with no Paxos read, so the scan reads at
    +//! quorum and the coordinator clock after it.
     
     use std::time::Duration;
     
    @@ -55,8 +70,9 @@ use cdrs_tokio::types::value::Value;
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, billing_mode_name, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock,
    +    StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, billing_mode_name,
    +    new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -387,6 +403,59 @@ impl JobClaim for CassandraBackupClaim {
     impl CassandraEngine {
         /// The index row `backup_arn` names in `account_id`, if there is one; an
         /// in-catalog row is `None`.
    +    /// Remove the three index entries of a `backups_by_arn` row this process
    +    /// has deleted: its status partition, its account listing and its table
    +    /// listing, in one logged batch.
    +    async fn remove_index_entries(&self, row: &BackupRow) -> Result<(), StorageError> {
    +        let catalog = self.catalog_keyspace();
    +        let created_at = seconds_to_millis(row.created_at);
    +        let batch = BatchQueryBuilder::new()
    +            .with_consistency(Consistency::LocalQuorum)
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backup_index_by_status \
    +                     WHERE store_root = ? AND backup_status = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.store.store_root.as_str(),
    +                    row.status.as_str(),
    +                    row.backup_arn.as_str()
    +                ),
    +            )
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backups_by_account \
    +                     WHERE account_id = ? AND created_at = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.account_id.as_str(),
    +                    created_at,
    +                    row.backup_arn.as_str()
    +                ),
    +            )
    +            .add_query(
    +                format!(
    +                    "DELETE FROM {catalog}.backups_by_table \
    +                     WHERE account_id = ? AND table_name = ? AND created_at = ? AND backup_arn = ?"
    +                ),
    +                cdrs_tokio::query_values!(
    +                    row.account_id.as_str(),
    +                    row.table_name.as_str(),
    +                    created_at,
    +                    row.backup_arn.as_str()
    +                ),
    +            );
    +        self.session
    +            .batch(
    +                batch
    +                    .build()
    +                    .map_err(|e| StorageError::Internal(e.to_string()))?,
    +            )
    +            .await
    +            .map_err(|e| StorageError::Internal(format!("Remove backup index rows: {e}")))?;
    +        Ok(())
    +    }
    +
         async fn read_index_row(
             &self,
             account_id: &str,
    @@ -735,6 +804,42 @@ impl BackupCatalog for CassandraEngine {
             })
         }
     
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // One partition read of the status index, then one row: the
    +            // entries are ordered by the instant minted into the backup id
    +            // (the row's `created_at` by the manifest rule), newest first,
    +            // and the first entry whose row confirms the status and the
    +            // store is the newest AVAILABLE row. An entry whose row moved
    +            // (a hint the deleter's batch has not yet moved) costs one more
    +            // read.
    +            let mut entries = self
    +                .status_index_arns(store, BackupStatus::Available)
    +                .await?;
    +            entries.sort_by(|(_, a), (_, b)| {
    +                let minted = |arn: &str| {
    +                    extenddb_core::backup_format::backup_creation_date_time(arn).unwrap_or(0.0)
    +                };
    +                minted(b)
    +                    .partial_cmp(&minted(a))
    +                    .unwrap_or(std::cmp::Ordering::Equal)
    +                    .then_with(|| b.cmp(a))
    +            });
    +            for (account_id, backup_arn) in entries {
    +                if let Some(row) = self.read_index_row(&account_id, &backup_arn).await?
    +                    && row.status == BackupStatus::Available
    +                    && row.store == *store
    +                {
    +                    return Ok(Some(row));
    +                }
    +            }
    +            Ok(None)
    +        })
    +    }
    +
         fn list_account_backup_rows<'a>(
             &'a self,
             account_id: &'a str,
    @@ -933,12 +1038,12 @@ impl BackupCatalog for CassandraEngine {
                 if !matches!(row.status, BackupStatus::Deleted | BackupStatus::Failed) {
                     return Ok(false);
                 }
    -            let catalog = self.catalog_keyspace();
                 // The authoritative row first, conditioned on the tombstone
                 // status read; then its index entries.
                 let delete = format!(
    -                "DELETE FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    -                 IF backup_status = ?"
    +                "DELETE FROM {}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    +                 IF backup_status = ?",
    +                self.catalog_keyspace()
                 );
                 let result = crate::cassandra_util::query_lwt(
                     &self.session,
    @@ -953,55 +1058,95 @@ impl BackupCatalog for CassandraEngine {
                 if !crate::cassandra_util::lwt_applied(&result)? {
                     return Ok(false);
                 }
    -            let created_at = seconds_to_millis(row.created_at);
    -            let batch = BatchQueryBuilder::new()
    -                .with_consistency(Consistency::LocalQuorum)
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backup_index_by_status \
    -                         WHERE store_root = ? AND backup_status = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        row.store.store_root.as_str(),
    -                        row.status.as_str(),
    -                        key.backup_arn.as_str()
    -                    ),
    -                )
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backups_by_account \
    -                         WHERE account_id = ? AND created_at = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        key.account_id.as_str(),
    -                        created_at,
    -                        key.backup_arn.as_str()
    -                    ),
    -                )
    -                .add_query(
    -                    format!(
    -                        "DELETE FROM {catalog}.backups_by_table \
    -                         WHERE account_id = ? AND table_name = ? AND created_at = ? AND backup_arn = ?"
    -                    ),
    -                    cdrs_tokio::query_values!(
    -                        key.account_id.as_str(),
    -                        row.table_name.as_str(),
    -                        created_at,
    -                        key.backup_arn.as_str()
    -                    ),
    -                );
    -            self.session
    -                .batch(
    -                    batch
    -                        .build()
    -                        .map_err(|e| StorageError::Internal(e.to_string()))?,
    -                )
    -                .await
    -                .map_err(|e| StorageError::Internal(format!("Remove backup index rows: {e}")))?;
    +            self.remove_index_entries(&row).await?;
                 Ok(true)
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let Some(row) = self
    +                .read_index_row(&key.account_id, &key.backup_arn)
    +                .await?
    +            else {
    +                return Ok(false);
    +            };
    +            if row.status != BackupStatus::Creating {
    +                return Ok(false);
    +            }
    +            let now = coordinator_now_millis(self).await?;
    +            let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +            let heartbeat_at = seconds_to_millis(row.heartbeat_at);
    +            if heartbeat_at >= now.saturating_sub(stale_ms) {
    +                return Ok(false);
    +            }
    +            // Conditioned on the status and the heartbeat read: a heartbeat
    +            // written since (the owner is alive after all) leaves the row.
    +            let delete = format!(
    +                "DELETE FROM {}.backups_by_arn WHERE account_id = ? AND backup_arn = ? \
    +                 IF backup_status = 'CREATING' AND heartbeat_at = ?",
    +                self.catalog_keyspace()
    +            );
    +            let result = crate::cassandra_util::query_lwt(
    +                &self.session,
    +                &delete,
    +                cdrs_tokio::query_values!(
    +                    key.account_id.as_str(),
    +                    key.backup_arn.as_str(),
    +                    heartbeat_at
    +                ),
    +            )
    +            .await?;
    +            if !crate::cassandra_util::lwt_applied(&result)? {
    +                return Ok(false);
    +            }
    +            self.remove_index_entries(&row).await?;
    +            Ok(true)
    +        })
    +    }
    +
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result> {
    +        rows_stream(async move {
    +            // The coordinator's clock is read once, before the status
    +            // partition is scanned, so a row's column is compared with the
    +            // clock that stamped it and never with this host's; a heartbeat
    +            // written while the rows are re-read is later than `now` and the
    +            // row is not listed. A row with no `completed_at` has no age on
    +            // that clock and is not listed.
    +            let now = coordinator_now_millis(self).await?;
    +            let age_ms = i64::try_from(age.as_millis()).unwrap_or(i64::MAX);
    +            let cutoff = now.saturating_sub(age_ms);
    +            let mut rows = Vec::new();
    +            for (account_id, backup_arn) in self.status_index_arns(store, status).await? {
    +                let Some(row) = self.read_index_row(&account_id, &backup_arn).await? else {
    +                    continue;
    +                };
    +                if row.status != status || row.store != *store {
    +                    continue;
    +                }
    +                let at = match clock {
    +                    StaleClock::Heartbeat => Some(seconds_to_millis(row.heartbeat_at)),
    +                    StaleClock::Completed => row.completed_at.map(seconds_to_millis),
    +                };
    +                if at.is_some_and(|at| at < cutoff) {
    +                    rows.push(row);
    +                }
    +            }
    +            sort_listing(&mut rows);
    +            Ok(rows)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -1075,9 +1220,6 @@ impl BackupCatalog for CassandraEngine {
             stale_after: Duration,
         ) -> BoxFuture<'_, Result, StorageError>> {
             Box::pin(async move {
    -            let now = coordinator_now_millis(self).await?;
    -            #[allow(clippy::cast_possible_truncation, clippy::cast_possible_wrap)]
    -            let cutoff = now - stale_after.as_millis() as i64;
                 // Two filtered scans over the tables (CQL takes one equality on a
                 // non-key column per scan): RUNNING records, dead when their
                 // heartbeat is older than the cutoff; and FAILED records whose
    @@ -1086,11 +1228,15 @@ impl BackupCatalog for CassandraEngine {
                 // DELETING with its registration: the removal's claim landed and
                 // its drop did not), dead when their `restore_completed_at` (or,
                 // where none was stamped, their heartbeat) is that old. Both
    -            // kinds are few.
    +            // kinds are few. The clock is read after the scans, so no
    +            // heartbeat a scan returned is later than `now` (with a zero
    +            // window, every RUNNING record the scans saw is listed). A range
    +            // scan has no Paxos read (a serial level resolves to a local
    +            // quorum there), so it stays at quorum.
                 let columns = "account_id, table_name, table_id, table_status, \
                                restore_source_backup_arn, restore_kind, restore_status, \
                                restore_owner_generation, restore_heartbeat_at, restore_completed_at";
    -            let mut dead = Vec::new();
    +            let mut scanned = Vec::new();
                 for status in ["RUNNING", "FAILED"] {
                     let query = format!(
                         "SELECT {columns} FROM {}.tables WHERE restore_status = ? ALLOW FILTERING",
    @@ -1103,6 +1249,13 @@ impl BackupCatalog for CassandraEngine {
                         "dead restore jobs",
                     )
                     .await?;
    +                scanned.push((status, rows));
    +            }
    +            let now = coordinator_now_millis(self).await?;
    +            #[allow(clippy::cast_possible_truncation, clippy::cast_possible_wrap)]
    +            let cutoff = now - stale_after.as_millis() as i64;
    +            let mut dead = Vec::new();
    +            for (status, rows) in scanned {
                     for row in rows {
                         let heartbeat_at: Option =
                             row.get_by_name("restore_heartbeat_at").ok().flatten();
    @@ -1168,6 +1321,52 @@ impl BackupCatalog for CassandraEngine {
             })
         }
     
    +    fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The RUNNING scan of `list_dead_restore_jobs` with no cutoff: by
    +            // status alone, so a record stamped in the scan's millisecond, or
    +            // ahead of the coordinator's clock (skew between coordinators),
    +            // is listed too.
    +            let query = format!(
    +                "SELECT account_id, table_name, table_id, restore_source_backup_arn, \
    +                 restore_kind, restore_owner_generation, restore_heartbeat_at \
    +                 FROM {}.tables WHERE restore_status = 'RUNNING' ALLOW FILTERING",
    +                self.catalog_keyspace()
    +            );
    +            let rows = crate::cassandra_util::query_rows_quorum::(
    +                &self.session,
    +                &query,
    +                cdrs_tokio::query_values!(),
    +                "running restore jobs",
    +            )
    +            .await?;
    +            let mut running = Vec::with_capacity(rows.len());
    +            for row in rows {
    +                let column = |name: &str| -> Result {
    +                    crate::cassandra_util::get_column(&row, name, "running restore jobs")
    +                };
    +                let kind = column("restore_kind")?;
    +                let generation: Option =
    +                    row.get_by_name("restore_owner_generation").ok().flatten();
    +                let heartbeat_at: Option =
    +                    row.get_by_name("restore_heartbeat_at").ok().flatten();
    +                running.push(RestoreJobRow {
    +                    target: RestoreTargetKey {
    +                        account_id: column("account_id")?,
    +                        table_name: column("table_name")?,
    +                        table_id: column("table_id")?,
    +                    },
    +                    source_backup_arn: column("restore_source_backup_arn")?,
    +                    kind: RestoreKind::parse(&kind)?,
    +                    status: JobStatus::Running,
    +                    owner_generation: generation_from_column(generation),
    +                    heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds),
    +                });
    +            }
    +            Ok(running)
    +        })
    +    }
    +
         /// Conditioned on RUNNING, on the listed `owner_generation` and on the
         /// listed `restore_heartbeat_at` (a millisecond timestamp, rebuilt from
         /// the listed epoch seconds exactly), and written with a fresh generation
    @@ -1206,20 +1405,269 @@ impl BackupCatalog for CassandraEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
             Box::pin(async move {
    +            // `backup_index_by_status` is partitioned by (root, status), so a
    +            // root's rows are its five status partitions; the entry carries
    +            // the kind and id, which is all this read needs (an orphan entry
    +            // left by a crash names the same store its row did).
                 let query = format!(
    -                "SELECT claim FROM {}.backup_maintenance LIMIT 1",
    +                "SELECT store_kind, store_id FROM {}.backup_index_by_status \
    +                 WHERE store_root = ? AND backup_status = ?",
                     self.catalog_keyspace()
                 );
    -            let rows = crate::cassandra_util::query_rows_quorum::(
    +            let mut ids = Vec::new();
    +            for status in [
    +                BackupStatus::Creating,
    +                BackupStatus::Available,
    +                BackupStatus::Deleting,
    +                BackupStatus::Deleted,
    +                BackupStatus::Failed,
    +            ] {
    +                let rows = crate::cassandra_util::query_rows_quorum::(
    +                    &self.session,
    +                    &query,
    +                    cdrs_tokio::query_values!(store_root, status.as_str()),
    +                    "status index",
    +                )
    +                .await?;
    +                for row in rows {
    +                    let kind: Option = row.get_by_name("store_kind").ok().flatten();
    +                    let id: Option = row.get_by_name("store_id").ok().flatten();
    +                    if kind.as_deref() == Some(store_kind.as_str())
    +                        && let Some(id) = id
    +                        && !ids.contains(&id)
    +                    {
    +                        ids.push(id);
    +                    }
    +                }
    +            }
    +            ids.sort();
    +            Ok(ids)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The row is the claim: `INSERT ... IF NOT EXISTS` takes it when
    +            // there is none; otherwise the row is read, and one whose
    +            // heartbeat is older than `stale_after` on the coordinator's clock
    +            // is taken over with `IF owner_generation = `, so two
    +            // commands racing for a dead holder's row see one win.
    +            let catalog = self.catalog_keyspace();
    +            let generation = new_generation();
    +            let insert = format!(
    +                "INSERT INTO {catalog}.backup_maintenance \
    +                 (claim, holder, owner_generation, claimed_at, heartbeat_at) \
    +                 VALUES (?, ?, ?, toTimestamp(now()), toTimestamp(now())) IF NOT EXISTS"
    +            );
    +            let inserted = conditional_write(
    +                &self.session,
    +                &insert,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    holder,
    +                    generation_column(generation)
    +                ),
    +            )
    +            .await?;
    +            // A test's lost answer: the insert ran, the caller sees no result.
    +            let inserted = if crate::test_hooks::maintenance_insert_answer_lost() {
    +                WriteOutcome::Unknown
    +            } else {
    +                inserted
    +            };
    +            if inserted != WriteOutcome::Matched {
    +                let read = format!(
    +                    "SELECT holder, owner_generation, heartbeat_at FROM \
    +                     {catalog}.backup_maintenance WHERE claim = ?"
    +                );
    +                let rows = crate::cassandra_util::query_rows_serial::(
    +                    &self.session,
    +                    &read,
    +                    cdrs_tokio::query_values!(MAINTENANCE_CLAIM_KEY),
    +                    "maintenance claim",
    +                )
    +                .await?;
    +                let row = rows.first();
    +                let current_holder: Option =
    +                    row.and_then(|r| r.get_by_name("holder").ok().flatten());
    +                let current_generation: Option =
    +                    row.and_then(|r| r.get_by_name("owner_generation").ok().flatten());
    +                // An insert whose answer was lost (`Unknown`) may have
    +                // committed: a row carrying this call's generation is this
    +                // call's claim.
    +                if inserted == WriteOutcome::Unknown
    +                    && current_generation == Some(generation_column(generation))
    +                {
    +                    return Ok(Box::new(CassandraMaintenanceClaim {
    +                        engine: self.clone(),
    +                        generation,
    +                    }) as Box);
    +                }
    +                let heartbeat_at: Option =
    +                    row.and_then(|r| r.get_by_name("heartbeat_at").ok().flatten());
    +                let now = coordinator_now_millis(self).await?;
    +                let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +                let stale = heartbeat_at.is_none_or(|at| at < now.saturating_sub(stale_ms));
    +                let refused = || {
    +                    StorageError::AlreadyExists(format!(
    +                        "the backup maintenance claim is held by {}",
    +                        current_holder
    +                            .as_deref()
    +                            .unwrap_or("another maintenance command")
    +                    ))
    +                };
    +                if !stale {
    +                    return Err(refused());
    +                }
    +                let take_over = format!(
    +                    "UPDATE {catalog}.backup_maintenance SET holder = ?, owner_generation = ?, \
    +                     claimed_at = toTimestamp(now()), heartbeat_at = toTimestamp(now()) \
    +                     WHERE claim = ? IF owner_generation = ?"
    +                );
    +                // A test's pause between the read and the takeover, so two
    +                // takers both read the stale row before either writes.
    +                crate::test_hooks::before_maintenance_takeover().await;
    +                let taken = conditional_write(
    +                    &self.session,
    +                    &take_over,
    +                    cdrs_tokio::query_values!(
    +                        holder,
    +                        generation_column(generation),
    +                        MAINTENANCE_CLAIM_KEY,
    +                        current_generation
    +                    ),
    +                )
    +                .await?;
    +                match taken {
    +                    WriteOutcome::Matched => {}
    +                    WriteOutcome::NotMatched => return Err(refused()),
    +                    WriteOutcome::Unknown => {
    +                        // The takeover's answer was lost; the update may
    +                        // have committed. Re-read at `LOCAL_SERIAL`, as the
    +                        // insert path does: a row carrying this call's
    +                        // generation is this call's claim, and answering
    +                        // "held by" the old holder would leave a row with
    +                        // this generation and a fresh heartbeat that nobody
    +                        // renews, refusing every job until it goes stale.
    +                        let rows = crate::cassandra_util::query_rows_serial::(
    +                            &self.session,
    +                            &read,
    +                            cdrs_tokio::query_values!(MAINTENANCE_CLAIM_KEY),
    +                            "maintenance claim",
    +                        )
    +                        .await?;
    +                        let now_generation: Option = rows
    +                            .first()
    +                            .and_then(|r| r.get_by_name("owner_generation").ok().flatten());
    +                        if now_generation != Some(generation_column(generation)) {
    +                            return Err(refused());
    +                        }
    +                    }
    +                }
    +            }
    +            Ok(Box::new(CassandraMaintenanceClaim {
    +                engine: self.clone(),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "SELECT heartbeat_at FROM {}.backup_maintenance WHERE claim = ?",
    +                self.catalog_keyspace()
    +            );
    +            let rows = crate::cassandra_util::query_rows_serial::(
                     &self.session,
                     &query,
    -                cdrs_tokio::query_values!(),
    +                cdrs_tokio::query_values!(MAINTENANCE_CLAIM_KEY),
                     "maintenance claim",
                 )
                 .await?;
    -            Ok(!rows.is_empty())
    +            let Some(row) = rows.first() else {
    +                return Ok(false);
    +            };
    +            let heartbeat_at: Option = row.get_by_name("heartbeat_at").ok().flatten();
    +            let Some(heartbeat_at) = heartbeat_at else {
    +                return Ok(false);
    +            };
    +            let now = coordinator_now_millis(self).await?;
    +            let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX);
    +            Ok(heartbeat_at >= now.saturating_sub(stale_ms))
    +        })
    +    }
    +}
    +
    +/// The partition key of the one maintenance claim row.
    +const MAINTENANCE_CLAIM_KEY: &str = "claim";
    +
    +/// The holder's handle on the `backup_maintenance` row.
    +struct CassandraMaintenanceClaim {
    +    engine: CassandraEngine,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for CassandraMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "UPDATE {}.backup_maintenance SET heartbeat_at = toTimestamp(now()) \
    +                 WHERE claim = ? IF owner_generation = ?",
    +                self.engine.catalog_keyspace()
    +            );
    +            let result = crate::cassandra_util::query_lwt(
    +                &self.engine.session,
    +                &query,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    generation_column(self.generation)
    +                ),
    +            )
    +            .await?;
    +            Ok(if crate::cassandra_util::lwt_applied(&result)? {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            let query = format!(
    +                "DELETE FROM {}.backup_maintenance WHERE claim = ? IF owner_generation = ?",
    +                self.engine.catalog_keyspace()
    +            );
    +            // Not applied means another holder took the row over; it is
    +            // theirs to release.
    +            conditional_write(
    +                &self.engine.session,
    +                &query,
    +                cdrs_tokio::query_values!(
    +                    MAINTENANCE_CLAIM_KEY,
    +                    generation_column(self.generation)
    +                ),
    +            )
    +            .await?;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-cassandra/src/cassandra_util.rs b/crates/storage-cassandra/src/cassandra_util.rs
    index 22c70a02a..2b846cc79 100644
    --- a/crates/storage-cassandra/src/cassandra_util.rs
    +++ b/crates/storage-cassandra/src/cassandra_util.rs
    @@ -147,6 +147,39 @@ pub async fn query_rows_quorum(
         Ok(body.into_rows().unwrap_or_default())
     }
     
    +/// Execute a query at `LOCAL_SERIAL` and return all rows: the read sees
    +/// every Paxos round that committed, so a row an LWT wrote (a claim taken,
    +/// a heartbeat refreshed) is read as written. For the reads whose answer
    +/// decides whether another process's claim is live.
    +pub async fn query_rows_serial(
    +    session: &Arc,
    +    query: &str,
    +    values: QueryValues,
    +    context: &str,
    +) -> Result, E> {
    +    use cdrs_tokio::consistency::Consistency;
    +    use cdrs_tokio::statement::StatementParamsBuilder;
    +
    +    let params = StatementParamsBuilder::new()
    +        .with_consistency(Consistency::LocalSerial)
    +        .with_values(values)
    +        .build();
    +    let result = session
    +        .query_with_params(query, params)
    +        .await
    +        .map_err(|e| {
    +            tracing::error!("{context} serial query failed: {e}");
    +            E::db_error(format!("{context}: {e}"))
    +        })?;
    +
    +    let body = result.response_body().map_err(|e| {
    +        tracing::error!("{context} response_body failed: {e}");
    +        E::db_error(format!("{context} response_body: {e}"))
    +    })?;
    +
    +    Ok(body.into_rows().unwrap_or_default())
    +}
    +
     /// Execute a query and return the first row, if any.
     ///
     /// Similar to `sqlx::query().fetch_optional()`.
    diff --git a/crates/storage-cassandra/src/lib.rs b/crates/storage-cassandra/src/lib.rs
    index 3f0489402..bff16c1d4 100644
    --- a/crates/storage-cassandra/src/lib.rs
    +++ b/crates/storage-cassandra/src/lib.rs
    @@ -316,11 +316,14 @@ fn server_components_factory(
                 .instance_id
                 .clone()
                 .unwrap_or_else(|| "unknown".to_owned());
    -        let registration = migrations::register_server(
    +        // Every caller registers, a command too, so a migration does not run
    +        // under one; a command's registration carries its own name.
    +        let registration = migrations::register_server_as(
                 status_session,
                 &cassandra_config.keyspace_prefix,
                 &host,
                 options.release,
    +            options.command.unwrap_or(migrations::SERVE_COMMAND),
             )
             .await
             .map_err(BackendError::Catalog)?;
    @@ -340,14 +343,19 @@ fn server_components_factory(
             let control_plane_notify = engine.control_plane_notify.clone();
             let engine = Arc::new(engine);
     
    -        match engine.process_control_plane_transitions().await {
    -            Ok(ref t) if t.is_empty() => {}
    -            Ok(transitions) => {
    -                for (name, transition) in &transitions {
    -                    tracing::info!("Recovered table '{name}': {transition}");
    +        // The startup recovery is a server's; a command reading beside one,
    +        // or one that writes beside the live servers (`backup purge`,
    +        // `options.startup_recovery` off), leaves it to the server.
    +        if !options.beside_server && options.startup_recovery {
    +            match engine.process_control_plane_transitions().await {
    +                Ok(ref t) if t.is_empty() => {}
    +                Ok(transitions) => {
    +                    for (name, transition) in &transitions {
    +                        tracing::info!("Recovered table '{name}': {transition}");
    +                    }
                     }
    +                Err(e) => tracing::error!("Failed to recover control plane transitions: {e}"),
                 }
    -            Err(e) => tracing::error!("Failed to recover control plane transitions: {e}"),
             }
     
             // Load encryption key from catalog.
    diff --git a/crates/storage-cassandra/src/migrations/mod.rs b/crates/storage-cassandra/src/migrations/mod.rs
    index 3e2b6178c..7015b82f6 100644
    --- a/crates/storage-cassandra/src/migrations/mod.rs
    +++ b/crates/storage-cassandra/src/migrations/mod.rs
    @@ -57,7 +57,7 @@ pub(crate) use connect::{LOCKED_DDL_TIMEOUT, REQUEST_TIMEOUT};
     pub(crate) use locks::LeaseHandle;
     pub use locks::{LeaseTimes, MigrationLock};
     pub use registrar::ServerRegistration;
    -pub use registrations::{RegistrationTimes, ServingRegistration};
    +pub use registrations::{RegistrationTimes, SERVE_COMMAND, ServingRegistration};
     
     use crate::cassandra_util::CassandraSession;
     
    @@ -720,11 +720,31 @@ pub async fn register_server(
         host: &str,
         release: &str,
     ) -> Result {
    -    register_server_timed(
    +    register_server_as(session, keyspace_prefix, host, release, SERVE_COMMAND).await
    +}
    +
    +/// [`register_server`] for the process `command` names: `extenddb serve`
    +/// ([`SERVE_COMMAND`]) or a command that opens the catalog as a server does
    +/// (`extenddb backup list`), whose row carries its name (the registrations
    +/// module doc says where) and whose refusal while a migration holds the
    +/// lock names the holder without the server's advice to start the server.
    +///
    +/// # Errors
    +///
    +/// As [`register_server`].
    +pub async fn register_server_as(
    +    session: Arc,
    +    keyspace_prefix: &str,
    +    host: &str,
    +    release: &str,
    +    command: &str,
    +) -> Result {
    +    register_server_with(
             session,
             keyspace_prefix,
             host,
             release,
    +        command,
             RegistrationTimes::DEFAULT,
         )
         .await
    @@ -742,6 +762,26 @@ pub async fn register_server_timed(
         release: &str,
         times: RegistrationTimes,
     ) -> Result {
    +    register_server_with(
    +        session,
    +        keyspace_prefix,
    +        host,
    +        release,
    +        SERVE_COMMAND,
    +        times,
    +    )
    +    .await
    +}
    +
    +async fn register_server_with(
    +    session: Arc,
    +    keyspace_prefix: &str,
    +    host: &str,
    +    release: &str,
    +    command: &str,
    +    times: RegistrationTimes,
    +) -> Result {
    +    let release = ®istrations::release_cell(release, command);
         let catalog = format!("{keyspace_prefix}_catalog");
         agreement::wait_for_agreement(&session, AGREEMENT_TIMEOUT).await?;
         if !connect::table_exists(&session, &catalog, "extenddb_servers").await? {
    @@ -760,9 +800,15 @@ pub async fn register_server_timed(
                 .await?;
         let checked = async {
             if let Some(holder) = locks::lock_holder(&session, &catalog).await? {
    +            // A command's refusal ends at the holder; the command's caller
    +            // adds that it is retried once the migration has finished.
    +            let then = if command == SERVE_COMMAND {
    +                "; start the server again when it finishes"
    +            } else {
    +                ""
    +            };
                 return Err(MigrationError::Locked(format!(
    -                "a migration holds the catalog's migration lock (holder {holder}); start the \
    -                 server again when it finishes"
    +                "a migration holds the catalog's migration lock (holder {holder}){then}"
                 )));
             }
             match catalog_status(session.clone(), keyspace_prefix).await? {
    diff --git a/crates/storage-cassandra/src/migrations/registrations.rs b/crates/storage-cassandra/src/migrations/registrations.rs
    index 33d011215..5b5b798e8 100644
    --- a/crates/storage-cassandra/src/migrations/registrations.rs
    +++ b/crates/storage-cassandra/src/migrations/registrations.rs
    @@ -14,6 +14,15 @@
     //! `migrate --yes`, which waits while a row exists, can see it gone. The server then stops admitting requests and stops its
     //! background workers, and registers again with a new row
     //! ([`super::registrar`]).
    +//!
    +//! A command that opens the catalog as a server does (`extenddb backup list`,
    +//! `extenddb backup purge`) registers too, so a migration waits for it. The
    +//! table has no column for the process's name, and a column would be a
    +//! catalog migration, so a command's row carries its name in `release`,
    +//! after the release and ` as ` (`1.0.0 as extenddb backup purge`;
    +//! [`release_cell`] writes it and [`registered_servers`] reads it back), and
    +//! the migration's wait prints the name. A row whose `release` has no
    +//! ` as ` is a server's.
     
     use std::sync::Arc;
     use std::sync::atomic::{AtomicBool, Ordering};
    @@ -58,9 +67,33 @@ impl RegistrationTimes {
         };
     }
     
    -/// One running server, as `migrate --yes` names it.
    +/// The name a server's registration carries; a command's carries its own
    +/// (`extenddb backup list`).
    +pub const SERVE_COMMAND: &str = "extenddb serve";
    +
    +/// What a registration writes in `release`: the release for a server, the
    +/// release, ` as ` and the command's name for a command.
    +pub(crate) fn release_cell(release: &str, command: &str) -> String {
    +    if command == SERVE_COMMAND {
    +        release.to_owned()
    +    } else {
    +        format!("{release} as {command}")
    +    }
    +}
    +
    +/// The release and the process name a `release` cell carries.
    +fn split_release_cell(cell: &str) -> (String, String) {
    +    match cell.split_once(" as ") {
    +        Some((release, command)) => (release.to_owned(), command.to_owned()),
    +        None => (cell.to_owned(), SERVE_COMMAND.to_owned()),
    +    }
    +}
    +
    +/// One running server, or a command registered as one, as `migrate --yes`
    +/// names it.
     #[derive(Debug, Clone, PartialEq, Eq)]
     pub(crate) struct Server {
    +    pub(crate) command: String,
         pub(crate) host: String,
         pub(crate) pid: i32,
         pub(crate) release: String,
    @@ -71,8 +104,8 @@ impl std::fmt::Display for Server {
         fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
             write!(
                 f,
    -            "extenddb serve {} on {} (process {}, started {})",
    -            self.release, self.host, self.pid, self.started_at
    +            "{} {} on {} (process {}, started {})",
    +            self.command, self.release, self.host, self.pid, self.started_at
             )
         }
     }
    @@ -101,10 +134,14 @@ pub(crate) async fn registered_servers(
         found
             .iter()
             .map(|row| {
    +            let (release, command) = split_release_cell(
    +                &nullable::(row, "release", "read a server")?.unwrap_or_default(),
    +            );
                 Ok(Server {
    +                command,
                     host: nullable::(row, "host", "read a server")?.unwrap_or_default(),
                     pid: nullable::(row, "pid", "read a server")?.unwrap_or_default(),
    -                release: nullable::(row, "release", "read a server")?.unwrap_or_default(),
    +                release,
                     started_at: nullable::(row, "started_at", "read a server")?
                         .unwrap_or_default(),
                 })
    @@ -513,4 +550,31 @@ mod tests {
                 }
             );
         }
    +
    +    /// A server's `release` cell is the release; a command's carries its
    +    /// name after ` as `, and the read gives both back, so the wait names
    +    /// the command where it names a server `extenddb serve`.
    +    #[test]
    +    fn a_commands_release_cell_carries_its_name() {
    +        use super::{SERVE_COMMAND, Server, release_cell, split_release_cell};
    +        assert_eq!(release_cell("1.0.0", SERVE_COMMAND), "1.0.0");
    +        let cell = release_cell("1.0.0", "extenddb backup purge");
    +        assert_eq!(cell, "1.0.0 as extenddb backup purge");
    +        let (release, command) = split_release_cell(&cell);
    +        let server = Server {
    +            command,
    +            host: "host-a".to_owned(),
    +            pid: 7,
    +            release,
    +            started_at: "2026-11-03T00:00:00Z".to_owned(),
    +        };
    +        assert_eq!(
    +            server.to_string(),
    +            "extenddb backup purge 1.0.0 on host-a (process 7, started 2026-11-03T00:00:00Z)"
    +        );
    +        assert_eq!(
    +            split_release_cell("1.0.0"),
    +            ("1.0.0".to_owned(), SERVE_COMMAND.to_owned())
    +        );
    +    }
     }
    diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs
    index c4daf7440..961226110 100644
    --- a/crates/storage-cassandra/src/test_hooks.rs
    +++ b/crates/storage-cassandra/src/test_hooks.rs
    @@ -24,6 +24,10 @@
     //!   between the read of a CREATING row and the fence that moves it to
     //!   DELETING, so a test can heartbeat the row in that gap and see the
     //!   fence's own `heartbeat_at = ?` condition leave it alone.
    +//! * a lost answer on the maintenance claim's insert, so a test can see the
    +//!   re-read settle a claim whose `INSERT ... IF NOT EXISTS` committed;
    +//! * a pause between a maintenance claim takeover's read and its write, so
    +//!   a test can hold two takers of one stale row there and see one win.
     
     use std::time::Duration;
     
    @@ -74,6 +78,8 @@ mod armed {
             pub(super) heartbeat_errors: HashMap,
             pub(super) before_delete_claim: HashMap>,
             pub(super) before_abandoned_fence: HashMap>,
    +        pub(super) lose_maintenance_insert_answer: bool,
    +        pub(super) before_maintenance_takeover: Option>,
         }
     
         pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default);
    @@ -198,3 +204,50 @@ pub(crate) async fn before_abandoned_fence(backup_arn: &str) {
         }
         let _ = backup_arn;
     }
    +
    +/// Make the next `maintenance_claim`'s `INSERT ... IF NOT EXISTS` lose its
    +/// answer: the statement runs and commits, and the caller sees
    +/// `WriteOutcome::Unknown`, as after a write timeout on the Paxos round. The
    +/// hook is taken once and then disarmed.
    +#[cfg(feature = "test-hooks")]
    +pub fn lose_next_maintenance_insert_answer() {
    +    armed::with_hooks(|hooks| hooks.lose_maintenance_insert_answer = true);
    +}
    +
    +/// Arm a pause in every `maintenance_claim` takeover, between the read of
    +/// the stale row and the `UPDATE ... IF owner_generation = `, so a test
    +/// can hold two takers there and release both: each signals `reached` and
    +/// waits for one `release`. Disarmed by [`clear_maintenance_takeover_pause`].
    +#[cfg(feature = "test-hooks")]
    +pub fn pause_before_maintenance_takeover() -> std::sync::Arc {
    +    let pause = std::sync::Arc::new(Pause::new());
    +    armed::with_hooks(|hooks| {
    +        hooks.before_maintenance_takeover = Some(std::sync::Arc::clone(&pause));
    +    });
    +    pause
    +}
    +
    +/// Disarm [`pause_before_maintenance_takeover`].
    +#[cfg(feature = "test-hooks")]
    +pub fn clear_maintenance_takeover_pause() {
    +    armed::with_hooks(|hooks| hooks.before_maintenance_takeover = None);
    +}
    +
    +/// Hold here if a test armed the takeover pause.
    +pub(crate) async fn before_maintenance_takeover() {
    +    #[cfg(feature = "test-hooks")]
    +    if let Some(pause) = armed::with_hooks(|hooks| hooks.before_maintenance_takeover.clone()) {
    +        pause.hold().await;
    +    }
    +}
    +
    +/// Whether a test asked for the maintenance claim insert's answer to be
    +/// lost ([`lose_next_maintenance_insert_answer`]); disarms the hook.
    +pub(crate) fn maintenance_insert_answer_lost() -> bool {
    +    #[cfg(feature = "test-hooks")]
    +    {
    +        armed::with_hooks(|hooks| std::mem::take(&mut hooks.lose_maintenance_insert_answer))
    +    }
    +    #[cfg(not(feature = "test-hooks"))]
    +    false
    +}
    diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs
    index 6c8c4da0d..a63f70ba7 100644
    --- a/crates/storage-cassandra/tests/direct/backup_engine.rs
    +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs
    @@ -20,8 +20,8 @@ use extenddb_core::types::{
         TimeToLiveStatus,
     };
     use extenddb_storage::backup_catalog::{
    -    BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StatusFields,
    -    StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
    +    BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StaleClock,
    +    StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
     };
     use extenddb_storage::error::StorageError;
     use extenddb_storage::migrations::{CatalogStatus, MigrationError, MigrationId};
    @@ -2122,7 +2122,6 @@ async fn test_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             uuid::Uuid::parse_str(&engine.catalog_id().await.expect("id")).is_ok(),
             "the catalog has an id from the migration"
         );
    -    assert!(!engine.check_maintenance_claim().await.expect("check"));
         let store = unique_store();
         let row = index_row(&account_id, "Music", "nightly", &store);
         let claim = engine.claim_backup(&row).await.expect("claim");
    @@ -4825,3 +4824,646 @@ async fn test_the_failed_write_matches_only_the_listed_generation_and_heartbeat(
             .await
             .expect("the backup is free");
     }
    +
    +/// `read_store_ids`: the distinct ids of the rows for a root, of every
    +/// status, scoped to the kind and the root, read from the status partitions.
    +#[tokio::test]
    +async fn test_read_store_ids_lists_the_distinct_ids_of_a_root() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    let root = store.store_root.clone();
    +    let ids = |kind: StoreKind| {
    +        let engine = &engine;
    +        let root = root.clone();
    +        async move { engine.read_store_ids(kind, &root).await.expect("ids") }
    +    };
    +    assert!(ids(StoreKind::Filesystem).await.is_empty());
    +    let first = index_row(&account_id, "Music", "a", &store);
    +    let _c1 = engine.claim_backup(&first).await.expect("claim");
    +    let second = index_row(&account_id, "Music", "b", &store);
    +    let c2 = engine.claim_backup(&second).await.expect("claim");
    +    assert_eq!(
    +        c2.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields::default()
    +        )
    +        .await
    +        .expect("fail"),
    +        WriteOutcome::Matched
    +    );
    +    assert_eq!(
    +        ids(StoreKind::Filesystem).await,
    +        ["store-1"],
    +        "two rows of one store, of two statuses, are one id"
    +    );
    +    let mut other_store = store.clone();
    +    other_store.store_id = "store-2".to_owned();
    +    let other = index_row(&account_id, "Songs", "c", &other_store);
    +    let _c3 = engine.claim_backup(&other).await.expect("claim");
    +    assert_eq!(ids(StoreKind::Filesystem).await, ["store-1", "store-2"]);
    +    assert!(
    +        ids(StoreKind::S3).await.is_empty(),
    +        "another kind at the same root is another store"
    +    );
    +    assert!(
    +        engine
    +            .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +            .await
    +            .expect("ids")
    +            .is_empty()
    +    );
    +}
    +
    +/// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older
    +/// than the window is removed outright, its status-index entry with it; one
    +/// with a live owner, one that is not CREATING, and one that is gone are left,
    +/// answering `false`.
    +#[tokio::test]
    +async fn test_a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    let stale = Duration::from_secs(300);
    +    let ten_minutes_ago = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        - 600_000;
    +    let age = |backup_arn: String, generation: u64| {
    +        let engine = &engine;
    +        let account_id = account_id.clone();
    +        async move {
    +            apply(
    +                engine,
    +                &format!(
    +                    "UPDATE {}.backups_by_arn SET heartbeat_at = ? \
    +                     WHERE account_id = ? AND backup_arn = ? IF owner_generation = ?",
    +                    engine.catalog_keyspace()
    +                ),
    +                cdrs_tokio::query_values!(
    +                    ten_minutes_ago,
    +                    account_id.as_str(),
    +                    backup_arn.as_str(),
    +                    i64::try_from(generation).unwrap()
    +                ),
    +            )
    +            .await;
    +        }
    +    };
    +    let creating = index_row(&account_id, "Music", "dead-a", &store);
    +    let key = creating.key();
    +    let claim = engine.claim_backup(&creating).await.expect("claim");
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "a live owner keeps its row"
    +    );
    +    age(creating.backup_arn.clone(), claim.generation()).await;
    +    assert!(
    +        engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap()
    +    );
    +    assert!(
    +        engine
    +            .read_backup_row(&arn_of(&creating))
    +            .await
    +            .unwrap()
    +            .is_none()
    +    );
    +    let creating_rows: Vec = engine
    +        .list_backup_rows(Some(BackupStatus::Creating), &store)
    +        .try_collect()
    +        .await
    +        .unwrap();
    +    assert!(
    +        creating_rows.is_empty(),
    +        "the status index entry went with the row: {creating_rows:?}"
    +    );
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "gone"
    +    );
    +    drop(claim);
    +
    +    let available = index_row(&account_id, "Music", "dead-b", &store);
    +    let claim = engine.claim_backup(&available).await.expect("claim");
    +    assert_eq!(
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default()
    +            )
    +            .await
    +            .expect("available"),
    +        WriteOutcome::Matched
    +    );
    +    age(available.backup_arn.clone(), claim.generation()).await;
    +    assert!(
    +        !engine
    +            .remove_dead_creating_backup_row(&available.key(), stale)
    +            .await
    +            .unwrap(),
    +        "only a CREATING row"
    +    );
    +    assert_eq!(
    +        engine
    +            .read_backup_row(&arn_of(&available))
    +            .await
    +            .unwrap()
    +            .unwrap()
    +            .status,
    +        BackupStatus::Available
    +    );
    +    drop(claim);
    +}
    +
    +/// `list_backup_rows_older_than`: the rows of a store with a status whose
    +/// named clock column is older than the age on the coordinator's clock. A
    +/// row FAILED after its heartbeat aged is listed on the heartbeat clock and
    +/// not on the completed clock until that window too has passed (waited out
    +/// on the coordinator's clock, not hand-dated); a row with no `completed_at`
    +/// is never listed on the completed clock; another status and another store
    +/// are not listed.
    +#[tokio::test]
    +async fn test_rows_older_than_are_listed_on_the_named_clock() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    let window = Duration::from_secs(1);
    +    let listed = |status: BackupStatus, clock: StaleClock, age: Duration| {
    +        let engine = &engine;
    +        let store = &store;
    +        async move {
    +            let mut arns: Vec = engine
    +                .list_backup_rows_older_than(status, store, clock, age)
    +                .map_ok(|row| row.backup_arn)
    +                .try_collect()
    +                .await
    +                .expect("listing");
    +            arns.sort();
    +            arns
    +        }
    +    };
    +    let completed_failure = StatusFields {
    +        failure_reason: Some("owner dead".to_owned()),
    +        completed: true,
    +        ..StatusFields::default()
    +    };
    +    let open_failure = StatusFields {
    +        failure_reason: Some("owner dead".to_owned()),
    +        completed: false,
    +        ..StatusFields::default()
    +    };
    +    let stamped = index_row(&account_id, "Music", "stamped", &store);
    +    let unstamped = index_row(&account_id, "Music", "unstamped", &store);
    +    let creating = index_row(&account_id, "Music", "creating", &store);
    +    let mut other_store = store.clone();
    +    other_store.store_id = "store-2".to_owned();
    +    let elsewhere = index_row(&account_id, "Music", "elsewhere", &other_store);
    +    let stamped_claim = engine.claim_backup(&stamped).await.expect("claim");
    +    let unstamped_claim = engine.claim_backup(&unstamped).await.expect("claim");
    +    let _creating_claim = engine.claim_backup(&creating).await.expect("claim");
    +    let elsewhere_claim = engine.claim_backup(&elsewhere).await.expect("claim");
    +    assert_eq!(
    +        elsewhere_claim
    +            .set_status(JobStatus::Creating, JobStatus::Failed, &completed_failure)
    +            .await
    +            .expect("write"),
    +        WriteOutcome::Matched
    +    );
    +    assert!(
    +        listed(BackupStatus::Failed, StaleClock::Heartbeat, Duration::ZERO)
    +            .await
    +            .is_empty(),
    +        "nothing of this store is FAILED yet"
    +    );
    +
    +    // The heartbeats age past the window; then the two rows fail, one with a
    +    // completion time and one without.
    +    tokio::time::sleep(Duration::from_millis(1_300)).await;
    +    assert_eq!(
    +        stamped_claim
    +            .set_status(JobStatus::Creating, JobStatus::Failed, &completed_failure)
    +            .await
    +            .expect("write"),
    +        WriteOutcome::Matched
    +    );
    +    assert_eq!(
    +        unstamped_claim
    +            .set_status(JobStatus::Creating, JobStatus::Failed, &open_failure)
    +            .await
    +            .expect("write"),
    +        WriteOutcome::Matched
    +    );
    +    let mut both = vec![stamped.backup_arn.clone(), unstamped.backup_arn.clone()];
    +    both.sort();
    +    assert_eq!(
    +        listed(BackupStatus::Failed, StaleClock::Heartbeat, window).await,
    +        both,
    +        "a row just failed is listed on the heartbeat clock: its dead owner's beat is stale"
    +    );
    +    assert!(
    +        listed(BackupStatus::Failed, StaleClock::Completed, window)
    +            .await
    +            .is_empty(),
    +        "the completion time is an instant old"
    +    );
    +    tokio::time::sleep(Duration::from_millis(50)).await;
    +    assert_eq!(
    +        listed(BackupStatus::Failed, StaleClock::Completed, Duration::ZERO).await,
    +        std::slice::from_ref(&stamped.backup_arn),
    +        "a row with no completed_at is not listed on the completed clock"
    +    );
    +    assert!(
    +        listed(
    +            BackupStatus::Failed,
    +            StaleClock::Heartbeat,
    +            Duration::from_secs(300)
    +        )
    +        .await
    +        .is_empty(),
    +        "a long window lists nothing"
    +    );
    +    assert_eq!(
    +        listed(BackupStatus::Creating, StaleClock::Heartbeat, window).await,
    +        std::slice::from_ref(&creating.backup_arn),
    +        "the CREATING row is listed under its own status only"
    +    );
    +    assert!(
    +        listed(
    +            BackupStatus::Creating,
    +            StaleClock::Completed,
    +            Duration::ZERO
    +        )
    +        .await
    +        .is_empty(),
    +        "a CREATING row has no completion time"
    +    );
    +    assert!(
    +        listed(BackupStatus::Deleted, StaleClock::Completed, Duration::ZERO)
    +            .await
    +            .is_empty()
    +    );
    +
    +    // The completed window crossed on the coordinator's clock.
    +    tokio::time::sleep(Duration::from_millis(1_300)).await;
    +    assert_eq!(
    +        listed(BackupStatus::Failed, StaleClock::Completed, window).await,
    +        std::slice::from_ref(&stamped.backup_arn)
    +    );
    +    assert_eq!(
    +        listed(BackupStatus::Failed, StaleClock::Heartbeat, window).await,
    +        both,
    +        "the other store's FAILED row is not listed"
    +    );
    +}
    +
    +/// The maintenance claim: taken once, refused to a second holder naming the
    +/// first, heartbeated on the coordinator's clock, released; a holder whose
    +/// heartbeat is older than the stale window is dead, so its claim is not
    +/// reported held and is taken over with a lightweight transaction on its
    +/// generation, and its handle finds the claim lost. The catalog is shared by
    +/// this binary's tests, so the test releases what it takes.
    +#[tokio::test]
    +async fn test_the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let stale = Duration::from_secs(300);
    +    let check = || engine.check_maintenance_claim(stale);
    +    // Another test of this binary never holds the claim; a stale row an
    +    // interrupted run left is taken over below like any dead holder's.
    +    let first = engine
    +        .maintenance_claim("purge on host-a", stale)
    +        .await
    +        .expect("claim");
    +    assert_ne!(first.generation(), UNOWNED_GENERATION);
    +    assert!(check().await.expect("check"), "a live claim is held");
    +    match engine.maintenance_claim("purge on host-b", stale).await {
    +        Err(StorageError::AlreadyExists(m)) => assert!(m.contains("purge on host-a"), "{m}"),
    +        Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +        Ok(_) => panic!("a second holder is refused"),
    +    }
    +    assert_eq!(
    +        first.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    first.release().await.expect("release");
    +    assert!(!check().await.expect("check"), "released");
    +    first.release().await.expect("a second release is a no-op");
    +
    +    // A dead holder: its heartbeat aged past the window.
    +    let dead = engine
    +        .maintenance_claim("purge on host-c", stale)
    +        .await
    +        .expect("claim");
    +    let aged = format!(
    +        "UPDATE {}.backup_maintenance SET heartbeat_at = ? WHERE claim = 'claim' \
    +         IF owner_generation = ?",
    +        engine.catalog_keyspace()
    +    );
    +    let ten_minutes_ago = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        - 600_000;
    +    apply(
    +        &engine,
    +        &aged,
    +        cdrs_tokio::query_values!(ten_minutes_ago, i64::try_from(dead.generation()).unwrap()),
    +    )
    +    .await;
    +    assert!(!check().await.expect("check"), "a stale claim is not held");
    +    let taker = engine
    +        .maintenance_claim("purge on host-d", stale)
    +        .await
    +        .expect("a dead holder's claim is taken over");
    +    assert_ne!(taker.generation(), dead.generation());
    +    assert!(check().await.expect("check"));
    +    assert_eq!(
    +        dead.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Lost,
    +        "the dead holder's handle finds the claim lost"
    +    );
    +    dead.release()
    +        .await
    +        .expect("a release by a lost holder changes nothing");
    +    assert!(check().await.expect("check"), "the taker still holds it");
    +    taker.release().await.expect("release");
    +    assert!(!check().await.expect("check"));
    +}
    +
    +/// Two takers racing for one dead holder's row see one win: each reads the
    +/// stale row (the test hook holds both after their read, so neither sees
    +/// the other's write first) and runs `UPDATE ... IF owner_generation = `,
    +/// and the lightweight transaction matches once. With `IF EXISTS` in its
    +/// place both would win and two commands would hold one claim. The loser is
    +/// refused naming a holder, and the winner's handle heartbeats.
    +#[cfg(feature = "test-hooks")]
    +#[tokio::test]
    +async fn test_two_takers_of_one_stale_maintenance_claim_see_one_win() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let stale = Duration::from_secs(300);
    +    let dead = engine
    +        .maintenance_claim("purge on host-dead", stale)
    +        .await
    +        .expect("claim");
    +    let aged = format!(
    +        "UPDATE {}.backup_maintenance SET heartbeat_at = ? WHERE claim = 'claim' \
    +         IF owner_generation = ?",
    +        engine.catalog_keyspace()
    +    );
    +    let ten_minutes_ago = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        - 600_000;
    +    apply(
    +        &engine,
    +        &aged,
    +        cdrs_tokio::query_values!(ten_minutes_ago, i64::try_from(dead.generation()).unwrap()),
    +    )
    +    .await;
    +    assert!(
    +        !engine.check_maintenance_claim(stale).await.expect("check"),
    +        "a stale claim is not held"
    +    );
    +    let pause = extenddb_storage_cassandra::test_hooks::pause_before_maintenance_takeover();
    +    let (a, b, ()) = tokio::join!(
    +        engine.maintenance_claim("purge on host-a", stale),
    +        engine.maintenance_claim("purge on host-b", stale),
    +        async {
    +            pause.reached().await;
    +            pause.reached().await;
    +            extenddb_storage_cassandra::test_hooks::clear_maintenance_takeover_pause();
    +            pause.release();
    +            pause.release();
    +        },
    +    );
    +    let (winner, loser) = match (a, b) {
    +        (Ok(w), Err(l)) | (Err(l), Ok(w)) => (w, l),
    +        (Ok(_), Ok(_)) => panic!("two takers both won one stale claim"),
    +        (Err(a), Err(b)) => panic!("neither taker won: {a:?} / {b:?}"),
    +    };
    +    match loser {
    +        StorageError::AlreadyExists(m) => assert!(m.contains("purge on host-"), "{m}"),
    +        other => panic!("the loser is refused with AlreadyExists: {other:?}"),
    +    }
    +    assert_ne!(winner.generation(), dead.generation());
    +    assert_eq!(
    +        winner.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    assert_eq!(
    +        dead.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Lost,
    +        "the dead holder's handle finds the claim lost"
    +    );
    +    assert!(engine.check_maintenance_claim(stale).await.expect("check"));
    +    winner.release().await.expect("release");
    +    assert!(!engine.check_maintenance_claim(stale).await.expect("check"));
    +}
    +
    +/// An insert whose answer is lost after it committed (the test hook turns
    +/// the applied answer into `Unknown`) is settled by the re-read: the row
    +/// carries the call's generation, so the call holds the claim, heartbeats it
    +/// and releases it. Without the re-read the command would be refused while
    +/// its own row, which nobody heartbeats, refused every job until it went
    +/// stale.
    +#[cfg(feature = "test-hooks")]
    +#[tokio::test]
    +async fn test_a_maintenance_insert_whose_answer_is_lost_is_settled_by_the_re_read() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let stale = Duration::from_secs(300);
    +    assert!(
    +        !engine.check_maintenance_claim(stale).await.expect("check"),
    +        "no claim is held before the test"
    +    );
    +    extenddb_storage_cassandra::test_hooks::lose_next_maintenance_insert_answer();
    +    let claim = engine
    +        .maintenance_claim("purge on host-lost", stale)
    +        .await
    +        .expect("an insert whose answer was lost is this call's claim once re-read");
    +    assert!(
    +        engine.check_maintenance_claim(stale).await.expect("check"),
    +        "the committed insert is held"
    +    );
    +    let rows = extenddb_storage_cassandra::cassandra_util::query_rows::(
    +        &engine.session_arc(),
    +        &format!(
    +            "SELECT holder, owner_generation FROM {}.backup_maintenance WHERE claim = 'claim'",
    +            engine.catalog_keyspace()
    +        ),
    +        cdrs_tokio::query_values!(),
    +        "maintenance claim",
    +    )
    +    .await
    +    .expect("read");
    +    use cdrs_tokio::types::IntoRustByName;
    +    let holder: Option = rows
    +        .first()
    +        .and_then(|r| r.get_by_name("holder").ok().flatten());
    +    let generation: Option = rows
    +        .first()
    +        .and_then(|r| r.get_by_name("owner_generation").ok().flatten());
    +    assert_eq!(holder.as_deref(), Some("purge on host-lost"));
    +    assert_eq!(generation, Some(i64::try_from(claim.generation()).unwrap()));
    +    assert_eq!(
    +        claim.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    claim.release().await.expect("release");
    +    assert!(!engine.check_maintenance_claim(stale).await.expect("check"));
    +}
    +
    +/// `newest_available_backup_row` reads the status index's AVAILABLE
    +/// partition and one row: the newest AVAILABLE row in the listing's order,
    +/// none while the store has no AVAILABLE row, and never a row of another
    +/// store or status. `list_running_restore_jobs` lists a RUNNING record by
    +/// status alone, one whose heartbeat is ahead of the coordinator's clock
    +/// included, which the dead-job listing under a zero window does not list.
    +#[tokio::test]
    +async fn test_the_newest_available_row_and_the_running_restores_are_read_by_status() {
    +    if crate::helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    let engine = setup_engine().await;
    +    let account = TestAccount::new(&engine, "extenddb_ttl_test").await;
    +    let account_id = account.account_id.clone();
    +    let store = unique_store();
    +    assert!(
    +        engine
    +            .newest_available_backup_row(&store)
    +            .await
    +            .expect("read")
    +            .is_none(),
    +        "no AVAILABLE row yet"
    +    );
    +    let publish = |row: BackupRow| {
    +        let engine = &engine;
    +        async move {
    +            let claim = engine.claim_backup(&row).await.expect("claim");
    +            assert_eq!(
    +                claim
    +                    .set_status(
    +                        JobStatus::Creating,
    +                        JobStatus::Available,
    +                        &StatusFields::default()
    +                    )
    +                    .await
    +                    .expect("publish"),
    +                WriteOutcome::Matched
    +            );
    +        }
    +    };
    +    publish(index_row(&account_id, "Music", "older", &store)).await;
    +    tokio::time::sleep(Duration::from_millis(5)).await;
    +    publish(index_row(&account_id, "Music", "newer", &store)).await;
    +    let creating = index_row(&account_id, "Music", "creating", &store);
    +    let _creating = engine.claim_backup(&creating).await.expect("claim");
    +    let mut other_store = store.clone();
    +    other_store.store_id = "store-2".to_owned();
    +    publish(index_row(&account_id, "Songs", "elsewhere", &other_store)).await;
    +    let newest = engine
    +        .newest_available_backup_row(&store)
    +        .await
    +        .expect("read")
    +        .expect("one AVAILABLE row");
    +    let listed: Vec = engine
    +        .list_backup_rows(Some(BackupStatus::Available), &store)
    +        .try_collect()
    +        .await
    +        .expect("list");
    +    assert_eq!(listed.len(), 2);
    +    assert_eq!(newest.backup_arn, listed.last().unwrap().backup_arn);
    +    assert_eq!(newest.backup_name, "newer");
    +
    +    // A RUNNING restore record stamped a minute ahead of the coordinator.
    +    let table = TestTable::with_account(&engine, &account_id, "rr_target", false).await;
    +    let table_id = table.key_info.table_id.clone();
    +    let ahead = i64::try_from(
    +        std::time::SystemTime::now()
    +            .duration_since(std::time::UNIX_EPOCH)
    +            .unwrap()
    +            .as_millis(),
    +    )
    +    .unwrap()
    +        + 60_000;
    +    apply(
    +        &engine,
    +        &format!(
    +            "UPDATE {}.tables SET restore_status = 'RUNNING', restore_kind = 'legacy', \
    +             restore_owner_generation = 7, restore_heartbeat_at = ?, \
    +             restore_source_backup_arn = ? WHERE account_id = ? AND table_name = ? \
    +             IF table_id = ?",
    +            engine.catalog_keyspace()
    +        ),
    +        cdrs_tokio::query_values!(
    +            ahead,
    +            newest.backup_arn.as_str(),
    +            account_id.as_str(),
    +            "rr_target",
    +            table_id.as_str()
    +        ),
    +    )
    +    .await;
    +    let running = engine
    +        .list_running_restore_jobs()
    +        .await
    +        .expect("list running");
    +    let ours: Vec<&extenddb_storage::backup_catalog::RestoreJobRow> = running
    +        .iter()
    +        .filter(|job| job.target.account_id == account_id)
    +        .collect();
    +    assert_eq!(ours.len(), 1, "{running:?}");
    +    assert_eq!(ours[0].target.table_name, "rr_target");
    +    assert_eq!(ours[0].target.table_id, table_id);
    +    assert_eq!(ours[0].status, JobStatus::Running);
    +    assert!(
    +        !engine
    +            .list_dead_restore_jobs(Duration::ZERO)
    +            .await
    +            .expect("list dead")
    +            .iter()
    +            .any(|job| job.target.account_id == account_id),
    +        "the age-based listing misses a heartbeat ahead of the clock"
    +    );
    +}
    diff --git a/crates/storage-cassandra/tests/migrations/lock.rs b/crates/storage-cassandra/tests/migrations/lock.rs
    index b56c2d441..eceac2106 100644
    --- a/crates/storage-cassandra/tests/migrations/lock.rs
    +++ b/crates/storage-cassandra/tests/migrations/lock.rs
    @@ -12,7 +12,7 @@ use extenddb_storage::migrations::{CatalogMigrations, CatalogStatus, MigrationEr
     use extenddb_storage_cassandra::cassandra_util::CassandraSession;
     use extenddb_storage_cassandra::migrations::{
         CassandraCatalogMigrations, LeaseTimes, MigrationLock, RegistrationTimes, ServerRegistration,
    -    ServingRegistration, register_server, schema_statement_bounds,
    +    ServingRegistration, register_server, register_server_as, schema_statement_bounds,
     };
     
     use super::{deployment, target_of};
    @@ -197,6 +197,96 @@ async fn a_server_and_a_migration_refuse_each_other() {
         assert!(rows.is_empty(), "a released server leaves no row");
     }
     
    +/// The `release` cells of the catalog's `extenddb_servers` rows.
    +async fn server_releases(session: &CassandraSession, catalog: &str) -> Vec {
    +    use cdrs_tokio::types::IntoRustByName;
    +    session
    +        .query(format!("SELECT release FROM {catalog}.extenddb_servers"))
    +        .await
    +        .expect("read")
    +        .response_body()
    +        .expect("body")
    +        .into_rows()
    +        .unwrap_or_default()
    +        .iter()
    +        .map(|row| {
    +            let release: Option = row.get_by_name("release").expect("release");
    +            release.unwrap_or_default()
    +        })
    +        .collect()
    +}
    +
    +/// A command that opens the catalog as a server does (`extenddb backup
    +/// list`) registers under its own name: the row's `release` carries it
    +/// after the release, a migration's wait names the command where it names
    +/// a server `extenddb serve`, the name is kept when the registration is taken
    +/// again, and the command's refusal while a migration holds the lock names
    +/// the holder without the server's advice to start the server.
    +#[tokio::test]
    +async fn a_command_registers_under_its_own_name() {
    +    if helpers::skip_without_cassandra() {
    +        return;
    +    }
    +    const COMMAND: &str = "extenddb backup list";
    +    let session = session().await;
    +    let (prefix, _drops) = new_prefix("command");
    +    deployment(&session, &prefix).await;
    +    let catalog = format!("{prefix}_catalog");
    +    let registration = register_server_as(session.clone(), &prefix, "127.0.0.1:1", "test", COMMAND)
    +        .await
    +        .expect("register");
    +    assert_eq!(
    +        server_releases(&session, &catalog).await,
    +        vec![format!("test as {COMMAND}")]
    +    );
    +    let migrations = CassandraCatalogMigrations;
    +    match migrations
    +        .lock(&target_of(&prefix), Duration::from_secs(2))
    +        .await
    +    {
    +        Err(MigrationError::Locked(message)) => {
    +            assert!(
    +                message.contains(&format!("{COMMAND} test on 127.0.0.1:1")),
    +                "{message}"
    +            );
    +            assert!(!message.contains("extenddb serve"), "{message}");
    +        }
    +        other => panic!("{other:?}"),
    +    }
    +    let again = registration.register_again().await;
    +    assert!(matches!(again, Reregistration::Registered), "{again:?}");
    +    assert!(
    +        server_releases(&session, &catalog)
    +            .await
    +            .contains(&format!("test as {COMMAND}")),
    +        "taken again under the command's name"
    +    );
    +    registration.release().await;
    +
    +    let lock = migrations
    +        .lock(&target_of(&prefix), Duration::from_secs(5))
    +        .await
    +        .expect("no server is registered");
    +    let refused = register_server_as(session.clone(), &prefix, "127.0.0.1:2", "test", COMMAND)
    +        .await
    +        .err()
    +        .expect("a migration holds the lock");
    +    assert_eq!(refused.exit_code(), 6);
    +    let why = refused.to_string();
    +    assert!(why.contains("holds the catalog's migration lock"), "{why}");
    +    assert!(!why.contains("start the server"), "{why}");
    +    let served = register_server(session.clone(), &prefix, "127.0.0.1:3", "test")
    +        .await
    +        .err()
    +        .expect("a server is refused too");
    +    assert!(served.to_string().contains("start the server"), "{served}");
    +    migrations.release(lock).await;
    +    assert!(
    +        server_releases(&session, &catalog).await.is_empty(),
    +        "a refused command leaves no row"
    +    );
    +}
    +
     /// A code body's request to a node that stops answering fails at the request
     /// timeout, through the store the body is handed, rather than when the lease
     /// cuts the body 35 to 65 s later (SPEC.md A6: every request under the lock
    diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs
    index 1b960b93f..752d0e106 100644
    --- a/crates/storage-mongodb/src/backup_catalog.rs
    +++ b/crates/storage-mongodb/src/backup_catalog.rs
    @@ -24,14 +24,22 @@
     //! (`WriteOutcome::Unknown`: the statement may have reached the server);
     //! a server error (a command or write error) is `Err`, as a statement error
     //! is on PostgreSQL.
    +//!
    +//! The maintenance claim is one document (`_id: "claim"`) in
    +//! `backup_maintenance`: a record whose heartbeat is older than the stale
    +//! window is taken over by a conditional update, a missing one is inserted by
    +//! an upsert whose filter a live record never matches (its insert then trips
    +//! the `_id` index and the claim is refused), and heartbeats and the release
    +//! are conditioned on the generation. `$expr` is not allowed in an upsert's
    +//! filter, which is why the takeover and the insert are two writes.
     
     use std::time::Duration;
     
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock,
    +    StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::TryStreamExt;
    @@ -196,6 +204,31 @@ fn key_filter(key: &BackupKey) -> Document {
         doc! { "account_id": &key.account_id, "backup_id": &key.backup_id }
     }
     
    +/// The filter for the rows of `store`.
    +fn store_filter(store: &StoreIdentity) -> Document {
    +    doc! {
    +        "store_kind": store.store_kind.as_str(),
    +        "store_root": &store.store_root,
    +        "store_id": &store.store_id,
    +    }
    +}
    +
    +/// The `$expr` of [`BackupCatalog::list_backup_rows_older_than`]: the
    +/// `clock` column older than `age` on the server's clock. `age_is_stale`
    +/// reads a missing column as the epoch, which is the rule the dead-job
    +/// listing wants and not this one: a row with `completed_at` unset has not
    +/// reached its terminal status's clock, so the column must also be present.
    +fn older_than_filter(clock: StaleClock, age: Duration) -> Document {
    +    let column = match clock {
    +        StaleClock::Heartbeat => "heartbeat_at",
    +        StaleClock::Completed => "completed_at",
    +    };
    +    doc! {
    +        column: { "$type": "date" },
    +        "$expr": age_is_stale(&[column], age),
    +    }
    +}
    +
     /// The sort every listing applies: `(created_at, backup_arn)`.
     fn listing_sort() -> Document {
         doc! { "created_at": 1, "backup_arn": 1 }
    @@ -244,6 +277,28 @@ pub(crate) fn dead_restore_job_filter(stale_after: Duration) -> Document {
         }
     }
     
    +/// A restore job record as the two listings read a table document.
    +fn restore_job_row(doc: &Document) -> Result {
    +    let id = doc
    +        .get_document("_id")
    +        .map_err(|_| StorageError::Internal("table is missing _id".to_owned()))?;
    +    let job = doc.get_document(RESTORE_JOB).map_err(|_| {
    +        StorageError::Internal("restore target is missing its job record".to_owned())
    +    })?;
    +    Ok(RestoreJobRow {
    +        target: RestoreTargetKey {
    +            account_id: required_str(id, "account_id")?,
    +            table_name: required_str(id, "table_name")?,
    +            table_id: required_str(doc, "table_id")?,
    +        },
    +        source_backup_arn: required_str(doc, "restore_source_backup_arn")?,
    +        kind: RestoreKind::parse(&required_str(job, "kind")?)?,
    +        status: JobStatus::parse(&required_str(job, "status")?)?,
    +        owner_generation: generation_from_field(job, "owner_generation"),
    +        heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0),
    +    })
    +}
    +
     /// A backup job's handle on its `backup_index` document.
     struct MongoBackupClaim {
         index: mongodb::Collection,
    @@ -530,11 +585,7 @@ impl BackupCatalog for MongoEngine {
             store: &'a StoreIdentity,
         ) -> BoxStream<'a, Result> {
             rows_stream(async move {
    -            let mut filter = doc! {
    -                "store_kind": store.store_kind.as_str(),
    -                "store_root": &store.store_root,
    -                "store_id": &store.store_id,
    -            };
    +            let mut filter = store_filter(store);
                 if let Some(status) = status {
                     filter.insert("status", status.as_str());
                 }
    @@ -549,6 +600,24 @@ impl BackupCatalog for MongoEngine {
             })
         }
     
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            let mut filter = store_filter(store);
    +            filter.insert("status", BackupStatus::Available.as_str());
    +            self.backup_index()
    +                .find_one(filter)
    +                .sort(doc! { "created_at": -1, "backup_arn": -1 })
    +                .await
    +                .map_err(internal)?
    +                .as_ref()
    +                .map(row_from_doc)
    +                .transpose()
    +        })
    +    }
    +
         fn list_account_backup_rows<'a>(
             &'a self,
             account_id: &'a str,
    @@ -648,6 +717,46 @@ impl BackupCatalog for MongoEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let mut filter = key_filter(key);
    +            filter.insert("status", BackupStatus::Creating.as_str());
    +            filter.insert("$expr", age_is_stale(&["heartbeat_at"], stale_after));
    +            let result = self
    +                .backup_index()
    +                .delete_one(filter)
    +                .await
    +                .map_err(internal)?;
    +            Ok(result.deleted_count == 1)
    +        })
    +    }
    +
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result> {
    +        rows_stream(async move {
    +            let mut filter = store_filter(store);
    +            filter.insert("status", status.as_str());
    +            filter.extend(older_than_filter(clock, age));
    +            self.backup_index()
    +                .find(filter)
    +                .sort(listing_sort())
    +                .await
    +                .map_err(internal)?
    +                .try_collect()
    +                .await
    +                .map_err(internal)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -729,44 +838,36 @@ impl BackupCatalog for MongoEngine {
                 // cannot read, so one such document does not stop the cycle.
                 Ok(docs
                     .iter()
    -                .filter_map(|doc| {
    -                    let read = || -> Result {
    -                        let id = doc.get_document("_id").map_err(|_| {
    -                            StorageError::Internal("table is missing _id".to_owned())
    -                        })?;
    -                        let job = doc.get_document(RESTORE_JOB).map_err(|_| {
    -                            StorageError::Internal(
    -                                "restore target is missing its job record".to_owned(),
    -                            )
    -                        })?;
    -                        Ok(RestoreJobRow {
    -                            target: RestoreTargetKey {
    -                                account_id: required_str(id, "account_id")?,
    -                                table_name: required_str(id, "table_name")?,
    -                                table_id: required_str(doc, "table_id")?,
    -                            },
    -                            source_backup_arn: required_str(doc, "restore_source_backup_arn")?,
    -                            kind: RestoreKind::parse(&required_str(job, "kind")?)?,
    -                            status: JobStatus::parse(&required_str(job, "status")?)?,
    -                            owner_generation: generation_from_field(job, "owner_generation"),
    -                            heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0),
    -                        })
    -                    };
    -                    match read() {
    -                        Ok(row) => Some(row),
    -                        Err(e) => {
    -                            tracing::warn!(
    -                                "reconciler: restore job record {:?} skipped: {e}",
    -                                doc.get("_id")
    -                            );
    -                            None
    -                        }
    +                .filter_map(|doc| match restore_job_row(doc) {
    +                    Ok(row) => Some(row),
    +                    Err(e) => {
    +                        tracing::warn!(
    +                            "reconciler: restore job record {:?} skipped: {e}",
    +                            doc.get("_id")
    +                        );
    +                        None
                         }
                     })
                     .collect())
             })
         }
     
    +    fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>> {
    +        Box::pin(async move {
    +            // By status alone: a record stamped in this read's instant, or
    +            // ahead of the server's clock, is RUNNING too.
    +            let docs: Vec = self
    +                .tables()
    +                .find(doc! { RESTORE_JOB_STATUS: JobStatus::Running.as_str() })
    +                .await
    +                .map_err(internal)?
    +                .try_collect()
    +                .await
    +                .map_err(internal)?;
    +            docs.iter().map(restore_job_row).collect()
    +        })
    +    }
    +
         /// Conditioned on RUNNING, on the listed `owner_generation` and on the
         /// listed `heartbeat_at` (a `Date`, millisecond precision, rebuilt from
         /// the listed epoch seconds exactly), and written with a fresh generation
    @@ -802,12 +903,122 @@ impl BackupCatalog for MongoEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            let ids = self
    +                .backup_index()
    +                .distinct(
    +                    "store_id",
    +                    doc! { "store_kind": store_kind.as_str(), "store_root": store_root },
    +                )
    +                .await
    +                .map_err(internal)?;
    +            let mut ids: Vec = ids
    +                .into_iter()
    +                .filter_map(|v| match v {
    +                    Bson::String(s) => Some(s),
    +                    _ => None,
    +                })
    +                .collect();
    +            ids.sort();
    +            Ok(ids)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The record is the claim, as a backup row is: one document under
    +            // a fixed `_id`. Two writes: the first takes over a record whose
    +            // heartbeat is stale (a dead holder's), the second inserts the
    +            // record when there is none. A live record matches neither
    +            // filter, so the second write's upsert trips the `_id` index and
    +            // the claim is refused; two commands racing for a stale record
    +            // both run the first write, and the one whose filter no longer
    +            // matches falls to the second and is refused the same way.
    +            // (`$expr` is not allowed in an upsert's filter, which is why
    +            // the two are not one write.)
    +            let generation = new_generation();
    +            let collection = self.maintenance();
    +            let set = doc! { "$set": {
    +                "holder": { "$literal": holder },
    +                "owner_generation": { "$literal": generation_field(generation) },
    +                "claimed_at": "$$NOW",
    +                "heartbeat_at": "$$NOW",
    +            } };
    +            let refused = |current: Option| {
    +                StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {}",
    +                    current.as_deref().unwrap_or("another maintenance command")
    +                ))
    +            };
    +            let taken_over = collection
    +                .update_one(
    +                    doc! {
    +                        "_id": MAINTENANCE_CLAIM_ID,
    +                        "$expr": age_is_stale(&["heartbeat_at"], stale_after),
    +                    },
    +                    vec![set.clone()],
    +                )
    +                .await
    +                .map_err(internal)?
    +                .matched_count
    +                == 1;
    +            if !taken_over {
    +                match collection
    +                    .update_one(
    +                        doc! {
    +                            "_id": MAINTENANCE_CLAIM_ID,
    +                            "owner_generation": { "$exists": false },
    +                        },
    +                        vec![set],
    +                    )
    +                    .upsert(true)
    +                    .await
    +                {
    +                    // The upsert matched a document with no
    +                    // `owner_generation` (an operator's insert, say, not a
    +                    // release's) and overwrote it with this call's
    +                    // generation: the claim is this call's, and refusing here
    +                    // would leave a document nobody heartbeats until it goes
    +                    // stale.
    +                    Ok(result) if result.upserted_id.is_some() || result.matched_count == 1 => {}
    +                    Ok(_) => return Err(refused(None)),
    +                    Err(e) if is_duplicate_key(&e) => {
    +                        let current = collection
    +                            .find_one(doc! { "_id": MAINTENANCE_CLAIM_ID })
    +                            .await
    +                            .map_err(internal)?
    +                            .and_then(|d| d.get_str("holder").ok().map(str::to_owned));
    +                        return Err(refused(current));
    +                    }
    +                    Err(e) => return Err(internal(e)),
    +                }
    +            }
    +            Ok(Box::new(MongoMaintenanceClaim {
    +                collection,
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
             Box::pin(async move {
                 let held = self
    -                .catalog_db
    -                .collection::(BACKUP_MAINTENANCE)
    -                .find_one(doc! {})
    +                .maintenance()
    +                .find_one(doc! {
    +                    "$nor": [ { "$expr": age_is_stale(&["heartbeat_at"], stale_after) } ],
    +                })
                     .await
                     .map_err(internal)?;
                 Ok(held.is_some())
    @@ -815,6 +1026,64 @@ impl BackupCatalog for MongoEngine {
         }
     }
     
    +/// The `_id` of the one maintenance claim document.
    +const MAINTENANCE_CLAIM_ID: &str = "claim";
    +
    +impl MongoEngine {
    +    fn maintenance(&self) -> mongodb::Collection {
    +        self.catalog_db.collection::(BACKUP_MAINTENANCE)
    +    }
    +}
    +
    +/// The holder's handle on the maintenance claim document.
    +struct MongoMaintenanceClaim {
    +    collection: mongodb::Collection,
    +    generation: u64,
    +}
    +
    +impl MongoMaintenanceClaim {
    +    fn owned_filter(&self) -> Document {
    +        doc! {
    +            "_id": MAINTENANCE_CLAIM_ID,
    +            "owner_generation": generation_field(self.generation),
    +        }
    +    }
    +}
    +
    +impl MaintenanceClaim for MongoMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let result = self
    +                .collection
    +                .update_one(
    +                    self.owned_filter(),
    +                    vec![doc! { "$set": { "heartbeat_at": "$$NOW" } }],
    +                )
    +                .await
    +                .map_err(|e| StorageError::Internal(format!("maintenance heartbeat: {e}")))?;
    +            Ok(if result.matched_count == 1 {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            self.collection
    +                .delete_one(self.owned_filter())
    +                .await
    +                .map_err(internal)?;
    +            Ok(())
    +        })
    +    }
    +}
    +
     #[cfg(test)]
     mod tests {
         use super::*;
    @@ -906,6 +1175,33 @@ mod tests {
             assert!(!outcome_unknown(&command));
         }
     
    +    /// The older-than filter reads the named column alone against the
    +    /// server's clock and requires the column to be a date, so a row whose
    +    /// `completed_at` was never stamped is not read as old as the epoch.
    +    #[test]
    +    fn the_older_than_filter_names_one_clock_column_and_requires_it_set() {
    +        let heartbeat = older_than_filter(StaleClock::Heartbeat, Duration::from_secs(90));
    +        assert_eq!(
    +            heartbeat.get_document("heartbeat_at").unwrap(),
    +            &doc! { "$type": "date" }
    +        );
    +        assert!(heartbeat.get("completed_at").is_none());
    +        let expr = heartbeat.get_document("$expr").unwrap().to_string();
    +        assert!(expr.contains("$heartbeat_at"), "{expr}");
    +        assert!(!expr.contains("completed_at"), "{expr}");
    +        assert!(expr.contains("$$NOW") && expr.contains("90000"), "{expr}");
    +
    +        let completed = older_than_filter(StaleClock::Completed, Duration::from_secs(90));
    +        assert_eq!(
    +            completed.get_document("completed_at").unwrap(),
    +            &doc! { "$type": "date" }
    +        );
    +        assert!(completed.get("heartbeat_at").is_none());
    +        let expr = completed.get_document("$expr").unwrap().to_string();
    +        assert!(expr.contains("$completed_at"), "{expr}");
    +        assert!(!expr.contains("heartbeat_at"), "{expr}");
    +    }
    +
         /// The dead-job filter selects, by the server's clock, RUNNING records
         /// with a stale heartbeat and FAILED records of a CREATING table whose
         /// `completed_at`, or heartbeat where none, is stale.
    diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs
    index e396ace38..ab4ad92c4 100644
    --- a/crates/storage-mongodb/src/backup_live_tests.rs
    +++ b/crates/storage-mongodb/src/backup_live_tests.rs
    @@ -31,7 +31,8 @@ use extenddb_core::types::{
     };
     use extenddb_storage::backup_catalog::{
         BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, RestoreTargetKey,
    -    StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
    +    StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION,
    +    WriteOutcome,
     };
     use extenddb_storage::error::StorageError;
     use extenddb_storage::{
    @@ -528,7 +529,12 @@ async fn live_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             uuid::Uuid::parse_str(&fx.engine.catalog_id().await.expect("id")).is_ok(),
             "the catalog has an id from the migration step"
         );
    -    assert!(!fx.engine.check_maintenance_claim().await.expect("check"));
    +    assert!(
    +        !fx.engine
    +            .check_maintenance_claim(std::time::Duration::from_secs(300))
    +            .await
    +            .expect("check")
    +    );
         let row = row(ACCOUNT, "Music", "nightly");
         let claim = fx.engine.claim_backup(&row).await.expect("claim");
         assert_ne!(claim.generation(), UNOWNED_GENERATION);
    @@ -672,14 +678,384 @@ async fn live_claim_backup_excludes_a_second_owner_and_fences_its_writes() {
             "{third:?}"
         );
     
    -    // A maintenance record, once written, is seen by every job.
    -    fx.engine
    +    fx.finish().await;
    +}
    +
    +/// `read_store_ids`: the distinct ids of the rows for a root, of every
    +/// status, scoped to the kind and the root.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_read_store_ids_lists_the_distinct_ids_of_a_root() {
    +    let fx = Fixture::new().await;
    +    let root = store().store_root.clone();
    +    let ids = |kind: StoreKind, root: String| {
    +        let engine = &fx.engine;
    +        async move { engine.read_store_ids(kind, &root).await.expect("ids") }
    +    };
    +    assert!(ids(StoreKind::Filesystem, root.clone()).await.is_empty());
    +    let first = row(ACCOUNT, "Music", "a");
    +    let _c1 = fx.engine.claim_backup(&first).await.expect("claim");
    +    let second = row(ACCOUNT, "Music", "b");
    +    let c2 = fx.engine.claim_backup(&second).await.expect("claim");
    +    c2.set_status(
    +        JobStatus::Creating,
    +        JobStatus::Failed,
    +        &StatusFields::default(),
    +    )
    +    .await
    +    .expect("fail");
    +    assert_eq!(
    +        ids(StoreKind::Filesystem, root.clone()).await,
    +        ["store-1"],
    +        "two rows of one store, of two statuses, are one id"
    +    );
    +    let mut other = row(ACCOUNT, "Songs", "c");
    +    other.store.store_id = "store-2".to_owned();
    +    let _c3 = fx.engine.claim_backup(&other).await.expect("claim");
    +    assert_eq!(
    +        ids(StoreKind::Filesystem, root.clone()).await,
    +        ["store-1", "store-2"]
    +    );
    +    assert!(
    +        ids(StoreKind::S3, root.clone()).await.is_empty(),
    +        "another kind at the same root is another store"
    +    );
    +    assert!(
    +        ids(StoreKind::Filesystem, "/elsewhere".to_owned())
    +            .await
    +            .is_empty()
    +    );
    +    fx.finish().await;
    +}
    +
    +/// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is older
    +/// than the window is removed outright; one with a live owner, one that is
    +/// not CREATING, and one that is gone are left, answering `false`.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +    let fx = Fixture::new().await;
    +    let stale = std::time::Duration::from_secs(300);
    +    let index = fx.engine.backup_index();
    +    let age = |backup_id: String| {
    +        let index = index.clone();
    +        async move {
    +            index
    +                .update_one(
    +                    doc! { "backup_id": backup_id },
    +                    vec![doc! { "$set": { "heartbeat_at": { "$subtract": [ "$$NOW", 600_000_i64 ] } } }],
    +                )
    +                .await
    +                .expect("age");
    +        }
    +    };
    +    let creating = row(ACCOUNT, "Music", "a");
    +    let key = creating.key();
    +    let claim = fx.engine.claim_backup(&creating).await.expect("claim");
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "a live owner keeps its row"
    +    );
    +    age(creating.backup_id.clone()).await;
    +    assert!(
    +        fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap()
    +    );
    +    assert!(
    +        fx.engine
    +            .read_backup_row(&arn_of(&creating))
    +            .await
    +            .unwrap()
    +            .is_none()
    +    );
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&key, stale)
    +            .await
    +            .unwrap(),
    +        "gone"
    +    );
    +    drop(claim);
    +
    +    let available = row(ACCOUNT, "Music", "b");
    +    let claim = fx.engine.claim_backup(&available).await.expect("claim");
    +    claim
    +        .set_status(
    +            JobStatus::Creating,
    +            JobStatus::Available,
    +            &StatusFields::default(),
    +        )
    +        .await
    +        .expect("available");
    +    age(available.backup_id.clone()).await;
    +    assert!(
    +        !fx.engine
    +            .remove_dead_creating_backup_row(&available.key(), stale)
    +            .await
    +            .unwrap(),
    +        "only a CREATING row"
    +    );
    +    assert_eq!(
    +        fx.engine
    +            .read_backup_row(&arn_of(&available))
    +            .await
    +            .unwrap()
    +            .unwrap()
    +            .status,
    +        BackupStatus::Available
    +    );
    +    drop(claim);
    +    fx.finish().await;
    +}
    +
    +/// The `backup_index` collection read at `majority`, for the observations
    +/// a test makes between the writes under test.
    +fn index_majority(engine: &MongoEngine) -> mongodb::Collection {
    +    engine.catalog_db.collection_with_options(
    +        BACKUP_INDEX,
    +        mongodb::options::CollectionOptions::builder()
    +            .read_concern(mongodb::options::ReadConcern::majority())
    +            .build(),
    +    )
    +}
    +
    +async fn index_doc(engine: &MongoEngine, row: &BackupRow) -> Document {
    +    index_majority(engine)
    +        .find_one(doc! { "account_id": &row.account_id, "backup_id": &row.backup_id })
    +        .await
    +        .expect("read the row")
    +        .expect("the row exists")
    +}
    +
    +/// `list_backup_rows_older_than`: the rows of the store with the status
    +/// whose named clock column is older than the age on the server's clock. A
    +/// row FAILED just now by a dead owner is listed on the heartbeat clock (its
    +/// heartbeat is the owner's last) and not yet on the completed clock, which
    +/// tells the two columns apart; the completed window is then crossed by
    +/// waiting. A FAILED row with no `completed_at` is never listed on the
    +/// completed clock. A CREATING row, and a FAILED row of another store, are
    +/// not listed under FAILED; a DELETED row is listed under DELETED on its
    +/// `completed_at`. The listing keeps the catalog's order.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_rows_older_than_are_listed_on_the_named_clock() {
    +    let fx = Fixture::new().await;
    +    let window = Duration::from_secs(3);
    +    let names = |listed: Vec| -> Vec {
    +        listed.into_iter().map(|r| r.backup_name).collect()
    +    };
    +    let older = |status: BackupStatus, store: StoreIdentity, clock: StaleClock| {
    +        let engine = &fx.engine;
    +        async move {
    +            let rows: Vec = engine
    +                .list_backup_rows_older_than(status, &store, clock, window)
    +                .try_collect()
    +                .await
    +                .expect("listing");
    +            names(rows)
    +        }
    +    };
    +    let mut other_store = store();
    +    other_store.store_id = "store-2".to_owned();
    +
    +    let completed = row(ACCOUNT, "Music", "completed");
    +    let completed_claim = fx.engine.claim_backup(&completed).await.expect("claim");
    +    tokio::time::sleep(Duration::from_millis(5)).await;
    +    let unstamped = row(ACCOUNT, "Music", "unstamped");
    +    let unstamped_claim = fx.engine.claim_backup(&unstamped).await.expect("claim");
    +    tokio::time::sleep(Duration::from_millis(5)).await;
    +    let creating = row(ACCOUNT, "Music", "creating");
    +    let _creating_claim = fx.engine.claim_backup(&creating).await.expect("claim");
    +    let mut elsewhere = row(ACCOUNT, "Music", "elsewhere");
    +    elsewhere.store = other_store.clone();
    +    let elsewhere_claim = fx.engine.claim_backup(&elsewhere).await.expect("claim");
    +    let deleted = row(ACCOUNT, "Music", "deleted");
    +    let deleted_claim = fx.engine.claim_backup(&deleted).await.expect("claim");
    +    let done = StatusFields {
    +        completed: true,
    +        ..StatusFields::default()
    +    };
    +    assert_eq!(
    +        deleted_claim
    +            .set_status(JobStatus::Creating, JobStatus::Available, &done)
    +            .await
    +            .unwrap(),
    +        WriteOutcome::Matched
    +    );
    +    assert_eq!(
    +        fx.engine.start_backup_delete(&deleted.key()).await.unwrap(),
    +        DeleteStart::Deleting
    +    );
    +    assert!(fx.engine.set_backup_deleted(&deleted.key()).await.unwrap());
    +    assert!(
    +        older(BackupStatus::Failed, store(), StaleClock::Heartbeat)
    +            .await
    +            .is_empty(),
    +        "no row is FAILED yet"
    +    );
    +
    +    tokio::time::sleep(window + Duration::from_millis(300)).await;
    +    // The owners die here: FAILED writes with the heartbeat left as it was.
    +    for (claim, fields) in [
    +        (&completed_claim, done.clone()),
    +        (&unstamped_claim, StatusFields::default()),
    +        (&elsewhere_claim, done.clone()),
    +    ] {
    +        assert_eq!(
    +            claim
    +                .set_status(JobStatus::Creating, JobStatus::Failed, &fields)
    +                .await
    +                .unwrap(),
    +            WriteOutcome::Matched
    +        );
    +    }
    +    let unstamped_doc = index_doc(&fx.engine, &unstamped).await;
    +    assert!(
    +        unstamped_doc.get("completed_at").is_none(),
    +        "a write with `completed: false` stamps nothing: {unstamped_doc}"
    +    );
    +    assert_eq!(
    +        older(BackupStatus::Failed, store(), StaleClock::Heartbeat).await,
    +        ["completed", "unstamped"],
    +        "rows FAILED just now are listed on their dead owners' heartbeats, in the catalog's order"
    +    );
    +    assert!(
    +        older(BackupStatus::Failed, store(), StaleClock::Completed)
    +            .await
    +            .is_empty(),
    +        "a `completed_at` stamped just now is within the window; an unset one is never older"
    +    );
    +    assert_eq!(
    +        older(BackupStatus::Deleted, store(), StaleClock::Completed).await,
    +        ["deleted"]
    +    );
    +    assert_eq!(
    +        older(BackupStatus::Creating, store(), StaleClock::Heartbeat).await,
    +        ["creating"]
    +    );
    +    assert_eq!(
    +        older(
    +            BackupStatus::Failed,
    +            other_store.clone(),
    +            StaleClock::Heartbeat
    +        )
    +        .await,
    +        ["elsewhere"]
    +    );
    +
    +    tokio::time::sleep(window + Duration::from_millis(300)).await;
    +    assert_eq!(
    +        older(BackupStatus::Failed, store(), StaleClock::Completed).await,
    +        ["completed"],
    +        "the completed window crossed on the server's clock; no `completed_at` is never older"
    +    );
    +    assert_eq!(
    +        older(BackupStatus::Failed, store(), StaleClock::Heartbeat).await,
    +        ["completed", "unstamped"]
    +    );
    +    fx.finish().await;
    +}
    +
    +/// The maintenance claim: taken once, refused to a second holder naming the
    +/// first, heartbeated on the server's clock, released; a holder whose
    +/// heartbeat is older than the stale window is dead, so its claim is not
    +/// reported held and is taken over, and its handle finds the claim lost. A
    +/// record without a heartbeat (no release of this server wrote one) is
    +/// stale.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +    let fx = Fixture::new().await;
    +    let stale = std::time::Duration::from_secs(300);
    +    let check = || fx.engine.check_maintenance_claim(stale);
    +    let maintenance = fx
    +        .engine
             .catalog_db
    -        .collection::(crate::backup_catalog::BACKUP_MAINTENANCE)
    -        .insert_one(doc! { "holder": "backup purge", "owner_generation": 1_i64 })
    +        .collection::(crate::backup_catalog::BACKUP_MAINTENANCE);
    +    assert!(!check().await.expect("check"));
    +    let first = fx
    +        .engine
    +        .maintenance_claim("purge on host-a", stale)
    +        .await
    +        .expect("claim");
    +    assert_ne!(first.generation(), UNOWNED_GENERATION);
    +    assert!(check().await.expect("check"), "a live claim is held");
    +    match fx.engine.maintenance_claim("purge on host-b", stale).await {
    +        Err(StorageError::AlreadyExists(m)) => assert!(m.contains("purge on host-a"), "{m}"),
    +        Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +        Ok(_) => panic!("a second holder is refused"),
    +    }
    +    assert_eq!(
    +        first.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    first.release().await.expect("release");
    +    assert!(!check().await.expect("check"), "released");
    +    assert_eq!(
    +        maintenance.count_documents(doc! {}).await.expect("count"),
    +        0
    +    );
    +    first.release().await.expect("a second release is a no-op");
    +
    +    // A dead holder: its heartbeat aged past the window.
    +    let dead = fx
    +        .engine
    +        .maintenance_claim("purge on host-c", stale)
    +        .await
    +        .expect("claim");
    +    maintenance
    +        .update_one(
    +            doc! {},
    +            vec![doc! { "$set": { "heartbeat_at": { "$subtract": [ "$$NOW", 600_000_i64 ] } } }],
    +        )
    +        .await
    +        .expect("age");
    +    assert!(!check().await.expect("check"), "a stale claim is not held");
    +    let taker = fx
    +        .engine
    +        .maintenance_claim("purge on host-d", stale)
    +        .await
    +        .expect("a dead holder's claim is taken over");
    +    assert_ne!(taker.generation(), dead.generation());
    +    assert!(check().await.expect("check"));
    +    assert_eq!(
    +        dead.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Lost,
    +        "the dead holder's handle finds the claim lost"
    +    );
    +    dead.release()
    +        .await
    +        .expect("a release by a lost holder changes nothing");
    +    assert!(check().await.expect("check"), "the taker still holds it");
    +    let holder = maintenance
    +        .find_one(doc! {})
    +        .await
    +        .expect("find")
    +        .expect("record");
    +    assert_eq!(holder.get_str("holder").unwrap(), "purge on host-d");
    +    taker.release().await.expect("release");
    +    assert!(!check().await.expect("check"));
    +
    +    // A record with no heartbeat at all is stale: taken over, never held.
    +    maintenance
    +        .insert_one(
    +            doc! { "_id": "claim", "holder": "an older release", "owner_generation": 1_i64 },
    +        )
             .await
    -        .expect("maintenance record");
    -    assert!(fx.engine.check_maintenance_claim().await.expect("check"));
    +        .expect("record");
    +    assert!(!check().await.expect("check"));
    +    let over = fx
    +        .engine
    +        .maintenance_claim("purge on host-e", stale)
    +        .await
    +        .expect("taken over");
    +    assert!(check().await.expect("check"));
    +    over.release().await.expect("release");
         fx.finish().await;
     }
     
    @@ -3034,3 +3410,167 @@ async fn live_the_failed_write_matches_only_the_listed_generation_and_heartbeat(
         assert!(fx.engine.remove_restore_target(&key).await.expect("remove"));
         fx.finish().await;
     }
    +/// A claim document with a fresh `heartbeat_at` and no `owner_generation`
    +/// (an operator's insert, not a release's): the upsert's filter matches it
    +/// and overwrites it with this call's generation, so the claim is this
    +/// call's and is heartbeated and released by it, where refusing it would
    +/// leave a document nobody heartbeats until it went stale. A document with
    +/// a generation and a fresh heartbeat (a live holder's) is still refused.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_a_claim_document_without_a_generation_is_taken_not_refused() {
    +    let fx = Fixture::new().await;
    +    let stale = std::time::Duration::from_secs(300);
    +    let maintenance = fx
    +        .engine
    +        .catalog_db
    +        .collection::(crate::backup_catalog::BACKUP_MAINTENANCE);
    +    maintenance
    +        .insert_one(doc! { "_id": "claim", "holder": "an operator's insert" })
    +        .await
    +        .expect("insert");
    +    maintenance
    +        .update_one(doc! {}, vec![doc! { "$set": { "heartbeat_at": "$$NOW" } }])
    +        .await
    +        .expect("stamp");
    +    let claim = fx
    +        .engine
    +        .maintenance_claim("purge on host-a", stale)
    +        .await
    +        .expect("a document with no owner_generation is taken");
    +    let document = maintenance
    +        .find_one(doc! { "_id": "claim" })
    +        .await
    +        .expect("read")
    +        .expect("one document");
    +    assert_eq!(document.get_str("holder").unwrap(), "purge on host-a");
    +    assert!(
    +        document.get("owner_generation").is_some(),
    +        "the document carries the taker's generation: {document:?}"
    +    );
    +    assert!(
    +        fx.engine
    +            .check_maintenance_claim(stale)
    +            .await
    +            .expect("check")
    +    );
    +    assert_eq!(
    +        claim.heartbeat().await.expect("heartbeat"),
    +        Heartbeat::Confirmed
    +    );
    +    match fx.engine.maintenance_claim("purge on host-b", stale).await {
    +        Err(StorageError::AlreadyExists(m)) => assert!(m.contains("purge on host-a"), "{m}"),
    +        Err(other) => panic!("a live holder's document is refused with AlreadyExists: {other:?}"),
    +        Ok(_) => panic!("a live holder's document is refused"),
    +    }
    +    claim.release().await.expect("release");
    +    assert!(
    +        !fx.engine
    +            .check_maintenance_claim(stale)
    +            .await
    +            .expect("check")
    +    );
    +    fx.finish().await;
    +}
    +
    +/// `newest_available_backup_row` reads the last AVAILABLE row in the
    +/// listing's order, and none when the store has no AVAILABLE row;
    +/// `list_running_restore_jobs` lists a RUNNING record by status alone, one
    +/// whose heartbeat is ahead of the server's clock included, which the
    +/// dead-job listing under a zero window does not list.
    +#[tokio::test(flavor = "multi_thread")]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_the_newest_available_row_and_the_running_restores_are_read_by_status() {
    +    let fx = Fixture::new().await;
    +    let store = store();
    +    assert!(
    +        fx.engine
    +            .newest_available_backup_row(&store)
    +            .await
    +            .expect("read")
    +            .is_none(),
    +        "no AVAILABLE row yet"
    +    );
    +    let older = row(ACCOUNT, "Music", "older");
    +    let newer = row(ACCOUNT, "Music", "newer");
    +    for row in [&older, &newer] {
    +        let claim = fx.engine.claim_backup(row).await.expect("claim");
    +        assert_eq!(
    +            claim
    +                .set_status(
    +                    JobStatus::Creating,
    +                    JobStatus::Available,
    +                    &StatusFields::default()
    +                )
    +                .await
    +                .expect("publish"),
    +            WriteOutcome::Matched
    +        );
    +    }
    +    let newest = fx
    +        .engine
    +        .newest_available_backup_row(&store)
    +        .await
    +        .expect("read")
    +        .expect("one AVAILABLE row");
    +    let listed: Vec = fx
    +        .engine
    +        .list_backup_rows(Some(BackupStatus::Available), &store)
    +        .try_collect()
    +        .await
    +        .expect("list");
    +    assert_eq!(
    +        newest.backup_arn,
    +        listed.last().unwrap().backup_arn,
    +        "the newest row is the last in the listing's order"
    +    );
    +
    +    // A RUNNING restore record stamped 60 s ahead of the server's clock.
    +    source_table_with(&fx.engine, "rr_target", 1).await;
    +    let table_id = table_doc(&fx.engine, "rr_target")
    +        .await
    +        .expect("the table")
    +        .get_str("table_id")
    +        .expect("table_id")
    +        .to_owned();
    +    fx.engine
    +        .catalog_db
    +        .collection::("tables")
    +        .update_one(
    +            doc! { "_id": { "account_id": ACCOUNT, "table_name": "rr_target" } },
    +            vec![doc! { "$set": {
    +                "table_status": { "$literal": "CREATING" },
    +                "status_transition_at": { "$literal": bson::Bson::Null },
    +                "restore_source_backup_arn": { "$literal": &older.backup_arn },
    +                "restore_source_table_arn": {
    +                    "$literal": "arn:aws:dynamodb:us-east-1:123456789012:table/Music"
    +                },
    +                "restore_date_time": "$$NOW",
    +                RESTORE_JOB: {
    +                    "kind": { "$literal": "legacy" },
    +                    "status": { "$literal": "RUNNING" },
    +                    "owner_generation": { "$literal": 7_i64 },
    +                    "heartbeat_at": { "$add": [ "$$NOW", 60_000_i64 ] },
    +                },
    +            } }],
    +        )
    +        .await
    +        .expect("stamp a RUNNING record ahead of the clock");
    +    let running = fx
    +        .engine
    +        .list_running_restore_jobs()
    +        .await
    +        .expect("list running");
    +    assert_eq!(running.len(), 1, "{running:?}");
    +    assert_eq!(running[0].target.table_name, "rr_target");
    +    assert_eq!(running[0].target.table_id, table_id);
    +    assert!(
    +        fx.engine
    +            .list_dead_restore_jobs(std::time::Duration::ZERO)
    +            .await
    +            .expect("list dead")
    +            .is_empty(),
    +        "the age-based listing misses a heartbeat ahead of the clock"
    +    );
    +    fx.finish().await;
    +}
    diff --git a/crates/storage-mongodb/src/data_engine_key_repair.rs b/crates/storage-mongodb/src/data_engine_key_repair.rs
    index a7eef75cc..2d2dac083 100644
    --- a/crates/storage-mongodb/src/data_engine_key_repair.rs
    +++ b/crates/storage-mongodb/src/data_engine_key_repair.rs
    @@ -30,7 +30,8 @@
     //! [`spawn_key_repair`] runs the whole pass as a background task, so `serve`
     //! opens its listener without waiting for it (see `build_components` in
     //! `lib.rs`, which spawns it only once `migrations::register_and_check` has
    -//! accepted the catalog). Until the repair reaches a given document,
    +//! accepted the catalog, and only for a server, not for a command that opens
    +//! the catalog beside one). Until the repair reaches a given document,
     //! requests on that document's key answer as they did before this repair
     //! existed. Progress is logged every [`PROGRESS_EVERY`] documents and once
     //! at the end. A failure on one document is logged and skipped; the summary
    diff --git a/crates/storage-mongodb/src/data_engine_live_tests_repair_pass.rs b/crates/storage-mongodb/src/data_engine_live_tests_repair_pass.rs
    index c378233f7..85e91421d 100644
    --- a/crates/storage-mongodb/src/data_engine_live_tests_repair_pass.rs
    +++ b/crates/storage-mongodb/src/data_engine_live_tests_repair_pass.rs
    @@ -11,6 +11,7 @@ use std::sync::Arc;
     
     use bson::{Bson, Document, doc};
     use extenddb_core::types::TableKeyInfo;
    +use extenddb_storage::server_components::ServerComponentsOptions;
     use futures::TryStreamExt;
     
     use super::{
    @@ -314,3 +315,66 @@ async fn live_a_refused_catalog_starts_no_key_repair() {
             .expect("the finished task dropped its engine");
         finish(engine).await;
     }
    +
    +/// A command that opens the catalog beside a server (`backup list`, the
    +/// purge's dry run: `beside_server`) starts no key repair. The factory's
    +/// `build_components` decides through `start_key_repair`, which this test
    +/// calls with the engine the factory would hand it (the factory opens the
    +/// default databases). With a data document stored without its key, the
    +/// call for a command starts no task and the document is still in the shape
    +/// the pre-#436 upsert stored two seconds later, far past a one document
    +/// repair (the refusal test measures one in milliseconds). The same call for
    +/// a server then starts the repair, which rewrites the document.
    +#[tokio::test]
    +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"]
    +async fn live_an_open_beside_a_server_starts_no_key_repair() {
    +    use std::time::{Duration, Instant};
    +
    +    let engine = scratch_engine().await;
    +    let key_info = table(&engine, "pass-beside", false);
    +    let key = item_key(false);
    +    engine
    +        .catalog_db
    +        .collection::("tables")
    +        .insert_one(catalog_row("beside", &key_info.table_id, &key_info))
    +        .await
    +        .expect("catalog row");
    +    store_without_key(&engine, &key_info, &key, Some("upd")).await;
    +    let unrepaired = |document: &Document| document.get_object_id("_id").is_ok();
    +    let engine = Arc::new(engine);
    +
    +    let mut beside = ServerComponentsOptions::default();
    +    beside.beside_server = true;
    +    assert!(
    +        crate::start_key_repair(&engine, &beside).is_none(),
    +        "an open beside a server started a key repair"
    +    );
    +    tokio::time::sleep(Duration::from_secs(2)).await;
    +    assert!(
    +        unrepaired(&stored(&engine, &key_info, &key).await),
    +        "an open beside a server rewrote a data document"
    +    );
    +
    +    let started = Instant::now();
    +    let repair = crate::start_key_repair(&engine, &ServerComponentsOptions::default())
    +        .expect("a server's open starts the repair");
    +    let deadline = started + Duration::from_secs(30);
    +    while unrepaired(&stored(&engine, &key_info, &key).await) {
    +        assert!(
    +            Instant::now() < deadline,
    +            "the repair the server's open started did not rewrite the document"
    +        );
    +        tokio::time::sleep(Duration::from_millis(20)).await;
    +    }
    +    let want = with(key.clone(), "w", s("upd"));
    +    assert_eq!(
    +        stored(&engine, &key_info, &key).await,
    +        expected_document(&want, &key_info, 1),
    +        "the server's repair rebuilt the document"
    +    );
    +    let _ = repair.await;
    +    let engine = Arc::try_unwrap(engine)
    +        .ok()
    +        .expect("the finished task dropped its engine");
    +    finish(engine).await;
    +}
    diff --git a/crates/storage-mongodb/src/lib.rs b/crates/storage-mongodb/src/lib.rs
    index 3580b5fb5..14c735a72 100644
    --- a/crates/storage-mongodb/src/lib.rs
    +++ b/crates/storage-mongodb/src/lib.rs
    @@ -133,9 +133,12 @@ fn server_components_factory(
             // Register as the catalog's server, then read the migration lock and
             // the catalog's status, before the engine writes anything: a server
             // that refuses has deleted its registration and written nothing else.
    -        let registration = migrations::register_and_check(
    +        // Every caller registers, a command too, so a migration does not run
    +        // under one; a command's registration carries its own name.
    +        let registration = migrations::register_and_check_as(
                 &migrations::MongoTarget::new(&connection_string),
                 options.release,
    +            options.command.unwrap_or(migrations::SERVE_COMMAND),
                 migrations::Timing::PRODUCTION,
             )
             .await
    @@ -148,6 +151,7 @@ fn server_components_factory(
                 max_catalog_connections,
                 tx_read_concern,
                 Arc::clone(®istration),
    +            &options,
             )
             .await;
             if built.is_err() {
    @@ -157,6 +161,33 @@ fn server_components_factory(
         })
     }
     
    +/// Start the startup key repair, for a server only; returns its task.
    +///
    +/// An UpdateItem on a build before the fix for issue #436 could create
    +/// an item whose document has no key. The repair rewrites any such
    +/// document, so that reads return the key and PutItem on it succeeds (see
    +/// `data_engine::key_repair`). It runs in the background: the pass costs
    +/// time in proportion to the number of affected documents, and the
    +/// listener must not wait for it. A key the pass has not reached yet
    +/// answers as it did before the repair existed, and each batch is its own
    +/// transaction, so running it while another process serves the same data
    +/// is safe.
    +///
    +/// The repair is a server's: a command that opens the catalog beside a
    +/// server (`options.beside_server`: `extenddb backup list` and the purge's
    +/// dry run), or one that writes beside the live servers with
    +/// `options.startup_recovery` off (`purge --confirm`), starts none, as on
    +/// PostgreSQL, where such a command runs none of the startup recovery; it
    +/// would otherwise start a write pass over the data and exit while the pass
    +/// may be mid-batch.
    +fn start_key_repair(
    +    engine: &Arc,
    +    options: &extenddb_storage::server_components::ServerComponentsOptions,
    +) -> Option> {
    +    (!options.beside_server && options.startup_recovery)
    +        .then(|| data_engine::key_repair::spawn_key_repair(engine.clone()))
    +}
    +
     /// The engine and stores `serve` runs on, built once the server holds its
     /// registration.
     async fn build_components(
    @@ -166,6 +197,7 @@ async fn build_components(
         max_catalog_connections: Option,
         tx_read_concern: mongodb::options::ReadConcern,
         registration: Arc,
    +    options: &extenddb_storage::server_components::ServerComponentsOptions,
     ) -> Result {
         // Create MongoEngine
         let engine = MongoEngine::new(connection_string, region, max_connections, tx_read_concern)
    @@ -177,16 +209,7 @@ async fn build_components(
     
         let engine = Arc::new(engine);
     
    -    // An UpdateItem on a build before the fix for issue #436 could create
    -    // an item whose document has no key. Rewrite any such document, so
    -    // that reads return the key and PutItem on it succeeds (see
    -    // `data_engine::key_repair`). This runs in the background: the pass
    -    // costs time in proportion to the number of affected documents, and
    -    // the listener must not wait for it. A key the pass has not reached
    -    // yet answers as it did before the repair existed, and each batch is
    -    // its own transaction, so running this while another process serves
    -    // the same data is safe.
    -    data_engine::key_repair::spawn_key_repair(engine.clone());
    +    start_key_repair(&engine, options);
     
         // Create one shared catalog/management client. MongoDB Client clones
         // share the underlying pool, so max_catalog_connections limits the
    diff --git a/crates/storage-mongodb/src/migrations/mod.rs b/crates/storage-mongodb/src/migrations/mod.rs
    index dfa01dcee..89f8f18f7 100644
    --- a/crates/storage-mongodb/src/migrations/mod.rs
    +++ b/crates/storage-mongodb/src/migrations/mod.rs
    @@ -62,7 +62,7 @@ pub use locks::{MigrationLock, Timing};
     #[cfg(feature = "test-util")]
     pub use postcheck::{Collection, Definitions, Index, describe_store};
     pub use registrar::ServerRegistration;
    -pub use registrations::ServingRegistration;
    +pub use registrations::{SERVE_COMMAND, ServingRegistration};
     
     /// The catalog database every MongoDB deployment uses.
     pub const CATALOG_DATABASE: &str = "extenddb_catalog";
    @@ -348,7 +348,24 @@ pub async fn register_and_check(
         release: &str,
         timing: Timing,
     ) -> Result {
    -    let registration = ServingRegistration::register(target, release, timing).await?;
    +    register_and_check_as(target, release, SERVE_COMMAND, timing).await
    +}
    +
    +/// [`register_and_check`] for the process `command` names: `extenddb serve`
    +/// ([`SERVE_COMMAND`]) or a command that opens the catalog as a server does
    +/// (`extenddb backup list`), whose registration carries its name
    +/// ([`ServingRegistration::register_as`]).
    +///
    +/// # Errors
    +///
    +/// As [`register_and_check`].
    +pub async fn register_and_check_as(
    +    target: &MongoTarget,
    +    release: &str,
    +    command: &'static str,
    +    timing: Timing,
    +) -> Result {
    +    let registration = ServingRegistration::register_as(target, release, command, timing).await?;
         let status = match catalog_status(target).await {
             Ok(status) => status,
             Err(e) => {
    @@ -372,6 +389,7 @@ pub async fn register_and_check(
         Ok(ServerRegistration::new(
             target,
             release,
    +        command,
             timing,
             status,
             registration,
    diff --git a/crates/storage-mongodb/src/migrations/registrar.rs b/crates/storage-mongodb/src/migrations/registrar.rs
    index 9ff1d5cf5..9cf857329 100644
    --- a/crates/storage-mongodb/src/migrations/registrar.rs
    +++ b/crates/storage-mongodb/src/migrations/registrar.rs
    @@ -25,11 +25,12 @@ use super::registrations::ServingRegistration;
     use super::{MongoTarget, catalog_status};
     
     /// The serving registration of a running server, with what it needs to
    -/// take it again: the catalog, the release it registers as, its timings and
    -/// the status the server started with.
    +/// take it again: the catalog, the release and the process name it
    +/// registers as, its timings and the status the server started with.
     pub struct ServerRegistration {
         target: MongoTarget,
         release: String,
    +    command: &'static str,
         timing: Timing,
         started: CatalogStatus,
         current: Mutex>,
    @@ -39,6 +40,7 @@ impl ServerRegistration {
         pub(super) fn new(
             target: &MongoTarget,
             release: &str,
    +        command: &'static str,
             timing: Timing,
             started: CatalogStatus,
             registration: ServingRegistration,
    @@ -46,6 +48,7 @@ impl ServerRegistration {
             Self {
                 target: target.clone(),
                 release: release.to_owned(),
    +            command,
                 timing,
                 started,
                 current: Mutex::new(Arc::new(registration)),
    @@ -86,12 +89,18 @@ impl ServingRegistrar for ServerRegistration {
         }
     
         async fn register_again(&self) -> Reregistration {
    -        let registration =
    -            match ServingRegistration::register(&self.target, &self.release, self.timing).await {
    -                Ok(registration) => registration,
    -                Err(MigrationError::Locked(why)) => return Reregistration::Refused(why),
    -                Err(e) => return Reregistration::Unavailable(e.to_string()),
    -            };
    +        let registration = match ServingRegistration::register_as(
    +            &self.target,
    +            &self.release,
    +            self.command,
    +            self.timing,
    +        )
    +        .await
    +        {
    +            Ok(registration) => registration,
    +            Err(MigrationError::Locked(why)) => return Reregistration::Refused(why),
    +            Err(e) => return Reregistration::Unavailable(e.to_string()),
    +        };
             let status = match catalog_status(&self.target).await {
                 Ok(status) => status,
                 Err(e) => {
    diff --git a/crates/storage-mongodb/src/migrations/registrations.rs b/crates/storage-mongodb/src/migrations/registrations.rs
    index 8fd1c29e8..1d756a861 100644
    --- a/crates/storage-mongodb/src/migrations/registrations.rs
    +++ b/crates/storage-mongodb/src/migrations/registrations.rs
    @@ -5,6 +5,12 @@
     //! `extenddb_servers` while it serves, and the read of every live one that
     //! `migrate --yes` waits on. The lock it is paired with is in `locks.rs`,
     //! whose module doc describes both.
    +//!
    +//! A command that opens the catalog as a server does (`extenddb backup list`,
    +//! `extenddb backup purge`) registers too, so a migration waits for it, and
    +//! its document's `command` field carries its name, which the migration's
    +//! wait prints. A document without the field (an earlier build's) is a
    +//! server's.
     
     use std::sync::atomic::{AtomicBool, Ordering};
     use std::time::Instant;
    @@ -38,7 +44,8 @@ pub(crate) async fn live_registrations(
             .iter()
             .map(|d| {
                 format!(
    -                "pid {} on {} (release {}, registered {}, expires {})",
    +                "{}, pid {} on {} (release {}, registered {}, expires {})",
    +                d.get_str("command").unwrap_or(SERVE_COMMAND),
                     d.get("pid")
                         .map_or_else(|| "unknown".to_owned(), ToString::to_string),
                     d.get_str("host").unwrap_or("an unknown host"),
    @@ -52,6 +59,10 @@ pub(crate) async fn live_registrations(
             .collect())
     }
     
    +/// The name a server's registration carries; a command's carries its own
    +/// (`extenddb backup list`).
    +pub const SERVE_COMMAND: &str = "extenddb serve";
    +
     /// This host's name, for the registration.
     fn host_name() -> String {
         std::fs::read_to_string("/proc/sys/kernel/hostname")
    @@ -85,6 +96,24 @@ impl ServingRegistration {
             target: &MongoTarget,
             release: &str,
             timing: Timing,
    +    ) -> Result {
    +        Self::register_as(target, release, SERVE_COMMAND, timing).await
    +    }
    +
    +    /// [`ServingRegistration::register`] for the process `command` names
    +    /// ([`SERVE_COMMAND`], or a command such as `extenddb backup list`): the
    +    /// document's `command` field carries the name, and a command's refusal
    +    /// while a migration holds the lock names the holder without the
    +    /// server's advice to start the server.
    +    ///
    +    /// # Errors
    +    ///
    +    /// As [`ServingRegistration::register`].
    +    pub async fn register_as(
    +        target: &MongoTarget,
    +        release: &str,
    +        command: &str,
    +        timing: Timing,
         ) -> Result {
             let client = connect(&target.connection_string).await?;
             let db = super::catalog_handle(&client, &target.catalog);
    @@ -94,6 +123,7 @@ impl ServingRegistration {
                 .update_one(
                     doc! { "_id": &token },
                     vec![doc! { "$set": {
    +                    "command": command,
                         "host": host_name(),
                         "pid": i64::from(std::process::id()),
                         "release": release,
    @@ -135,9 +165,15 @@ impl ServingRegistration {
             };
             if let Some(lock) = lock {
                 registration.release().await;
    +            // A command's refusal ends at the holder; the command's caller
    +            // adds that it is retried once the migration has finished.
    +            let then = if command == SERVE_COMMAND {
    +                ", so the server does not start; start it again once the migration finishes"
    +            } else {
    +                ""
    +            };
                 return Err(MigrationError::Locked(format!(
    -                "{} holds the migration lock on this catalog, so the server does not start; \
    -                 start it again once the migration finishes",
    +                "{} holds the migration lock on this catalog{then}",
                     holder(Some(&lock))
                 )));
             }
    diff --git a/crates/storage-mongodb/tests/migrations_locks.rs b/crates/storage-mongodb/tests/migrations_locks.rs
    index 2978d3921..558128782 100644
    --- a/crates/storage-mongodb/tests/migrations_locks.rs
    +++ b/crates/storage-mongodb/tests/migrations_locks.rs
    @@ -13,7 +13,7 @@ use extenddb_storage::hooks::ServingRegistrar;
     use extenddb_storage::migrations::{CatalogStatus, MigrationError, MigrationId, Runner};
     use extenddb_storage_mongodb::migrations::{
         MigrationLock, MongoMigrations, ServingRegistration, Timing, lock, migrate_holding,
    -    register_and_check,
    +    register_and_check, register_and_check_as,
     };
     use migrations_support::{Deployment, fixture, server};
     
    @@ -239,6 +239,74 @@ async fn migrate_waits_for_a_running_server_and_a_server_refuses_while_a_migrati
         d.drop_databases().await;
     }
     
    +/// A command that opens the catalog as a server does (`extenddb backup
    +/// list`) registers under its own name: the document carries it, a
    +/// migration's wait names it, the name is kept when the registration is
    +/// taken again, and its refusal while a migration holds the lock names the
    +/// holder without the server's advice to start the server.
    +#[tokio::test(flavor = "multi_thread")]
    +async fn a_command_registers_under_its_own_name() {
    +    let Some(uri) = server() else { return };
    +    let d = Deployment::new(&uri, "command").await;
    +    d.create().await;
    +    const COMMAND: &str = "extenddb backup list";
    +
    +    let registered = register_and_check_as(&d.target, "test", COMMAND, Timing::PRODUCTION)
    +        .await
    +        .expect("registered");
    +    let documents = Deployment::documents(&d.catalog(), "extenddb_servers").await;
    +    assert_eq!(documents.len(), 1, "{documents:?}");
    +    assert_eq!(documents[0].get_str("command"), Ok(COMMAND));
    +    let error = lock(&d.target, false, Duration::from_secs(2), Timing::PRODUCTION)
    +        .await
    +        .err()
    +        .expect("the command's registration is waited for");
    +    assert_eq!(error.exit_code(), 6);
    +    let message = error.to_string();
    +    assert!(
    +        message.contains(&format!("{COMMAND}, pid {}", std::process::id())),
    +        "{message}"
    +    );
    +    let again = registered.register_again().await;
    +    assert!(
    +        matches!(again, extenddb_storage::hooks::Reregistration::Registered),
    +        "{again:?}"
    +    );
    +    let documents = Deployment::documents(&d.catalog(), "extenddb_servers").await;
    +    assert!(
    +        documents
    +            .iter()
    +            .any(|d| d.get_str("command") == Ok(COMMAND)),
    +        "taken again under the command's name: {documents:?}"
    +    );
    +    registered.release().await;
    +
    +    let (held, _) = lock(&d.target, false, Duration::from_secs(2), Timing::PRODUCTION)
    +        .await
    +        .expect("no server runs");
    +    let refused = register_and_check_as(&d.target, "test", COMMAND, Timing::PRODUCTION)
    +        .await
    +        .err()
    +        .expect("a migration holds the lock");
    +    assert_eq!(refused.exit_code(), 6);
    +    let why = refused.to_string();
    +    assert!(
    +        why.contains("holds the migration lock on this catalog"),
    +        "{why}"
    +    );
    +    assert!(!why.contains("the server does not start"), "{why}");
    +    let served = register_and_check(&d.target, "test", Timing::PRODUCTION)
    +        .await
    +        .err()
    +        .expect("a server is refused too");
    +    assert!(
    +        served.to_string().contains("the server does not start"),
    +        "{served}"
    +    );
    +    held.release().await;
    +    d.drop_databases().await;
    +}
    +
     #[tokio::test(flavor = "multi_thread")]
     async fn a_server_refuses_a_catalog_it_cannot_serve_and_deletes_its_registration() {
         let Some(uri) = server() else { return };
    diff --git a/crates/storage-postgres/Cargo.toml b/crates/storage-postgres/Cargo.toml
    index 81b205617..b2c47b590 100755
    --- a/crates/storage-postgres/Cargo.toml
    +++ b/crates/storage-postgres/Cargo.toml
    @@ -26,6 +26,11 @@ extenddb-storage = { workspace = true, features = ["test-util"] }
     # The crate's own tests need `test-hooks` and `test-util`; a dev-dependency on
     # itself turns both features on for test builds only.
     extenddb-storage-postgres = { path = ".", features = ["test-hooks", "test-util"] }
    +# The store identity tests run the engine's checks over two engines on one
    +# scratch catalog and two filesystem stores.
    +extenddb-engine = { workspace = true }
    +extenddb-backup-store = { workspace = true }
    +tempfile = "3"
     
     [dependencies]
     anyhow = { workspace = true }
    diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs
    index dcbcc68e4..58ffc1a39 100644
    --- a/crates/storage-postgres/src/backup_catalog.rs
    +++ b/crates/storage-postgres/src/backup_catalog.rs
    @@ -17,6 +17,19 @@
     //! registration is `FOR UPDATE` on the `backup_index` row. Staleness is
     //! measured on the catalog's clock.
     //!
    +//! The maintenance claim is the same mechanism on the `backup_maintenance`
    +//! row: an advisory lock (`backup_maintenance` in the same key space) on a
    +//! dedicated connection is the proof of a live holder, so a second command
    +//! finds it taken and a holder whose session ended has released it; the row
    +//! carries the holder, the generation and the heartbeat.
    +//! `check_maintenance_claim` reports the claim held while a session holds
    +//! the lock (a transaction-scoped shared try-lock in one autocommit
    +//! statement fails while the exclusive lock is held, and holds nothing past
    +//! its statement, so a request dropped mid-check leaves no lock on the
    +//! pool), whatever the row's heartbeat says, and otherwise while the row's
    +//! heartbeat is younger than the stale window, so a dead holder's claim
    +//! expires for the jobs that read it without anyone taking it over. `release` deletes the row and ends the session.
    +//!
     //! A conditional write whose error is of the connection class (an I/O or TLS
     //! error, a pool or worker failure, SQLSTATE class 08, or 57P01 to 57P03)
     //! has an unknown outcome (`WriteOutcome::Unknown`: the statement may have
    @@ -31,8 +44,8 @@ use std::time::Duration;
     use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock,
    +    StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -41,7 +54,9 @@ use sqlx::PgExecutor;
     use tokio::sync::Mutex;
     
     use crate::PostgresEngine;
    -use crate::backup_engine::{ClaimSession, backup_lock_key, generation_column, try_claim};
    +use crate::backup_engine::{
    +    ClaimSession, backup_lock_key, generation_column, try_claim, try_claim_with,
    +};
     
     /// The name of a CREATING restore target registered from `backup_arn`, if
     /// any: the one in-use rule DeleteBackup, a restore's registration and
    @@ -373,6 +388,26 @@ impl BackupCatalog for PostgresEngine {
             })
         }
     
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            let row: Option = sqlx::query_as(&format!(
    +                "SELECT {ROW_COLUMNS} FROM backup_index \
    +                 WHERE store_kind = $1 AND store_root = $2 AND store_id = $3 \
    +                 AND status = 'AVAILABLE' ORDER BY created_at DESC, backup_arn DESC LIMIT 1"
    +            ))
    +            .bind(store.store_kind.as_str())
    +            .bind(&store.store_root)
    +            .bind(&store.store_id)
    +            .fetch_optional(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            row.map(row_from_tuple).transpose()
    +        })
    +    }
    +
         fn list_account_backup_rows<'a>(
             &'a self,
             account_id: &'a str,
    @@ -499,6 +534,57 @@ impl BackupCatalog for PostgresEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let affected = sqlx::query(
    +                "DELETE FROM backup_index WHERE account_id = $1 AND backup_id = $2 \
    +                 AND status = 'CREATING' \
    +                 AND heartbeat_at < now() - make_interval(secs => $3)",
    +            )
    +            .bind(&key.account_id)
    +            .bind(&key.backup_id)
    +            .bind(stale_after.as_secs_f64())
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(affected == 1)
    +        })
    +    }
    +
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result> {
    +        let column = match clock {
    +            StaleClock::Heartbeat => "heartbeat_at",
    +            StaleClock::Completed => "completed_at",
    +        };
    +        rows_stream(async move {
    +            sqlx::query_as(&format!(
    +                "SELECT {ROW_COLUMNS} FROM backup_index \
    +                 WHERE store_kind = $1 AND store_root = $2 AND store_id = $3 AND status = $4 \
    +                 AND {column} < now() - make_interval(secs => $5) \
    +                 ORDER BY created_at, backup_arn"
    +            ))
    +            .bind(store.store_kind.as_str())
    +            .bind(&store.store_root)
    +            .bind(&store.store_id)
    +            .bind(status.as_str())
    +            .bind(age.as_secs_f64())
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -580,48 +666,48 @@ impl BackupCatalog for PostgresEngine {
                     .fetch_all(&self.pool)
                     .await
                     .map_err(db_err)?;
    +            // A row this build cannot read (a kind or status no release
    +            // writes) is logged and skipped, as ListBackups skips a row it
    +            // cannot read, so one such row does not stop the cycle.
                 Ok(rows
                     .into_iter()
    -                .filter_map(
    -                    |(
    -                        table_id,
    -                        account_id,
    -                        table_name,
    -                        source_backup_arn,
    -                        kind,
    -                        status,
    -                        owner_generation,
    -                        heartbeat_at,
    -                    )| {
    -                        let (kind, status) =
    -                            match (RestoreKind::parse(&kind), JobStatus::parse(&status)) {
    -                                (Ok(kind), Ok(status)) => (kind, status),
    -                                (Err(e), _) | (_, Err(e)) => {
    -                                    tracing::warn!(
    -                                        "reconciler: restore job record of {table_name} (account \
    -                                         {account_id}) skipped: {e}"
    -                                    );
    -                                    return None;
    -                                }
    -                            };
    -                        Some(RestoreJobRow {
    -                            target: RestoreTargetKey {
    -                                account_id,
    -                                table_name,
    -                                table_id,
    -                            },
    -                            source_backup_arn,
    -                            kind,
    -                            status,
    -                            owner_generation: u64::try_from(owner_generation).unwrap_or_default(),
    -                            heartbeat_at,
    -                        })
    -                    },
    -                )
    +                .filter_map(|row| {
    +                    let (account_id, table_name) = (row.1.clone(), row.2.clone());
    +                    match restore_job_row(row) {
    +                        Ok(job) => Some(job),
    +                        Err(e) => {
    +                            tracing::warn!(
    +                                "reconciler: restore job record of {table_name} (account \
    +                                 {account_id}) skipped: {e}"
    +                            );
    +                            None
    +                        }
    +                    }
    +                })
                     .collect())
             })
         }
     
    +    fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>> {
    +        Box::pin(async move {
    +            // By status alone: a record stamped in this statement's instant,
    +            // or ahead of the server's clock, is RUNNING too.
    +            #[allow(clippy::type_complexity)]
    +            let rows: Vec<(String, String, String, String, String, String, i64, f64)> =
    +                sqlx::query_as(
    +                    "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \
    +                     r.kind, r.status, r.owner_generation, \
    +                     EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8 \
    +                     FROM table_restores r JOIN tables t ON t.table_id = r.table_id \
    +                     WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name",
    +                )
    +                .fetch_all(&self.pool)
    +                .await
    +                .map_err(db_err)?;
    +            rows.into_iter().map(restore_job_row).collect()
    +        })
    +    }
    +
         /// Conditioned on RUNNING, on the listed `owner_generation` and on the
         /// listed `heartbeat_at` (compared through the same `EXTRACT(EPOCH ...)`
         /// expression the listing read it with, so the comparison is exact), and
    @@ -656,12 +742,239 @@ impl BackupCatalog for PostgresEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            sqlx::query_scalar(
    +                "SELECT DISTINCT store_id FROM backup_index \
    +                 WHERE store_kind = $1 AND store_root = $2 ORDER BY store_id",
    +            )
    +            .bind(store_kind.as_str())
    +            .bind(store_root)
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // The advisory lock is the live-holder proof, as a job's is: a
    +            // second command on any instance finds it taken while the first
    +            // runs, and a holder whose session ended has released it. The
    +            // row carries the generation the heartbeats and the release are
    +            // conditioned on, and the heartbeat age jobs read.
    +            let Some(mut session) = try_claim_with(
    +                &self.pool,
    +                MAINTENANCE_LOCK_KEY,
    +                None,
    +                MAINTENANCE_TAKE_TRIES,
    +                MAINTENANCE_TAKE_PAUSE,
    +            )
    +            .await?
    +            else {
    +                let current: Option =
    +                    sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +                        .fetch_optional(&self.pool)
    +                        .await
    +                        .map_err(db_err)?;
    +                return Err(StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {}",
    +                    current.as_deref().unwrap_or("another maintenance command")
    +                )));
    +            };
    +            // With the lock held, a row whose heartbeat is still young is a
    +            // holder whose lock session ended within the stale window: dead
    +            // by this backend's rule (4(e)), so the row is taken over
    +            // whatever its age. `stale_after` is the other backends' rule and
    +            // the one `check_maintenance_claim` applies on every backend.
    +            let _ = stale_after;
    +            let generation = session.generation();
    +            sqlx::query(
    +                "INSERT INTO backup_maintenance (singleton, holder, owner_generation) \
    +                 VALUES (TRUE, $1, $2) \
    +                 ON CONFLICT (singleton) DO UPDATE SET holder = EXCLUDED.holder, \
    +                 owner_generation = EXCLUDED.owner_generation, claimed_at = now(), \
    +                 heartbeat_at = now()",
    +            )
    +            .bind(holder)
    +            .bind(generation_column(generation))
    +            .execute(session.connection())
    +            .await
    +            .map_err(db_err)?;
    +            Ok(Box::new(PostgresMaintenanceClaim {
    +                session: Mutex::new(Some(session)),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            // A live holder's session holds the exclusive lock, so a shared
    +            // try-lock fails; it succeeds when no session does. The lock is
    +            // taken at transaction scope in one autocommit statement, so it
    +            // ends with the statement and nothing is held on the pooled
    +            // connection when the caller's future is dropped between awaits
    +            // (this runs inside every CreateBackup and RestoreTableFromBackup
    +            // request). Then the row a dead session left is held until its
    +            // heartbeat ages past the window.
    +            let unlocked: bool = sqlx::query_scalar(
    +                "SELECT pg_try_advisory_xact_lock_shared(hashtextextended($1, $2))",
    +            )
    +            .bind(MAINTENANCE_LOCK_KEY)
    +            .bind(crate::backup_engine::RESTORE_LOCK_SEED)
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            if !unlocked {
    +                return Ok(true);
    +            }
    +            sqlx::query_scalar(
    +                "SELECT EXISTS(SELECT 1 FROM backup_maintenance \
    +                 WHERE heartbeat_at >= now() - make_interval(secs => $1))",
    +            )
    +            .bind(stale_after.as_secs_f64())
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +}
    +
    +/// The advisory-lock key of the maintenance claim, in the one lock key space
    +/// (`RESTORE_LOCK_SEED`) the job claims use.
    +const MAINTENANCE_LOCK_KEY: &str = "backup_maintenance";
    +
    +/// A restore job record as the two listings select it.
    +#[allow(clippy::type_complexity)]
    +fn restore_job_row(
    +    (
    +        table_id,
    +        account_id,
    +        table_name,
    +        source_backup_arn,
    +        kind,
    +        status,
    +        owner_generation,
    +        heartbeat_at,
    +    ): (String, String, String, String, String, String, i64, f64),
    +) -> Result {
    +    Ok(RestoreJobRow {
    +        target: RestoreTargetKey {
    +            account_id,
    +            table_name,
    +            table_id,
    +        },
    +        source_backup_arn,
    +        kind: RestoreKind::parse(&kind)?,
    +        status: JobStatus::parse(&status)?,
    +        owner_generation: u64::try_from(owner_generation).unwrap_or_default(),
    +        heartbeat_at,
    +    })
    +}
    +
    +/// How many times `maintenance_claim` tries the exclusive lock (on its one
    +/// claim connection), and the pause between tries. `check_maintenance_claim` holds the key's shared lock for
    +/// one statement, and every CreateBackup and RestoreTableFromBackup runs one,
    +/// so a take that lands inside such a statement fails with no holder to
    +/// name; the tries span about 10 ms, past any such statement, so only a live
    +/// holder's session refuses the take.
    +const MAINTENANCE_TAKE_TRIES: u32 = 6;
    +const MAINTENANCE_TAKE_PAUSE: Duration = Duration::from_millis(2);
    +
    +impl crate::PostgresEngine {
    +    /// The lock step of `maintenance_claim` (the tries on one claim
    +    /// connection) with `tries` and `pause` in place of the take's own, so a
    +    /// test can hold the key's shared lock past the first try and release it
    +    /// while the take still tries. `true` when the lock was taken (it is
    +    /// released again as the session closes).
    +    ///
    +    /// # Errors
    +    ///
    +    /// As `maintenance_claim`'s connection and lock errors.
    +    #[cfg(feature = "test-util")]
    +    pub async fn maintenance_lock_for_tests(
    +        &self,
    +        tries: u32,
    +        pause: Duration,
    +    ) -> Result {
    +        Ok(
    +            try_claim_with(&self.pool, MAINTENANCE_LOCK_KEY, None, tries, pause)
    +                .await?
    +                .is_some(),
    +        )
    +    }
    +}
    +
    +/// The holder's handle on the `backup_maintenance` row: the lock session
    +/// (`None` once released) and the generation its writes are conditioned on.
    +struct PostgresMaintenanceClaim {
    +    session: Mutex>,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for PostgresMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let mut session = self.session.lock().await;
    +            let Some(session) = session.as_mut() else {
    +                return Ok(Heartbeat::Lost);
    +            };
    +            let confirmed = session
    +                .confirm(
    +                    "UPDATE backup_maintenance SET heartbeat_at = now() WHERE owner_generation = $1",
    +                    &[],
    +                )
    +                .await
    +                .map_err(|e| StorageError::Internal(format!("claim session: {e}")))?;
    +            Ok(if confirmed {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    /// Delete the row, release the advisory lock, then end the lock session.
    +    /// The unlock is explicit: `PgConnection::close` returns before the
    +    /// server backend exits and drops the session's locks, so a
    +    /// `check_maintenance_claim` on another connection right after the
    +    /// release would otherwise still find the lock held and answer held. A
    +    /// second release finds no session and does nothing.
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
             Box::pin(async move {
    -            sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM backup_maintenance)")
    -                .fetch_one(&self.pool)
    +            let mut slot = self.session.lock().await;
    +            let Some(mut session) = slot.take() else {
    +                return Ok(());
    +            };
    +            sqlx::query("DELETE FROM backup_maintenance WHERE owner_generation = $1")
    +                .bind(generation_column(self.generation))
    +                .execute(session.connection())
                     .await
    -                .map_err(db_err)
    +                .map_err(db_err)?;
    +            sqlx::query("SELECT pg_advisory_unlock(hashtextextended($1, $2))")
    +                .bind(MAINTENANCE_LOCK_KEY)
    +                .bind(crate::backup_engine::RESTORE_LOCK_SEED)
    +                .execute(session.connection())
    +                .await
    +                .map_err(db_err)?;
    +            session.close().await;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs
    index aaf35ce03..f4a5625ed 100755
    --- a/crates/storage-postgres/src/backup_engine.rs
    +++ b/crates/storage-postgres/src/backup_engine.rs
    @@ -112,6 +112,14 @@ impl ClaimSession {
             &mut self.conn
         }
     
    +    /// End the session, releasing its advisory lock; a close that fails is
    +    /// logged (the server reaps the connection either way).
    +    pub(crate) async fn close(self) {
    +        if let Err(e) = ::close(self.conn).await {
    +            tracing::warn!("closing a claim session: {e}");
    +        }
    +    }
    +
         /// Run `statement`, bound with the record's key and this generation, on
         /// the claim connection and report whether it refreshed exactly one row.
         pub(crate) async fn confirm(
    @@ -192,6 +200,18 @@ pub(crate) async fn try_claim(
         pool: &sqlx::PgPool,
         key: &str,
         permit: Option,
    +) -> Result, StorageError> {
    +    try_claim_with(pool, key, permit, 1, Duration::ZERO).await
    +}
    +
    +/// [`try_claim`] trying the lock up to `tries` times on the one connection,
    +/// `pause` apart, before answering `Ok(None)`.
    +pub(crate) async fn try_claim_with(
    +    pool: &sqlx::PgPool,
    +    key: &str,
    +    permit: Option,
    +    tries: u32,
    +    pause: Duration,
     ) -> Result, StorageError> {
         let options = pool.connect_options();
         let mut conn = ::connect_with(&options)
    @@ -214,12 +234,19 @@ pub(crate) async fn try_claim(
                  the server setting: {e}"
             );
         }
    -    let taken: bool = sqlx::query_scalar("SELECT pg_try_advisory_lock(hashtextextended($1, $2))")
    -        .bind(key)
    -        .bind(RESTORE_LOCK_SEED)
    -        .fetch_one(&mut conn)
    -        .await
    -        .map_err(|e| StorageError::Internal(format!("claim lock: {e}")))?;
    +    let mut taken = false;
    +    for attempt in 1..=tries.max(1) {
    +        taken = sqlx::query_scalar("SELECT pg_try_advisory_lock(hashtextextended($1, $2))")
    +            .bind(key)
    +            .bind(RESTORE_LOCK_SEED)
    +            .fetch_one(&mut conn)
    +            .await
    +            .map_err(|e| StorageError::Internal(format!("claim lock: {e}")))?;
    +        if taken || attempt == tries {
    +            break;
    +        }
    +        tokio::time::sleep(pause).await;
    +    }
         Ok(taken.then(|| ClaimSession {
             conn,
             generation: new_generation(),
    diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs
    index 7d0f9d28c..651c1072e 100755
    --- a/crates/storage-postgres/src/lib.rs
    +++ b/crates/storage-postgres/src/lib.rs
    @@ -824,9 +824,14 @@ fn server_components_factory(
         );
         let region = region.to_string();
         Box::pin(async move {
    -        let registration = migrations::ServerRegistration::register(&connection_string)
    -            .await
    -            .map_err(BackendError::Catalog)?;
    +        // Every caller registers, a command too: a migration must not run
    +        // under one. A command's session carries its own name.
    +        let registration = migrations::ServerRegistration::register_as(
    +            &connection_string,
    +            options.command.unwrap_or(migrations::SERVE_APPLICATION),
    +        )
    +        .await
    +        .map_err(BackendError::Catalog)?;
     
             // Build PostgresConfig from extracted values
             let pg_config = PostgresConfig {
    @@ -844,23 +849,37 @@ fn server_components_factory(
                 })?;
             engine.serving = Some(Arc::new(registration));
     
    -        // Rebuild any vector index a crash left CREATING, before serving: an index
    -        // in that state is not searchable, and nothing else will repair it.
    -        match crate::data::vector_index::reconcile_incomplete_vector_indexes(&engine).await {
    -            Ok(n) if n > 0 => {
    -                tracing::info!("Reconciled {n} incomplete vector index(es) at startup");
    +        // The startup recovery is a server's; a command reading beside one,
    +        // or one that writes beside the live servers (`backup purge`,
    +        // `options.startup_recovery` off), leaves it to the server.
    +        let recover = !options.beside_server && options.startup_recovery;
    +        if recover {
    +            // Rebuild any vector index a crash left CREATING, before serving:
    +            // an index in that state is not searchable, and nothing else will
    +            // repair it.
    +            match crate::data::vector_index::reconcile_incomplete_vector_indexes(&engine).await {
    +                Ok(n) if n > 0 => {
    +                    tracing::info!("Reconciled {n} incomplete vector index(es) at startup");
    +                }
    +                Ok(_) => {}
    +                Err(e) => tracing::error!("Failed to reconcile incomplete vector indexes: {e}"),
                 }
    -            Ok(_) => {}
    -            Err(e) => tracing::error!("Failed to reconcile incomplete vector indexes: {e}"),
    -        }
     
    -        // Recover control plane transitions (errors are logged, not fatal).
    -        // A drop deferred by a session still holding the table wakes the
    -        // poller, which `spawn_workers` starts later.
    -        workers::recover_control_plane_at_startup(&engine).await;
    +            // Recover control plane transitions (errors are logged, not fatal).
    +            // A drop deferred by a session still holding the table wakes the
    +            // poller, which `spawn_workers` starts later.
    +            workers::recover_control_plane_at_startup(&engine).await;
    +        }
     
    -        // Start GSI workers
    -        let engine = engine.start_gsi_workers();
    +        // Start GSI workers: a server's. A command (`backup list`, the
    +        // purge's dry run, `purge --confirm`) leaves the queue to the
    +        // server that owns the catalog, so the command applies no pending
    +        // index write and exits with none half-applied.
    +        let engine = if recover {
    +            engine.start_gsi_workers()
    +        } else {
    +            engine
    +        };
     
             // Get data database name for logging (before wrapping in Arc)
             let data_db_name = engine
    diff --git a/crates/storage-postgres/src/migrations/locks.rs b/crates/storage-postgres/src/migrations/locks.rs
    index 3d0e1b6b9..d5ac4e2fc 100644
    --- a/crates/storage-postgres/src/migrations/locks.rs
    +++ b/crates/storage-postgres/src/migrations/locks.rs
    @@ -54,6 +54,12 @@ pub(crate) const SERVING_KEY: i32 = 2;
     /// How often a holder runs `SELECT 1` on its lock connection.
     pub const KEEPALIVE_INTERVAL: Duration = Duration::from_secs(15);
     
    +/// The `application_name` of a server's serving session. A command that
    +/// opens the catalog as a server does (`extenddb backup list`) names its
    +/// own session after itself, so a migration waiting for the registrations
    +/// names the command rather than a server.
    +pub const SERVE_APPLICATION: &str = "extenddb serve";
    +
     /// Server-side keepalives of the serving session: the first probe after 10 s
     /// of silence, then one every 5 s, and the session ends after 4 unanswered.
     const TCP_KEEPALIVES: [(&str, &str); 3] = [
    @@ -413,16 +419,16 @@ pub struct ServingRegistration {
     }
     
     impl ServingRegistration {
    -    /// Register, with the session named `extenddb serve`. The key is taken
    -    /// with a try: a migration that holds or waits for it is never waited
    -    /// for.
    +    /// Register, with the session named `extenddb serve`
    +    /// ([`SERVE_APPLICATION`]). The key is taken with a try: a migration
    +    /// that holds or waits for it is never waited for.
         ///
         /// # Errors
         ///
         /// [`MigrationError::Locked`] (exit 6) when a migration is running, so a
         /// restart loop retries.
         pub async fn register(catalog: &PgConnectOptions) -> Result {
    -        Self::register_keepalive(catalog, KEEPALIVE_INTERVAL).await
    +        Self::register_keepalive(catalog, KEEPALIVE_INTERVAL, SERVE_APPLICATION).await
         }
     
         /// [`ServingRegistration::register`] with another keepalive interval, for
    @@ -436,15 +442,18 @@ impl ServingRegistration {
             catalog: &PgConnectOptions,
             keepalive: Duration,
         ) -> Result {
    -        Self::register_keepalive(catalog, keepalive).await
    +        Self::register_keepalive(catalog, keepalive, SERVE_APPLICATION).await
         }
     
    +    /// Register with the session named `application`: `extenddb serve` for
    +    /// a server, the command's name for a command.
         pub(crate) async fn register_keepalive(
             catalog: &PgConnectOptions,
             keepalive: Duration,
    +        application: &str,
         ) -> Result {
    -        let mut conn = connect(catalog, "extenddb serve").await?;
    -        match register_on(&mut conn).await {
    +        let mut conn = connect(catalog, application).await?;
    +        match register_on(&mut conn, application).await {
                 Ok(()) => {}
                 Err(e) => {
                     let _ = statement(&mut conn, "ROLLBACK").await;
    @@ -514,7 +523,15 @@ impl Drop for ServingRegistration {
         }
     }
     
    -async fn register_on(conn: &mut PgConnection) -> Result<(), MigrationError> {
    +async fn register_on(conn: &mut PgConnection, application: &str) -> Result<(), MigrationError> {
    +    // What the refusal tells a server's operator to do once the migration is
    +    // done; a command's refusal ends at the holders, and the command's
    +    // caller adds that it is retried once the migration has finished.
    +    let then = if application == SERVE_APPLICATION {
    +        "; start the server when it has finished"
    +    } else {
    +        ""
    +    };
         let fail = |e: sqlx::Error| MigrationError::store("register the server", e);
         statement(conn, "BEGIN").await.map_err(fail)?;
         // Inside the transaction that holds the key, so a transaction-mode pooler
    @@ -532,15 +549,14 @@ async fn register_on(conn: &mut PgConnection) -> Result<(), MigrationError> {
         {
             let names = holders(conn, SERVING_KEY).await.unwrap_or_default();
             return Err(MigrationError::Locked(format!(
    -            "a migration is running on this catalog ({}); start the server when it has finished",
    +            "a migration is running on this catalog ({}){then}",
                 listed(&names)
             )));
         }
         let migrators = holders(conn, MIGRATION_KEY).await.map_err(fail)?;
         if !migrators.is_empty() {
             return Err(MigrationError::Locked(format!(
    -            "a migration holds the migration lock on this catalog ({}); start the server when \
    -             it has finished",
    +            "a migration holds the migration lock on this catalog ({}){then}",
                 listed(&migrators)
             )));
         }
    diff --git a/crates/storage-postgres/src/migrations/mod.rs b/crates/storage-postgres/src/migrations/mod.rs
    index 1cf595927..1aa4c2ea5 100644
    --- a/crates/storage-postgres/src/migrations/mod.rs
    +++ b/crates/storage-postgres/src/migrations/mod.rs
    @@ -55,7 +55,7 @@ use sqlx::postgres::PgConnectOptions;
     
     pub use backend::{PostgresMigrations, PostgresStores};
     pub use commands::{PostgresCatalogMigrations, resolve};
    -pub use locks::{KEEPALIVE_INTERVAL, MigrationLock, ServingRegistration};
    +pub use locks::{KEEPALIVE_INTERVAL, MigrationLock, SERVE_APPLICATION, ServingRegistration};
     #[cfg(feature = "test-util")]
     pub use postcheck::{Column, Definitions, Index, compare, describe};
     pub use registrar::ServerRegistration;
    diff --git a/crates/storage-postgres/src/migrations/registrar.rs b/crates/storage-postgres/src/migrations/registrar.rs
    index 5bfbaeb52..f45e18fa0 100644
    --- a/crates/storage-postgres/src/migrations/registrar.rs
    +++ b/crates/storage-postgres/src/migrations/registrar.rs
    @@ -23,16 +23,17 @@ use async_trait::async_trait;
     use extenddb_storage::hooks::{Reregistration, ServingRegistrar};
     use extenddb_storage::migrations::{CatalogStatus, MigrationError, Target};
     
    -use super::locks::{KEEPALIVE_INTERVAL, ServingRegistration};
    +use super::locks::{KEEPALIVE_INTERVAL, SERVE_APPLICATION, ServingRegistration};
     use super::{PostgresTarget, catalog_status, require_data_connection_row, resolve};
     
     /// The serving registration of a running server, with what it needs to
    -/// take it again: the catalog's databases and the status the server started
    -/// with.
    +/// take it again: the catalog's databases, the status the server started
    +/// with, and the name its session carries.
     pub struct ServerRegistration {
         target: PostgresTarget,
         started: CatalogStatus,
         keepalive: Duration,
    +    application: &'static str,
         current: Mutex,
     }
     
    @@ -48,7 +49,24 @@ impl ServerRegistration {
         /// [`MigrationError::Locked`] (exit 6) while a migration runs, or the
         /// error of a database that cannot be reached or read.
         pub async fn register(connection_string: &str) -> Result {
    -        Self::register_keepalive(connection_string, KEEPALIVE_INTERVAL).await
    +        Self::register_as(connection_string, SERVE_APPLICATION).await
    +    }
    +
    +    /// [`ServerRegistration::register`] for the process `application` names
    +    /// (`extenddb serve`, or a command such as `extenddb backup list`): its
    +    /// session carries the name as `application_name`, which a migration
    +    /// waiting for the registrations prints, and a refusal while a
    +    /// migration runs names the migration's sessions without the server's
    +    /// advice to start the server.
    +    ///
    +    /// # Errors
    +    ///
    +    /// As [`ServerRegistration::register`].
    +    pub async fn register_as(
    +        connection_string: &str,
    +        application: &'static str,
    +    ) -> Result {
    +        Self::register_keepalive(connection_string, KEEPALIVE_INTERVAL, application).await
         }
     
         /// [`ServerRegistration::register`] with another keepalive interval, for
    @@ -62,18 +80,35 @@ impl ServerRegistration {
             connection_string: &str,
             keepalive: Duration,
         ) -> Result {
    -        Self::register_keepalive(connection_string, keepalive).await
    +        Self::register_keepalive(connection_string, keepalive, SERVE_APPLICATION).await
    +    }
    +
    +    /// [`ServerRegistration::register_as`] with another keepalive interval,
    +    /// for tests.
    +    ///
    +    /// # Errors
    +    ///
    +    /// As [`ServerRegistration::register`].
    +    #[cfg(feature = "test-util")]
    +    pub async fn register_as_with(
    +        connection_string: &str,
    +        application: &'static str,
    +        keepalive: Duration,
    +    ) -> Result {
    +        Self::register_keepalive(connection_string, keepalive, application).await
         }
     
         async fn register_keepalive(
             connection_string: &str,
             keepalive: Duration,
    +        application: &'static str,
         ) -> Result {
             let Some(target) = resolve(&Target::at(connection_string)).await? else {
                 return Err(MigrationError::Status(CatalogStatus::Uninitialized));
             };
             let registration =
    -            ServingRegistration::register_keepalive(&target.catalog, keepalive).await?;
    +            ServingRegistration::register_keepalive(&target.catalog, keepalive, application)
    +                .await?;
             let status = match catalog_status(&target).await {
                 Ok(status) => status,
                 Err(e) => {
    @@ -97,6 +132,7 @@ impl ServerRegistration {
                         target,
                         started: status,
                         keepalive,
    +                    application,
                         current: Mutex::new(registration),
                     })
                 }
    @@ -130,14 +166,17 @@ impl ServingRegistrar for ServerRegistration {
         }
     
         async fn register_again(&self) -> Reregistration {
    -        let registration =
    -            match ServingRegistration::register_keepalive(&self.target.catalog, self.keepalive)
    -                .await
    -            {
    -                Ok(registration) => registration,
    -                Err(MigrationError::Locked(why)) => return Reregistration::Refused(why),
    -                Err(e) => return Reregistration::Unavailable(e.to_string()),
    -            };
    +        let registration = match ServingRegistration::register_keepalive(
    +            &self.target.catalog,
    +            self.keepalive,
    +            self.application,
    +        )
    +        .await
    +        {
    +            Ok(registration) => registration,
    +            Err(MigrationError::Locked(why)) => return Reregistration::Refused(why),
    +            Err(e) => return Reregistration::Unavailable(e.to_string()),
    +        };
             let status = match catalog_status(&self.target).await {
                 Ok(status) => status,
                 Err(e) => {
    diff --git a/crates/storage-postgres/tests/backup_restore.rs b/crates/storage-postgres/tests/backup_restore.rs
    index 0493c34cd..2966ded7a 100644
    --- a/crates/storage-postgres/tests/backup_restore.rs
    +++ b/crates/storage-postgres/tests/backup_restore.rs
    @@ -57,6 +57,8 @@ struct Scratch {
         db: PgPool,
         catalog: PgPool,
         admin: PgPool,
    +    /// The catalog's URL, for a second engine on the same catalog.
    +    catalog_url: String,
         db_names: [String; 2],
         _guard: DbGuard,
         _serial: tokio::sync::MutexGuard<'static, ()>,
    @@ -216,7 +218,7 @@ async fn scratch_with_pool(pool_size: u32) -> Scratch {
         }
         let engine = PostgresEngine::new(
             &PostgresConfig {
    -            connection_string: catalog_url,
    +            connection_string: catalog_url.clone(),
                 pool_size,
                 max_item_size_bytes: 400_000,
             },
    @@ -229,6 +231,7 @@ async fn scratch_with_pool(pool_size: u32) -> Scratch {
             db,
             catalog,
             admin,
    +        catalog_url,
             db_names: [catalog_name, data_name],
             _guard: guard,
             _serial: serial,
    @@ -718,6 +721,14 @@ async fn indexed_source_with(s: &Scratch, name: &str, items: usize) -> String {
             .create_table(ACCOUNT, input)
             .await
             .expect("create the source table");
    +    put_indexed_items(s, name, &desc, items).await;
    +    desc.table_id
    +}
    +
    +/// Write `items` items through PutItem into the table `indexed_source`
    +/// created (`desc` its description): every item has `pk` and `sk`, every
    +/// second one `g` (the GSI's key), every third one `l` (the LSI's).
    +async fn put_indexed_items(s: &Scratch, name: &str, desc: &TableDescription, items: usize) {
         let key_info = TableKeyInfo {
             table_name: name.to_owned(),
             account_id: ACCOUNT.to_owned(),
    @@ -749,7 +760,6 @@ async fn indexed_source_with(s: &Scratch, name: &str, items: usize) -> String {
                 .await
                 .expect("put");
         }
    -    desc.table_id
     }
     
     /// `(index_name, index_id)` of a table's secondary indexes, by name.
    @@ -2722,12 +2732,22 @@ async fn a_definition_read_that_loses_its_connection_is_transient() {
     
     // --- the catalog index -----------------------------------------------------------
     
    +/// A catalog row for a backup of `table` in `account` minted now, with the
    +/// manifest fixture's source table, in the `catalog_index` module's store.
    +fn catalog_index_row(
    +    account: &str,
    +    table: &str,
    +    name: &str,
    +) -> extenddb_storage::backup_catalog::BackupRow {
    +    catalog_index::row(account, table, name)
    +}
    +
     mod catalog_index {
         use extenddb_core::backup_format::BackupArn;
         use extenddb_storage::BackupCatalog;
         use extenddb_storage::backup_catalog::{
    -        BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StatusFields,
    -        StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
    +        BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StaleClock,
    +        StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
         };
         use futures::TryStreamExt;
     
    @@ -2743,7 +2763,7 @@ mod catalog_index {
     
         /// A row for a backup of `table` in `account` minted now, with the
         /// manifest fixture's source table.
    -    fn row(account: &str, table: &str, name: &str) -> BackupRow {
    +    pub(super) fn row(account: &str, table: &str, name: &str) -> BackupRow {
             let backup_id = extenddb_storage::util::new_backup_id();
             let backup_arn = extenddb_storage::util::backup_arn(REGION, account, table, &backup_id);
             let manifest: serde_json::Value = serde_json::from_str(include_str!(
    @@ -2790,7 +2810,12 @@ mod catalog_index {
                 uuid::Uuid::parse_str(&s.engine.catalog_id().await.expect("id")).is_ok(),
                 "the catalog has an id from the migration"
             );
    -        assert!(!s.engine.check_maintenance_claim().await.expect("check"));
    +        assert!(
    +            !s.engine
    +                .check_maintenance_claim(std::time::Duration::from_secs(300))
    +                .await
    +                .expect("check")
    +        );
             let row = row(ACCOUNT, "Music", "nightly");
             assert_eq!(backup_lock_holder(&s, &row.backup_id).await, None);
             let claim = s.engine.claim_backup(&row).await.expect("claim");
    @@ -3855,6 +3880,1195 @@ mod catalog_index {
             assert!(record.1.is_some());
             s.cleanup().await;
         }
    +
    +    /// `read_store_ids`: the distinct ids of the rows for a root, of every
    +    /// status, scoped to the kind and the root.
    +    #[tokio::test]
    +    async fn read_store_ids_lists_the_distinct_ids_of_a_root() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP read_store_ids_lists_the_distinct_ids_of_a_root: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let root = &store().store_root;
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids")
    +                .is_empty()
    +        );
    +        let first = row(ACCOUNT, "Music", "a");
    +        let _c1 = s.engine.claim_backup(&first).await.expect("claim");
    +        let second = row(ACCOUNT, "Music", "b");
    +        let c2 = s.engine.claim_backup(&second).await.expect("claim");
    +        c2.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields::default(),
    +        )
    +        .await
    +        .expect("fail");
    +        assert_eq!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids"),
    +            ["store-1"],
    +            "two rows of one store, of two statuses, are one id"
    +        );
    +        let mut other = row(ACCOUNT, "Songs", "c");
    +        other.store.store_id = "store-2".to_owned();
    +        let _c3 = s.engine.claim_backup(&other).await.expect("claim");
    +        assert_eq!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, root)
    +                .await
    +                .expect("ids"),
    +            ["store-1", "store-2"]
    +        );
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::S3, root)
    +                .await
    +                .expect("ids")
    +                .is_empty(),
    +            "another kind at the same root is another store"
    +        );
    +        assert!(
    +            s.engine
    +                .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +                .await
    +                .expect("ids")
    +                .is_empty()
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    /// The maintenance claim: taken once, refused to a second holder naming
    +    /// the first, heartbeated, released; a holder whose session ended and
    +    /// whose heartbeat is older than the stale window is dead, so its claim
    +    /// is not reported held and is taken over, and its handle finds the claim
    +    /// lost.
    +    /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat is
    +    /// older than the window is removed outright; one with a live owner, one
    +    /// that is not CREATING, and one that is gone are left, answering
    +    /// `false`.
    +    #[tokio::test]
    +    async fn a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        let age = |backup_id: String| {
    +            let catalog = s.catalog.clone();
    +            async move {
    +                sqlx::query(
    +                    "UPDATE backup_index SET heartbeat_at = now() - interval '10 minutes' \
    +                     WHERE backup_id = $1",
    +                )
    +                .bind(backup_id)
    +                .execute(&catalog)
    +                .await
    +                .expect("age");
    +            }
    +        };
    +        let creating = row(ACCOUNT, "Music", "a");
    +        let key = creating.key();
    +        let claim = s.engine.claim_backup(&creating).await.expect("claim");
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap(),
    +            "a live owner keeps its row"
    +        );
    +        age(creating.backup_id.clone()).await;
    +        assert!(
    +            s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap()
    +        );
    +        assert!(
    +            s.engine
    +                .read_backup_row(&arn_of(&creating))
    +                .await
    +                .unwrap()
    +                .is_none()
    +        );
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&key, stale)
    +                .await
    +                .unwrap(),
    +            "gone"
    +        );
    +        drop(claim);
    +
    +        let available = row(ACCOUNT, "Music", "b");
    +        let claim = s.engine.claim_backup(&available).await.expect("claim");
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("available");
    +        age(available.backup_id.clone()).await;
    +        assert!(
    +            !s.engine
    +                .remove_dead_creating_backup_row(&available.key(), stale)
    +                .await
    +                .unwrap(),
    +            "only a CREATING row"
    +        );
    +        assert_eq!(
    +            s.engine
    +                .read_backup_row(&arn_of(&available))
    +                .await
    +                .unwrap()
    +                .unwrap()
    +                .status,
    +            BackupStatus::Available
    +        );
    +        drop(claim);
    +        s.cleanup().await;
    +    }
    +
    +    /// `list_backup_rows_older_than` compares the named column with the
    +    /// database's `now()`: a FAILED row whose heartbeat is older than the age
    +    /// but whose completion is not is listed on the heartbeat clock and not
    +    /// on the completed clock (the two columns are told apart by waiting
    +    /// between the last heartbeat and the FAILED write, on the database's
    +    /// clock); a row crosses the window by waiting; a row of another store,
    +    /// another status, or with `completed_at` unset on the completed clock
    +    /// is not listed.
    +    #[tokio::test]
    +    async fn rows_older_than_are_listed_on_the_named_clock() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP rows_older_than_are_listed_on_the_named_clock: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let age = std::time::Duration::from_secs(1);
    +        let engine = &s.engine;
    +        let list = move |status: BackupStatus, store: StoreIdentity, clock: StaleClock| async move {
    +            let rows: Vec = engine
    +                .list_backup_rows_older_than(status, &store, clock, age)
    +                .try_collect()
    +                .await
    +                .expect("list");
    +            rows.into_iter().map(|r| r.backup_name).collect::>()
    +        };
    +        // `aged`: heartbeat now, then FAILED after the age has passed, so its
    +        // heartbeat is older than the age and its completion is not.
    +        let aged = row(ACCOUNT, "Music", "aged");
    +        let aged_claim = s.engine.claim_backup(&aged).await.expect("claim");
    +        // `fresh`: FAILED at once; both columns within the age.
    +        let fresh = row(ACCOUNT, "Music", "fresh");
    +        let fresh_claim = s.engine.claim_backup(&fresh).await.expect("claim");
    +        let failed = StatusFields {
    +            failure_reason: Some("the test's".to_owned()),
    +            completed: true,
    +            ..StatusFields::default()
    +        };
    +        assert_eq!(
    +            fresh_claim
    +                .set_status(JobStatus::Creating, JobStatus::Failed, &failed)
    +                .await
    +                .unwrap(),
    +            WriteOutcome::Matched
    +        );
    +        // `open`: FAILED without a completion stamp (`completed: false`).
    +        let open = row(ACCOUNT, "Music", "open");
    +        let open_claim = s.engine.claim_backup(&open).await.expect("claim");
    +        assert_eq!(
    +            open_claim
    +                .set_status(
    +                    JobStatus::Creating,
    +                    JobStatus::Failed,
    +                    &StatusFields {
    +                        failure_reason: Some("the test's".to_owned()),
    +                        ..StatusFields::default()
    +                    }
    +                )
    +                .await
    +                .unwrap(),
    +            WriteOutcome::Matched
    +        );
    +        // `creating`: stays CREATING with its heartbeat ageing.
    +        let creating = row(ACCOUNT, "Music", "creating");
    +        let _creating_claim = s.engine.claim_backup(&creating).await.expect("claim");
    +        // `elsewhere`: FAILED at once, in another store.
    +        let mut elsewhere = row(ACCOUNT, "Music", "elsewhere");
    +        elsewhere.store.store_id = "store-2".to_owned();
    +        let elsewhere_claim = s.engine.claim_backup(&elsewhere).await.expect("claim");
    +        assert_eq!(
    +            elsewhere_claim
    +                .set_status(JobStatus::Creating, JobStatus::Failed, &failed)
    +                .await
    +                .unwrap(),
    +            WriteOutcome::Matched
    +        );
    +
    +        // Nothing is older than the age yet, on either clock.
    +        assert_eq!(
    +            list(BackupStatus::Failed, store(), StaleClock::Heartbeat).await,
    +            Vec::::new()
    +        );
    +        assert_eq!(
    +            list(BackupStatus::Failed, store(), StaleClock::Completed).await,
    +            Vec::::new()
    +        );
    +
    +        tokio::time::sleep(age + std::time::Duration::from_millis(300)).await;
    +        assert_eq!(
    +            aged_claim
    +                .set_status(JobStatus::Creating, JobStatus::Failed, &failed)
    +                .await
    +                .unwrap(),
    +            WriteOutcome::Matched
    +        );
    +        // Heartbeat clock: every FAILED row of the store heartbeated before
    +        // the wait (`open` has no completion but has a heartbeat); not the
    +        // CREATING row, not the other store's.
    +        assert_eq!(
    +            list(BackupStatus::Failed, store(), StaleClock::Heartbeat).await,
    +            ["aged", "fresh", "open"]
    +        );
    +        // Completed clock: `aged` completed just now, `open` never.
    +        assert_eq!(
    +            list(BackupStatus::Failed, store(), StaleClock::Completed).await,
    +            ["fresh"]
    +        );
    +        assert_eq!(
    +            list(BackupStatus::Creating, store(), StaleClock::Heartbeat).await,
    +            ["creating"]
    +        );
    +        let mut other = store();
    +        other.store_id = "store-2".to_owned();
    +        assert_eq!(
    +            list(BackupStatus::Failed, other, StaleClock::Heartbeat).await,
    +            ["elsewhere"]
    +        );
    +        // `aged` crosses the completed window by waiting.
    +        tokio::time::sleep(age + std::time::Duration::from_millis(300)).await;
    +        assert_eq!(
    +            list(BackupStatus::Failed, store(), StaleClock::Completed).await,
    +            ["aged", "fresh"]
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    #[tokio::test]
    +    async fn the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        let check = || s.engine.check_maintenance_claim(stale);
    +        assert!(!check().await.expect("check"));
    +        let first = s
    +            .engine
    +            .maintenance_claim("purge on host-a", stale)
    +            .await
    +            .expect("claim");
    +        assert_ne!(first.generation(), UNOWNED_GENERATION);
    +        assert!(check().await.expect("check"), "a live claim is held");
    +        match s.engine.maintenance_claim("purge on host-b", stale).await {
    +            Err(StorageError::AlreadyExists(m)) => {
    +                assert!(m.contains("purge on host-a"), "{m}");
    +            }
    +            Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +            Ok(_) => panic!("a second holder is refused"),
    +        }
    +        assert_eq!(
    +            first.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Confirmed
    +        );
    +        first.release().await.expect("release");
    +        assert!(!check().await.expect("check"), "released");
    +        first.release().await.expect("a second release is a no-op");
    +
    +        // A dead holder: its session ended (the handle dropped) and its
    +        // heartbeat aged past the window.
    +        let dead = s
    +            .engine
    +            .maintenance_claim("purge on host-c", stale)
    +            .await
    +            .expect("claim");
    +        let dead_generation = dead.generation();
    +        drop(dead);
    +        // The drop closes the holder's session; the backend reaps it a
    +        // moment later. Wait for the exclusive lock to be gone, so the check
    +        // below is answered by the heartbeat arm alone and not by a lock
    +        // still being released.
    +        let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
    +        loop {
    +            let exclusive: i64 = sqlx::query_scalar(
    +                "SELECT count(*) FROM pg_locks WHERE locktype = 'advisory' AND mode = \
    +                 'ExclusiveLock' AND database = (SELECT oid FROM pg_database WHERE datname = \
    +                 current_database())",
    +            )
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("pg_locks");
    +            if exclusive == 0 {
    +                break;
    +            }
    +            assert!(
    +                std::time::Instant::now() < deadline,
    +                "the dropped holder's session still holds the lock"
    +            );
    +            tokio::time::sleep(std::time::Duration::from_millis(20)).await;
    +        }
    +        // The heartbeat arm: the lock is free (the session ended with the
    +        // handle) and the row's heartbeat is young, so the jobs still see
    +        // the claim held, as they must for a command that died a moment ago
    +        // or lost its claim while its row is young.
    +        assert!(
    +            check().await.expect("check"),
    +            "a dead session's row is held by its heartbeat until it ages"
    +        );
    +        sqlx::query("UPDATE backup_maintenance SET heartbeat_at = now() - interval '10 minutes'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("age");
    +        assert!(!check().await.expect("check"), "a stale claim is not held");
    +        let taker = s
    +            .engine
    +            .maintenance_claim("purge on host-d", stale)
    +            .await
    +            .expect("a dead holder's claim is taken over");
    +        assert_ne!(taker.generation(), dead_generation);
    +        assert!(check().await.expect("check"));
    +        let holder: String = sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("holder");
    +        assert_eq!(holder, "purge on host-d");
    +        // A holder whose session is still open is alive whatever its
    +        // heartbeat says: the lock is the proof of life on this backend.
    +        sqlx::query("UPDATE backup_maintenance SET heartbeat_at = now() - interval '10 minutes'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("age");
    +        assert!(
    +            matches!(
    +                s.engine.maintenance_claim("purge on host-e", stale).await,
    +                Err(StorageError::AlreadyExists(_))
    +            ),
    +            "a live session keeps the claim"
    +        );
    +        assert!(
    +            check().await.expect("check"),
    +            "the jobs' check sees a live session's claim held through the lock, whatever the \
    +             row's heartbeat says"
    +        );
    +        // The check holds no lock past its statement: after it, the only
    +        // advisory lock in the catalog is the holder's exclusive one, so no
    +        // pooled connection carries a shared lock a dropped request left.
    +        let modes: Vec = sqlx::query_scalar(
    +            "SELECT mode FROM pg_locks WHERE locktype = 'advisory' AND database = \
    +             (SELECT oid FROM pg_database WHERE datname = current_database()) ORDER BY mode",
    +        )
    +        .fetch_all(&s.catalog)
    +        .await
    +        .expect("pg_locks");
    +        assert_eq!(
    +            modes,
    +            ["ExclusiveLock"],
    +            "the holder's lock and nothing else"
    +        );
    +        assert_eq!(
    +            taker.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Confirmed
    +        );
    +        assert!(check().await.expect("check"), "the heartbeat refreshed it");
    +        // The row's generation moved under the live holder (a takeover by
    +        // another writer): the holder's next heartbeat matches no row and
    +        // reports the claim lost, and its release leaves the other's row.
    +        sqlx::query("UPDATE backup_maintenance SET owner_generation = owner_generation + 1")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("move the generation");
    +        assert_eq!(
    +            taker.heartbeat().await.expect("heartbeat"),
    +            Heartbeat::Lost,
    +            "a heartbeat under a moved generation reports the claim lost"
    +        );
    +        taker.release().await.expect("release");
    +        let rows: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM backup_maintenance")
    +            .fetch_one(&s.catalog)
    +            .await
    +            .expect("count");
    +        assert_eq!(
    +            rows, 1,
    +            "a release by a holder that lost the claim leaves the row"
    +        );
    +        assert!(
    +            check().await.expect("check"),
    +            "the lock is free and the row young: held by the heartbeat arm"
    +        );
    +        sqlx::query("DELETE FROM backup_maintenance")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("clear");
    +        assert!(!check().await.expect("check"));
    +        s.cleanup().await;
    +    }
    +
    +    /// `check_maintenance_claim` holds the claim's key shared for one
    +    /// statement, and a take that lands inside it finds the exclusive lock
    +    /// refused with no holder, so the take tries again on its claim
    +    /// connection. Held across every try, the shared lock makes the take
    +    /// refuse, naming no holder (there is no row). With the lock step given
    +    /// a long span, a shared lock released once the claim connection has
    +    /// made a failed try (seen in `pg_stat_activity`) is taken by a later
    +    /// try.
    +    #[tokio::test]
    +    async fn a_maintenance_take_tries_again_past_a_checks_shared_lock() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP a_maintenance_take_tries_again_past_a_checks_shared_lock: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        const SHARED: &str =
    +            "SELECT pg_advisory_xact_lock_shared(hashtextextended('backup_maintenance', $1))";
    +
    +        let mut check = s.catalog.begin().await.expect("begin");
    +        sqlx::query(SHARED)
    +            .bind(RESTORE_LOCK_SEED)
    +            .execute(&mut *check)
    +            .await
    +            .expect("the check's shared lock");
    +        let started = std::time::Instant::now();
    +        match s.engine.maintenance_claim("purge on host-a", stale).await {
    +            Err(StorageError::AlreadyExists(m)) => {
    +                assert!(m.contains("another maintenance command"), "{m}");
    +            }
    +            Err(other) => panic!("refused otherwise: {other:?}"),
    +            Ok(_) => panic!("taken beside a held shared lock"),
    +        }
    +        assert!(
    +            started.elapsed() >= std::time::Duration::from_millis(10),
    +            "the take tried once: {:?}",
    +            started.elapsed()
    +        );
    +        check.rollback().await.expect("rollback");
    +
    +        let mut check = s.catalog.begin().await.expect("begin");
    +        sqlx::query(SHARED)
    +            .bind(RESTORE_LOCK_SEED)
    +            .execute(&mut *check)
    +            .await
    +            .expect("the check's shared lock");
    +        let tried = std::sync::atomic::AtomicBool::new(false);
    +        let (taken, ()) = tokio::join!(
    +            async {
    +                let taken = s
    +                    .engine
    +                    .maintenance_lock_for_tests(10_000, std::time::Duration::from_millis(1))
    +                    .await;
    +                tried.store(true, std::sync::atomic::Ordering::SeqCst);
    +                taken
    +            },
    +            async {
    +                let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
    +                loop {
    +                    let failed_try: bool = sqlx::query_scalar(
    +                        "SELECT EXISTS(SELECT 1 FROM pg_stat_activity \
    +                         WHERE pid <> pg_backend_pid() AND datname = current_database() \
    +                         AND query LIKE 'SELECT pg_try_advisory_lock(%')",
    +                    )
    +                    .fetch_one(&s.catalog)
    +                    .await
    +                    .expect("activity");
    +                    if failed_try
    +                        || tried.load(std::sync::atomic::Ordering::SeqCst)
    +                        || std::time::Instant::now() > deadline
    +                    {
    +                        break;
    +                    }
    +                    tokio::time::sleep(std::time::Duration::from_millis(1)).await;
    +                }
    +                check.rollback().await.expect("rollback");
    +            }
    +        );
    +        assert!(
    +            taken.expect("the lock step"),
    +            "a later try took the lock once the check's statement ended"
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    /// Purge's wait (`wait_for_quiet`) over this catalog, on a real claim.
    +    /// A FAILED restore record whose target is still CREATING (the state a
    +    /// stop between the FAILED write and the target's removal leaves) is
    +    /// returned by the zero-window dead-job listing with the RUNNING
    +    /// records; it is the reconciler's duty, not a running job, so the wait
    +    /// is quiet at its first read and runs no duty itself: the target is
    +    /// left CREATING for the server. A FAILED backup row whose heartbeat is
    +    /// within the window (a job that has not yet seen its claim lost may
    +    /// still write under its prefix) holds the wait to its limit, which
    +    /// names it; once the heartbeat has aged past the window it does not.
    +    #[tokio::test]
    +    async fn the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat() {
    +        use extenddb_engine::HeldClaim;
    +        use extenddb_engine::maintenance::{Interrupt, Wait, WaitOutcome, wait_for_quiet};
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let stale = std::time::Duration::from_secs(300);
    +        let s = scratch().await;
    +        indexed_source(&s, "wq_src").await;
    +        let backup = s
    +            .engine
    +            .create(ACCOUNT, "wq_src", "b", JobPermit::detached())
    +            .await
    +            .expect("backup");
    +        let desc = restore(&s, "wq_dst", &backup.backup_arn)
    +            .await
    +            .expect("restore");
    +        sqlx::query(
    +            "UPDATE tables SET table_status = 'CREATING', status_transition_at = NULL \
    +             WHERE table_id = $1",
    +        )
    +        .bind(&desc.table_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("the target back to CREATING");
    +        sqlx::query(
    +            "UPDATE table_restores SET status = 'FAILED', completed_at = NOW() - INTERVAL '1 hour' \
    +             WHERE table_id = $1",
    +        )
    +        .bind(&desc.table_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("the record FAILED an hour ago");
    +        let listed = s
    +            .engine
    +            .list_dead_restore_jobs(std::time::Duration::ZERO)
    +            .await
    +            .expect("list");
    +        assert_eq!(listed.len(), 1, "{listed:?}");
    +        assert_eq!(
    +            listed[0].status,
    +            JobStatus::Failed,
    +            "the zero-window listing returns the FAILED record the wait reads"
    +        );
    +
    +        let claim = s
    +            .engine
    +            .maintenance_claim("purge on host-a", stale)
    +            .await
    +            .expect("claim");
    +        let held = HeldClaim::start(claim, std::time::Duration::from_secs(1));
    +        let limit = std::time::Duration::from_secs(3);
    +        let wait = Wait {
    +            owner_window: stale,
    +            limit,
    +            poll: std::time::Duration::from_millis(200),
    +        };
    +        let announced: std::sync::Mutex>> = std::sync::Mutex::new(Vec::new());
    +        let announce = |jobs: &[String]| announced.lock().unwrap().push(jobs.to_vec());
    +        let (_tx, mut interrupt) = Interrupt::new();
    +        let started = std::time::Instant::now();
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Quiet,
    +            "a FAILED restore record is the reconciler's duty, not a running job"
    +        );
    +        assert!(
    +            started.elapsed() < std::time::Duration::from_secs(1),
    +            "quiet at the first read: {:?}",
    +            started.elapsed()
    +        );
    +        assert!(announced.lock().unwrap().is_empty());
    +        assert_eq!(
    +            table_status(&s, "wq_dst").await.as_deref(),
    +            Some("CREATING"),
    +            "the wait runs no duty: the target is left for the server's reconciler"
    +        );
    +        assert_eq!(
    +            job_status(&s, &desc.table_id).await.as_deref(),
    +            Some("FAILED")
    +        );
    +
    +        // A FAILED backup row of the store whose heartbeat is this moment's.
    +        let failed = row(ACCOUNT, "Music", "f");
    +        let job = s.engine.claim_backup(&failed).await.expect("claim");
    +        job.set_status(
    +            JobStatus::Creating,
    +            JobStatus::Failed,
    +            &StatusFields {
    +                failure_reason: Some("the test's".to_owned()),
    +                ..StatusFields::default()
    +            },
    +        )
    +        .await
    +        .expect("failed");
    +        drop(job);
    +        let started = std::time::Instant::now();
    +        let expected = vec![format!(
    +            "the FAILED backup {} has a live owner (heartbeat within stale_job_after_secs)",
    +            failed.backup_arn
    +        )];
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Busy(expected.clone()),
    +            "a FAILED row with a live heartbeat holds the wait and is named"
    +        );
    +        assert!(started.elapsed() >= limit, "{:?}", started.elapsed());
    +        assert_eq!(*announced.lock().unwrap(), vec![expected]);
    +        // Aged past the window on the catalog's clock: a dead owner.
    +        sqlx::query(
    +            "UPDATE backup_index SET heartbeat_at = now() - interval '10 minutes' \
    +             WHERE backup_id = $1",
    +        )
    +        .bind(&failed.backup_id)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("age");
    +        assert_eq!(
    +            wait_for_quiet(&s.engine, &store(), &held, wait, &mut interrupt, announce).await,
    +            WaitOutcome::Quiet,
    +            "a FAILED row whose heartbeat has aged is not waited on"
    +        );
    +        held.release().await.expect("release");
    +        assert!(
    +            !s.engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check")
    +        );
    +        s.cleanup().await;
    +    }
    +
    +    /// The factory (`create_server_components`) starts the GSI queue workers
    +    /// for a server only. Components opened `beside_server` (`backup list`,
    +    /// the purge's dry run) leave a pending index write where it is, for
    +    /// the server that owns the catalog; components opened as a server
    +    /// apply it.
    +    #[tokio::test]
    +    async fn components_opened_beside_a_server_apply_no_pending_index_write() {
    +        use extenddb_storage::server_components::ServerComponentsOptions;
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP components_opened_beside_a_server_apply_no_pending_index_write: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        indexed_source_with(&s, "gq_src", 0).await;
    +        // An asynchronous index: from here a write queues its index update
    +        // in `gsi_pending` for a worker, instead of applying it in its own
    +        // transaction.
    +        sqlx::query("UPDATE settings SET value = '50' WHERE key = 'index_propagation_delay_ms'")
    +            .execute(&s.catalog)
    +            .await
    +            .expect("an asynchronous index");
    +        let pending = || async {
    +            sqlx::query_scalar::<_, i64>("SELECT count(*) FROM gsi_pending")
    +                .fetch_one(&s.db)
    +                .await
    +                .expect("count gsi_pending")
    +        };
    +        assert_eq!(pending().await, 0);
    +        let desc = s
    +            .engine
    +            .describe_table(
    +                ACCOUNT,
    +                DescribeTableInput {
    +                    table_name: "gq_src".to_owned(),
    +                },
    +            )
    +            .await
    +            .expect("describe");
    +        put_indexed_items(&s, "gq_src", &desc, 1).await;
    +        assert_eq!(pending().await, 1, "the write queued its index update");
    +
    +        // The factory reads the encryption key `init` writes.
    +        sqlx::query(
    +            "INSERT INTO settings (key, value) VALUES ('encryption_key', 'test-key') \
    +             ON CONFLICT (key) DO NOTHING",
    +        )
    +        .execute(&s.catalog)
    +        .await
    +        .expect("encryption key");
    +        // Installed once per process; a second install is refused and that
    +        // is fine.
    +        let _ = extenddb_storage::set_backend(extenddb_storage_postgres::backend());
    +        let config = extenddb_storage_postgres::PostgresStorageConfig {
    +            connection_string: s.catalog_url.clone(),
    +            pool_size: 10,
    +            catalog_pool_size: Some(10),
    +        };
    +        let open = |beside_server: bool| {
    +            let mut options = ServerComponentsOptions::default();
    +            options.release = "test";
    +            options.beside_server = beside_server;
    +            extenddb_storage::create_server_components(&config, REGION, options)
    +        };
    +
    +        let beside = open(true).await.expect("open beside a server");
    +        tokio::time::sleep(std::time::Duration::from_secs(3)).await;
    +        assert_eq!(
    +            pending().await,
    +            1,
    +            "components opened beside a server start no worker: the row is left for the server"
    +        );
    +        drop(beside);
    +
    +        let server = open(false).await.expect("open as a server");
    +        let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10);
    +        while pending().await != 0 {
    +            assert!(
    +                std::time::Instant::now() < deadline,
    +                "a server's workers apply the pending row"
    +            );
    +            tokio::time::sleep(std::time::Duration::from_millis(100)).await;
    +        }
    +        drop(server);
    +        s.cleanup().await;
    +    }
    +
    +    /// `maintenance_claim` itself (its six tries 2 ms apart) takes the lock
    +    /// past a check's shared lock that ends before the tries do: the shared
    +    /// lock is held until the take's first try has failed (seen in
    +    /// `pg_stat_activity`) and then released, and a later try takes the lock.
    +    /// With one try in place of six the take would be refused with no holder
    +    /// to name.
    +    #[tokio::test]
    +    async fn a_maintenance_take_passes_a_shared_lock_shorter_than_its_span() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP a_maintenance_take_passes_a_shared_lock_shorter_than_its_span: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        let mut check = s.catalog.begin().await.expect("begin");
    +        sqlx::query(
    +            "SELECT pg_advisory_xact_lock_shared(hashtextextended('backup_maintenance', $1))",
    +        )
    +        .bind(RESTORE_LOCK_SEED)
    +        .execute(&mut *check)
    +        .await
    +        .expect("the check's shared lock");
    +        let tried = std::sync::atomic::AtomicBool::new(false);
    +        let (taken, ()) = tokio::join!(
    +            async {
    +                let taken = s.engine.maintenance_claim("purge on host-a", stale).await;
    +                tried.store(true, std::sync::atomic::Ordering::SeqCst);
    +                taken
    +            },
    +            async {
    +                let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30);
    +                loop {
    +                    let failed_try: bool = sqlx::query_scalar(
    +                        "SELECT EXISTS(SELECT 1 FROM pg_stat_activity \
    +                         WHERE pid <> pg_backend_pid() AND datname = current_database() \
    +                         AND query LIKE 'SELECT pg_try_advisory_lock(%')",
    +                    )
    +                    .fetch_one(&s.catalog)
    +                    .await
    +                    .expect("activity");
    +                    if failed_try
    +                        || tried.load(std::sync::atomic::Ordering::SeqCst)
    +                        || std::time::Instant::now() > deadline
    +                    {
    +                        break;
    +                    }
    +                    tokio::time::sleep(std::time::Duration::from_millis(1)).await;
    +                }
    +                check.rollback().await.expect("rollback");
    +            }
    +        );
    +        let claim = taken.expect("a later try of the take passed the check's shared lock");
    +        assert!(
    +            s.engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check"),
    +            "the take holds the claim"
    +        );
    +        claim.release().await.expect("release");
    +        s.cleanup().await;
    +    }
    +
    +    /// `release` unlocks the advisory lock before it closes the session, so a
    +    /// `check_maintenance_claim` on another connection right after the
    +    /// release reads the claim free, every time: a CreateBackup sent the
    +    /// moment a purge exits is not refused. Twenty rounds, with no wait
    +    /// between the release and the check.
    +    #[tokio::test]
    +    async fn a_released_claim_reads_free_at_once() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP a_released_claim_reads_free_at_once: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let stale = std::time::Duration::from_secs(300);
    +        for round in 0..20 {
    +            let claim = s
    +                .engine
    +                .maintenance_claim("purge on host-a", stale)
    +                .await
    +                .expect("claim");
    +            assert!(
    +                s.engine
    +                    .check_maintenance_claim(stale)
    +                    .await
    +                    .expect("check")
    +            );
    +            claim.release().await.expect("release");
    +            assert!(
    +                !s.engine
    +                    .check_maintenance_claim(stale)
    +                    .await
    +                    .expect("check"),
    +                "round {round}: the claim reads held right after its release"
    +            );
    +        }
    +        s.cleanup().await;
    +    }
    +
    +    /// The purge's wait set reads RUNNING restore records by status alone:
    +    /// a record whose heartbeat is ahead of the server's clock (coordinator
    +    /// skew, a backward clock step) or in this statement's instant is
    +    /// listed by `list_running_restore_jobs`, where the dead-job listing
    +    /// under a zero window, which compares the heartbeat with `now()`, does
    +    /// not list it.
    +    #[tokio::test]
    +    async fn running_restores_are_listed_by_status_whatever_their_heartbeat() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP running_restores_are_listed_by_status_whatever_their_heartbeat: no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        indexed_source(&s, "rr_target").await;
    +        let target_id = table_id_of(&s, "rr_target").await.unwrap();
    +        sqlx::query(
    +            "INSERT INTO table_restores (table_id, source_backup_arn, source_table_arn, \
    +             restore_date_time, status, kind, owner_generation, heartbeat_at) \
    +             VALUES ($1, $2, 'arn:aws:dynamodb:us-east-1:123456789012:table/Music', now(), \
    +             'RUNNING', 'legacy', 7, now() + INTERVAL '60 seconds')",
    +        )
    +        .bind(&target_id)
    +        .bind(&row(ACCOUNT, "Music", "nightly").backup_arn)
    +        .execute(&s.catalog)
    +        .await
    +        .expect("a RUNNING record stamped ahead of the clock");
    +        let running = s
    +            .engine
    +            .list_running_restore_jobs()
    +            .await
    +            .expect("list running");
    +        assert_eq!(running.len(), 1, "{running:?}");
    +        assert_eq!(running[0].target.table_name, "rr_target");
    +        assert_eq!(running[0].status, JobStatus::Running);
    +        assert!(
    +            s.engine
    +                .list_dead_restore_jobs(std::time::Duration::ZERO)
    +                .await
    +                .expect("list dead")
    +                .is_empty(),
    +            "the age-based listing misses a heartbeat ahead of the clock"
    +        );
    +        sqlx::query("UPDATE table_restores SET status = 'DONE' WHERE table_id = $1")
    +            .bind(&target_id)
    +            .execute(&s.catalog)
    +            .await
    +            .expect("settle");
    +        assert!(
    +            s.engine
    +                .list_running_restore_jobs()
    +                .await
    +                .expect("list running")
    +                .is_empty()
    +        );
    +        s.cleanup().await;
    +    }
    +}
    +
    +// --- the store identity file, over two engines on one catalog ----------------------
    +
    +/// The identity checks of a configured store (`extenddb_engine::store_identity`)
    +/// run against this catalog and real filesystem stores: two instances of one
    +/// catalog starting against one empty root, and two instances with
    +/// host-local roots under one `store_root` label.
    +mod store_identity {
    +    use std::sync::Arc;
    +
    +    use extenddb_backup_store::{BackupStore, FilesystemStore, byte_stream_from};
    +    use extenddb_core::backup_format::BackupArn;
    +    use extenddb_engine::StoreIdentityFile;
    +    use extenddb_engine::store_identity::{STORE_IDENTITY_KEY, StoreProblem, establish, recheck};
    +    use extenddb_storage::BackupCatalog;
    +    use extenddb_storage::backup_catalog::{JobStatus, StatusFields, StoreKind};
    +
    +    use super::*;
    +
    +    /// A private root: `tempfile` creates directories with the umask applied
    +    /// to 0777, which the store refuses.
    +    fn private_root() -> tempfile::TempDir {
    +        use std::os::unix::fs::PermissionsExt;
    +        let dir = tempfile::tempdir().expect("tempdir");
    +        std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o700))
    +            .expect("chmod");
    +        dir
    +    }
    +
    +    async fn second_engine(s: &Scratch) -> PostgresEngine {
    +        PostgresEngine::new(
    +            &PostgresConfig {
    +                connection_string: s.catalog_url.clone(),
    +                pool_size: 4,
    +                max_item_size_bytes: 400_000,
    +            },
    +            REGION,
    +        )
    +        .await
    +        .expect("a second engine on the same catalog")
    +    }
    +
    +    async fn read_file(store: &dyn BackupStore) -> StoreIdentityFile {
    +        use futures::TryStreamExt;
    +        let bytes: Vec = store
    +            .get(STORE_IDENTITY_KEY)
    +            .await
    +            .expect("the file")
    +            .try_fold(Vec::new(), |mut acc, chunk| async move {
    +                acc.extend_from_slice(&chunk);
    +                Ok(acc)
    +            })
    +            .await
    +            .expect("read");
    +        StoreIdentityFile::parse(&bytes).expect("parses")
    +    }
    +
    +    /// Two instances of one catalog started against one empty root end with
    +    /// one id in the file, and a row either writes carries it.
    +    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
    +    async fn two_instances_on_one_empty_root_end_with_one_id() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP two_instances_on_one_empty_root_end_with_one_id: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let other = second_engine(&s).await;
    +        let root = private_root();
    +        let store_a: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let store_b: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let label = root.path().canonicalize().unwrap().display().to_string();
    +        let (a, b) = tokio::join!(
    +            establish(
    +                &*store_a,
    +                StoreKind::Filesystem,
    +                &label,
    +                &s.engine,
    +                "postgres"
    +            ),
    +            establish(&*store_b, StoreKind::Filesystem, &label, &other, "postgres"),
    +        );
    +        let (a, b) = (a.expect("instance a"), b.expect("instance b"));
    +        assert_eq!(a, b, "both instances use one id");
    +        let file = read_file(&*store_a).await;
    +        assert_eq!(file.store_id, a.store_id);
    +        assert_eq!(file.catalog_id, s.engine.catalog_id().await.unwrap());
    +        assert_eq!(file.backend, "postgres");
    +        // A row written under the identity carries it, and both instances
    +        // read it back as the root's one id.
    +        let mut row = super::catalog_index_row(ACCOUNT, "Music", "nightly");
    +        row.store = a.clone();
    +        let claim = s.engine.claim_backup(&row).await.expect("claim");
    +        drop(claim);
    +        for engine in [&s.engine, &other] {
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &label)
    +                    .await
    +                    .unwrap(),
    +                std::slice::from_ref(&a.store_id)
    +            );
    +            recheck(&*store_a, &a, engine, "postgres")
    +                .await
    +                .expect("still healthy");
    +        }
    +        s.cleanup().await;
    +    }
    +
    +    /// Two instances of one catalog with host-local filesystem roots under
    +    /// one `store_root` label: the second, whose root has no file while the
    +    /// catalog has rows for the label, is unhealthy; with the first's file
    +    /// copied to it the manifest probe marks it unhealthy, because the
    +    /// newest AVAILABLE backup's manifest is not in its root.
    +    #[tokio::test]
    +    async fn host_local_roots_under_one_label_leave_the_second_instance_unhealthy() {
    +        if base_conn().is_none() {
    +            eprintln!(
    +                "SKIP host_local_roots_under_one_label_leave_the_second_instance_unhealthy: \
    +                 no PostgreSQL"
    +            );
    +            return;
    +        }
    +        let s = scratch().await;
    +        let other = second_engine(&s).await;
    +        let root_a = private_root();
    +        let root_b = private_root();
    +        let store_a: Arc =
    +            Arc::new(FilesystemStore::open(root_a.path()).await.unwrap());
    +        let store_b: Arc =
    +            Arc::new(FilesystemStore::open(root_b.path()).await.unwrap());
    +        // One label for both: the path both hosts configured.
    +        let label = "/var/lib/extenddb/backups";
    +
    +        // The first instance creates the file and an AVAILABLE backup with its
    +        // manifest in its own root.
    +        let a = establish(
    +            &*store_a,
    +            StoreKind::Filesystem,
    +            label,
    +            &s.engine,
    +            "postgres",
    +        )
    +        .await
    +        .expect("instance a");
    +        let mut row = super::catalog_index_row(ACCOUNT, "Music", "nightly");
    +        row.store = a.clone();
    +        let claim = s.engine.claim_backup(&row).await.expect("claim");
    +        let arn = BackupArn::parse(&row.backup_arn).unwrap();
    +        store_a
    +            .put(
    +                &format!("{}extenddb-manifest.json", arn.store_prefix()),
    +                byte_stream_from(b"{}".to_vec()),
    +            )
    +            .await
    +            .expect("manifest");
    +        claim
    +            .set_status(
    +                JobStatus::Creating,
    +                JobStatus::Available,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("available");
    +        recheck(&*store_a, &a, &s.engine, "postgres")
    +            .await
    +            .expect("a is healthy");
    +
    +        // The second instance: its root has no file and the catalog has rows
    +        // for the label.
    +        let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::FileMissingWithRows(vec![a.store_id.clone()])
    +        );
    +        assert!(
    +            store_b.head(STORE_IDENTITY_KEY).await.unwrap().is_none(),
    +            "nothing was created over the rows"
    +        );
    +
    +        // The operator copies the first's file to the second root: the ids
    +        // agree and the probe finds the manifest missing.
    +        let file = read_file(&*store_a).await;
    +        store_b
    +            .put(STORE_IDENTITY_KEY, byte_stream_from(file.to_bytes()))
    +            .await
    +            .unwrap();
    +        let err = establish(&*store_b, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::ManifestMissing {
    +                backup_arn: row.backup_arn.clone(),
    +            }
    +        );
    +        // The first instance is unaffected.
    +        recheck(&*store_a, &a, &s.engine, "postgres")
    +            .await
    +            .expect("a stays healthy");
    +        // A shared root (the second instance pointed at the first's
    +        // directory) passes.
    +        let shared: Arc =
    +            Arc::new(FilesystemStore::open(root_a.path()).await.unwrap());
    +        let b = establish(&*shared, StoreKind::Filesystem, label, &other, "postgres")
    +            .await
    +            .expect("a shared root is healthy");
    +        assert_eq!(b, a);
    +        s.cleanup().await;
    +    }
    +
    +    /// A root whose file names another catalog is unhealthy for this one
    +    /// until an operator changes the file; the first catalog's instance is
    +    /// unaffected.
    +    #[tokio::test]
    +    async fn a_root_owned_by_another_catalog_is_unhealthy() {
    +        if base_conn().is_none() {
    +            eprintln!("SKIP a_root_owned_by_another_catalog_is_unhealthy: no PostgreSQL");
    +            return;
    +        }
    +        let s = scratch().await;
    +        let root = private_root();
    +        let store: Arc =
    +            Arc::new(FilesystemStore::open(root.path()).await.unwrap());
    +        let label = root.path().canonicalize().unwrap().display().to_string();
    +        let foreign = StoreIdentityFile {
    +            format_version: 1,
    +            store_id: "s-foreign".to_owned(),
    +            catalog_id: "another-catalog".to_owned(),
    +            backend: "sqlite".to_owned(),
    +            created_at_ms: 0,
    +        };
    +        store
    +            .put(STORE_IDENTITY_KEY, byte_stream_from(foreign.to_bytes()))
    +            .await
    +            .unwrap();
    +        let err = establish(
    +            &*store,
    +            StoreKind::Filesystem,
    +            &label,
    +            &s.engine,
    +            "postgres",
    +        )
    +        .await
    +        .unwrap_err();
    +        assert_eq!(
    +            err,
    +            StoreProblem::ForeignCatalog {
    +                file_catalog_id: "another-catalog".to_owned(),
    +                catalog_id: s.engine.catalog_id().await.unwrap(),
    +            }
    +        );
    +        s.cleanup().await;
    +    }
     }
     
     /// The reconciler's fencing (design 4(e)): the FAILED write conditioned on
    diff --git a/crates/storage-postgres/tests/migrations_serving.rs b/crates/storage-postgres/tests/migrations_serving.rs
    index e3697db51..8a3b933a7 100644
    --- a/crates/storage-postgres/tests/migrations_serving.rs
    +++ b/crates/storage-postgres/tests/migrations_serving.rs
    @@ -13,7 +13,7 @@ mod migrations_support;
     use std::time::Duration;
     
     use extenddb_storage::hooks::{Reregistration, ServingRegistrar};
    -use extenddb_storage::migrations::{CatalogStatus, MigrationId, Runner};
    +use extenddb_storage::migrations::{CatalogStatus, MigrationError, MigrationId, Runner};
     use extenddb_storage_postgres::migrations::{
         MigrationLock, PostgresMigrations, ServerRegistration, migrate,
     };
    @@ -309,3 +309,64 @@ async fn an_attempt_dropped_while_its_status_read_waits_leaves_no_key() {
         assert_eq!(held, [], "a dropped attempt left the serving key held");
         admin.close().await;
     }
    +
    +/// A command that opens the catalog as a server does registers under its own
    +/// name: the serving session's `application_name` is the command's, a
    +/// migration that waits for the registrations names it, a registration
    +/// taken again keeps it, and a registration refused while a migration holds
    +/// the catalog names the migration without telling the operator to start a
    +/// server.
    +#[tokio::test]
    +async fn a_command_registers_under_its_own_name() {
    +    let Some(base) = base_conn() else {
    +        return skip("a command's registration");
    +    };
    +    let s = installed(&base, "pgs_command").await;
    +    let admin = pool(&format!("{base}/postgres"), 1).await;
    +    let registration =
    +        ServerRegistration::register_as_with(&s.catalog_url(), "extenddb backup list", KEEPALIVE)
    +            .await
    +            .expect("register");
    +    let sessions = serving_sessions(&admin, &s).await;
    +    assert_eq!(sessions.len(), 1, "{sessions:?}");
    +    assert_eq!(sessions[0].1, "extenddb backup list");
    +    let err = migrate(&s.target(), RELEASE, Duration::from_secs(1))
    +        .await
    +        .expect_err("a migration while the command is registered");
    +    assert_eq!(err.exit_code(), 6, "{err}");
    +    assert!(
    +        err.to_string().contains("\"extenddb backup list\""),
    +        "the migration names the command: {err}"
    +    );
    +
    +    end_the_serving_session(&admin, &s).await;
    +    lost(®istration).await;
    +    assert_eq!(
    +        registration.register_again().await,
    +        Reregistration::Registered
    +    );
    +    let sessions = serving_sessions(&admin, &s).await;
    +    assert_eq!(sessions.len(), 1, "{sessions:?}");
    +    assert_eq!(sessions[0].1, "extenddb backup list", "taken again");
    +    drop(registration);
    +
    +    let lock = MigrationLock::acquire(&s.options(), Duration::from_secs(5))
    +        .await
    +        .expect("the migration lock, once the command's key is gone");
    +    match ServerRegistration::register_as(&s.catalog_url(), "extenddb backup purge").await {
    +        Err(e @ MigrationError::Locked(_)) => {
    +            assert_eq!(e.exit_code(), 6, "{e}");
    +            assert!(
    +                e.to_string()
    +                    .contains("a migration is running on this catalog")
    +                    && !e.to_string().contains("start the server"),
    +                "{e}"
    +            );
    +        }
    +        Err(other) => panic!("refused otherwise: {other}"),
    +        Ok(_) => panic!("registered beside a migration"),
    +    }
    +    assert_eq!(serving_sessions(&admin, &s).await, []);
    +    lock.release().await;
    +    admin.close().await;
    +}
    diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs
    index ab1768d49..1629f2f53 100644
    --- a/crates/storage-sqlite/src/backup.rs
    +++ b/crates/storage-sqlite/src/backup.rs
    @@ -4327,7 +4327,8 @@ mod restore_tests {
             use extenddb_storage::BackupCatalog;
             use extenddb_storage::backup_catalog::{
                 BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey,
    -            StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome,
    +            StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION,
    +            WriteOutcome,
             };
             use futures::TryStreamExt;
     
    @@ -4388,7 +4389,12 @@ mod restore_tests {
                     uuid::Uuid::parse_str(&engine.catalog_id().await.expect("id")).is_ok(),
                     "the catalog has an id from the migration"
                 );
    -            assert!(!engine.check_maintenance_claim().await.expect("check"));
    +            assert!(
    +                !engine
    +                    .check_maintenance_claim(std::time::Duration::from_secs(300))
    +                    .await
    +                    .expect("check")
    +            );
                 let row = row(ACCOUNT, "Music", "nightly");
                 let claim = engine.claim_backup(&row).await.expect("claim");
                 assert_ne!(claim.generation(), UNOWNED_GENERATION);
    @@ -4994,6 +5000,503 @@ mod restore_tests {
                     .await
                     .expect("the backup is free");
             }
    +
    +        /// `read_store_ids`: the distinct ids of the rows for a root, of
    +        /// every status, scoped to the kind and the root.
    +        #[tokio::test]
    +        async fn read_store_ids_lists_the_distinct_ids_of_a_root() {
    +            let engine = engine().await;
    +            let root = store().store_root.clone();
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids")
    +                    .is_empty()
    +            );
    +            let first = row(ACCOUNT, "Music", "a");
    +            let _c1 = engine.claim_backup(&first).await.expect("claim");
    +            let second = row(ACCOUNT, "Music", "b");
    +            let c2 = engine.claim_backup(&second).await.expect("claim");
    +            c2.set_status(
    +                JobStatus::Creating,
    +                JobStatus::Failed,
    +                &StatusFields::default(),
    +            )
    +            .await
    +            .expect("fail");
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids"),
    +                ["store-1"],
    +                "two rows of one store, of two statuses, are one id"
    +            );
    +            let mut other = row(ACCOUNT, "Songs", "c");
    +            other.store.store_id = "store-2".to_owned();
    +            let _c3 = engine.claim_backup(&other).await.expect("claim");
    +            assert_eq!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, &root)
    +                    .await
    +                    .expect("ids"),
    +                ["store-1", "store-2"]
    +            );
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::S3, &root)
    +                    .await
    +                    .expect("ids")
    +                    .is_empty(),
    +                "another kind at the same root is another store"
    +            );
    +            assert!(
    +                engine
    +                    .read_store_ids(StoreKind::Filesystem, "/elsewhere")
    +                    .await
    +                    .expect("ids")
    +                    .is_empty()
    +            );
    +        }
    +
    +        /// `remove_dead_creating_backup_row`: a CREATING row whose heartbeat
    +        /// is older than the window is removed outright; one with a live
    +        /// owner, one that is not CREATING, and one that is gone are left,
    +        /// answering `false`.
    +        #[tokio::test]
    +        async fn a_dead_creating_row_is_removed_and_a_live_or_moved_one_is_kept() {
    +            let engine = engine().await;
    +            let stale = Duration::from_secs(300);
    +            let age = |backup_id: String| {
    +                let pool = engine.pool.clone();
    +                async move {
    +                    sqlx::query(
    +                        "UPDATE backup_index SET heartbeat_at = \
    +                         strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-10 minutes') WHERE backup_id = ?",
    +                    )
    +                    .bind(backup_id)
    +                    .execute(&pool)
    +                    .await
    +                    .expect("age");
    +                }
    +            };
    +            let creating = row(ACCOUNT, "Music", "a");
    +            let key = creating.key();
    +            let claim = engine.claim_backup(&creating).await.expect("claim");
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap(),
    +                "a live owner keeps its row"
    +            );
    +            age(creating.backup_id.clone()).await;
    +            assert!(
    +                engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap()
    +            );
    +            assert!(
    +                engine
    +                    .read_backup_row(&arn_of(&creating))
    +                    .await
    +                    .unwrap()
    +                    .is_none()
    +            );
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&key, stale)
    +                    .await
    +                    .unwrap(),
    +                "gone"
    +            );
    +            drop(claim);
    +
    +            let available = row(ACCOUNT, "Music", "b");
    +            let claim = engine.claim_backup(&available).await.expect("claim");
    +            claim
    +                .set_status(
    +                    JobStatus::Creating,
    +                    JobStatus::Available,
    +                    &StatusFields::default(),
    +                )
    +                .await
    +                .expect("available");
    +            age(available.backup_id.clone()).await;
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&available.key(), stale)
    +                    .await
    +                    .unwrap(),
    +                "only a CREATING row"
    +            );
    +            assert_eq!(
    +                engine
    +                    .read_backup_row(&arn_of(&available))
    +                    .await
    +                    .unwrap()
    +                    .unwrap()
    +                    .status,
    +                BackupStatus::Available
    +            );
    +        }
    +
    +        /// `list_backup_rows_older_than`: the rows of a store with a status
    +        /// whose clock column is older than the age on the database's clock.
    +        /// The two clocks are told apart by a row heartbeated at the start
    +        /// and FAILED after the window passed (as the reconciler fails a dead
    +        /// owner's row): listed on the heartbeat clock at once, not on the
    +        /// completed clock until its own window passes. A FAILED row without
    +        /// `completed_at` is never listed on the completed clock. A CREATING
    +        /// row and another store's row are not listed. The window is crossed
    +        /// by waiting, not by dating rows.
    +        #[tokio::test]
    +        async fn rows_older_than_are_listed_on_the_named_clock() {
    +            let engine = engine().await;
    +            let window = Duration::from_secs(1);
    +            // Names in the listing's order; two rows minted in one millisecond
    +            // share `created_at` and fall to ARN order, so the sets compare
    +            // sorted.
    +            let names = |listed: Vec| -> Vec {
    +                let mut names: Vec = listed.into_iter().map(|r| r.backup_name).collect();
    +                names.sort();
    +                names
    +            };
    +            let older = |status: BackupStatus, clock: StaleClock| {
    +                let engine = engine.clone();
    +                async move {
    +                    let rows: Vec = engine
    +                        .list_backup_rows_older_than(status, &store(), clock, window)
    +                        .try_collect()
    +                        .await
    +                        .expect("list");
    +                    names(rows)
    +                }
    +            };
    +            async fn fail(claim: &dyn extenddb_storage::backup_catalog::JobClaim, completed: bool) {
    +                let fields = StatusFields {
    +                    failure_reason: Some("test".to_owned()),
    +                    completed,
    +                    ..StatusFields::default()
    +                };
    +                assert_eq!(
    +                    claim
    +                        .set_status(JobStatus::Creating, JobStatus::Failed, &fields)
    +                        .await
    +                        .unwrap(),
    +                    WriteOutcome::Matched
    +                );
    +            }
    +
    +            // All heartbeated now. `late` stays CREATING until the window has
    +            // passed; `unfinished` is FAILED without `completed_at`;
    +            // `creating` stays CREATING; `elsewhere` is another store's;
    +            // `gone` is DELETED now.
    +            let late = row(ACCOUNT, "Music", "late");
    +            let late_claim = engine.claim_backup(&late).await.expect("claim");
    +            let unfinished = row(ACCOUNT, "Music", "unfinished");
    +            let unfinished_claim = engine.claim_backup(&unfinished).await.expect("claim");
    +            fail(unfinished_claim.as_ref(), false).await;
    +            let creating = row(ACCOUNT, "Music", "creating");
    +            let _creating_claim = engine.claim_backup(&creating).await.expect("claim");
    +            let mut elsewhere = row(ACCOUNT, "Music", "elsewhere");
    +            elsewhere.store.store_id = "store-2".to_owned();
    +            let elsewhere_claim = engine.claim_backup(&elsewhere).await.expect("claim");
    +            fail(elsewhere_claim.as_ref(), true).await;
    +            let gone = row(ACCOUNT, "Music", "gone");
    +            let gone_claim = engine.claim_backup(&gone).await.expect("claim");
    +            assert_eq!(
    +                gone_claim
    +                    .set_status(
    +                        JobStatus::Creating,
    +                        JobStatus::Available,
    +                        &StatusFields::default()
    +                    )
    +                    .await
    +                    .unwrap(),
    +                WriteOutcome::Matched
    +            );
    +            assert_eq!(
    +                engine.start_backup_delete(&gone.key()).await.unwrap(),
    +                DeleteStart::Deleting
    +            );
    +            assert!(engine.set_backup_deleted(&gone.key()).await.unwrap());
    +
    +            // Inside the window on every clock: nothing is older.
    +            assert!(
    +                older(BackupStatus::Failed, StaleClock::Heartbeat)
    +                    .await
    +                    .is_empty()
    +            );
    +            assert!(
    +                older(BackupStatus::Failed, StaleClock::Completed)
    +                    .await
    +                    .is_empty()
    +            );
    +            assert!(
    +                older(BackupStatus::Deleted, StaleClock::Completed)
    +                    .await
    +                    .is_empty()
    +            );
    +
    +            tokio::time::sleep(window + Duration::from_millis(300)).await;
    +            // The reconciler's shape: the owner's last beat is past the
    +            // window, the FAILED write is now.
    +            fail(late_claim.as_ref(), true).await;
    +            assert_eq!(
    +                older(BackupStatus::Failed, StaleClock::Heartbeat).await,
    +                ["late", "unfinished"],
    +                "both heartbeats are past the window; the just-failed row is listed on the \
    +                 heartbeat clock at once"
    +            );
    +            assert!(
    +                older(BackupStatus::Failed, StaleClock::Completed)
    +                    .await
    +                    .is_empty(),
    +                "completed just now, or never"
    +            );
    +            assert_eq!(
    +                older(BackupStatus::Deleted, StaleClock::Completed).await,
    +                ["gone"]
    +            );
    +            let elsewhere_rows: Vec = engine
    +                .list_backup_rows_older_than(
    +                    BackupStatus::Failed,
    +                    &elsewhere.store,
    +                    StaleClock::Heartbeat,
    +                    window,
    +                )
    +                .try_collect()
    +                .await
    +                .expect("list");
    +            assert_eq!(names(elsewhere_rows), ["elsewhere"], "scoped to the store");
    +
    +            tokio::time::sleep(window + Duration::from_millis(300)).await;
    +            assert_eq!(
    +                older(BackupStatus::Failed, StaleClock::Completed).await,
    +                ["late"],
    +                "the completed window crossed on the database's clock; a row without \
    +                 completed_at is never listed on this clock"
    +            );
    +            drop(late_claim);
    +            drop(unfinished_claim);
    +            drop(elsewhere_claim);
    +            drop(gone_claim);
    +        }
    +
    +        /// The maintenance claim: taken once, refused to a second holder
    +        /// naming the first, heartbeated, released; a holder whose heartbeat
    +        /// is older than the stale window is dead, so its claim is not
    +        /// reported held and is taken over, and its handle finds the claim
    +        /// lost.
    +        #[tokio::test]
    +        async fn the_maintenance_claim_excludes_a_second_holder_and_expires_when_stale() {
    +            let engine = engine().await;
    +            let stale = Duration::from_secs(300);
    +            let check = || engine.check_maintenance_claim(stale);
    +            assert!(!check().await.expect("check"));
    +            let first = engine
    +                .maintenance_claim("purge on host-a", stale)
    +                .await
    +                .expect("claim");
    +            assert_ne!(first.generation(), UNOWNED_GENERATION);
    +            assert!(check().await.expect("check"), "a live claim is held");
    +            match engine.maintenance_claim("purge on host-b", stale).await {
    +                Err(StorageError::AlreadyExists(m)) => {
    +                    assert!(m.contains("purge on host-a"), "{m}");
    +                }
    +                Err(other) => panic!("a second holder is refused with AlreadyExists: {other:?}"),
    +                Ok(_) => panic!("a second holder is refused"),
    +            }
    +            assert_eq!(
    +                first.heartbeat().await.expect("heartbeat"),
    +                Heartbeat::Confirmed
    +            );
    +            first.release().await.expect("release");
    +            assert!(!check().await.expect("check"), "released");
    +            first.release().await.expect("a second release is a no-op");
    +
    +            // A dead holder: its heartbeat aged past the window.
    +            let dead = engine
    +                .maintenance_claim("purge on host-c", stale)
    +                .await
    +                .expect("claim");
    +            sqlx::query(
    +                "UPDATE backup_maintenance SET heartbeat_at = \
    +                 strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-10 minutes')",
    +            )
    +            .execute(&engine.pool)
    +            .await
    +            .expect("age");
    +            assert!(!check().await.expect("check"), "a stale claim is not held");
    +            let taker = engine
    +                .maintenance_claim("purge on host-d", stale)
    +                .await
    +                .expect("a dead holder's claim is taken over");
    +            assert_ne!(taker.generation(), dead.generation());
    +            assert!(check().await.expect("check"));
    +            assert_eq!(
    +                dead.heartbeat().await.expect("heartbeat"),
    +                Heartbeat::Lost,
    +                "the dead holder's handle finds the claim lost"
    +            );
    +            dead.release()
    +                .await
    +                .expect("a release by a lost holder changes nothing");
    +            assert!(check().await.expect("check"), "the taker still holds it");
    +            let holder: String = sqlx::query_scalar("SELECT holder FROM backup_maintenance")
    +                .fetch_one(&engine.pool)
    +                .await
    +                .expect("holder");
    +            assert_eq!(holder, "purge on host-d");
    +            taker.release().await.expect("release");
    +            assert!(!check().await.expect("check"));
    +        }
    +
    +        /// An engine that holds the serve lock (a command that opened the
    +        /// file as `serve` does: `backup purge --confirm`) takes over a
    +        /// maintenance claim whose heartbeat is young: under the lock the
    +        /// row's holder is dead whatever its heartbeat says. The holder that
    +        /// was taken over finds its claim lost. The same take on an engine
    +        /// without the lock is refused naming the holder.
    +        #[cfg(unix)]
    +        #[tokio::test]
    +        async fn under_the_serve_lock_a_live_looking_claim_is_taken_over() {
    +            let stale = Duration::from_secs(300);
    +            let unlocked = engine().await;
    +            let killed = unlocked
    +                .maintenance_claim("extenddb backup purge on host-a (pid 1)", stale)
    +                .await
    +                .expect("claim");
    +            let refused = unlocked
    +                .maintenance_claim("extenddb backup purge on host-a (pid 2)", stale)
    +                .await
    +                .err()
    +                .expect("a live claim is refused without the serve lock");
    +            assert!(refused.to_string().contains("(pid 1)"), "{refused}");
    +
    +            let dir = std::env::temp_dir().join(format!(
    +                "extenddb-maintenance-serve-lock-{}",
    +                uuid::Uuid::new_v4().simple()
    +            ));
    +            std::fs::create_dir_all(&dir).expect("dir");
    +            let mut locked = unlocked.clone();
    +            locked.serve_lock = Some(std::sync::Arc::new(
    +                crate::serve_lock::ServeLock::try_acquire(&dir.join("db.sqlite"))
    +                    .unwrap_or_else(|_| panic!("the serve lock")),
    +            ));
    +            assert!(
    +                locked.check_maintenance_claim(stale).await.expect("check"),
    +                "the first claim's heartbeat is young"
    +            );
    +            let taken = locked
    +                .maintenance_claim("extenddb backup purge on host-a (pid 2)", stale)
    +                .await
    +                .expect("taken over under the serve lock");
    +            assert_ne!(taken.generation(), killed.generation());
    +            assert_eq!(killed.heartbeat().await.expect("beat"), Heartbeat::Lost);
    +            assert_eq!(taken.heartbeat().await.expect("beat"), Heartbeat::Confirmed);
    +            taken.release().await.expect("release");
    +            drop(locked);
    +            let _ = std::fs::remove_dir_all(&dir);
    +        }
    +
    +        /// A zero window selects by status alone: under the serve lock every
    +        /// owner is dead, so a FAILED or CREATING row whose heartbeat is
    +        /// stamped ahead of the database's clock (a backward clock step) is
    +        /// listed as dead and removed, where the age comparison would read it
    +        /// as a live owner and `backup purge` would wait on it and refuse.
    +        /// Under a non-zero window the same rows read as live. The RUNNING
    +        /// restore listing follows the same rule, and the wait set's listing
    +        /// by status sees such a record too.
    +        #[tokio::test]
    +        async fn a_zero_window_selects_dead_owners_by_status_alone() {
    +            let engine = engine().await;
    +            let failed = row(ACCOUNT, "Music", "failed-ahead");
    +            let failed_claim = engine.claim_backup(&failed).await.expect("claim");
    +            let fields = StatusFields {
    +                failure_reason: Some("test".to_owned()),
    +                completed: true,
    +                ..StatusFields::default()
    +            };
    +            assert_eq!(
    +                failed_claim
    +                    .set_status(JobStatus::Creating, JobStatus::Failed, &fields)
    +                    .await
    +                    .unwrap(),
    +                WriteOutcome::Matched
    +            );
    +            let creating = row(ACCOUNT, "Music", "creating-ahead");
    +            let _creating_claim = engine.claim_backup(&creating).await.expect("claim");
    +            // Both heartbeats a day ahead of the clock.
    +            let ahead: i64 = sqlx::query(
    +                "UPDATE backup_index SET heartbeat_at = strftime('%Y-%m-%dT%H:%M:%fZ', \
    +                 'now', '+1 day') WHERE backup_id IN (?, ?)",
    +            )
    +            .bind(&failed.backup_id)
    +            .bind(&creating.backup_id)
    +            .execute(&engine.pool)
    +            .await
    +            .expect("stamp ahead")
    +            .rows_affected()
    +            .try_into()
    +            .unwrap();
    +            assert_eq!(ahead, 2);
    +            let listed = |status: BackupStatus, window: Duration| {
    +                let engine = engine.clone();
    +                async move {
    +                    let rows: Vec = engine
    +                        .list_backup_rows_older_than(
    +                            status,
    +                            &store(),
    +                            StaleClock::Heartbeat,
    +                            window,
    +                        )
    +                        .try_collect()
    +                        .await
    +                        .expect("list");
    +                    rows.into_iter().map(|r| r.backup_name).collect::>()
    +                }
    +            };
    +            let window = Duration::from_secs(300);
    +            assert!(
    +                listed(BackupStatus::Failed, window).await.is_empty()
    +                    && listed(BackupStatus::Creating, window).await.is_empty(),
    +                "under a window a heartbeat ahead of the clock reads as live"
    +            );
    +            assert_eq!(
    +                listed(BackupStatus::Failed, Duration::ZERO).await,
    +                ["failed-ahead"]
    +            );
    +            assert_eq!(
    +                listed(BackupStatus::Creating, Duration::ZERO).await,
    +                ["creating-ahead"]
    +            );
    +            assert!(
    +                !engine
    +                    .remove_dead_creating_backup_row(&creating.key(), window)
    +                    .await
    +                    .unwrap(),
    +                "kept under a window"
    +            );
    +            assert!(
    +                engine
    +                    .remove_dead_creating_backup_row(&creating.key(), Duration::ZERO)
    +                    .await
    +                    .unwrap(),
    +                "removed under the zero window whatever the heartbeat says"
    +            );
    +            assert!(
    +                engine
    +                    .read_backup_row(&arn_of(&creating))
    +                    .await
    +                    .unwrap()
    +                    .is_none()
    +            );
    +            drop(failed_claim);
    +        }
         }
     
         /// The reconciler's fencing on SQLite (design 4(e)): the restore slot a
    diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs
    index 12dc1e63e..deba547c8 100644
    --- a/crates/storage-sqlite/src/backup_catalog.rs
    +++ b/crates/storage-sqlite/src/backup_catalog.rs
    @@ -22,6 +22,15 @@
     //! A conditional write never has an unknown outcome here (one process, one
     //! file: a statement either ran or errored), so `WriteOutcome::Unknown` is
     //! never returned.
    +//!
    +//! The maintenance claim is the `backup_maintenance` row, taken under the
    +//! engine write lock: refused while a row with a heartbeat younger than the
    +//! stale window exists, taken over (replaced) from a dead holder, heartbeated
    +//! and released by generation. One process serves a file, so a holder is a
    +//! command run while the server is stopped. Such a command passes a zero
    +//! stale window, which the dead-owner listings and removals read as "every
    +//! owner is dead" and select by status alone ([`zero_window_clause`]), so a
    +//! heartbeat stamped ahead of the clock is not read as a live owner.
     
     use std::collections::HashSet;
     use std::sync::{Arc, Mutex};
    @@ -31,8 +40,8 @@ use extenddb_core::backup_format::BackupArn;
     use extenddb_storage::JobPermit;
     use extenddb_storage::backup_catalog::{
         BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow,
    -    JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StatusFields, StoreIdentity,
    -    StoreKind, TimeBounds, WriteOutcome, new_generation,
    +    JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock,
    +    StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation,
     };
     use extenddb_storage::error::StorageError;
     use futures::future::BoxFuture;
    @@ -462,6 +471,49 @@ where
         )
     }
     
    +/// The age test of a dead-owner listing or removal, for one bound
    +/// parameter (`window` in seconds): `column` older than the window on the
    +/// database's clock, or, for a zero window, nothing (the bound is still
    +/// consumed). A zero window is `backup purge`'s under the serve lock: one
    +/// process has the file, so every owner is dead whatever its heartbeat says,
    +/// a heartbeat stamped ahead of the clock (a backward clock step) included,
    +/// where `julianday(x) < julianday('now')` would count it live.
    +fn zero_window_clause(window: Duration, column: &str) -> String {
    +    if window.is_zero() {
    +        "? = 0.0".to_owned()
    +    } else {
    +        format!("julianday({column}) < julianday('now') - ? / 86400.0")
    +    }
    +}
    +
    +/// A restore job record as the two listings select it.
    +#[allow(clippy::type_complexity)]
    +fn restore_job_row(
    +    (
    +        table_id,
    +        account_id,
    +        table_name,
    +        source_backup_arn,
    +        kind,
    +        status,
    +        owner_generation,
    +        heartbeat_at,
    +    ): (String, String, String, String, String, String, i64, String),
    +) -> Result {
    +    Ok(RestoreJobRow {
    +        target: RestoreTargetKey {
    +            account_id,
    +            table_name,
    +            table_id,
    +        },
    +        source_backup_arn,
    +        kind: RestoreKind::parse(&kind)?,
    +        status: JobStatus::parse(&status)?,
    +        owner_generation: u64::try_from(owner_generation).unwrap_or_default(),
    +        heartbeat_at: timestamp_epoch_seconds(&heartbeat_at),
    +    })
    +}
    +
     impl SqliteEngine {
         /// Every RUNNING restore job record at startup was owned by the process
         /// that stopped: one process serves a file, so a restart is proof of
    @@ -489,6 +541,20 @@ impl SqliteEngine {
             tx.commit().await.map_err(db_err)?;
             Ok(names)
         }
    +
    +    /// Remove the maintenance claim, whoever holds it: the server's startup
    +    /// step under the serve lock, where one process has the file and so no
    +    /// holder can be alive (a `backup purge` that died, or was killed, left
    +    /// the row with a heartbeat the jobs' check would otherwise honour for
    +    /// the whole stale window). Returns the holder's label when a row was
    +    /// removed.
    +    pub(crate) async fn clear_maintenance_claim(&self) -> Result, StorageError> {
    +        let _writer = self.write_lock.lock().await;
    +        sqlx::query_scalar("DELETE FROM backup_maintenance RETURNING holder")
    +            .fetch_optional(&self.pool)
    +            .await
    +            .map_err(db_err)
    +    }
     }
     
     impl BackupCatalog for SqliteEngine {
    @@ -592,6 +658,26 @@ impl BackupCatalog for SqliteEngine {
             })
         }
     
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            let row: Option = sqlx::query_as(&format!(
    +                "SELECT {ROW_COLUMNS} FROM backup_index \
    +                 WHERE store_kind = ? AND store_root = ? AND store_id = ? \
    +                 AND status = 'AVAILABLE' ORDER BY created_at DESC, backup_arn DESC LIMIT 1"
    +            ))
    +            .bind(store.store_kind.as_str())
    +            .bind(&store.store_root)
    +            .bind(&store.store_id)
    +            .fetch_optional(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            row.map(row_from_tuple).transpose()
    +        })
    +    }
    +
         fn list_account_backup_rows<'a>(
             &'a self,
             account_id: &'a str,
    @@ -721,6 +807,68 @@ impl BackupCatalog for SqliteEngine {
             })
         }
     
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result> {
    +        Box::pin(async move {
    +            let _writer = self.write_lock.lock().await;
    +            // A zero window selects by status alone (`zero_window_clause`):
    +            // the caller holds the serve lock and every owner is dead,
    +            // whatever the heartbeat says, a heartbeat ahead of the clock
    +            // (a backward clock step) included.
    +            let affected = sqlx::query(&format!(
    +                "DELETE FROM backup_index WHERE account_id = ? AND backup_id = ? \
    +                 AND status = 'CREATING' \
    +                 AND {}",
    +                zero_window_clause(stale_after, "heartbeat_at")
    +            ))
    +            .bind(&key.account_id)
    +            .bind(&key.backup_id)
    +            .bind(stale_after.as_secs_f64())
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(affected == 1)
    +        })
    +    }
    +
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result> {
    +        let column = match clock {
    +            StaleClock::Heartbeat => "heartbeat_at",
    +            StaleClock::Completed => "completed_at",
    +        };
    +        rows_stream(async move {
    +            // On the database's clock, as `remove_dead_creating_backup_row`
    +            // measures; `julianday(NULL)` is NULL, so a row with
    +            // `completed_at` unset is never older than any age on that clock.
    +            // A zero window selects by status alone (`zero_window_clause`).
    +            sqlx::query_as(&format!(
    +                "SELECT {ROW_COLUMNS} FROM backup_index \
    +                 WHERE store_kind = ? AND store_root = ? AND store_id = ? AND status = ? \
    +                 AND {} \
    +                 ORDER BY created_at, backup_arn",
    +                zero_window_clause(age, column)
    +            ))
    +            .bind(store.store_kind.as_str())
    +            .bind(&store.store_root)
    +            .bind(&store.store_id)
    +            .bind(status.as_str())
    +            .bind(age.as_secs_f64())
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
         fn backup_in_use<'a>(
             &'a self,
             key: &'a BackupKey,
    @@ -785,14 +933,8 @@ impl BackupCatalog for SqliteEngine {
                 // the copy's own failure path stamp `completed_at`, and a record
                 // without one is aged by its heartbeat). The delay is what leaves
                 // a task still on its own failure path to finish it. Both ages
    -            // are measured on the database's clock (`julianday('now')`). A
    -            // RUNNING record whose in-process slot a task holds is then
    -            // skipped, however old its heartbeat: one process serves the
    -            // file, so the slot is the liveness rule and the heartbeat only
    -            // what a restart ages (design 4(e)). A row this build cannot read
    -            // (a kind or status no release writes) is logged and skipped, as
    -            // ListBackups skips a row it cannot read, so one such row does
    -            // not stop the cycle.
    +            // are measured on the database's clock (`julianday('now')`); a
    +            // zero window selects by status alone (`zero_window_clause`).
                 #[allow(clippy::type_complexity)]
                 let rows: Vec<(
                     String,
    @@ -803,68 +945,81 @@ impl BackupCatalog for SqliteEngine {
                     String,
                     i64,
                     String,
    -            )> = sqlx::query_as(
    +            )> = sqlx::query_as(&format!(
                     "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \
                          r.kind, r.status, r.owner_generation, r.heartbeat_at \
                          FROM table_restores r JOIN tables t ON t.table_id = r.table_id \
    -                     WHERE (r.status = 'RUNNING' \
    -                            AND julianday(r.heartbeat_at) < julianday('now') - ?1 / 86400.0) \
    +                     WHERE (r.status = 'RUNNING' AND {}) \
                             OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \
    -                            AND julianday(COALESCE(r.completed_at, r.heartbeat_at)) \
    -                                < julianday('now') - ?1 / 86400.0) \
    +                            AND {}) \
                          ORDER BY t.account_id, t.table_name",
    -            )
    +                zero_window_clause(stale_after, "r.heartbeat_at"),
    +                zero_window_clause(stale_after, "COALESCE(r.completed_at, r.heartbeat_at)"),
    +            ))
    +            .bind(stale_after.as_secs_f64())
                 .bind(stale_after.as_secs_f64())
                 .fetch_all(&self.pool)
                 .await
                 .map_err(db_err)?;
    +            // A RUNNING record whose in-process slot a task holds is skipped,
    +            // however old its heartbeat: one process serves the file, so the
    +            // slot is the liveness rule and the heartbeat only what a restart
    +            // ages (design 4(e)). A row this build cannot read (a kind or
    +            // status no release writes) is logged and skipped, as ListBackups
    +            // skips a row it cannot read, so one such row does not stop the
    +            // cycle.
                 Ok(rows
                     .into_iter()
    -                .filter_map(
    -                    |(
    -                        table_id,
    -                        account_id,
    -                        table_name,
    -                        source_backup_arn,
    -                        kind,
    -                        status,
    -                        owner_generation,
    -                        heartbeat_at,
    -                    )| {
    -                        let (kind, status) =
    -                            match (RestoreKind::parse(&kind), JobStatus::parse(&status)) {
    -                                (Ok(kind), Ok(status)) => (kind, status),
    -                                (Err(e), _) | (_, Err(e)) => {
    -                                    tracing::warn!(
    -                                        "reconciler: restore job record of {table_name} (account \
    -                                     {account_id}) skipped: {e}"
    -                                    );
    -                                    return None;
    -                                }
    -                            };
    -                        if status == JobStatus::Running
    -                            && self.job_slots.holds(&restore_slot_key(&table_id))
    -                        {
    -                            return None;
    +                .filter_map(|row| {
    +                    let (table_id, account_id, table_name, status) =
    +                        (row.0.clone(), row.1.clone(), row.2.clone(), row.5.clone());
    +                    if status == JobStatus::Running.as_str()
    +                        && self.job_slots.holds(&restore_slot_key(&table_id))
    +                    {
    +                        return None;
    +                    }
    +                    match restore_job_row(row) {
    +                        Ok(job) => Some(job),
    +                        Err(e) => {
    +                            tracing::warn!(
    +                                "reconciler: restore job record of {table_name} (account \
    +                                 {account_id}) skipped: {e}"
    +                            );
    +                            None
                             }
    -                        Some(RestoreJobRow {
    -                            target: RestoreTargetKey {
    -                                account_id,
    -                                table_name,
    -                                table_id,
    -                            },
    -                            source_backup_arn,
    -                            kind,
    -                            status,
    -                            owner_generation: u64::try_from(owner_generation).unwrap_or_default(),
    -                            heartbeat_at: timestamp_epoch_seconds(&heartbeat_at),
    -                        })
    -                    },
    -                )
    +                    }
    +                })
                     .collect())
             })
         }
     
    +    fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>> {
    +        Box::pin(async move {
    +            // By status alone: a record stamped in this statement's instant,
    +            // or ahead of the database's clock, is RUNNING too.
    +            #[allow(clippy::type_complexity)]
    +            let rows: Vec<(
    +                String,
    +                String,
    +                String,
    +                String,
    +                String,
    +                String,
    +                i64,
    +                String,
    +            )> = sqlx::query_as(
    +                "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \
    +                     r.kind, r.status, r.owner_generation, r.heartbeat_at \
    +                     FROM table_restores r JOIN tables t ON t.table_id = r.table_id \
    +                     WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name",
    +            )
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            rows.into_iter().map(restore_job_row).collect()
    +        })
    +    }
    +
         /// Conditioned on RUNNING, on the listed `owner_generation` and on the
         /// listed `heartbeat_at` (compared as instants through `julianday`, so
         /// the text shape of the stored value does not matter), and written with
    @@ -905,12 +1060,130 @@ impl BackupCatalog for SqliteEngine {
             })
         }
     
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result> {
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
             Box::pin(async move {
    -            sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM backup_maintenance)")
    -                .fetch_one(&self.pool)
    +            sqlx::query_scalar(
    +                "SELECT DISTINCT store_id FROM backup_index \
    +                 WHERE store_kind = ? AND store_root = ? ORDER BY store_id",
    +            )
    +            .bind(store_kind.as_str())
    +            .bind(store_root)
    +            .fetch_all(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>> {
    +        Box::pin(async move {
    +            // One process serves a file (the serve lock), so the row is the
    +            // claim: taken under the engine write lock, refused while a row
    +            // with a live heartbeat exists, taken over from a dead one. An
    +            // engine that holds the serve lock (a command that opened the
    +            // file as `serve` does) is the only process on the file, so the
    +            // holder of any row is dead whatever its heartbeat says, as the
    +            // server's startup decides (`clear_maintenance_claim`): the row
    +            // is taken over.
    +            let _writer = self.write_lock.lock().await;
    +            let live: Option = if self.serve_lock.is_some() {
    +                None
    +            } else {
    +                sqlx::query_scalar(
    +                    "SELECT holder FROM backup_maintenance \
    +                     WHERE julianday(heartbeat_at) >= julianday('now') - ? / 86400.0",
    +                )
    +                .bind(stale_after.as_secs_f64())
    +                .fetch_optional(&self.pool)
                     .await
    -                .map_err(db_err)
    +                .map_err(db_err)?
    +            };
    +            if let Some(current) = live {
    +                return Err(StorageError::AlreadyExists(format!(
    +                    "the backup maintenance claim is held by {current}"
    +                )));
    +            }
    +            let generation = new_generation();
    +            sqlx::query(&format!(
    +                "INSERT OR REPLACE INTO backup_maintenance \
    +                 (singleton, holder, owner_generation, claimed_at, heartbeat_at) \
    +                 VALUES (1, ?, ?, {NOW}, {NOW})"
    +            ))
    +            .bind(holder)
    +            .bind(generation_column(generation))
    +            .execute(&self.pool)
    +            .await
    +            .map_err(db_err)?;
    +            Ok(Box::new(SqliteMaintenanceClaim {
    +                engine: self.clone(),
    +                generation,
    +            }) as Box)
    +        })
    +    }
    +
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            sqlx::query_scalar(
    +                "SELECT EXISTS(SELECT 1 FROM backup_maintenance \
    +                 WHERE julianday(heartbeat_at) >= julianday('now') - ? / 86400.0)",
    +            )
    +            .bind(stale_after.as_secs_f64())
    +            .fetch_one(&self.pool)
    +            .await
    +            .map_err(db_err)
    +        })
    +    }
    +}
    +
    +/// The holder's handle on the `backup_maintenance` row.
    +struct SqliteMaintenanceClaim {
    +    engine: SqliteEngine,
    +    generation: u64,
    +}
    +
    +impl MaintenanceClaim for SqliteMaintenanceClaim {
    +    fn generation(&self) -> u64 {
    +        self.generation
    +    }
    +
    +    fn heartbeat(&self) -> BoxFuture<'_, Result> {
    +        Box::pin(async move {
    +            let _writer = self.engine.write_lock.lock().await;
    +            let affected = sqlx::query(&format!(
    +                "UPDATE backup_maintenance SET heartbeat_at = {NOW} WHERE owner_generation = ?"
    +            ))
    +            .bind(generation_column(self.generation))
    +            .execute(&self.engine.pool)
    +            .await
    +            .map_err(db_err)?
    +            .rows_affected();
    +            Ok(if affected == 1 {
    +                Heartbeat::Confirmed
    +            } else {
    +                Heartbeat::Lost
    +            })
    +        })
    +    }
    +
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>> {
    +        Box::pin(async move {
    +            let _writer = self.engine.write_lock.lock().await;
    +            sqlx::query("DELETE FROM backup_maintenance WHERE owner_generation = ?")
    +                .bind(generation_column(self.generation))
    +                .execute(&self.engine.pool)
    +                .await
    +                .map_err(db_err)?;
    +            Ok(())
             })
         }
     }
    diff --git a/crates/storage-sqlite/src/config.rs b/crates/storage-sqlite/src/config.rs
    index 6ba092de0..b0d947d56 100644
    --- a/crates/storage-sqlite/src/config.rs
    +++ b/crates/storage-sqlite/src/config.rs
    @@ -69,6 +69,10 @@ impl StorageConfig for SqliteConfig {
             &self.path
         }
     
    +    fn database_file(&self) -> Option {
    +        crate::serve_lock::database_file(&self.path).ok().flatten()
    +    }
    +
         fn max_connections(&self) -> u32 {
             self.pool_size
         }
    diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs
    index 2e1401039..7c2188096 100644
    --- a/crates/storage-sqlite/src/lib.rs
    +++ b/crates/storage-sqlite/src/lib.rs
    @@ -212,18 +212,39 @@ fn sqlite_server_components_factory(
             // One server per database file, taken before the database is opened:
             // startup recovery below assumes no other server is using it, and a
             // migration holds the same lock. A file database only; an in-memory
    -        // one belongs to this process alone.
    -        let serve_lock = serve_lock::ServeLock::try_acquire_for_location(&db_path)
    -            .map_err(|failure| {
    -                BackendError::Catalog(match failure {
    -                    serve_lock::LockFailure::Held(m) => MigrationError::Locked(m),
    -                    serve_lock::LockFailure::Failed(m) => MigrationError::Store(m),
    +        // one belongs to this process alone. A command reading beside a
    +        // server (`options.beside_server`) takes no lock and runs none of
    +        // the recovery: the file is a served database and the server owns
    +        // both.
    +        let serve_lock = if options.beside_server {
    +            // The catalog's status is still read, on a read-only
    +            // connection: a catalog this binary cannot serve (behind, or
    +            // ahead with a migration that is not additive) is reported as
    +            // the server would report it, not as a missing table.
    +            match migrations::catalog_status(&db_path)
    +                .await
    +                .map_err(BackendError::Catalog)?
    +            {
    +                extenddb_storage::migrations::CatalogStatus::Current { .. }
    +                | extenddb_storage::migrations::CatalogStatus::AheadCompatible { .. } => {}
    +                status => {
    +                    return Err(BackendError::Catalog(MigrationError::Status(status)));
    +                }
    +            }
    +            None
    +        } else {
    +            serve_lock::ServeLock::try_acquire_for_location(&db_path)
    +                .map_err(|failure| {
    +                    BackendError::Catalog(match failure {
    +                        serve_lock::LockFailure::Held(m) => MigrationError::Locked(m),
    +                        serve_lock::LockFailure::Failed(m) => MigrationError::Store(m),
    +                    })
    +                })?
    +                .map(|lock| {
    +                    tracing::debug!("holding serve lock {}", lock.path().display());
    +                    std::sync::Arc::new(lock)
                     })
    -            })?
    -            .map(|lock| {
    -                tracing::debug!("holding serve lock {}", lock.path().display());
    -                std::sync::Arc::new(lock)
    -            });
    +        };
             // The catalog's status is read before the engine opens the file, so
             // a server that refuses has written nothing. A dev build migrates
             // instead, as `migrate --yes` would, copy included.
    @@ -240,6 +261,9 @@ fn sqlite_server_components_factory(
                         backend: "sqlite".to_owned(),
                         details: e.to_string(),
                     })?;
    +        if options.beside_server {
    +            return components_of(engine, &db_path).await;
    +        }
     
             // An in-memory database exists only on the engine's single
             // connection, so its catalog is created there, by the runner, every
    @@ -276,6 +300,17 @@ fn sqlite_server_components_factory(
                 }
             }
     
    +        // A command that writes under the serve lock (`backup purge
    +        // --confirm`, `options.startup_recovery` off) holds the file alone
    +        // and resolves what it needs itself (the reconciler's duties with a
    +        // zero window; its take of the maintenance claim takes over any row
    +        // under the lock): none of the server's recovery below runs, so the
    +        // command exits with none of it half done.
    +        if !options.startup_recovery {
    +            engine.serve_lock = serve_lock;
    +            return components_of(engine, &db_path).await;
    +        }
    +
             // Recover any in-flight control-plane transitions from a prior run.
             match engine.process_control_plane_transitions().await {
                 Ok(transitions) => {
    @@ -306,6 +341,19 @@ fn sqlite_server_components_factory(
                 Err(e) => tracing::error!("Failed to mark interrupted restores: {e}"),
             }
     
    +        // A maintenance claim (`backup purge`) the previous holder left: one
    +        // process has this file, so the holder is dead whatever the row's
    +        // heartbeat says, and the row would otherwise make the jobs refuse
    +        // for the whole stale window.
    +        match engine.clear_maintenance_claim().await {
    +            Ok(Some(holder)) => tracing::warn!(
    +                "the maintenance claim held by {holder} was left behind; cleared (one process \
    +                 serves this file, so its holder is gone)"
    +            ),
    +            Ok(None) => {}
    +            Err(e) => tracing::error!("Failed to clear a left-behind maintenance claim: {e}"),
    +        }
    +
             // Rebuild any GSI left mid-backfill (status CREATING) by a prior crash.
             match engine.reconcile_incomplete_gsis().await {
                 Ok(n) if n > 0 => tracing::info!("Reconciled {n} incomplete GSI(s) at startup"),
    @@ -325,59 +373,72 @@ fn sqlite_server_components_factory(
             }
     
             engine.serve_lock = serve_lock;
    -        let control_plane_notify = engine.control_plane_notify();
    -        let engine = Arc::new(engine);
    -
    -        // Catalog + auth queries: a file-backed database opens a dedicated
    -        // catalog pool over the same file; an in-memory database must reuse the
    -        // engine's single shared connection (its data lives only there).
    -        let catalog_pool = if in_memory {
    -            engine.pool.clone()
    -        } else {
    -            catalog_pool(&db_path)
    -                .await
    -                .map_err(|e| BackendError::ConnectionFailed {
    -                    backend: "sqlite".to_owned(),
    -                    details: format!("catalog pool: {e}"),
    -                })?
    -        };
    +        components_of(engine, &db_path).await
    +    })
    +}
     
    -        let enc_key: Option =
    -            sqlx::query_scalar("SELECT value FROM settings WHERE key = 'encryption_key'")
    -                .fetch_optional(&catalog_pool)
    -                .await
    -                .map_err(|e| {
    -                    BackendError::InitializationFailed(format!("fetch encryption key: {e}"))
    -                })?;
    -        let enc_key = enc_key.ok_or(BackendError::MissingEncryptionKey)?;
    -
    -        // The catalog store writes through the second pool over the same
    -        // file, so its writers must hold the SAME lock instance as the
    -        // engine's (design decision D1); an unshared lock would leave the two
    -        // pools contending at the SQLite level, where a slow commit exhausts
    -        // a concurrent writer's busy_timeout ("database is locked" -> 500).
    -        // For an in-memory database the pools are one and the sharing is
    -        // harmless. The credential store only reads and needs no lock.
    -        let catalog_store: Arc =
    -            Arc::new(SqliteCatalogStore::with_encryption_key(
    -                catalog_pool.clone(),
    -                enc_key.clone(),
    -                Arc::clone(&engine.write_lock),
    -            ));
    -        let credential_store: Arc =
    -            Arc::new(SqliteCredentialStore::new(catalog_pool, enc_key));
    -
    -        let runtime_hooks = Box::new(SqliteRuntimeHooks {
    -            engine: engine.clone(),
    -            control_plane_notify,
    -        });
    -
    -        Ok(ServerComponents {
    -            engine,
    -            catalog_store,
    -            credential_store,
    -            runtime_hooks: Some(runtime_hooks),
    -        })
    +/// The server components around an opened engine: the catalog and
    +/// credential stores over a second pool on the same file (the engine's own
    +/// connection for an in-memory database, whose data lives only there) and
    +/// the runtime hooks.
    +async fn components_of(
    +    engine: SqliteEngine,
    +    db_path: &str,
    +) -> Result {
    +    let in_memory = engine.in_memory;
    +    let control_plane_notify = engine.control_plane_notify();
    +    let engine = Arc::new(engine);
    +
    +    // Catalog + auth queries: a file-backed database opens a dedicated
    +    // catalog pool over the same file; an in-memory database must reuse the
    +    // engine's single shared connection (its data lives only there).
    +
    +    let catalog_pool = if in_memory {
    +        engine.pool.clone()
    +    } else {
    +        catalog_pool(db_path)
    +            .await
    +            .map_err(|e| BackendError::ConnectionFailed {
    +                backend: "sqlite".to_owned(),
    +                details: format!("catalog pool: {e}"),
    +            })?
    +    };
    +
    +    let enc_key: Option =
    +        sqlx::query_scalar("SELECT value FROM settings WHERE key = 'encryption_key'")
    +            .fetch_optional(&catalog_pool)
    +            .await
    +            .map_err(|e| {
    +                BackendError::InitializationFailed(format!("fetch encryption key: {e}"))
    +            })?;
    +    let enc_key = enc_key.ok_or(BackendError::MissingEncryptionKey)?;
    +
    +    // The catalog store writes through the second pool over the same
    +    // file, so its writers must hold the SAME lock instance as the
    +    // engine's (design decision D1); an unshared lock would leave the two
    +    // pools contending at the SQLite level, where a slow commit exhausts
    +    // a concurrent writer's busy_timeout ("database is locked" -> 500).
    +    // For an in-memory database the pools are one and the sharing is
    +    // harmless. The credential store only reads and needs no lock.
    +    let catalog_store: Arc =
    +        Arc::new(SqliteCatalogStore::with_encryption_key(
    +            catalog_pool.clone(),
    +            enc_key.clone(),
    +            Arc::clone(&engine.write_lock),
    +        ));
    +    let credential_store: Arc =
    +        Arc::new(SqliteCredentialStore::new(catalog_pool, enc_key));
    +
    +    let runtime_hooks = Box::new(SqliteRuntimeHooks {
    +        engine: engine.clone(),
    +        control_plane_notify,
    +    });
    +
    +    Ok(ServerComponents {
    +        engine,
    +        catalog_store,
    +        credential_store,
    +        runtime_hooks: Some(runtime_hooks),
         })
     }
     
    @@ -416,4 +477,63 @@ mod serve_lock_tests {
             drop(again);
             let _ = std::fs::remove_dir_all(&dir);
         }
    +
    +    /// A maintenance claim a killed `backup purge` left (its row's heartbeat
    +    /// young) is cleared when a server starts on the file: under the serve
    +    /// lock no holder can be alive, so the jobs' check answers free at once
    +    /// rather than after the stale window.
    +    #[tokio::test]
    +    async fn a_server_start_clears_a_left_behind_maintenance_claim() {
    +        let dir = std::env::temp_dir().join(format!(
    +            "extenddb-serve-clears-claim-{}",
    +            uuid::Uuid::new_v4().simple()
    +        ));
    +        std::fs::create_dir_all(&dir).expect("dir");
    +        let config = crate::config::SqliteConfig {
    +            path: dir.join("db.sqlite").to_string_lossy().into_owned(),
    +            pool_size: 2,
    +        };
    +        let mut options = extenddb_storage::server_components::ServerComponentsOptions::default();
    +        options.bootstrap_if_uninitialized = true;
    +        let stale = std::time::Duration::from_secs(300);
    +        let first = super::sqlite_server_components_factory(&config, "us-east-1", options)
    +            .await
    +            .expect("first server");
    +        let claim = first
    +            .engine
    +            .maintenance_claim("purge on this host (pid 1)", stale)
    +            .await
    +            .expect("claim");
    +        assert!(
    +            first
    +                .engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check")
    +        );
    +        // The holder dies without releasing: the row stays, heartbeat young.
    +        drop(claim);
    +        drop(first);
    +
    +        let second = super::sqlite_server_components_factory(&config, "us-east-1", options)
    +            .await
    +            .expect("starts once the first is gone");
    +        assert!(
    +            !second
    +                .engine
    +                .check_maintenance_claim(stale)
    +                .await
    +                .expect("check"),
    +            "the claim a dead holder left is cleared at startup, not honoured for the window"
    +        );
    +        // The claim is free to a new holder at once.
    +        let taken = second
    +            .engine
    +            .maintenance_claim("purge on this host (pid 2)", stale)
    +            .await
    +            .expect("a new holder takes the cleared claim");
    +        taken.release().await.expect("release");
    +        drop(second);
    +        let _ = std::fs::remove_dir_all(&dir);
    +    }
     }
    diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs
    index b30b79054..cd81e4e39 100644
    --- a/crates/storage/src/backup_catalog.rs
    +++ b/crates/storage/src/backup_catalog.rs
    @@ -542,6 +542,17 @@ pub struct StatusFields {
         pub completed: bool,
     }
     
    +/// The column an age in [`BackupCatalog::list_backup_rows_older_than`] is
    +/// measured on, against the database's own `now()`.
    +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
    +pub enum StaleClock {
    +    /// `heartbeat_at`: the owner's last sign of life.
    +    Heartbeat,
    +    /// `completed_at`: when the row reached its terminal status. A row with
    +    /// `completed_at` unset is never older than any age on this clock.
    +    Completed,
    +}
    +
     /// What a conditional write did.
     #[derive(Debug, Clone, Copy, PartialEq, Eq)]
     pub enum WriteOutcome {
    @@ -735,6 +746,16 @@ pub trait BackupCatalog: Send + Sync {
             store: &'a StoreIdentity,
         ) -> BoxStream<'a, Result>;
     
    +    /// The newest AVAILABLE row of `store` (the last in the
    +    /// (`created_at`, `backup_arn`) order [`Self::list_backup_rows`] lists
    +    /// in), or `None` when the store has none. The store's manifest probe
    +    /// reads it each reconciler cycle, so it reads one row and not the
    +    /// whole listing.
    +    fn newest_available_backup_row<'a>(
    +        &'a self,
    +        store: &'a StoreIdentity,
    +    ) -> BoxFuture<'a, Result, StorageError>>;
    +
         /// The account's rows, of `table_name` when given, created within
         /// `bounds`, of every status: the caller decides which statuses are
         /// listed ([`BackupStatus::is_listed`]).
    @@ -777,6 +798,32 @@ pub trait BackupCatalog: Send + Sync {
             key: &'a BackupKey,
         ) -> BoxFuture<'a, Result>;
     
    +    /// Remove a CREATING row whose owner is dead: `heartbeat_at` older than
    +    /// `stale_after` on the catalog's clock. `false` when the row is not
    +    /// CREATING, is gone, or has a live owner. The maintenance command's
    +    /// write for a CREATING row its wait saw no live owner for, after it has
    +    /// deleted the row's prefix; the row's own job, were it alive, would
    +    /// find its claim lost at its next heartbeat.
    +    fn remove_dead_creating_backup_row<'a>(
    +        &'a self,
    +        key: &'a BackupKey,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result>;
    +
    +    /// The rows of `store` with `status` whose `clock` column is older than
    +    /// `age` on the database's clock: the maintenance command's test of a
    +    /// live owner (a CREATING or FAILED row it does not list under the
    +    /// window on [`StaleClock::Heartbeat`] has one). The comparison is the
    +    /// backend's, in its own query, so the engine never compares a row's
    +    /// timestamp with a clock of its own.
    +    fn list_backup_rows_older_than<'a>(
    +        &'a self,
    +        status: BackupStatus,
    +        store: &'a StoreIdentity,
    +        clock: StaleClock,
    +        age: Duration,
    +    ) -> BoxStream<'a, Result>;
    +
         /// The name of a CREATING restore target registered from the backup, if
         /// any: the rule under which DeleteBackup and a second restore of the
         /// backup are refused with `BackupInUseException`. Reads the target's
    @@ -818,6 +865,12 @@ pub trait BackupCatalog: Send + Sync {
             stale_after: Duration,
         ) -> BoxFuture<'_, Result, StorageError>>;
     
    +    /// Every RUNNING restore job record, by status alone, whatever its
    +    /// heartbeat says: the maintenance command's wait set, which must see a
    +    /// record stamped in the listing's own instant (or ahead of the
    +    /// listing's clock) as well as an old one. Not a liveness test.
    +    fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>>;
    +
         /// RUNNING to FAILED with `reason` on the target's job record, conditional
         /// on RUNNING and on the record still carrying the `owner_generation` and
         /// `heartbeat_at` the reconciler listed (design 4(e)): a heartbeat that
    @@ -834,10 +887,69 @@ pub trait BackupCatalog: Send + Sync {
             reason: &'a str,
         ) -> BoxFuture<'a, Result>;
     
    -    /// Whether the catalog-level maintenance claim is held. Jobs check it
    -    /// after their own claim and row exist and refuse while it is held; the
    -    /// reconciler runs only the duties the holder waits on.
    -    fn check_maintenance_claim(&self) -> BoxFuture<'_, Result>;
    +    /// The distinct `store_id`s the catalog's rows record for a root, of
    +    /// every status: the store identity check reads them before it knows an
    +    /// id. Empty when no row names the root.
    +    fn read_store_ids<'a>(
    +        &'a self,
    +        store_kind: StoreKind,
    +        store_root: &'a str,
    +    ) -> BoxFuture<'a, Result, StorageError>>;
    +
    +    /// Take the catalog-level maintenance claim for `holder` (a label naming
    +    /// the command and the host it runs on, for the operator who finds it
    +    /// held). The claim is one record, `backup_maintenance`, born owned
    +    /// with a fresh generation and heartbeated through the handle; a holder
    +    /// whose heartbeat is older than `stale_after` on the catalog's clock is
    +    /// dead and its claim is taken over. PostgreSQL is the exception to the
    +    /// liveness rule on the holder's side: there the holder's session holds
    +    /// an advisory lock, a holder whose session ended is dead whatever its
    +    /// heartbeat says, and `stale_after` is read by [`check_maintenance_claim`]
    +    /// alone; on SQLite a server's startup under the serve lock removes the
    +    /// record outright, since no holder can be alive there.
    +    ///
    +    /// [`check_maintenance_claim`]: BackupCatalog::check_maintenance_claim
    +    ///
    +    /// # Errors
    +    ///
    +    /// `AlreadyExists` naming the holder when a live one holds the claim: a
    +    /// second maintenance command fails at once.
    +    fn maintenance_claim<'a>(
    +        &'a self,
    +        holder: &'a str,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'a, Result, StorageError>>;
    +
    +    /// Whether the catalog-level maintenance claim is held by a live holder:
    +    /// a record whose heartbeat is younger than `stale_after` on the
    +    /// catalog's clock. Jobs check it and refuse while it is held; the
    +    /// reconciler runs only the duties the holder waits on. Where in the
    +    /// job's order the check runs depends on the job: the in-catalog create
    +    /// and restore of this release commit their row inside the backend and
    +    /// detach, so the engine checks after every other refusal and before the
    +    /// row exists (the engine's backup module states it); the store-backed
    +    /// job of a later release owns its row from the engine and checks after
    +    /// its own claim and row exist (the design's 5.1 step 2).
    +    fn check_maintenance_claim(
    +        &self,
    +        stale_after: Duration,
    +    ) -> BoxFuture<'_, Result>;
    +}
    +
    +/// The holder's handle on the catalog-level maintenance claim: the
    +/// heartbeat that keeps it, and the release that ends it. Dropping the handle
    +/// without `release` leaves the record for the stale rule to expire.
    +pub trait MaintenanceClaim: Send + Sync {
    +    /// The fencing token the claim was taken with.
    +    fn generation(&self) -> u64;
    +
    +    /// Refresh `heartbeat_at` on the catalog's clock and confirm the claim is
    +    /// still this holder's. The caller treats an `Err` as [`Heartbeat::Lost`].
    +    fn heartbeat(&self) -> BoxFuture<'_, Result>;
    +
    +    /// Remove the record, conditioned on the generation; a record another
    +    /// holder took over is left to it. Idempotent.
    +    fn release(&self) -> BoxFuture<'_, Result<(), StorageError>>;
     }
     
     /// A job's handle on its row: the claim that excludes a second owner and
    diff --git a/crates/storage/src/config.rs b/crates/storage/src/config.rs
    index 64ffba9f5..21ff4914a 100644
    --- a/crates/storage/src/config.rs
    +++ b/crates/storage/src/config.rs
    @@ -59,6 +59,16 @@ pub trait StorageConfig: Send + Sync + std::fmt::Debug {
             Vec::new()
         }
     
    +    /// The local file the database lives in, for a backend whose database is
    +    /// one file on this host (SQLite with a file path); `None` for a server
    +    /// database and for an in-memory database. The backup configuration
    +    /// derives its per-backend defaults from it: a store root and a restore
    +    /// spool beside the file, and the rule that neither may sit in the
    +    /// file's directory. The default is `None`.
    +    fn database_file(&self) -> Option {
    +        None
    +    }
    +
         /// Clone this config into a boxed trait object.
         fn clone_box(&self) -> Box;
     
    diff --git a/crates/storage/src/lib.rs b/crates/storage/src/lib.rs
    index 7e1036021..79f8665a0 100755
    --- a/crates/storage/src/lib.rs
    +++ b/crates/storage/src/lib.rs
    @@ -38,7 +38,9 @@ pub mod vector_catalog;
     pub mod vector_lifecycle;
     
     pub use backend::{Backend, BackendAlreadySet, backend_name, set_backend, try_backend};
    -pub use backup_catalog::{BackupCatalog, JobClaim, JobPermit, LegacyBackupReader};
    +pub use backup_catalog::{
    +    BackupCatalog, JobClaim, JobPermit, LegacyBackupReader, MaintenanceClaim,
    +};
     pub use backup_definition::{BackupRestoreSource, RestoreRequest, RestoreTableOverrides};
     pub use snapshot::{SnapshotEngine, SnapshotInfo, TableSnapshot};
     
    diff --git a/crates/storage/src/server_components.rs b/crates/storage/src/server_components.rs
    index 427fec383..c685e383a 100644
    --- a/crates/storage/src/server_components.rs
    +++ b/crates/storage/src/server_components.rs
    @@ -105,6 +105,33 @@ pub struct ServerComponentsOptions {
         /// sizes it from this ([`crate::backup_catalog::pool_size_for_jobs`]);
         /// `0` (the default outside a server) asks for no job capacity.
         pub max_concurrent_jobs: usize,
    +    /// The caller is a command that reads the catalog beside a running
    +    /// server (`extenddb backup list`), not a server: no serve lock is taken
    +    /// (on SQLite the file is opened as another reader of a served
    +    /// database) and none of the startup recovery a server runs (see
    +    /// `startup_recovery`, which this implies off), since the server that
    +    /// owns the catalog does that. A command that writes (`backup purge`)
    +    /// leaves this `false`: on SQLite it takes the serve lock, as a server
    +    /// does.
    +    pub beside_server: bool,
    +    /// Whether the factory runs the startup recovery a server runs
    +    /// (interrupted control-plane transitions, incomplete indexes,
    +    /// PostgreSQL's GSI queue workers, MongoDB's repair of documents stored
    +    /// without their key, SQLite's expiry of interrupted restores and of a
    +    /// left-behind maintenance claim). `true` for a server; a command that
    +    /// writes (`backup purge`) sets it `false` beside `beside_server =
    +    /// false`, so on PostgreSQL, MongoDB and Cassandra it runs none of that
    +    /// work beside the live servers and exits with none of it half done,
    +    /// and on SQLite it holds the serve lock and resolves what it needs
    +    /// itself. Read only when `beside_server` is `false`.
    +    pub startup_recovery: bool,
    +    /// The command that opens the catalog, when it is not `serve`
    +    /// (`extenddb backup list`, `extenddb backup purge`). A backend whose
    +    /// factory takes a serving registration names it after the command
    +    /// (PostgreSQL's session `application_name`, MongoDB's and Cassandra's
    +    /// `extenddb_servers` entry), which a migration waiting for the
    +    /// registrations prints; `None` is `extenddb serve`.
    +    pub command: Option<&'static str>,
     }
     
     impl Default for ServerComponentsOptions {
    @@ -114,6 +141,9 @@ impl Default for ServerComponentsOptions {
                 max_item_size_bytes: extenddb_core::limits::LimitsConfig::default().max_item_size_bytes,
                 release: "",
                 max_concurrent_jobs: 0,
    +            beside_server: false,
    +            startup_recovery: true,
    +            command: None,
             }
         }
     }
    diff --git a/devtools/catalog_checks.py b/devtools/catalog_checks.py
    index b687def35..3dc664bf8 100644
    --- a/devtools/catalog_checks.py
    +++ b/devtools/catalog_checks.py
    @@ -13,6 +13,7 @@
     command's output indented under a FAIL; `finish` prints the count of failed
     checks or `all checks passed` and returns the script's exit code.
     """
    +import os
     import signal
     import subprocess
     import time
    @@ -125,6 +126,20 @@ def stop(proc: subprocess.Popen) -> None:
                 proc.wait()
     
     
    +def with_filesystem_store(config: str, work: str) -> tuple[str, str]:
    +    """A copy of `config` with a filesystem backup store at `/backups`
    +    (created `0700`), for the backup commands' store-configured runs; the
    +    copy's path and the root's canonical path, as the commands print it."""
    +    root = os.path.realpath(os.path.join(work, "backups"))
    +    os.mkdir(root, 0o700)
    +    store_config = os.path.join(work, "extenddb-store.toml")
    +    with open(config, encoding="utf-8") as f:
    +        base = f.read()
    +    with open(store_config, "w", encoding="utf-8") as f:
    +        f.write(f'{base}\n[backup.store]\nkind = "filesystem"\n\n[backup.store.filesystem]\npath = "{root}"\n')
    +    return store_config, root
    +
    +
     def finish() -> int:
         """Print the outcome of every check reported, and return the exit code."""
         if failures:
    @@ -162,7 +177,9 @@ def start_mongo(container: str, port: int, mongod_args: tuple[str, ...] = (), mo
         if started.returncode != 0:
             return report(False, f"start {container}", started.stderr)
         for _ in range(60):
    -        if mongosh(container, port, "db.runCommand({ping: 1}).ok")[1].strip().endswith("1"):
    +        # The whole output must be the `1`: a refused connection's error
    +        # names the port, and a port ending in 1 would pass an `endswith`.
    +        if mongosh(container, port, "db.runCommand({ping: 1}).ok")[1].strip() == "1":
                 break
             time.sleep(1)
         mongosh(container, port, f"rs.initiate({{_id: 'rs0', members: [{{_id: 0, host: 'localhost:{port}'}}]}})")
    diff --git a/devtools/check-cassandra-catalog-lock b/devtools/check-cassandra-catalog-lock
    index d27061206..a0d578823 100755
    --- a/devtools/check-cassandra-catalog-lock
    +++ b/devtools/check-cassandra-catalog-lock
    @@ -49,7 +49,17 @@ import tempfile
     import time
     import tomllib
     
    -from catalog_checks import expect_exit, finish, healthy, report, run, start, stop, watch_lost_registration
    +from catalog_checks import (
    +    expect_exit,
    +    finish,
    +    healthy,
    +    report,
    +    run,
    +    start,
    +    stop,
    +    watch_lost_registration,
    +    with_filesystem_store,
    +)
     
     ADMIN_USER = os.environ.get("CASSANDRA_ADMIN_USER", "cassandra")
     ADMIN_PASSWORD = os.environ.get("CASSANDRA_ADMIN_PASSWORD", "cassandra")
    @@ -230,6 +240,47 @@ def main(binary: str, container: str, contact_point: str, port: int) -> int:
             )
             report(time.monotonic() - started < 30, "the stopped server's registration was removed")
     
    +        # The backup commands register as a server does and release the
    +        # registration when they exit, whatever the outcome, so a migration
    +        # right after them does not wait for the row's TTL. With no store
    +        # configured the command releases on its no-store path before it
    +        # prints; with a filesystem store (a root under the work directory)
    +        # it opens the store, runs, and releases when it closes the catalog,
    +        # the path a store-backed deployment takes.
    +        backup = [binary, "backup", "--config", config]
    +        expect_exit(0, "backup list with no server running", backup + ["list"], env, contains="none (")
    +        expect_exit(
    +            0,
    +            "backup purge's dry run with no server running",
    +            backup + ["purge"],
    +            env,
    +            contains="Dry run: nothing was deleted",
    +        )
    +        store_config, root = with_filesystem_store(config, work)
    +        backup = [binary, "backup", "--config", store_config]
    +        expect_exit(
    +            0,
    +            "backup list with a filesystem store configured",
    +            backup + ["list"],
    +            env,
    +            contains=f"=== backups in the filesystem store {root}",
    +        )
    +        expect_exit(
    +            0,
    +            "backup purge's dry run with a filesystem store configured",
    +            backup + ["purge"],
    +            env,
    +            contains="Dry run: nothing was deleted. Run again with --confirm",
    +        )
    +        started = time.monotonic()
    +        expect_exit(
    +            0,
    +            "migrate --yes right after the backup commands, without waiting for a registration's TTL",
    +            migrate + ["--yes", "--lock-timeout", "5"],
    +            env,
    +        )
    +        report(time.monotonic() - started < 30, "the backup commands' registration was removed")
    +
             keyspace = f"{stem}_catalog"
             code, output = cqlsh(
                 container,
    diff --git a/devtools/check-mongodb-catalog-lock b/devtools/check-mongodb-catalog-lock
    index f13f764e6..762b23fdc 100755
    --- a/devtools/check-mongodb-catalog-lock
    +++ b/devtools/check-mongodb-catalog-lock
    @@ -14,7 +14,9 @@ extenddb-check-lock-, published on 127.0.0.1 only:
     - the migrate preview and verify run beside a server and exit 0, and
       migrate --yes exits 6 naming the server's host and pid;
     - a server stopped with SIGTERM deletes its registration, so migrate --yes
    -  then runs at once;
    +  then runs at once; `backup list` and `backup purge`'s dry run, with and
    +  without a store configured, release the registration they take, so
    +  migrate --yes right after them runs at once too;
     - while a migration holds the lock, a server exits 6 and a second
       migrate --yes exits 6, and the preview still runs;
     - a server refuses a catalog behind this binary with 3, and the command it
    @@ -51,6 +53,7 @@ from catalog_checks import (
         start_mongo,
         stop,
         watch_lost_registration,
    +    with_filesystem_store,
     )
     
     
    @@ -115,6 +118,30 @@ def main(binary: str, mongo_port: int, api_port: int) -> int:
             report(live_servers(container, mongo_port) == 0, "the stopped server deleted its registration")
             expect_exit(0, "migrate --yes once the server stopped", migrate + ["--yes", "--lock-timeout", "2"], env)
     
    +        # The backup commands register as a server does and release the
    +        # registration when they exit, whatever the outcome, so a migration
    +        # right after them does not wait for the document to expire 45 s
    +        # after its last renewal. With no store configured the command
    +        # releases on its no-store path before it prints; with a filesystem
    +        # store (a root under the work directory) it opens the store, runs,
    +        # and releases when it closes the catalog, the path a store-backed
    +        # deployment takes.
    +        backup = [binary, "backup", "--config", "extenddb.toml"]
    +        expect_exit(0, "backup list with no server running", backup + ["list"], env, contains="none (")
    +        expect_exit(0, "backup purge's dry run with no server running", backup + ["purge"], env,
    +                    contains="Dry run: nothing was deleted")
    +        store_config, root = with_filesystem_store(os.path.join(work, "extenddb.toml"), work)
    +        backup = [binary, "backup", "--config", store_config]
    +        expect_exit(0, "backup list with a filesystem store configured", backup + ["list"], env,
    +                    contains=f"=== backups in the filesystem store {root}")
    +        expect_exit(0, "backup purge's dry run with a filesystem store configured", backup + ["purge"], env,
    +                    contains="Dry run: nothing was deleted. Run again with --confirm")
    +        report(live_servers(container, mongo_port) == 0, "the backup commands released their registration")
    +        started = time.monotonic()
    +        expect_exit(0, "migrate --yes right after the backup commands", migrate + ["--yes", "--lock-timeout", "2"],
    +                    env)
    +        report(time.monotonic() - started < 20, "migrate --yes did not wait for a backup command's registration")
    +
             # Another migration's lock, as its claim writes it.
             mongosh(container, mongo_port, "db.getSiblingDB('extenddb_catalog').settings.insertOne("
                     "{_id: 'migration_lock', owner: 'check', expires_at: new Date(Date.now() + 120000)})")
    diff --git a/devtools/check-postgres-catalog-lock b/devtools/check-postgres-catalog-lock
    index 81a76f794..6fbb5693e 100755
    --- a/devtools/check-postgres-catalog-lock
    +++ b/devtools/check-postgres-catalog-lock
    @@ -45,7 +45,7 @@ import urllib.parse
     
     import psycopg2
     
    -from catalog_checks import expect_exit, finish, healthy, report, run, start, stop
    +from catalog_checks import expect_exit, finish, healthy, report, run, start, stop, with_filesystem_store
     
     NAMESPACE = 0x0045_4442  # the advisory key namespace of crates/storage-postgres
     MIGRATION_KEY = 1
    @@ -174,6 +174,46 @@ def main(binary: str, admin_url: str, port: int) -> int:
             )
             stop(server)
     
    +        # The backup commands register as a server does (the session's
    +        # application_name) and release the registration when they exit,
    +        # whatever the outcome, so a migration right after them finds no
    +        # session to wait for. With no store configured the command releases
    +        # on its no-store path before it prints; with a filesystem store (a
    +        # root under the work directory) it opens the store, runs, and
    +        # releases when it closes the catalog, the path a store-backed
    +        # deployment takes.
    +        backup = [binary, "backup", "--config", config]
    +        expect_exit(0, "backup list with no server running", backup + ["list"], env, contains="none (")
    +        expect_exit(
    +            0,
    +            "backup purge's dry run with no server running",
    +            backup + ["purge"],
    +            env,
    +            contains="Dry run: nothing was deleted",
    +        )
    +        store_config, root = with_filesystem_store(config, work)
    +        backup = [binary, "backup", "--config", store_config]
    +        expect_exit(
    +            0,
    +            "backup list with a filesystem store configured",
    +            backup + ["list"],
    +            env,
    +            contains=f"=== backups in the filesystem store {root}",
    +        )
    +        expect_exit(
    +            0,
    +            "backup purge's dry run with a filesystem store configured",
    +            backup + ["purge"],
    +            env,
    +            contains="Dry run: nothing was deleted. Run again with --confirm",
    +        )
    +        expect_exit(
    +            0,
    +            "migrate --yes right after the backup commands",
    +            migrate + ["--yes", "--lock-timeout", "1"],
    +            env,
    +        )
    +
             holder = psycopg2.connect(f"{admin_url.rstrip('/')}/{stem}_catalog")
             with holder.cursor() as cur:
                 cur.execute("SELECT pg_advisory_xact_lock(%s, %s)", (NAMESPACE, MIGRATION_KEY))
    diff --git a/devtools/run-tests b/devtools/run-tests
    index ae7984896..f923281c3 100755
    --- a/devtools/run-tests
    +++ b/devtools/run-tests
    @@ -467,8 +467,10 @@ if $RUN_PYTEST; then
             echo "  endpoint: $EXTENDDB_TEST_ENDPOINT"
         fi
         echo "  AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}"
    -    # CLI lifecycle tests manage their own server; exclude from main suite
    -    PYTEST_ARGS=(python3 -m pytest tests/ -v --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py)
    +    # CLI lifecycle tests manage their own server; exclude from main suite.
    +    # tests/test_backup_store_config.py does too: it runs below, in the CLI
    +    # lifecycle list on PostgreSQL and in its own step on a SQLite build.
    +    PYTEST_ARGS=(python3 -m pytest tests/ -v --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py --ignore=tests/test_backup_store_config.py)
         # Parallel execution (--parallel): distribute by file so module/class-scoped
         # fixtures stay within a single worker.
         if [[ -n "$PARALLEL" ]] && python3 -c "import xdist" 2>/dev/null; then
    @@ -560,7 +562,7 @@ if $RUN_PYTEST \
             && -n "${EXTENDDB_TEST_PG_CONNECTION_STRING:-}" ]]; then
         CLI_OUTFILE="discussions/test-cli-${HASH}.txt"
         echo "=== CLI lifecycle tests → ${CLI_OUTFILE} ==="
    -    CLI_ARGS=(python3 -m pytest tests/test_cli_lifecycle.py tests/test_cli_container_readiness.py tests/test_cli_migrate_concurrency.py tests/test_cli_vector_catalog_migration.py -v)
    +    CLI_ARGS=(python3 -m pytest tests/test_cli_lifecycle.py tests/test_cli_container_readiness.py tests/test_cli_migrate_concurrency.py tests/test_cli_vector_catalog_migration.py tests/test_backup_store_config.py -v)
         if [[ -n "$FILTER" ]]; then
             CLI_ARGS+=(-k "$FILTER")
         fi
    @@ -573,6 +575,37 @@ if $RUN_PYTEST \
         echo ""
     fi
     
    +# --- Backup store wire tests on a SQLite build (run last — they start/stop their own servers) ---
    +# tests/test_backup_store_config.py's SQLite cells: the default root, the
    +# S3 store's health against moto (skipped with a reason unless
    +# EXTENDDB_TEST_S3_ENDPOINT names a moto endpoint), `backup list` and
    +# `backup purge` with their signals and exit codes. Its PostgreSQL cells run
    +# in the CLI lifecycle list above. The build decides: the step runs when the
    +# binary reports itself a SQLite build.
    +if $RUN_PYTEST && [[ "$TARGET" != "real-dynamodb" ]]; then
    +    WIRE_BINARY="${EXTENDDB_BINARY:-target/release/$TARGET}"
    +    if [[ ! -x "$WIRE_BINARY" ]]; then
    +        WIRE_BINARY="target/debug/$TARGET"
    +    fi
    +    if [[ -x "$WIRE_BINARY" ]] && "$WIRE_BINARY" version 2>/dev/null | grep -q '(sqlite)'; then
    +        WIRE_OUTFILE="discussions/test-backup-store-wire-${HASH}.txt"
    +        echo "=== Backup store wire tests (SQLite build) → ${WIRE_OUTFILE} ==="
    +        echo "  binary: $WIRE_BINARY"
    +        echo "  S3 endpoint: ${EXTENDDB_TEST_S3_ENDPOINT:-none (the S3 cells skip)}"
    +        WIRE_ARGS=(python3 -m pytest tests/test_backup_store_config.py -v)
    +        if [[ -n "$FILTER" ]]; then
    +            WIRE_ARGS+=(-k "$FILTER")
    +        fi
    +        if EXTENDDB_BINARY="$WIRE_BINARY" "${WIRE_ARGS[@]}" 2>&1 | tee "$WIRE_OUTFILE"; then
    +            echo "  ✓ Backup store wire tests passed"
    +        else
    +            echo "  ✗ Backup store wire tests failed"
    +            PASS=false
    +        fi
    +        echo ""
    +    fi
    +fi
    +
     # --- Catalog check ---
     if $RUN_CATALOG_CHECK; then
         if [[ "$TARGET" == "real-dynamodb" ]]; then
    diff --git a/docs/design/13-storage-mongodb.md b/docs/design/13-storage-mongodb.md
    index b1219ccbc..cc4f74af3 100644
    --- a/docs/design/13-storage-mongodb.md
    +++ b/docs/design/13-storage-mongodb.md
    @@ -837,6 +837,13 @@ opens the engine, and refuses a catalog with pending migrations or with a
     breaking migration this binary does not know, as on every backend.
     [ADR-0023](../adr/0023-catalog-migrations.md) records the design.
     
    +The factory registers and reads the status before it builds the engine
    +or starts the startup key repair, so a server that refuses has written
    +nothing but its registration, which it deletes. A command that opens
    +the catalog beside a server (`ServerComponentsOptions::beside_server`:
    +`extenddb backup list`, the purge's dry run) starts no repair; the
    +server does.
    +
     ## 6. Crate Structure
     
     ```
    diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md
    index 06c13ece0..020f068d0 100644
    --- a/docs/design/15-backup-restore.md
    +++ b/docs/design/15-backup-restore.md
    @@ -100,7 +100,7 @@ The invariants and state machines every later PR honours, by lettered part; the
     
     (e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the one liveness rule the three share. On PostgreSQL the advisory lock is the claim's proof against a second owner, not the liveness test: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler reads the heartbeat alone and never probes the lock. Staleness is always measured on the database's clock, never on an application host's. The reconciler takes the same claim as a worker, with a fresh `owner_generation` that fences the dead job's generation out, leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write errors with an unknown outcome, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7). A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state.
     
    -(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING backup row with a live owner, no DELETING row of the configured store and no RUNNING `restore_jobs` row remains, and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other.
    +(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4; this release's in-catalog create and restore check before their row, since the backends commit it and detach, revision item 29) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other.
     
     (g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and marks it DELETING only with `--prune-missing`; without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root.
     
    @@ -115,7 +115,7 @@ Per operation, the contract: the request checks in their order, the error each r
     #### 5.1 CreateBackup
     
     1. Validate the request as #384 does and in its order, every check before the permit: `TableName` resolved (a table ARN is accepted), the table exists and is ACTIVE for the account (`TableNotFoundException`, "Table not found: ", otherwise, whether or not a permit is free), the server's region fits a backup ARN (`check_backup_arn_fits`: a `server.region` outside `[a-z0-9-]` is a ValidationException naming the setting, since no backup ARN can be formed for any table under it). The store-prefix rules on the name (over 255 bytes, or made only of dots; `check_store_prefix_table_name`) are the store-backed writer's (PR 8), applied before it writes and naming the rule whatever the name's length: the in-catalog create writes no prefix, and the service backs up any table it can create (measured: a table named `...` backs up), so the engine applies no name rule in front of `LegacyBackupReader::create`. #384's `refuse_backup_of_vector_table` is not among the off-host path's checks (the manifest records vector indexes, section 1); the engine keeps the refusal in front of `LegacyBackupReader::create`, the path of every backend without `[backup.store]` and of Cassandra always (section 7), and the tests that change are PR 8's and PR 9a's sections. A second CreateBackup on a table that already has a CREATING backup is allowed and takes its own snapshot; the `BackupInUseException` the service models on CreateBackup is not raised. Then reserve a job slot: `max_concurrent_jobs` is a per-instance semaphore the request acquires a permit from, waiting up to 30 s for one (`JobPermit::acquire_within`, `PERMIT_WAIT`; the bound the PostgreSQL and MongoDB restore slots of earlier releases waited, and the service accepts bursts over the count: five restores and six CreateBackup calls sent together were all accepted), the maintenance claim of section 4 being the cross-instance exclusion, and the permit lives with the job; if none is freed within the bound return `LimitExceededException` before opening anything. The request holds nothing while it waits, since the permit is the last of its checks and the snapshot is opened after it; RestoreTableFromBackup and the legacy create take a permit from the same semaphore and return the same error (5.3 step 1, section 3). The default is 4 (section 7), so one running restore does not turn every other backup operation on the instance into `LimitExceededException`.
    -2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, so the backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` taken from the snapshot's description; then check the maintenance claim (section 4): if it is held, set the row FAILED with reason `maintenance`, release the snapshot, the lock and the permit, and return `LimitExceededException`; otherwise return `BackupDetails` with BackupStatus CREATING. Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written.
    +2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, so the backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` taken from the snapshot's description; then check the maintenance claim (section 4; this release's in-catalog create checks it before its row, revision item 29): if it is held, set the row FAILED with reason `maintenance`, release the snapshot, the lock and the permit, and return `LimitExceededException`; otherwise return `BackupDetails` with BackupStatus CREATING. Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written.
     3. The claim: section 4(e) states the rules; what the claim is on each backend, the claim connection's settings and the hand-off to the worker are PR 6's and PR 8's sections.
     4. The job streams items from the snapshot into gzip data files through `DataFileEncoder`; a file closes once its `uncompressed_bytes` reaches `data_file_target_bytes` (default 64 MiB), so a file exceeds the knob by less than one line, at most `MAX_LINE_BYTES` (32 MiB, section 1), which is also the knob's minimum. Every item is checked, before it is encoded, with the item validator PR 8 adds to `extenddb-core` (`extenddb_core::validation::validate_restorable_item`, with this instance's `max_item_size_bytes`; the restore binds the same function, 5.3 step 5), so no backup that no restore can load is reported AVAILABLE; that is a regression from main for a table that already holds a value #618 let in (Drawbacks), and the reason reaches the operator only (`backup list --failed` and the log); the client sees CREATING and then BackupNotFound. The heartbeat is written on a timer (`heartbeat_interval_secs`, default 15) for as long as the job task is alive, independent of upload progress; before each put the job checks that its latest heartbeat confirmed the claim within `heartbeat_interval_secs` and starts no put otherwise; a heartbeat that does not confirm the claim, or that has not returned within `heartbeat_interval_secs`, aborts the job without further store writes, deletes included. `stale_job_after_secs` must exceed several heartbeat periods; startup refuses a value under three times the period the jobs run at, which until the knob reaches the jobs is the larger of `heartbeat_interval_secs` and the 15 s constant `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL` every backend's heartbeat loop reads (so 45 s at the least), which is what makes that detection precede a reconciler's deletion. After the last file: write `manifest-files.json`, `manifest-summary.json`, `extenddb-manifest.sha256` (the order #394's `BackupDocuments::build` doc gives); re-check the claim; then write `extenddb-manifest.json` and update the row to AVAILABLE with size and count (`JobClaim::set_status(CREATING, AVAILABLE)`, conditional on the generation and on `status = CREATING`). An update whose outcome is unknown is followed by a re-read of the row, not treated as a failure; one that matches zero rows ends the job with no store access. Store writes are not fenced by the generation; the re-check narrows the window in which a fenced-out job can still write a manifest, and the FAILED tombstone (section 4) closes what remains: sync never adopts a prefix whose row is FAILED, and the reconciler never deletes a prefix while the row's `heartbeat_at` is younger than `stale_job_after_secs` on the catalog database's clock, so a late manifest from a fenced-out job lands under a tombstone that sync will not adopt.
     5. On any failure, including `SnapshotExpired`: set the row FAILED with the reason, conditional on the generation and on `status = CREATING`, so a job whose AVAILABLE update committed with its response lost cannot turn a backup a client has seen AVAILABLE into a deleted one. Only if that write reports exactly one row matched does the job stop its uploads, await its in-flight puts and `delete_prefix` the partial backup (which on S3 also aborts any upload still open under it); a write that matches zero rows, or any error on the claim connection, ends the job with no further store access of any kind, because the holder of the claim owns the prefix. If the delete fails the reconciler retries it (section 4). A backup never resumes from a cursor position, so a lost snapshot means a new CreateBackup. DescribeBackup returns BackupNotFound; ListBackups omits it.
    @@ -124,7 +124,7 @@ Per operation, the contract: the request checks in their order, the error each r
     #### 5.2 DescribeBackup, ListBackups, DeleteBackup
     
     Served from the catalog index with the legacy fallback. DescribeBackup reads the row by the ARN's account and backup id (`BackupCatalog::read_backup_row`) and returns BackupNotFoundException unless the row's `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, as 5.3 step 1 does. ListBackups implements the full request as #384 implements it (the members, the (`created_at`, `backup_arn`) order and the cursor rules with #384's two exceptions), except that `TableName` may be a table name or a table ARN, as the service accepts it (measured on DynamoDB) and as main's CreateBackup already does; the differences doc's `ListBackups` paging row changes with PR 6 (section 10).
    -DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock, so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; if any step fails the row stays DELETING and the API returns InternalServerError, and the reconciler finishes the deletion later (section 4). The request returns the description with BackupStatus DELETED after the row reaches DELETED.
    +DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock, so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; the store calls run under the store's check bound and in-flight rule (section 7); the request answers the description with BackupStatus DELETED once the DELETING write has committed, whatever the deletion does (every reader reports the backup deleted from that write on, and the service has no counterpart for a failed object deletion); if a step fails the row stays DELETING, the failure is logged, and the reconciler finishes the deletion later (section 4) (revision item 30).
     
     #### 5.3 RestoreTableFromBackup
     
    @@ -133,7 +133,7 @@ The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the
     1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name's syntax; the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`; then the target name under the account (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target), after the backup is known to exist and before its in-use rule and the overrides are checked, the order the service uses (a restore of a missing backup into an existing table, into a table CreateTable is still creating or into a CREATING restore target is `BackupNotFoundException`, an in-use backup into an existing table `TableAlreadyExistsException` and a backup into its own CREATING target `TableInUseException`, all measured); then, for a row, a backup in use (the rule of 5.2) returns `BackupInUseException` naming the backup, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is freed within the 30 s wait of 5.1). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`.
     2. Validate that the target backend supports every feature in `source_table` that a restore reproduces (a non-empty `vector_indexes` into a backend without vector support is the one case, section 1); refuse before creating anything otherwise. Unlike #384 (`ensure_single_part_base_key`), this path restores multi-part-key tables (as `unordered` backups, section 1): `bulk_load` keys each row and derives its index rows with the functions the backend's item write path uses (section 3), so a restored multi-part-key table is laid out as a natively created one.
     3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place and no copy is made. A transport error during a download is retried with backoff for up to 15 minutes (a constant, not a knob in v1), the row kept RUNNING and heartbeated, before the restore fails (step 7); a checksum or size mismatch is not retried and fails the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section.
    -4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle.
    +4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 29); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle.
     5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed`, apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through `bulk_load(target_table_id, claim, file_index, start_at, items)` (section 3), with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise. Each backend writes base rows and index rows together and records progress so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL and SQLite, and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra (PR 9a's section: the batch sizes and progress records). The restored table has no streams and no TTL, so there is nothing to bypass.
     6. (Job.) After all files, and for an unordered backup after the count of section 1 matched, the job flips the target to ACTIVE conditional on the table still being CREATING as a restore target and on the job's `owner_generation`; on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE. MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction, and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today.
     7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with only its spool removed, because the claim holder owns the table. DescribeTable then returns ResourceNotFound. Crash mid-restore: the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (section 4(e)) and resumes from the recorded progress, reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not, and resuming nothing while the configured store is unhealthy; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). Shutdown: a restore job stops at its next batch without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule.
    @@ -167,11 +167,11 @@ spool_path = ""            # restore download area for an S3 store; d
     
     `[backup.store]` and its `filesystem` and `s3` tables are the crate's `BackupStoreConfig` (`crates/backup-store/src/config.rs` module doc, #393: the TOML block with every key, its default and `open`'s refusals); `kind = "disabled"`, the per-backend default root, the spool and `backup purge` are PR 7's section.
     
    -On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there.
    +On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there.
     
     Memory per running job, which operators size hosts from: a backup holds the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store), the read path, the gzip window and the manifest's data-file entries until the manifest is written; with the byte budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) the read path holds at most one page, the budget plus one row of text, plus its decoded form, and neither term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today.
     
    -A configuration error (a missing or denied bucket, a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above, where it is a warning. A transport failure (no region, no credentials, an unreachable endpoint), and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy, the operations that touch the store refuse with a ValidationException that names the store's kind and not its root (CreateBackup on the off-host path, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and Cassandra's legacy create are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes.
    +A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (RestoreTableFromBackup and DeleteBackup of an off-host backup, and CreateBackup once it writes to the store in PR 8; Describe and List, legacy backups and this release's in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes.
     
     Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job keys; PR 7 adds `store` and its validation) and `BackupStoreConfig` in `crates/backup-store`; defaults above are the ones they encode.
     
    @@ -253,7 +253,7 @@ backup_id         14-digit zero-padded epoch millis of the snapshot instant, `-`
     account_id, table_name, backup_arn, backup_name,
     status            CREATING | AVAILABLE | DELETING | DELETED | FAILED
     store_kind        filesystem | s3
    -store_root        the configured root at creation time: the canonical directory for a filesystem store, `s3:///` for S3, with `?endpoint=` appended when an endpoint is configured (the URL without userinfo, query or fragment)
    +store_root        the configured root at creation time: the canonical directory for a filesystem store, `s3:///` for S3, with `?endpoint=` appended when the client uses an endpoint other than the SDK's default (configured, or from the SDK's environment and shared config), the URL in one spelling: lowercase scheme and host, no default port, no `/` for an empty path, no userinfo, query or fragment
     store_id          the id in the store's `extenddb-store.json` (store identity, below)
     size_bytes, item_count, created_at (= backup_creation_date_time, the id's instant), completed_at, failure_reason,
     source_table_details_json   the manifest's source_table object, verbatim
    @@ -310,7 +310,7 @@ pub trait BackupCatalog: Send + Sync {
             // RUNNING to FAILED with the reason, conditional on RUNNING and on the record still carrying the `owner_generation`
             // and `heartbeat_at` the reconciler listed (4(e): a heartbeat that lands after the listing makes it match nothing),
             // written with a fresh generation; `false` when not RUNNING under those values or gone; for a dead legacy restore
    -    fn check_maintenance_claim(&self) -> BoxFuture>;   // true while one is held; jobs call it
    +    fn check_maintenance_claim(&self, stale_after: Duration) -> BoxFuture>;   // true while a live holder has it; jobs call it with stale_job_after_secs
     }
     
     pub trait LegacyBackupReader: Send + Sync {
    @@ -337,7 +337,7 @@ Documentation rows it corrects: the differences doc's `ListBackups` paging row's
     ### PR 7: configuration, the store identity file, `backup purge`
     
     Goal: the `[backup.store]` configuration with its validation, the store identity file and the health component, and `backup purge`. Its line names the interfaces and carries the acceptance:
    -- PR 7: configuration: `store` (`[backup.store]`) added to the `[backup]` struct PR 6 adds, with its validation (section 7, including the absent-section default and its warning-only validation, `kind = "disabled"`, the legacy create as the fallback without a store, the spool path rule and the spool `.lock`, the pool-size check), init template, both banners, store `validate()` at startup with the store-identity check around it (`extenddb-store.json` created with `put_if_absent`, which this PR adds to the `BackupStore` trait and both stores, then read and compared with the rows through `read_store_ids` and with the catalog's `catalog_id`; the manifest probe; both run again each reconciler cycle, and a failure marks the store unhealthy rather than stopping startup), the configuration-error versus transport-failure split, `/health` component, destroy semantics, `backup purge` deleting each configured-store row's prefix and the row. Acceptance: a bad bucket or an explicitly configured unwritable root fails startup with the name in the message; a config without `[backup]` starts and creates the default root `0700`; a config without `[backup]` on SQLite under a group-writable parent starts, logs the warning and leaves the store unconfigured, so CreateBackup there uses the legacy create with its warning, while the same root configured explicitly fails startup; an S3 store whose endpoint is unreachable at startup starts with the store component unhealthy and CreateBackup refusing, and recovers when the endpoint answers; a spool path inside or containing the root is refused, a spool path another live instance holds is refused, and on PostgreSQL an S3 store without `spool_path` fails startup; a backup created on `sqlite-memory` with `[backup.store.filesystem] path` set restores, and without `[backup.store]` CreateBackup there creates a legacy backup with the warning; a root whose `extenddb-store.json` names another catalog, carries another id than the rows, or is missing while rows exist starts with the store unhealthy (logged, `/health` unhealthy, CreateBackup, an off-host restore and DeleteBackup of an off-host backup refused naming the store's kind and not its root, the reconciler acting on no row of the store) and recovers once the file is restored; two instances of one catalog started against one empty root end with one id in the file and on the rows; two instances of one catalog with host-local filesystem roots at one path: the second, with no file, is unhealthy, and with the first's file copied to it the manifest probe marks it unhealthy; on either store a `put_if_absent` of a key that exists returns `None` and leaves the first object; purge takes the maintenance claim, waits for a CreateBackup running on a second instance, no job starts while it holds the claim, and afterwards no row of the configured store remains while `extenddb-store.json` and a prefix with no row do; PostgreSQL without `[backup.store]` creates an in-catalog backup, logs the deprecation warning naming `[backup.store]` and keeps serving legacy backups. Also `extenddb backup list` and `--failed`, printing `store_root`. Depends on PR 3.
    +- PR 7: configuration: `store` (`[backup.store]`) added to the `[backup]` struct PR 6 adds, with its validation (section 7, including the absent-section default and its warning-only validation, `kind = "disabled"`, the legacy create as the fallback without a store, the spool path rule and the spool `.lock`, the pool-size check), init template, both banners, store `validate()` with the store-identity check around it, before the listener for a filesystem store and in the reconciler's first cycle for an S3 store, the store `checking` until then (`extenddb-store.json` created with `put_if_absent`, which this PR adds to the `BackupStore` trait and both stores, then read and compared with the rows through `read_store_ids` and with the catalog's `catalog_id`; the manifest probe; both run again each reconciler cycle, and a failure marks the store unhealthy rather than stopping startup), the configuration-error versus transport-failure split, `/health` component, destroy semantics, `backup purge` deleting each configured-store row's prefix and the row. Acceptance: a bucket name `open` refuses or an explicitly configured unwritable root fails startup with the name in the message, and a missing or denied bucket makes the store unhealthy after the listener; a config without `[backup]` starts and creates the default root `0700`; a config without `[backup]` on SQLite under a group-writable parent starts, logs the warning and leaves the store unconfigured, so CreateBackup there uses the legacy create with its warning, while the same root configured explicitly fails startup; a row seeded under the defaulted SQLite root with its manifest missing starts the server with the store configured and unhealthy (`/health` degraded, `manifest probe failed`), the row's restore and delete refused naming the kind and the category, `backup purge --confirm` and `backup list` reaching the store, and the store healthy again once the manifest is back; a relative `[backup.store.filesystem] path` under daemon-mode `serve` resolves against the shell's directory; an S3 store whose endpoint is unreachable at startup serves with the store component `checking` and then unhealthy, the in-catalog CreateBackup still answering, a stalled endpoint holding the listener for no longer than its own startup and each reconciler cycle for no longer than the check's 30 s bound (the restore of a seeded row refused naming the pending check, then `unreachable`; the next cycle's check reaching the endpoint within seconds of the abandoned one), and recovers when the endpoint answers; a spool path inside or containing the root is refused, a spool path another live instance holds is refused, and on PostgreSQL an S3 store without `spool_path` fails startup; a backup created on `sqlite-memory` with `[backup.store.filesystem] path` set restores, and without `[backup.store]` CreateBackup there creates a legacy backup with the warning; a root whose `extenddb-store.json` names another catalog, carries another id than the rows, or is missing while rows exist starts with the store unhealthy (logged, `/health` unhealthy with the category as its reason, an off-host restore and DeleteBackup of an off-host backup refused naming the store's kind and the category and not its root, the in-catalog CreateBackup unaffected, the reconciler acting on no row of the store) and recovers once the file is restored, on an explicitly configured root and on the defaulted SQLite root once the catalog has rows for it; two instances of one catalog started against one empty root end with one id in the file and on the rows; two instances of one catalog with host-local filesystem roots at one path: the second, with no file, is unhealthy, and with the first's file copied to it the manifest probe marks it unhealthy; on either store a `put_if_absent` of a key that exists returns `None` and leaves the first object; purge takes the maintenance claim and heartbeats it for its whole run, waits for a CreateBackup running on a second instance (a CREATING or FAILED row with a live heartbeat), no job starts while it holds the claim (the in-catalog create and restore answer LimitExceededException), a CREATING row with a dead owner is purged with the rest, and afterwards no row of the configured store remains while `extenddb-store.json` and a prefix with no row do; PostgreSQL without `[backup.store]` creates an in-catalog backup, logs the deprecation warning naming `[backup.store]` and keeps serving legacy backups. Also `extenddb backup list` and `--failed`, printing `store_root` and `store_id`, a read beside a server (no SQLite serve lock, nothing created, an unhealthy store listed with its problem), as the purge's dry run is. Depends on PR 3.
     
     Invariants honoured: 4(c) (store identity and health, the shared-root rule), 4(f) (purge takes the maintenance claim), section 7's rules (an upgrade never stops a server that never configured backups; a configuration error stops startup, a transport or identity failure degrades the store).
     Interfaces it adds. `put_if_absent`, which PR 7 adds to the trait and both stores for the store-identity file (section 4), creates the object only if no object is under the key and returns `None`, leaving the existing object untouched, if one is: S3 sends `PutObject` with `If-None-Match: *` (a 412 is the key present; a 409 `ConditionalRequestConflict` from a concurrent conditional write is retried), and the filesystem store writes and syncs its temporary as `put` does, the temporary created with `O_CREAT | O_EXCL`, then `link`s it to the key, which fails if the key exists, so the object appears complete or not at all. `deny_unknown_fields` applies to every section; the server's `[backup]` struct holds the crate's `BackupStoreConfig` under its own key, `store`, which keeps one definition of the store shape, in the crate.
    @@ -345,11 +345,18 @@ Interfaces it adds. `put_if_absent`, which PR 7 adds to the trait and both store
         fn read_store_ids(&self, store_kind: StoreKind, store_root: &str) -> BoxFuture, StorageError>>;
             // the distinct `store_id`s recorded for a root: the identity check (section 4), before it knows an id; PR 7
         // Maintenance claim (section 4). PR 7 for purge, PR 10 for sync.
    -    fn maintenance_claim(&self) -> BoxFuture, StorageError>>;
    +    fn maintenance_claim(&self, holder: &str, stale_after: Duration) -> BoxFuture, StorageError>>;
    +        // `holder` labels the command and host for the operator who finds the claim held; `MaintenanceClaim` is
    +        // the handle's generation, heartbeat and release (the claim has no status machine, so it is not a `JobClaim`)
    +    fn list_backup_rows_older_than(&self, status: BackupStatus, store: &StoreIdentity, clock: StaleClock, age: Duration)
    +        -> BoxStream>;   // PR 7
    +        // the rows of a store with a status whose `heartbeat_at` (`StaleClock::Heartbeat`) or `completed_at`
    +        // (`StaleClock::Completed`, never older when unset) is older than `age`, compared in the backend's own query
    +        // on the database's clock (4(e)); purge's test of a live owner (4(f)), and PR 8's reconciler listings
     ```
     The identity file (4(c) states what makes a store unhealthy): On a root with no `extenddb-store.json` and a catalog with no rows for it, the server creates the file with `BackupStore::put_if_absent` (section 2: `If-None-Match: *` on S3, a temporary created `O_CREAT | O_EXCL` and `link`ed to the key on the filesystem store), then reads the file and uses the id it holds. Of two instances of one catalog starting against one empty root, one creates the file and the other's create finds it present, and both use the winner's id. An S3-compatible server that ignores `If-None-Match` on `PutObject` makes the create a plain put, last writer wins: startup's re-read then catches a race lost before it and marks the store unhealthy, naming both ids, and a race lost after it is caught by the next check (a sync, a startup or a reconciler cycle), which finds rows carrying an id the file does not. Startup runs the store's `validate` (its checks are #393's README and `validate` doc) and, on both store kinds, the identity check and the manifest probe of section 4, which run again each reconciler cycle; a root on a filesystem that reports a permissive mode (`0777` from a mount without Unix permission semantics) fails `validate` with the path, and one that reports `0700` while enforcing nothing passes, the case the store cannot observe and the deployment guide names.
     The spool: The spool: restores from an S3 store download to `spool_path`, created `0700` with files `0600`. The default is per backend: on file-backed SQLite it is `backup-spool/` beside the database file; on PostgreSQL, MongoDB, Cassandra and `sqlite-memory` with an S3 store `spool_path` is required and startup refuses without it (a filesystem store needs no spool). Startup refuses a `spool_path` that contains or is contained in the filesystem store root (the store would list and delete it, and sync would read its first component as an account id) and one that is or contains the database file's directory, and the reconciler's spool sweep removes only entries whose names match the job-entry pattern (`restore-
    `), never other files under `spool_path`. `spool_host` is `:`, and startup refuses a `spool_path` whose `.lock` another live instance holds (5.3 step 3). Disk cost of a restore: S3 store, the compressed backup size at `spool_path` while the restore runs; filesystem store, none for the restore, because files are verified in place (section 5.3 step 3). The filesystem store itself holds every backup's compressed size under its root, beside the database on the SQLite default, kept through `extenddb destroy` (below); that is the disk statement a SQLite operator sizes from. -Destroy and purge: `extenddb destroy` removes the catalog and data databases and leaves the backup store untouched: it is the copy that survives destroy. A separate `extenddb backup purge --confirm` deletes the prefix of each of the configured store's rows (`BackupArn::store_prefix`), never a whole account prefix, so a prefix with no row (another catalog's, or a copy) survives, and deletes those catalog rows outright, leaving no tombstones (a row over no bytes would be a backup Describe reports and Restore cannot read); it leaves `extenddb-store.json`, which is under no `/` prefix, so the root keeps its identity, and the `exports/` component (5.4). It takes the maintenance claim of section 4 and refuses if a backup row is still CREATING or DELETING or a `restore_jobs` row RUNNING when the claim's wait ends, so no job on any instance starts between the check and the deletion (the job permits are per instance and would exclude nothing elsewhere), prints what it will delete before deleting, and is audit-logged (section 8). It lands in PR 7. +Destroy and purge: `extenddb destroy` removes the catalog and data databases and leaves the backup store untouched: it is the copy that survives destroy. A separate `extenddb backup purge --confirm` deletes the prefix of each of the configured store's rows (`BackupArn::store_prefix`), never a whole account prefix, so a prefix with no row (another catalog's, or a copy) survives, and deletes those catalog rows outright, leaving no tombstones (a row over no bytes would be a backup Describe reports and Restore cannot read); it leaves `extenddb-store.json`, which is under no `/` prefix, so the root keeps its identity, and the `exports/` component (5.4). It takes the maintenance claim of section 4 and refuses if a CREATING or FAILED backup row still has a live owner or a `restore_jobs` row is still RUNNING when the claim's wait ends (4(f)), so no job on any instance starts between the check and the deletion (the job permits are per instance and would exclude nothing elsewhere); a CREATING row whose owner is dead is deleted with the rest, a DELETING row is finished, and a row a DeleteBackup moves or removes between the command's read and its turn is finished and counted as deleted; it prints what it will delete before deleting, and is audit-logged (section 8). It lands in PR 7. State transitions it owns: the store's health (healthy, unhealthy for each of 4(c)'s conditions, and back once the check passes); purge deletes rows outright, leaving no tombstone. Documentation rows it corrects: limits doc backup rows (the knobs); upgrade manual: CreateBackup without `[backup.store]` keeps creating in-catalog backups with a deprecation warning until PR 14, so a store is to be configured before that release, and `kind = "disabled"` is the opt-out (section 7); the rule the SQLite default root must meet and what the warning means when it does not (section 7); instances that share a catalog must share one store root (section 4). @@ -411,13 +418,14 @@ Interfaces it adds: fn insert_backup_row(&self, row: &BackupRow) -> BoxFuture>; // sync's unowned insert-if-absent fn list_backup_rows_by_root(&self, store_kind: StoreKind, store_root: &str, status: Option) -> BoxStream>; // a root's rows whatever their `store_id`: `--forget-store` and `--move-store` (section 4), which get a root and no id; PR 10 - // on Cassandra this and `read_store_ids` read `backup_index_by_status` one partition per status for the root, - // then each row's `backups_by_arn` row, since the index carries neither `store_kind` nor `store_id` + // on Cassandra this and `read_store_ids` read `backup_index_by_status` one partition per status for the root; + // migration 6's entry carries `store_kind` and `store_id`, so `read_store_ids` needs no second read, and this + // listing reads each row's `backups_by_arn` row for the rest of the row fn rewrite_store(&self, keys: &[BackupKey], store: &StoreIdentity) -> BoxFuture>; // `--move-store`, PR 10 ``` `backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
    //` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts; the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. `migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). -The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup-maintenance', RESTORE_LOCK_SEED)`, held exclusively on the command's dedicated connection; a job tests it with `pg_try_advisory_lock_shared` and releases the shared lock at once. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, so before the wait such a command runs the reconciler's dead-owner resolutions itself (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished), and a job the stop interrupted never makes it wait or refuse; `backup list` is a read and runs beside a server. +The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`; a command's take tries the exclusive lock a few times over about 10 ms on its connection, so a job's one-statement shared lock does not refuse it. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, and so is the holder of any maintenance claim row, which such a command takes over whatever its heartbeat says; before the wait it runs the reconciler's dead-owner resolutions itself, as the claim's holder (the cycle takes the held branch without reading the claim back through the zero window) (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 29 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. State transitions it owns: a prefix to an AVAILABLE row (adoption); CREATING or DELETING rows of a moved root resolved by `--move-store`; AVAILABLE to DELETING by `--prune-missing`; rows deleted outright by `--forget-store` and `--forget-backup`; a FAILED restore's CREATING target removed by `remove-target`; legacy rows replaced by an AVAILABLE row by `migrate-legacy`. Documentation rows it corrects: upgrade manual: on SQLite the backup CLI commands that write the catalog take the serve lock, so the server is stopped for `backup sync` and its options and for `purge` (section 4); legacy backups are carried forward with `backup migrate-legacy` (above), and one it refuses (an id with no instant, an item the validator refuses, a pre-0.0.4 SQLite definition) is carried forward by restoring it and taking a new backup, which has a new ARN; a store owned by another catalog is adopted only with `backup sync --adopt-store`. @@ -539,3 +547,6 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 26. (2026-10-30, PR 6 as implemented on #666's registry.) PR 6's migration is id 5 on PostgreSQL and SQLite (one `MigrationId` list for both, ids 1 to 4 taken), and the Cassandra file is V006 and the MongoDB step `002_backup_catalog_index`, both writing `0.0.5` so that one number names one change on every backend; the literals in 4(i) and PR 6's section are updated, PR 1's line keeps its own numbers and renumbers if it lands after this. 27. (2026-10-30, PR 6.) The FAILED-target duty lands in PR 6: `list_dead_restore_jobs` also returns a FAILED `restore_jobs` row whose target is still CREATING once the row is stale, and the reconciler removes the target without a second FAILED write, which closes the state the job's or the reconciler's failure path leaves when the FAILED write lands and the removal does not (4(e), PR 6, PR 8). PostgreSQL's dead rule is the heartbeat alone, as on MongoDB and Cassandra: the advisory lock is the claim's exclusion, not the liveness test, and the `IS NULL` arm item 21 named does not exist, the column being NOT NULL (4(e), 5.3 step 7, PR 6). The stale floor is computed from the period the jobs run at, `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, the one constant every backend's heartbeat loop reads until the knob reaches the jobs, and `max_concurrent_jobs` has an upper bound (5.1 step 4, section 7, PR 6). SQLite's `backup_server_slot_count` is kept for snapshot opens until PR 8, the legacy create trying it without waiting and answering the one `LimitExceededException` text (PR 6). The engine's `BackupArnSpellings` carries an optional parsed ARN per spelling, so a legacy backup of a name the store-prefix rules refuse (dot-only, over 255 bytes) still describes, restores and deletes through the legacy reader (4(d), PR 6). DeleteBackup over a row runs the in-use check before the other-store check, as a restore does, builds the description before the DELETING write, and a row whose source table the build cannot read is left out of ListBackups rather than failing the listing (5.2, PR 6). The acceptance cell running the two orderings over a `backup_index` row is deferred to PR 9a, with the row lock observed by a two-transaction test of `start_backup_delete` in PR 6 (PR 6). On Cassandra an index row's list rows exist for as long as the row does, whatever its status, readers re-reading the authoritative row; the legacy create keeps its publish-time rule (PR 6). The reconciler records the worker metrics of section 9 (PR 6). On Cassandra a migration 6 that fails in its backfill leaves the migration unrecorded, so `serve` keeps refusing the catalog and `migrate --yes` runs the body again; a Cassandra job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and the fence the reconciler puts on a dead create's row takes a fresh `owner_generation` (4(e), PR 6). MongoDB's RestoreTableFromBackup waits a bounded time for the registration and answers `InternalServerError` past it while the registration goes on; what the request leaves is stated in 5.3 and the differences document (5.3, PR 6). CreateBackup checks `server.region` before forming a backup ARN and refuses, naming the setting, under a region outside `[a-z0-9-]`, where main created a backup no operation could then address; the store-prefix rules on the name are applied before the ARN is formed so a name over 255 bytes is refused naming the limit whatever its length (5.1 step 1, PR 6). The `LimitExceededException` text names the bound that was met beside its number (`max_concurrent_jobs`, SQLite's snapshot slots, or PostgreSQL's connection slots for the claim connection), one implementation (`limit_exceeded_message`) every refusal binds (5.1 step 1, section 7, PR 6). Section 7 states, where it bounds snapshots by half the pool, that the pool is the raised one (section 7, PR 6). Item 21's `IS NULL` arm is withdrawn by this item alone; item 21 keeps its wording, as item 22 requires. On Cassandra, where the removal is a claim and then a drop, the FAILED-target duty also lists a FAILED record whose target is DELETING with its registration, and `remove_restore_target` resumes the removal from the drop (4(e), PR 6). On MongoDB the in-use check of a restore's registration and of DeleteBackup reads at the transaction read concern, majority-committed data, while DescribeTable reads at `local`, so a request issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer `BackupInUseException` and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag (5.3, PR 6). The rule that a job performs no further operation after a write it cannot confirm is scoped to the conditional write itself: a legacy restore whose claim connection (the heartbeat's) errors runs 5.3 step 7, the conditional FAILED write on a pooled connection and the removal only when that matched, so on PostgreSQL the target is removed at the next heartbeat rather than by the reconciler once the record is stale (4(e), 5.3 step 7, PR 6). On MongoDB `remove_restore_target` deletes the target's catalog document first and returns the target removed once that is gone: a collection drop that fails after it is logged naming the collections and leaves them for `backup sync` (PR 10), there being no record for a later reconciler cycle to retry from (4(e), PR 6). PR 6's interface block lists `list_dead_restore_jobs`, `fail_restore_job` and `LegacyBackupReader::abandoned_creating_backups` with their contracts, as the trait ships them (PR 6). On MongoDB `serve` registers and reads the catalog's status (`register_and_check`, #681) before it builds the data engine or starts the startup key repair of documents stored without their key (#473), so a server that refuses the catalog has written nothing, as on PostgreSQL (PR 6). After #335 and #681 merged, PR 6 is rebased onto main: PR 1 merged as #335 with migration 5, `drop_continuous_backups`, on every backend, so PR 6's migration is id 6 and PR 9a's `restore_progress` id 7; Cassandra and MongoDB migrate through the numbered runner, where PR 6's change is a code body (Cassandra's executing `0006_backup_catalog_index.cql` before the backfill and the catalog id), and no backend keeps a catalog version string, so the version-last rule of Cassandra's former runner, MongoDB's versioned step and their shared version check are withdrawn, the runner's status check being `serve`'s refusal of a catalog without migration 6 on every backend; item 26's numbers are superseded by this sentence, and item 26 keeps its wording, as item 22 requires (4(i), PR 1, PR 6, PR 9a). 28. (2026-11-04, PR 6 after the eighth review round, measured against DynamoDB.) The engine checks a restore's target name after the backup resolves and before the overrides are checked against it, the service's order, so a restore of a missing backup is `BackupNotFoundException` whatever the target name's state (5.3 step 1). `check_backup_arn_fits` keeps the region rule only; the store-prefix rules on the name are the store-backed writer's (PR 8), and the in-catalog create backs up a dot-only or over-255-byte name as main did, as the service backs up any table it can create (5.1 step 1). A table ARN in CreateBackup's or ListBackups's `TableName` whose region is not `server.region` is `ValidationException` "1 validation error detected: Invalid AWS region in ''", the service's text. The job permit is waited for up to 30 s (`JobPermit::acquire_within`, `PERMIT_WAIT`) rather than refused at once, the request holding nothing meanwhile (5.1 step 1, 5.3 step 1, section 7). `fail_restore_job` takes the listed `owner_generation` and `heartbeat_at` and writes FAILED with a fresh generation, so the reconciler's write cannot land over a heartbeat that followed its listing (4(e), the PR 6 interface block). The SQLite restore claim holds a `JobSlots` slot for the copy's life, and `list_dead_restore_jobs` skips a RUNNING record whose slot is held, the SQLite rule of 4(e): the heartbeat queues behind the write lock and is not that process's liveness test. SQLite's `abandoned_creating_backups` removes the dead CREATING rows itself, in one write under the engine write lock limited to rows whose slot no task holds (`AbandonedLegacyBackup::removed`), so a create that commits AVAILABLE between a read and the removal keeps its backup. The Cassandra restore heartbeat requires `restore_status = 'RUNNING'`, so a FAILED write stops the copy as on the other backends. `BackupInUseException` for a restore of an in-use backup carries the service's text, `Backup is being used to restore another table: `, the target's name going to the log. `SourceTableFeatureDetails` is always present, `{}` when the source recorded no feature. The MongoDB registration stamps its first heartbeat with `$$NOW` inside the registration's transaction. A restore job record this build cannot read is logged and skipped by `list_dead_restore_jobs` on every backend, and the reconciler's once-per-record log set lives in `serve`, outside the workers it restarts (4(e), PR 6). +29. (2026-10-30, PR 7 as implemented.) A store problem never gates the in-catalog create: with a configured store that is unhealthy, CreateBackup writes the in-catalog format as it does with no store, and only the store-backed create of PR 8 is refused by `require_healthy_store`; a defaulted SQLite root whose identity check fails (a catalog re-created by `destroy` and `init` under the old root's `extenddb-store.json`) is the same warning as a failed validation, leaving the store unconfigured, and `destroy` and the admin guide say the file must go with the catalog (section 7, PR 7). An S3 store's `validate` and identity check run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports; a filesystem store's run before the listener; the configuration errors that need no request still stop startup; a `validate` refusal of an S3 bucket after the listener makes the store unhealthy (`failed validation`) rather than stopping startup (section 7, 9, PR 7). `/health`'s `reason` is the problem's category alone; the ids and ARNs go to the operator's log line (section 9, PR 7). The identity file is read up to 64 KiB. The S3 root label is built from the endpoint the client uses, in one spelling. A check that cannot read the catalog leaves the store's health unchanged. An existing spool must be owner-only. The maintenance claim is heartbeated on a timer from the take to the release, and PostgreSQL's `check_maintenance_claim` tests the advisory lock before the heartbeat's age (4(f), PR 7). The jobs' side of the claim is implemented in the engine for the in-catalog create and restore, before the permit and, since those paths commit their row inside the backend, before the row exists; the ordering 5.1 step 2 states applies to the store-backed job PR 8 adds (5.1, 5.3, PR 7). `backup list` opens the SQLite catalog without the serve lock, reads the identity file without creating it and runs on an unhealthy store; on SQLite `backup purge` runs the reconciler's duties with a zero window before its wait; a CREATING row with a dead owner is purged as any row, through `remove_dead_creating_backup_row`, a `BackupCatalog` method of PR 7; a FAILED row with a young heartbeat is waited on; the dry run exits 0 (PR 7, PR 10). Cassandra's claim reads run at `LOCAL_SERIAL` and an insert whose answer was lost is settled by a re-read of the generation (PR 7). The PR 10 comment on `backup_index_by_status` is corrected: migration 6's entry carries `store_kind` and `store_id`. The body sentences that paraphrased the earlier rules are edited with this item (section 7's configuration-error paragraph, 4(f) and 5.1 step 2 and 5.3 step 4's claim check, PR 6's `store_root` row, PR 7's interfaces and acceptance), and the filesystem store's startup checks are bounded by a timeout that leaves the store `checking` when a mount stops answering after the root is opened (a root that does not answer at startup holds the server before the listener, as the database directory would). Also: the PostgreSQL check takes its shared lock at transaction scope so a cancelled request leaves no lock on the pool; a dropped `HeldClaim` stops its timer; the purge's dry run is a read like `list`; a CREATING row is removed before its prefix; an empty `filesystem.path` is left for `open` to refuse; the spool meets the root's whole rule, parents included; `backup list` reads the catalog's status without the serve lock. The defaulted SQLite root's warning holds only while the catalog has no backup rows for the root (`read_store_ids(Filesystem, root)`: the upgrade, fresh and re-init cases); with rows for it the root has been used and is handled as an explicitly configured one, so its backups never vanish behind a warning: a failure of its identity check or manifest probe is an unhealthy store (reported, its rows' restores and deletes refused, rechecked each cycle), and a refusal of its `open` or `validate` a startup error naming the root (section 7, PR 7's acceptance). The server's `configure` carries the reason for an absent store out (`NotConfigured`), and `backup list` and `backup purge` print it, so a defaulted root that failed its check is no longer reported as "[backup.store] is absent or disabled". The store's check runs under a per-kind bound in every reconciler cycle (`check_configured_store`: 5 s for a filesystem root, the startup bound; 30 s for a bucket, under the SDK's 60 s per-attempt timeout so a timed-out check is dropped before the SDK retries its request), recording a transport failure naming the bound, and the cycle selects the check against the shutdown token; an unbounded check let a stalled S3 endpoint hold each cycle for the SDK's whole retry budget and a stalled mount hold it for good, with shutdown hitting the drain timeout (section 7). A filesystem root's `validate` reports no transport class; an I/O failure of its probe write or of resolving the root at startup degrades the store (`failed validation`) as the same condition one cycle later does, while the owner-only and existence refusals stay configuration errors (section 7). A transport failure's operator line carries the store error's whole cause chain (the SDK's top link is a bare class). Every store error of a check that is neither a transport failure nor a denial is `failed validation`, the category `validate` gives it, so a bucket deleted under a running server reads as a bucket missing at startup. A filesystem root that is or contains the database directory is refused in `resolve_store`, beside the spool rule (section 7). A filesystem store's synchronous root checks run on the blocking pool so the startup bound fires whichever call stalls. The checking store's refusal says its first check is pending; `/health`'s reason list carries `catalog unreadable`; `record_check` reports whether the category changed. Purge's wait set is CREATING and FAILED rows with a live heartbeat and RUNNING restores, DELETING rows being finished rather than waited on, and its refusal rule is that set at the wait's end; 4(f), the PR 7 "Destroy and purge" paragraph and the PR 10 claim paragraph (`MAINTENANCE_LOCK_KEY`, `pg_try_advisory_xact_lock_shared` then the heartbeat age, `LOCAL_SERIAL`) and the PR 6 and PR 7 signatures (`check_maintenance_claim(stale_after)`, `maintenance_claim(holder, stale_after) -> Box`) are edited to the code with this item. A row a DeleteBackup moves to DELETING or DELETED, or removes, between purge's read and its turn is finished and counted as deleted, not left (the deletion is one idempotent sequence on both sides). On SQLite purge waits and removes CREATING rows with a zero stale window (under the serve lock every owner is dead, so a young heartbeat proves nothing), and the server's startup under the serve lock removes a maintenance claim a stopped command left, so a killed purge costs no stale window; on every backend purge releases the claim on `SIGINT` or `SIGTERM`, at once during the wait and before the next row during the deletion. The wait's three failures (jobs still running at the limit, the claim lost, the catalog unreadable) and the signal are reported each as itself. PostgreSQL's factory starts the GSI queue workers only for a server, not for a command reading beside one. Purge's restore part of the wait set is the RUNNING `restore_jobs` records alone: the dead-job listing it reads with a zero window also returns the FAILED records whose target is still CREATING (the FAILED-target duty's, 4(e)), which are the reconciler's to finish and no job runs under, so they are not waited on; the wait finishes at once on a stopped deployment that holds one. On SQLite the zero-window pass resolves the kinds this release writes; a RUNNING `restore_jobs` record of kind `manifest`, which PR 9a introduces, is left for the reconciler that resumes it and purge waits on it and refuses naming the table, the one state the PR 10 paragraph's "never makes it wait or refuse" does not cover. The reconciler finishes the configured store's DELETING rows while the maintenance claim is held, as 4(f) lists. The binary ends its tokio runtime with a bounded shutdown (5 s) after `serve` and after every command: a filesystem store's check abandoned at its bound leaves its call parked on a blocking thread until the mount answers, and a plain drop of the runtime joined that thread, so a server whose root had stopped answering drained its workers at SIGTERM and then stayed alive until SIGKILL. Each call of a store check runs as a task of its own that an abandoned check does not take with it, and the store records that a call is in flight; no check starts while one is (the store is recorded `unreachable` again, with the reason), so a mount that stays stalled parks one thread rather than one per cycle. A command's identity read (`backup list`, the dry run) runs under the store's check bound as the server's check does, where the unbounded read waited out the SDK's three attempts of 60 s on an endpoint that accepts and never answers. An empty `filesystem.path` is compared with nothing in `resolve_store` (compared as written it was the working directory, refused by name as the database directory when the server starts there) and left for `open` to refuse. A root that refuses `link(2)` with `EPERM` is reported `failed validation` (the README and the deployment guide said `unreachable`). `tests/test_backup_store_config.py` is excluded from `devtools/run-tests`'s pytest run, as its docstring says; a catalog whose rows cannot be read keeps a defaulted root configured, and that arm is tested. On SQLite the purge's own reconciler cycle is told that its caller holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) and takes the held branch without reading the claim; read through the zero dead-owner window, the command's own claim was seen held only when its heartbeat and the check fell in one millisecond, so the branch was decided by scheduler timing (PR 7, PR 10). Purge's test of a live owner is the backend's `list_backup_rows_older_than` on the heartbeat clock, a `BackupCatalog` method of PR 7 that PR 8's reconciler also lists by, where it compared `heartbeat_at` with this host's clock, which a host running ahead of the database read as a shorter wait, not a longer one (4(e), 4(f), PR 7). On SQLite a command under the serve lock takes over a maintenance claim row whatever its heartbeat says, and PostgreSQL's take tries the exclusive lock again over about 10 ms, so a job's one-statement shared lock does not refuse it (PR 10's claim paragraph). On MongoDB a command that opens the catalog beside a server (`backup list`, the dry run) starts no key repair of documents stored without their key (#473); the server does. On PostgreSQL every `backup` command registers as a server does and its registration's session carries the command's name as `application_name`, so a migration that waits for it names it, and a catalog the factory refuses ends the command with the message and exit code `serve` gives (3 behind, 6 while a migration runs). The jobs' refusal while the claim is held names a backup maintenance command, not `purge`, since every maintenance command holds the one claim. The console banner shows a filesystem root in the canonical spelling the rows and the log line use, with the configured spelling only where it differs (PR 7). After #335 and #681 merged, PR 7 is rebased onto PR 6's rebased head: MongoDB's and Cassandra's `serve` register through the numbered runner (#681), so the backup commands register there too, as on PostgreSQL, named after the command (MongoDB's `extenddb_servers` document carries a `command` field; Cassandra's `extenddb_servers` row, which has no column for it, carries the name in `release` after the release and ` as `, a column being a catalog migration), refuse with the same exit codes, and release the registration when they exit, where a registration left to the runtime hooks the command drops would be deleted only by a task the command's exit may never run, and `migrate --yes` would then wait out the row's or document's 45 s; MongoDB's factory starts the key repair in its `build_components`, after `register_and_check`, only for a server (`start_key_repair`); the reconciler, which takes the backup runtime, runs in the worker set the registration supervisor stops and starts (#666) (PR 7). + +30. (2026-11-04, PR 7, second review round.) DeleteBackup of a store-backed row answers the DELETED description once the DELETING write has committed and leaves a failed object deletion to the reconciler, where it answered InternalServerError (5.2). The store's bound covers the store's calls alone, each through one adapter (`ConfiguredStore::calls`): the catalog's reads run beside them, so a slow catalog costs the check time and never marks the store unhealthy, and the deletions DeleteBackup and the reconciler run go through the same adapter, under the same bound and the in-flight rule (section 7; the in-flight mark now means a call abandoned at its bound has not returned). After the second round's review: a filesystem root's calls run one at a time (`ConfiguredStore::call_slot`), a call that finds another running waiting for it under the bound and then having the bound for its own call (third round: one bound over the wait and the call abandoned a 3 s call behind a 3 s call on a healthy root), so two DeleteBackup requests on a stalled mount park one thread between them where each parked its own (the in-flight mark alone was set only once the first call reached its bound); the mark is set under the call task's own lock, only while the task still runs, so a task that ended between the bound's last poll and the mark leaves no mark with nothing to clear it. A store whose startup `validate` failed is built unvalidated, so the next check runs `validate` again rather than reporting the store healthy on a re-read of the identity file. A defaulted SQLite root whose startup checks did not finish keeps its rule (`ConfiguredStore::defaulted`): the reconciler applies it to the first check's outcome and sets a failing root with no rows aside (`set_aside`), as the startup check would have (section 7). The manifest probe reads the newest AVAILABLE row through `BackupCatalog::newest_available_backup_row` rather than listing every one. Purge's wait reads RUNNING restores by status (`BackupCatalog::list_running_restore_jobs`), with no age, so a record stamped in the listing's instant or ahead of the clock holds the wait (4(f)); its deletion loop lives in the engine (`maintenance::purge_rows`) and stops before its next row at a lost claim or a signal; `HeldClaim::release` is bounded (10 s). On SQLite a zero window selects the dead-owner listings and removals by status alone, so a heartbeat stamped ahead of the clock is not a live owner under the serve lock (PR 10's claim paragraph). The backup commands run none of the startup recovery a server runs (`ServerComponentsOptions::startup_recovery`), `purge --confirm` included, which still takes SQLite's serve lock; the signal listener is up before the open in every subcommand, so a signal during the open ends the command with its registration released. PostgreSQL's `release` unlocks the advisory lock before closing the session, so a check right after the release reads the claim free; Cassandra's takeover re-reads on a lost answer as the insert does; MongoDB's claim upsert that matched a document with no `owner_generation` is taken. `tests/test_backup_store_config.py` runs in CI: on the SQLite job with moto, and in the PostgreSQL CLI lifecycle list. Two rows join `docs/differences-from-dynamodb.md`: the maintenance claim's LimitExceededException and the unhealthy store's ValidationException. diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 0a09f691a..a6088e3fa 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -127,6 +127,8 @@ Behaviour stated as measured, and every number below, was observed with release | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace. Take database-level backups for off-host copies until the backup store program ships | | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE` on all four backends. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw it `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB reports `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s); a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed writer of a later release refuses it before it writes, and the in-catalog format of this release carries it as earlier releases did. Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the check runs after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | | `DeleteBackup` response | The full `DescribeBackup` description with `BackupStatus` `DELETED`, including `SourceTableDetails.ProvisionedThroughput` (a required member) and `SourceTableFeatureDetails` (measured) | The same on all four backends: the engine reads the backup's recorded definition before the delete and fills the description as `DescribeBackup` does, with `BackupStatus` `DELETED`. A backup whose definition this build cannot read or that does not parse is still deleted and described without feature details (logged at warn), which means a damaged `PROVISIONED` backup is described with on-demand `ProvisionedThroughput` `{0, 0}`: ExtendDB's choice, deletability over the throughput member of a row that is being removed, where `DescribeBackup` on the same row is an `InternalServerError`; a failure to reach the database fails the request and deletes nothing. | +| A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | +| A store-backed backup while the configured store is unhealthy | Not applicable: managed storage | `DeleteBackup` and `RestoreTableFromBackup` of a backup in the configured `[backup.store]`, while that store is unhealthy or its first check is pending, answer HTTP 400 `ValidationException` "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", the kind (`filesystem`, `s3`) and the category in parentheses being the store's (`/health` reports the same category); before the first check of a store checked after the listener: "The s3 backup store configured on this server has not passed a check yet (its first check is pending); the operation is refused until it does". `DescribeBackup` and `ListBackups` still describe such a backup. Once `DeleteBackup`'s DELETING write has committed the request answers the description with `BackupStatus` `DELETED` whatever the object deletion does; a deletion the store does not answer within its bound is left DELETING for the reconciler, logged, and the request still answers DELETED. No backup lives in a store on this release's write path (CreateBackup writes the in-catalog format), so these answers need a row a later release wrote. | | `DescribeBackup` when the definition cannot be read | Not applicable | A definition row that does not parse, a renamed column, or a revoked privilege (`Internal` from the backend) is an `InternalServerError`, as it is for `RestoreTableFromBackup` on the same read; only a definition version this build does not know is described without `SourceTableFeatureDetails`. | | Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot. How *(from the code)*: PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a MongoDB or Cassandra table while it is not being written if the backup must be consistent | | `ListBackups` paging | At most 100 summaries per page, with `LastEvaluatedBackupArn`; `Limit` 1 to 100. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list. `TableName` is checked as the model says (3 to 255, `[a-zA-Z0-9_.-]+`, the "N validation error(s) detected" texts). An `ExclusiveStartBackupArn` of another account is `ValidationException` "Invalid Request: The supplied ExclusiveStartBackupArn does not match the account"; of another region, or of a table other than `TableName`'s, `ValidationException` ("does not match the region", "does not match the table name predicate"); a time bound before the epoch is `ValidationException`; a lower bound above the upper bound is `ValidationException` "Time range lower bound must be less than or equal to upper bound" (all measured) | The same paging on all four backends: `Limit` outside 1 to 100 is `ValidationException`, `ExclusiveStartBackupArn` resumes after the given backup, and summaries are ordered by creation time then ARN. Summaries carry `TableId`. `TimeRangeLowerBound` (inclusive) and `TimeRangeUpperBound` (exclusive) keep backups created in that range, and `BackupType` keeps backups of that type (`USER` when absent, every type for `ALL`); every ExtendDB backup is `USER`, so `SYSTEM` and `AWS_BACKUP` list nothing. The filters apply before paging, so `Limit` and the cursor count only backups that pass. Measured on SQLite and PostgreSQL with whole-second bounds between two backups taken 2 s apart; the millisecond edges are from the engine's unit tests. The request checks above are the service's, with its texts where captured: the account text in full; the region and table-name texts complete the captured fragments on the account text's pattern ("Invalid Request: The supplied ExclusiveStartBackupArn does not match the region" / "... the table name predicate"), checked in that order after the ARN's shape; a bound before the epoch is refused with ExtendDB's own text naming the member; equal bounds list nothing. An `ExclusiveStartBackupArn` that is not in the listing (excluded by the filters, or deleted since the previous page) is placed by the creation time in its id; one whose id carries no time (the 13-digit ids of earlier builds, with or without the hex suffix: only the 14-digit form carries one) is then a `ValidationException`. A cursor in another region is accepted only when it is the stored ARN of a backup of the account (or of `TableName`), which is what `LastEvaluatedBackupArn` hands out for a backup created under an earlier `server.region`, so paging passes such a backup; any other cursor in another region is refused ("does not match the region"). If that backup is deleted between two pages, its ARN is no longer a stored one and the next page is refused the same way (a same-region cursor would be placed by its time instead); start the listing again. Under a `server.region` outside `[a-z0-9-]` the cursor is still the stored ARN: a listing of backups created under a valid region pages, and a cursor that is the ARN of a backup an earlier release created under the invalid region is refused with "Invalid Request: BackupArn is not valid". `TableName` is a table name or a table ARN, as `CreateBackup` accepts it (the service accepts both, measured on DynamoDB): the ARN form lists what the name lists, an ARN whose region is not the server's (compared as written) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice), another account's ARN is `AccessDeniedException`, and the cursor's table check runs against the resolved name. A numeric `TableName` is refused with the text the service uses for a map ("Start of structure or map found where not expected"), not measured for a number. Every `ListBackups` request reads all of the account's backups (or all of the table's, with `TableName`) from storage and pages them in memory, so walking N backups 100 at a time reads about N²/100 rows in total *(from the code)* | diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index ae4e703c5..9caaaa099 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -128,10 +128,11 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | Limit | DynamoDB Value | Status | Notes | |-------|---------------|--------|-------| -| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | +| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | | `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed and backed up whatever its name, as on the service: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed writer of a later release refuses it before it writes, and the in-catalog format of this release carries it (see the differences document's backup creation row). Both members may also be the table's ARN; an ARN whose region is not the server's is `ValidationException` "1 validation error detected: Invalid AWS region in ''", the service's text. The value is echoed in the message as sent | | `BackupArn` | 37 to 1024 characters | Maximum enforced; minimum implied | Surrounding ASCII whitespace is trimmed first, then over 1024 bytes, not of the backup ARN shape, a control or non-ASCII character, or a region outside `[a-z0-9-]`, an account that is not digits, or a table name outside `[a-zA-Z0-9_.-]` is `ValidationException` "Invalid Request: BackupArn is not valid"; the right shape with an id that is not a backup id is "Invalid Backup ARN"; a digit-only account that is not the caller's is `AccessDeniedException`. No minimum is checked: the shortest string of the right shape is longer than 37 characters. Another region of the caller's account resolves (the server's region is tried first, then the ARN as written). `ListBackups`'s `ExclusiveStartBackupArn` must also match the account, the region, and `TableName` when given | | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | +| Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where ExtendDB's backups will live from a later release's CreateBackup on (this release's writes the in-catalog format and the store serves restores and deletes of the backups it holds): a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses the restore and delete of store-backed backups (CreateBackup, which writes the in-catalog format in this release, is not refused) with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | | Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service. Backups are stored inside the database the server uses; take database-level backups for off-host copies | | Point-in-time recovery | 35-day window | N/A | Not supported: `UpdateContinuousBackups` refuses to enable it (`ContinuousBackupsUnavailableException`) and `RestoreTableToPointInTime` refuses every restore (`PointInTimeRecoveryUnavailableException`); use on-demand backups (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index aa2b9b10b..047494eed 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -53,9 +53,34 @@ SIGTERM directly. Alternatively start it with ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health -# {"status":"healthy"} +# {"status":"healthy","components":{}} +# with a backup store configured: +# {"status":"healthy","components":{"backup_store":{"status":"healthy","kind":"filesystem"}}} +# while it cannot be reached: +# {"status":"degraded","components":{"backup_store":{"status":"unhealthy","kind":"s3","reason":"unreachable"}}} +# right after startup, before an S3 store's first check has returned: +# {"status":"healthy","components":{"backup_store":{"status":"checking","kind":"s3"}}} ``` +`status` is `healthy`, or `degraded` when a component is unhealthy; the HTTP +status stays 200 either way, because a component problem never takes the data +plane out of rotation. The one component is `backup_store`, present whenever +the server has a store: one `[backup.store]` configures, or the default root +beside a file-backed SQLite database when the section is absent (absent when +no store is configured, when `kind = "disabled"`, and when a default root +failed its check with no rows for it); its `reason` is the problem's category in +one or two words (`unreachable`, `permission denied`, `failed validation`, +`identity file unreadable`, `identity file missing`, `identity mismatch`, +`owned by another catalog`, `manifest probe failed`, `catalog unreadable`) and +nothing else: the endpoint is unauthenticated, so the ids, ARNs and table +names a problem involves are in the server's log line only. An S3 store's +first check runs after the listener is up (the next section); until it +returns the component is `checking`, with the server `healthy`. The +reconciler re-runs the store's checks every `reconcile_interval_secs`, each +run bounded (5 s for a filesystem root, 30 s for a bucket, after which the +check is abandoned and the store is `unreachable` until a later check +answers), and the component recovers on its own. + Or without `curl` — useful inside a minimal container image, and what a Docker `HEALTHCHECK` should call: @@ -67,9 +92,8 @@ echo $? # 0 = healthy, 1 = not `healthcheck` reads the port from the config file, or takes an explicit `--endpoint https://host:port`. It accepts the server's self-signed certificate. -This checks liveness, not readiness. `/health` is a static handler that does not -query PostgreSQL, so it reports healthy even when the database has become -unreachable since startup. That is intentional for a liveness probe: one that +This checks liveness, not readiness. `/health` does not query the database, so +it reports healthy even when the database has become unreachable since startup. That is intentional for a liveness probe: one that failed on a database outage would restart every replica at once and prolong the outage. A database that is unreachable at startup stops the server from listening at all, so that case is still caught. @@ -129,7 +153,25 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau | `failed_row_retention_secs` | `604800` | How long FAILED backup rows are kept for the operator (7 days) | | `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup | | `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller | -| `spool_path` | (unset) | Restore download area for an S3 store, used by a later release | +| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | + +#### [backup.store] + +The store CreateBackup's backups will live in, and the one restores and deletes of store-backed backups read. The section's keys, defaults and the refusals of a bad value are the backup store crate's (`crates/backup-store/src/config.rs`); the table below is what the server adds. One definition of the shape exists, so an unknown key anywhere under `[backup]` is refused at load. + +| Key | Default | Description | +|-----|---------|-------------| +| `kind` | see below | `filesystem`, `s3`, or `disabled` | +| `filesystem.path` | `backups/` beside a SQLite database file | The root directory of a filesystem store. It must exist, be owned by the extenddb user with mode `0700`, and sit under directories owned by root or that user which others cannot write to; `validate` refuses anything else at startup. A relative path, like a relative `spool_path`, is resolved against the working directory the server was started from (the daemon's own working directory is `/`) | +| `s3.bucket`, `s3.prefix`, `s3.region`, `s3.endpoint`, `s3.force_path_style`, `s3.allow_insecure_endpoint`, `s3.upload_part_size_bytes`, `s3.max_concurrent_uploads` | | The S3 store; see the crate's `config` module doc. Credentials come from the AWS SDK's chain and are never in the file | + +When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call; with one configured, this release's CreateBackup still writes the in-catalog format (the store-backed writer is a later release's) and the store serves restores and deletes of the backups it holds. Backups of either format stay listable, describable, restorable and deletable. + +Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check's store calls are bounded (5 s for a filesystem root, 30 s for a bucket); the catalog's own reads (the catalog id, the rows' store ids, the newest AVAILABLE row) run beside the calls and not under the bound, so a slow catalog costs the check time and never marks the store unhealthy. A store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), a root's calls run one at a time so that calls started before the first reached its bound (two `DeleteBackup` requests together, a `DeleteBackup` beside a check) wait for it under the bound, then have the bound for their own call (a slow call behind another finishes, where one bound over both would abandon it), and park no thread of their own, and `stop` or SIGTERM ends the server within 5 s with that thread still parked. The deletions `DeleteBackup` and the reconciler run against the store go through the same bound and rule: `DeleteBackup` answers the description with `BackupStatus` `DELETED` once its DELETING write has committed, and a deletion the store does not answer within the bound, or that leaves an object under the prefix, leaves the row DELETING for the reconciler and is logged, never an error to the client. A default SQLite root whose startup checks did not finish starts `checking` with its rule kept: a first check that fails while the catalog has no rows for it is the same warning and leaves the store unconfigured. CreateBackup is not refused by an unhealthy store in this release: it writes the in-catalog format, which touches no store; the store-backed create of a later release is. + +The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no rows for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id); point every instance at one shared root before the first backup. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. + +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. Neither command runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database under the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse; the one exception is a RUNNING restore of kind `manifest`, which no release before PR 9a writes: the command leaves it for the reconciler that resumes it, waits on it as on any RUNNING restore and refuses naming the table; and a server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra both commands open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. #### [auth] @@ -889,9 +931,12 @@ JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health -# {"status":"healthy"} +# {"status":"healthy","components":{}} ``` +The `components` object carries the `backup_store` component when a backup +store is configured (see [Health Check](#health-check) under Server Lifecycle). + ## Troubleshooting ### Vector Index Support (PostgreSQL) diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index b50c36fd6..8593fded4 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -6,6 +6,8 @@ This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 6, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), and migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)). +Configure `[backup.store]` before the release that stops creating in-catalog backups. In this release CreateBackup writes the in-catalog format whether or not a store is configured, and logs a deprecation warning naming `[backup.store]` on each call when none is; a later release's CreateBackup writes to the store and refuses without one. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. + A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. The [general upgrade procedure](#general-upgrade-procedure) gives the sequence. The [version history](#version-history) records the 0.1.x catalog versions and what the 0.1.x binaries print, and [Behavior Changes by Release](#behavior-changes-by-release) what each release changes in behavior. @@ -206,7 +208,7 @@ Adds the table definition a backup records and the registration of a running res A backup taken before this upgrade has no recorded definition. It restores with its keys, items, and billing mode but no secondary indexes; a provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra. On PostgreSQL a pre-upgrade backup of a table with vector indexes is refused at restore; on SQLite it restores without the vector indexes. Tables restored before this upgrade have no registration and report no `RestoreSummary`. -The upgrade is not reversible in place: a 0.0.3 PostgreSQL or SQLite binary refuses to start against a 0.0.4 catalog (`Catalog version mismatch: expected 0.0.3, found 0.0.4`). Roll back by restoring what you saved before the upgrade. Backups live in the catalog, so that also removes every backup created after the upgrade. +The upgrade is not reversible in place: a 0.0.3 PostgreSQL or SQLite binary refuses to start against a 0.0.4 catalog (`Catalog version mismatch: expected 0.0.3, found 0.0.4`). Roll back by restoring what you saved before the upgrade. Backups live in the catalog, so that also removes every backup created after the upgrade (this is the in-catalog format; a backup in a `[backup.store]` store of a later catalog is outside the database, and a catalog restored from a pre-upgrade dump loses its `backup_index` rows while the store keeps the bytes). #### PostgreSQL diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index bc0b59626..d8e664c5c 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -277,7 +277,8 @@ Run one extenddb instance per catalog. Multiple instances behind a load balancer - Credentials, policies, and table metadata are cached in each process (see the cache TTLs in the admin guide). Nothing invalidates a cache on another instance, so a key revoked or a policy changed through one instance stays in force on the others until the TTL expires. - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. -- `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. +- `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. It does report the backup store: `components.backup_store` is `unhealthy` and `status` is `degraded` while the configured store cannot be reached or fails its identity check, with HTTP 200 kept, so a store outage never drains a replica. +- Instances that share a catalog must share one backup store root (`[backup.store]`: one S3 bucket and prefix, or one shared mount that supports hard links, which the store's conditional create needs; some CIFS and FUSE mounts refuse `link(2)` with `EPERM`, and the server then reports the store as having failed validation, with the `link` error in the log line). A host-local directory configured at the same path on each host is not one root: once the catalog has rows for the root, the second instance finds the root's identity file missing or the newest backup's manifest absent and starts with the store unhealthy. While the catalog has no rows for the root (a fresh deployment, or one whose backups were purged), nothing tells the two directories apart: each instance creates its own identity file and store id and both start healthy, and the first backups then split the rows between two ids, which the next check reports as an identity mismatch that only `backup purge` resolves. Point every instance at one root before the first backup. Each instance on a host needs its own `spool_path`; the spool's `.lock` refuses a second instance. An existing spool directory must be owned by the server's user with mode `0700`, as the root must; a spool created `0755` stops startup naming it. On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL, MongoDB, and Cassandra it is not enforced. On PostgreSQL, if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. diff --git a/extenddb.sample.toml b/extenddb.sample.toml index 2443ccd3d..34d07a562 100755 --- a/extenddb.sample.toml +++ b/extenddb.sample.toml @@ -188,7 +188,30 @@ # failed_row_retention_secs = 604800 # FAILED tombstones kept for the operator (7 days) # reconcile_interval_secs = 60 # Reconciler cycle # max_concurrent_jobs = 4 # Backup and restore jobs running at once on this instance -# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store +# spool_path = "/var/lib/extenddb/spool" # Restore download area for an S3 store; required +# # with an S3 store except on file-backed SQLite, where +# # it defaults to backup-spool/ beside the database file + +# [backup.store] +# Where CreateBackup's backups will live, from a later release (this release's +# CreateBackup writes in-catalog backups with any store, and the store serves +# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem +# store at backups/ beside the database file. Absent on PostgreSQL, MongoDB, +# Cassandra and an in-memory SQLite database: no store, and CreateBackup +# writes in-catalog backups with a deprecation warning. kind = "disabled" +# opts out on any backend. Instances that share a catalog must share one +# store root (an S3 bucket and prefix, or a shared mount). +# kind = "filesystem" # "filesystem" | "s3" | "disabled" +# [backup.store.filesystem] +# path = "/var/lib/extenddb/backups" # Must exist, owned by the extenddb user, mode 0700 +# [backup.store.s3] +# bucket = "my-extenddb-backups" +# prefix = "extenddb/" # Optional key prefix +# region = "us-east-1" # Optional; the AWS SDK's chain otherwise +# endpoint = "http://127.0.0.1:9000" # Optional, for an S3-compatible server +# force_path_style = true # Required with an http:// endpoint +# upload_part_size_bytes = 8388608 # 5 MiB to 5 GiB +# max_concurrent_uploads = 4 # Import/Export file operations are disabled by default (secure default). # Configure allowed directories separately for imports and exports. diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py new file mode 100644 index 000000000..174c7216e --- /dev/null +++ b/tests/test_backup_store_config.py @@ -0,0 +1,1519 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 + +"""The `[backup.store]` configuration at the server's boundary. + +Each test starts its own `extenddb serve --foreground` on an isolated +deployment and reads `/health`, the server's log and the API's answers: + +- the component `backup_store` on `/health`: absent without a store, healthy + with a filesystem store, `checking` while an S3 store's first check is in + flight, unhealthy (status `degraded`) while an S3 endpoint refuses + connections or stalls, and healthy again once it answers, without a restart; +- the restore and delete of a store-backed backup refused with a + ValidationException naming the store's kind and the problem's category and + not its root while the store is checking or unhealthy; CreateBackup served + throughout (this release's create writes the in-catalog format, which + touches no store); +- a stalled S3 endpoint holding the listener for no longer than the server's + own startup, and each reconciler cycle for no longer than the check's bound; +- the configuration errors that stop startup with the key, path or bucket in + the message: an S3 store without `spool_path` on PostgreSQL, a `spool_path` + that is the database directory, a filesystem root that does not exist, and + a defaulted SQLite root under a group-writable parent configured explicitly + (the same root defaulted is a warning that leaves the store unconfigured + while the catalog has no rows for it; with rows for it the defaulted root + is handled as a configured one: unhealthy, reported, its rows refused); +- the default root beside a SQLite database file created `0700` with + `extenddb-store.json` in it, and a relative root resolved against the + shell's directory in daemon mode; +- `extenddb backup list` and `backup purge --confirm` on an empty store, and + `backup purge` refused while a server holds the SQLite serve lock (exit 6); +- on PostgreSQL, the backup commands ending with the exit code `serve` gives + for a catalog the factory refuses: 6 while a migration holds the serving + key, 3 on a catalog behind the binary. + +The binary at `EXTENDDB_BINARY` decides which cells run: a SQLite build runs +the SQLite cells, a PostgreSQL build (with +`EXTENDDB_TEST_PG_CONNECTION_STRING`) the PostgreSQL ones. The S3 cells need +moto at `EXTENDDB_TEST_S3_ENDPOINT` (an `http://127.0.0.1:` endpoint) +and skip with a reason otherwise. Excluded from the backend-agnostic suite: +`devtools/run-tests` runs this file after it, in its own step on a SQLite +build (CI starts moto for it) and in the CLI lifecycle list on PostgreSQL. +""" + +from __future__ import annotations + +import json +import os +import shutil +import signal +import socket +import ssl +import subprocess +import threading +import time +import urllib.request +import uuid + +import boto3 +import pytest +from botocore.exceptions import ClientError + +import lifecycle_helpers as lh + +# Absolute, since the deployments run the binary with `cwd` set to their own +# directory (the relative-root cell needs the shell's directory): CI names +# the binary as `target/release/extenddb`, which would otherwise be resolved +# against that directory. +EXTENDDB_BINARY = os.path.abspath(lh.EXTENDDB_BINARY) +S3_ENDPOINT = os.environ.get("EXTENDDB_TEST_S3_ENDPOINT", "").strip() +ADMIN_PASSWORD = "TestPass1!" +_DEVTOOLS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "devtools") + + +def _health(port: int) -> dict: + ctx = ssl.create_default_context() + ctx.check_hostname = False + ctx.verify_mode = ssl.CERT_NONE + req = urllib.request.Request(f"https://127.0.0.1:{port}/health") + with urllib.request.urlopen(req, context=ctx, timeout=5) as resp: + assert resp.status == 200 + return json.loads(resp.read()) + + +def _wait_health(port: int, predicate, timeout: float = 20.0, server: "_Server | None" = None) -> dict: + deadline = time.monotonic() + timeout + last = None + while time.monotonic() < deadline: + if server is not None and server.proc.poll() is not None: + raise AssertionError(f"the server exited {server.proc.returncode}:\n{server.log_text()}") + try: + last = _health(port) + if predicate(last): + return last + except Exception: # noqa: BLE001 - the server may not listen yet + pass + time.sleep(0.25) + log = server.log_text() if server is not None else "" + raise AssertionError(f"/health did not reach the expected state; last: {last!r}\n{log}") + + +def _append(config_path: str, toml: str) -> None: + with open(config_path, "a") as f: + f.write("\n" + toml + "\n") + + +def _free_port() -> int: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +class _Server: + """A `serve --foreground` process with its log in a file.""" + + def __init__(self, config_path: str, log_path: str, env: dict | None = None): + self.log_path = log_path + self.log = open(log_path, "wb") # noqa: SIM115 - closed in stop() + # A SQLite deployment's paths are relative to the config's directory. + self.proc = subprocess.Popen( + [EXTENDDB_BINARY, "serve", "--foreground", "--config", config_path], + stdout=self.log, + stderr=subprocess.STDOUT, + stdin=subprocess.DEVNULL, + env=env, + cwd=os.path.dirname(os.path.abspath(config_path)), + ) + + def log_text(self) -> str: + self.log.flush() + with open(self.log_path, errors="replace") as f: + return f.read() + + def wait_exit(self, timeout: float = 20.0) -> int: + return self.proc.wait(timeout=timeout) + + def stop(self) -> None: + if self.proc.poll() is None: + self.proc.terminate() + try: + self.proc.wait(timeout=20) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait(timeout=10) + self.log.close() + + +def _provision(port: int, cert_path: str, admin_password: str) -> dict: + env = os.environ.copy() + env.update( + { + "EXTENDDB_TEST_ENDPOINT": f"https://127.0.0.1:{port}", + "EXTENDDB_ADMIN_USER": "admin", + "EXTENDDB_ADMIN_PASSWORD": admin_password, + "EXTENDDB_CA_CERT": cert_path, + } + ) + res = subprocess.run( + ["python3", os.path.join(_DEVTOOLS, "provision-test-credentials")], + capture_output=True, + text=True, + env=env, + check=True, + timeout=60, + ) + creds = {} + for line in res.stdout.splitlines(): + line = line.strip() + if line.startswith("export "): + k, _, v = line[len("export "):].partition("=") + creds[k] = v.strip().strip('"').strip("'") + assert "AWS_ACCESS_KEY_ID" in creds, f"provisioning failed: {res.stdout}\n{res.stderr}" + return creds + + +def _client(port: int, creds: dict): + return boto3.client( + "dynamodb", + endpoint_url=f"https://127.0.0.1:{port}", + region_name="us-east-1", + aws_access_key_id=creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=creds["AWS_SECRET_ACCESS_KEY"], + verify=False, + ) + + +def _create_table(ddb, name: str) -> None: + ddb.create_table( + TableName=name, + KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}], + AttributeDefinitions=[{"AttributeName": "pk", "AttributeType": "S"}], + BillingMode="PAY_PER_REQUEST", + ) + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + if ddb.describe_table(TableName=name)["Table"]["TableStatus"] == "ACTIVE": + return + time.sleep(0.05) + raise AssertionError("table did not become ACTIVE") + + +class _Forwarder: + """A TCP forwarder from a local port to moto, started when the test wants + the S3 endpoint to start answering.""" + + def __init__(self, listen_port: int, upstream: tuple[str, int]): + self.listen_port = listen_port + self.upstream = upstream + self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.sock.bind(("127.0.0.1", listen_port)) + self.sock.listen(16) + self.stopping = threading.Event() + self.thread = threading.Thread(target=self._accept, daemon=True) + self.thread.start() + + def _pump(self, a: socket.socket, b: socket.socket) -> None: + try: + while not self.stopping.is_set(): + data = a.recv(65536) + if not data: + break + b.sendall(data) + except OSError: + pass + finally: + for s in (a, b): + try: + s.shutdown(socket.SHUT_RDWR) + except OSError: + pass + + def _accept(self) -> None: + self.sock.settimeout(0.5) + while not self.stopping.is_set(): + try: + client, _ = self.sock.accept() + except socket.timeout: + continue + except OSError: + break + try: + upstream = socket.create_connection(self.upstream, timeout=5) + except OSError: + client.close() + continue + threading.Thread(target=self._pump, args=(client, upstream), daemon=True).start() + threading.Thread(target=self._pump, args=(upstream, client), daemon=True).start() + + def stop(self) -> None: + self.stopping.set() + self.sock.close() + + +def _moto_or_skip() -> tuple[str, int]: + if not S3_ENDPOINT.startswith("http://"): + pytest.skip("EXTENDDB_TEST_S3_ENDPOINT is not an http:// moto endpoint") + host_port = S3_ENDPOINT[len("http://"):].rstrip("/") + host, _, port = host_port.partition(":") + return host, int(port) + + +def _s3_env() -> dict: + env = os.environ.copy() + env.update( + { + "AWS_ACCESS_KEY_ID": "testing", + "AWS_SECRET_ACCESS_KEY": "testing", + "AWS_REGION": "us-east-1", + "AWS_EC2_METADATA_DISABLED": "true", + } + ) + env.pop("AWS_SESSION_TOKEN", None) + env.pop("AWS_PROFILE", None) + return env + + +def _s3_client(): + return boto3.client( + "s3", + endpoint_url=S3_ENDPOINT, + region_name="us-east-1", + aws_access_key_id="testing", + aws_secret_access_key="testing", + ) + + +@pytest.fixture() +def moto_bucket(): + """A bucket of this test's own on the shared moto, emptied and deleted at + teardown so the endpoint does not accumulate one per run.""" + _moto_or_skip() + bucket = f"extenddb-wire-{uuid.uuid4().hex[:12]}" + s3 = _s3_client() + s3.create_bucket(Bucket=bucket) + yield bucket + try: + listed = s3.list_objects_v2(Bucket=bucket).get("Contents", []) + if listed: + s3.delete_objects(Bucket=bucket, Delete={"Objects": [{"Key": o["Key"]} for o in listed]}) + for upload in s3.list_multipart_uploads(Bucket=bucket).get("Uploads", []): + s3.abort_multipart_upload(Bucket=bucket, Key=upload["Key"], UploadId=upload["UploadId"]) + s3.delete_bucket(Bucket=bucket) + except ClientError: + pass + + +def _assert_store_refusal(exc: ClientError, kind: str, root_fragment: str, category: str = "unreachable") -> None: + """The refusal of a store-backed backup's restore or delete: a + ValidationException naming the store's kind and the problem's category + (`unreachable`, `manifest probe failed`, `first check is pending`) and + never the root.""" + err = exc.response["Error"] + assert err["Code"] == "ValidationException", err + assert f"{kind} backup store" in err["Message"], err + assert category in err["Message"], err + assert root_fragment not in err["Message"], err + + +_SOURCE_TABLE_JSON = json.dumps({ + "table_name": "Music", "table_id": "t-1", "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music", + "key_schema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "attribute_definitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "billing_mode": "PAY_PER_REQUEST", "provisioned_throughput": None, "on_demand_throughput": None, + "global_secondary_indexes": [], "local_secondary_indexes": [], "vector_indexes": None, + "stream_specification": None, "time_to_live": None, "sse_description": None, + "item_count": 0, "table_size_bytes": 0, "table_creation_date_time": 0.0, +}) + + +def _sqlite_module(): + try: + # The catalog's schema needs a SQLite newer than some hosts' stdlib + # module links; the wheel carries a current library. + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + return sqlite3 + + +def _seed_backup_row(db_path: str, backup_id: str, status: str, store_kind: str, store_root: str, store_id: str) -> str: + """One `backup_index` row of account 123456789012's `Music` table under + the given store, as a server that used the store leaves it; returns the + backup's ARN.""" + account = "123456789012" + arn = f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" + conn = _sqlite_module().connect(db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', ?, ?, ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, '2026-10-01T00:00:00.000Z')", + (account, backup_id, arn, status, store_kind, store_root, store_id, _SOURCE_TABLE_JSON), + ) + conn.commit() + finally: + conn.close() + return arn + + +# --- SQLite --------------------------------------------------------------------------- + + +def _sqlite_build() -> bool: + if not os.path.isfile(EXTENDDB_BINARY): + return False + res = subprocess.run( + [EXTENDDB_BINARY, "version"], capture_output=True, text=True, timeout=30 + ) + return "(sqlite)" in res.stdout + + +class _SqliteDeployment: + def __init__(self, tmp_dir: str): + self.dir = tmp_dir + self.config = os.path.join(tmp_dir, "extenddb.toml") + self.db_path = os.path.join(tmp_dir, "extenddb.sqlite") + self.port = _free_port() + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "sqlite", "--config", self.config], + cwd=tmp_dir, + capture_output=True, + text=True, + stdin=subprocess.DEVNULL, + timeout=60, + check=True, + env=env, + ) + lh._patch_config_port(self.config, self.port) + self.cert = os.path.expanduser("~/.extenddb/tls/cert.pem") + + def run(self, *args, env=None): + return subprocess.run( + [EXTENDDB_BINARY, args[0], "--config", self.config, *args[1:]], + cwd=self.dir, + capture_output=True, + text=True, + stdin=subprocess.DEVNULL, + timeout=180, + env=env, + ) + + +@pytest.fixture() +def sqlite_deployment(tmp_path): + if not _sqlite_build(): + pytest.skip("EXTENDDB_BINARY is not a SQLite build") + return _SqliteDeployment(str(tmp_path)) + + +class TestSqlite: + def test_default_root_is_created_private_and_reported_healthy(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health["status"] == "healthy", health + component = health["components"]["backup_store"] + assert component == {"status": "healthy", "kind": "filesystem"}, component + root = os.path.join(d.dir, "backups") + assert os.path.isdir(root) + assert (os.stat(root).st_mode & 0o777) == 0o700 + identity = os.path.join(root, "extenddb-store.json") + with open(identity) as f: + file = json.load(f) + assert file["format_version"] == 1 + assert file["backend"] == "sqlite" + assert file["store_id"] and file["catalog_id"] + assert (os.stat(identity).st_mode & 0o777) == 0o600 + log = server.log_text() + assert "backup store: filesystem" in log and root in log, log + assert f"store id {file['store_id']}" in log, log + # The console banner names the store beside the storage line. + assert " storage: sqlite" in log, log + # In the spelling the log line and the rows use (the canonical + # directory), with the configured spelling only where it differs. + canonical = os.path.realpath(root) + configured = f" (configured as {root})" if canonical != root else "" + assert f" backup store: filesystem {canonical}{configured} (the default beside the database file)" in log, log + assert f"backup store: filesystem {canonical} (store id" in log, log + # CreateBackup takes the in-catalog path and says the store is configured. + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + assert "writing the in-catalog backup format; the configured filesystem backup store" in server.log_text() + assert "configure [backup.store]" not in server.log_text() + finally: + server.stop() + + def test_disabled_leaves_no_component_and_the_legacy_create_warns(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + _append(d.config, '[backup.store]\nkind = "disabled"\n') + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health["status"] == "healthy" + assert "backup_store" not in health["components"], health + assert not os.path.exists(os.path.join(d.dir, "backups")) + assert 'backup store: disabled' in server.log_text() + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + ddb.create_backup(TableName="Music", BackupName="nightly") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + + def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sqlite_deployment, tmp_path): + """`sqlite-memory`'s shape (`[storage.sqlite] path = ":memory:"`, the + default of that build) on the SQLite build: with `[backup.store.filesystem] + path` set the store is configured and healthy, its identity file is + written, and a backup created there (in-catalog in this release, the + log naming the configured store) restores with its item; without + `[backup.store]` there is no default root (no database file to put it + beside), no component, and CreateBackup writes a legacy backup with the + warning naming `[backup.store]`.""" + d = sqlite_deployment + env = os.environ.copy() + env["EXTENDDB__STORAGE__SQLITE__PATH"] = ":memory:" + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + plain_config = os.path.join(d.dir, "plain.toml") + shutil.copyfile(d.config, plain_config) + root = tmp_path / "memory-root" + root.mkdir(mode=0o700) + _append(d.config, f'[backup.store]\nkind = "filesystem"\n\n[backup.store.filesystem]\npath = "{root}"\n') + + server = _Server(d.config, str(tmp_path / "serve-memory-store.log"), env=env) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["components"]["backup_store"] == {"status": "healthy", "kind": "filesystem"}, health + assert " storage: sqlite (:memory:)" in server.log_text(), "the server runs on an in-memory database" + assert os.path.isfile(root / "extenddb-store.json") + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + ddb.put_item(TableName="Music", Item={"pk": {"S": "a"}, "v": {"N": "1"}}) + arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] + deadline = time.monotonic() + 60 + while ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] != "AVAILABLE": + assert time.monotonic() < deadline, "the backup did not become AVAILABLE" + time.sleep(0.1) + assert "the configured filesystem backup store" in server.log_text(), server.log_text() + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + deadline = time.monotonic() + 60 + while ddb.describe_table(TableName="Restored")["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the restore did not finish" + time.sleep(0.1) + item = ddb.get_item(TableName="Restored", Key={"pk": {"S": "a"}})["Item"] + assert item == {"pk": {"S": "a"}, "v": {"N": "1"}}, item + finally: + server.stop() + + server = _Server(plain_config, str(tmp_path / "serve-memory-plain.log"), env=env) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert "backup_store" not in health["components"], health + assert " storage: sqlite (:memory:)" in server.log_text(), server.log_text() + assert not os.path.exists(os.path.join(d.dir, "backups")), "no default root beside an in-memory database" + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), details + assert "configure [backup.store] to write backups to a backup store" in server.log_text(), server.log_text() + finally: + server.stop() + + def test_default_root_under_a_group_writable_parent_warns_and_the_same_root_explicit_fails( + self, sqlite_deployment, tmp_path + ): + d = sqlite_deployment + os.chmod(d.dir, 0o775) + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert "backup_store" not in health["components"], health + log = server.log_text() + # `open` applies the ancestor rule; `validate` would report the + # same root the same way. Either is the warning path. + assert ("is not usable" in log or "failed its validation" in log) and "unconfigured" in log, log + assert "writable by group or others" in log, log + assert "kind = \"disabled\" to silence this" in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + ddb.create_backup(TableName="Music", BackupName="nightly") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + # The same root configured explicitly is a startup error naming it. + root = os.path.join(d.dir, "backups") + _append(d.config, f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n') + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log or "failed its startup validation" in log, log + assert "writable by group or others" in log and d.dir in log, log + finally: + server.stop() + + def test_a_missing_explicit_root_and_a_spool_in_the_database_directory_stop_startup( + self, sqlite_deployment, tmp_path + ): + d = sqlite_deployment + missing = os.path.join(d.dir, "nowhere") + _append(d.config, f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{missing}"\n') + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log and missing in log, log + finally: + server.stop() + # A spool that is the database directory. + (tmp_path / "second").mkdir() + d2 = _SqliteDeployment(str(tmp_path / "second")) + _append( + d2.config, + f'[backup]\nspool_path = "{d2.dir}"\n[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "b"\n', + ) + server = _Server(d2.config, str(tmp_path / "serve2.log")) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "backup.spool_path" in log and "database directory" in log, log + finally: + server.stop() + + def test_an_unreachable_s3_endpoint_starts_the_server_unhealthy_and_recovers( + self, sqlite_deployment, tmp_path, moto_bucket + ): + d = sqlite_deployment + moto_host, moto_port = _moto_or_skip() + bucket = moto_bucket + dead_port = _free_port() + _append( + d.config, + "[backup]\nreconcile_interval_secs = 1\n" + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "{bucket}"\nendpoint = "http://127.0.0.1:{dead_port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n', + ) + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + forwarder = None + try: + # The first check runs after the listener is up: the component is + # `checking` first, then `unhealthy` when the connection is refused. + health = _wait_health( + d.port, lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "unhealthy", + server=server, + ) + assert health["status"] == "degraded", health + component = health["components"]["backup_store"] + assert component["status"] == "unhealthy" and component["kind"] == "s3", component + assert component["reason"] == "unreachable", component + assert bucket not in json.dumps(component) + spool = os.path.join(d.dir, "backup-spool") + assert (os.stat(spool).st_mode & 0o777) == 0o700 + assert (os.stat(os.path.join(spool, ".lock")).st_mode & 0o777) == 0o600 + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + # The in-catalog create touches no store: it runs on an unhealthy + # one, as it does on a server with none. + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + log = server.log_text() + assert "is unhealthy: the store could not be reached" in log, log + assert bucket in log, "the operator's line names the bucket" + + # The endpoint starts answering: the next reconciler cycle recovers. + forwarder = _Forwarder(dead_port, (moto_host, moto_port)) + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "healthy", timeout=30, server=server + ) + assert health["status"] == "healthy" + assert "is healthy again" in server.log_text() + details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + # The identity file was created on the first passing check. + body = _s3_client().get_object(Bucket=bucket, Key="extenddb-store.json")["Body"].read() + assert json.loads(body)["backend"] == "sqlite" + + # And degrades again once the endpoint is gone. + forwarder.stop() + forwarder = None + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "unhealthy", timeout=30, server=server + ) + assert health["status"] == "degraded" + details = ddb.create_backup(TableName="Music", BackupName="again")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), "the in-catalog create is not gated" + finally: + if forwarder is not None: + forwarder.stop() + server.stop() + + def test_a_stalled_s3_endpoint_holds_neither_the_listener_nor_the_reconciler(self, sqlite_deployment, tmp_path): + """An endpoint that accepts connections and never answers. Checked: + the server binds and serves before the store's first check, which runs + after the listener is up (`/health` answers well within the SDK's + attempt timeout with the component `checking`); the in-catalog + CreateBackup is served meanwhile; the restore of a seeded store-backed + row is refused naming the pending check; the check is abandoned at its + 30 s bound, the component turns `unhealthy` (`unreachable`) without a + restart and the refusal names that; and the reconciler keeps cycling + while the endpoint stalls (the next cycle's check opens a new + connection to the tarpit within seconds of the abandoned one).""" + d = sqlite_deployment + tarpit = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + tarpit.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + tarpit.bind(("127.0.0.1", 0)) + tarpit.listen(16) + tarpit_port = tarpit.getsockname()[1] + held = [] + accepted_at = [] + + def accept_and_hold(): + while True: + try: + conn, _ = tarpit.accept() + except OSError: + return + held.append(conn) + accepted_at.append(time.monotonic()) + + threading.Thread(target=accept_and_hold, daemon=True).start() + # A row of the store, as a server that used it leaves one; the root + # label is the bucket and the endpoint in the spelling the store uses. + store_root = f"s3://tarpit-bucket/?endpoint=http://127.0.0.1:{tarpit_port}" + arn = _seed_backup_row(d.db_path, "01791976431438-00000001", "AVAILABLE", "s3", store_root, "store-seeded") + _append( + d.config, + "[backup]\nreconcile_interval_secs = 1\n" + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "tarpit-bucket"\nendpoint = "http://127.0.0.1:{tarpit_port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n', + ) + started = time.monotonic() + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + try: + health = _wait_health(d.port, lambda h: "components" in h, timeout=30, server=server) + answered_after = time.monotonic() - started + # The first answer is the server's own startup (TLS, the catalog + # open), about a second here; a check awaited before the listener + # would add the SDK's 60 s attempt timeout at the least. + assert answered_after < 30.0, f"/health first answered after {answered_after:.1f} s" + assert health["status"] == "healthy", health + assert health["components"]["backup_store"] == {"status": "checking", "kind": "s3"}, health + deadline = time.monotonic() + 20 + while not held and time.monotonic() < deadline: + time.sleep(0.1) + assert held, "the store's check reached the endpoint" + log = server.log_text() + assert "listening on" in log and "run after the listener is up" in log, log + assert "backup store: s3" in log and "tarpit-bucket" in log, log + # Requests are served while the check hangs: the in-catalog create + # runs, and the seeded row's restore is refused naming the pending + # check and the store's kind, not the bucket. + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="first check is pending") + assert "Restored" not in [t for t in ddb.list_tables()["TableNames"]] + + # The check is abandoned at its bound (30 s, under the SDK's 60 s + # attempt timeout): the component turns unhealthy without a + # restart, the log says the check was abandoned and the cycle's + # duties ran. + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "unhealthy", timeout=45, server=server + ) + assert health["status"] == "degraded", health + assert health["components"]["backup_store"]["reason"] == "unreachable", health + log = server.log_text() + assert "check did not finish within 30 s and was abandoned" in log, log + assert "is unhealthy: the store could not be reached: the check did not finish within 30 s" in log, log + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="unreachable") + # The reconciler keeps cycling: the next cycle's check opens a new + # connection to the tarpit within seconds of the abandoned one + # (the SDK's own retry would come 60 s after the first). + first = accepted_at[0] + deadline = time.monotonic() + 15 + while len(accepted_at) < 2 and time.monotonic() < deadline: + time.sleep(0.2) + assert len(accepted_at) >= 2, "the next cycle's check reached the endpoint" + assert accepted_at[1] - first < 45.0, f"the second connection came {accepted_at[1] - first:.1f} s after the first" + assert "is unhealthy" in log and log.count("is unhealthy: the store could not be reached") == 1, \ + "the transition is logged once" + details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + finally: + server.stop() + tarpit.close() + for conn in held: + conn.close() + # Shutdown did not wait on the stalled check. + with open(server.log_path, errors="replace") as f: + final_log = f.read() + assert "did not drain" not in final_log, final_log + + def test_a_default_root_with_rows_is_unhealthy_rather_than_unconfigured(self, sqlite_deployment, tmp_path): + """The default root's leniency ends with the catalog's first row for + it: a row seeded under the default root with its manifest missing + starts the server with the store configured and unhealthy (`/health` + degraded, `manifest probe failed`), the row's restore refused naming + the kind and the category and not the root, recovery when the manifest + is put back, and `backup purge --confirm` and `backup list` reaching + the store (the purge refuses on the unhealthy store, not for want of + one).""" + d = sqlite_deployment + _append(d.config, "[backup]\nreconcile_interval_secs = 1\n") + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + root = os.path.realpath(os.path.join(d.dir, "backups")) + with open(os.path.join(root, "extenddb-store.json")) as f: + store_id = json.load(f)["store_id"] + backup_id = "01791976431438-00000001" + arn = _seed_backup_row(d.db_path, backup_id, "AVAILABLE", "filesystem", root, store_id) + manifest_dir = os.path.join(root, "123456789012", "Music", backup_id) + manifest = os.path.join(manifest_dir, "extenddb-manifest.json") + + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["status"] == "degraded", health + assert health["components"]["backup_store"] == { + "status": "unhealthy", "kind": "filesystem", "reason": "manifest probe failed", + }, health + log = server.log_text() + assert "is unhealthy: the manifest of " + arn + " is not under the root" in log, log + assert "backup rows for it, so it is handled as a configured root" in log, log + assert "unconfigured" not in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") + with pytest.raises(ClientError) as refused: + ddb.delete_backup(BackupArn=arn) + _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") + # The in-catalog create is served on the unhealthy store. + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + # The manifest put back: the next cycle recovers the store. + for level in (os.path.dirname(os.path.dirname(manifest_dir)), os.path.dirname(manifest_dir), manifest_dir): + if not os.path.isdir(level): + os.mkdir(level, 0o700) + with open(manifest, "wb") as f: + f.write(b"{}") + health = _wait_health( + d.port, lambda h: h["components"]["backup_store"]["status"] == "healthy", timeout=30, server=server + ) + assert health["status"] == "healthy", health + assert "is healthy again" in server.log_text() + finally: + server.stop() + + # The commands reach the store: with the manifest gone again, the + # listing (a read of the file and the rows, no probe) shows the row + # under the store, and the purge, which runs the check, refuses on + # the unhealthy store and deletes nothing. + os.remove(manifest) + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert backup_id in res.stdout and f"backups in the filesystem store {root}" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + out = res.stdout + res.stderr + assert res.returncode != 0, out + assert "is unhealthy" in out and "manifest" in out, out + assert "no backup store is configured" not in out, out + res = d.run("backup", "list") + assert backup_id in res.stdout, "nothing was deleted: " + res.stdout + + def test_a_relative_root_resolves_against_the_shell_in_daemon_mode(self, sqlite_deployment, tmp_path): + """A relative `[backup.store.filesystem] path` is made absolute at + config load, against the shell's directory, so the daemonized child + (whose working directory is `/`) opens the same root the operator + named: the root beside the config gets the identity file and nothing + appears under `/`.""" + d = sqlite_deployment + os.mkdir(os.path.join(d.dir, "rel-backups"), 0o700) + run_dir = os.path.join(d.dir, "run") + with open(d.config) as f: + text = f.read() + import re + text, n = re.subn(r'(?m)^[ \t]*#?[ \t]*run_dir[ \t]*=.*$', f'run_dir = "{run_dir}"', text, count=1) + assert n == 1, text + with open(d.config, "w") as f: + f.write(text) + _append(d.config, '[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "rel-backups"\n') + res = subprocess.run( + [EXTENDDB_BINARY, "serve", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=60, + ) + try: + assert res.returncode == 0, res.stdout + res.stderr + # Absolute at load; the banner spells it as the rows do (the + # canonical directory), the configured spelling beside it only + # where the two differ. + absolute = os.path.join(d.dir, "rel-backups") + canonical = os.path.realpath(absolute) + configured = f" (configured as {absolute})" if canonical != absolute else "" + assert f"backup store: filesystem {canonical}{configured}\n" in res.stdout, res.stdout + _wait_health(d.port, lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy") + assert os.path.isfile(os.path.join(d.dir, "rel-backups", "extenddb-store.json")) + assert not os.path.exists("/rel-backups") + finally: + # Stop the daemon whatever the outcome above; a daemon that never + # started has no PID file and `stop` says so. + subprocess.run( + [EXTENDDB_BINARY, "stop", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=60, + ) + deadline = time.monotonic() + 30 + while time.monotonic() < deadline: + try: + _health(d.port) + except Exception: # noqa: BLE001 - the listener is gone + break + time.sleep(0.25) + + def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(self, sqlite_deployment, tmp_path): + """`destroy` then `init --overwrite` on the same directory: the default + root's identity file names the old catalog. The server starts with the + store unconfigured and a warning naming the file (not degraded, not a + startup error), CreateBackup keeps the in-catalog path, and `destroy`'s + output said to remove the file.""" + d = sqlite_deployment + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + identity = os.path.join(d.dir, "backups", "extenddb-store.json") + with open(identity) as f: + old_catalog = json.load(f)["catalog_id"] + res = d.run("destroy", "--yes") + assert res.returncode == 0, res.stdout + res.stderr + assert "needs that file removed" in res.stdout, res.stdout + assert os.path.isfile(identity) + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "sqlite", "--config", d.config, "--overwrite"], + cwd=d.dir, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=60, check=True, env=env, + ) + lh._patch_config_port(d.config, d.port) + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "components" in h, server=server) + assert health == {"status": "healthy", "components": {}}, health + log = server.log_text() + assert "failed its check" in log and "extenddb-store.json names catalog" in log, log + assert old_catalog in log and "destroy" in log and "unconfigured" in log, log + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + finally: + server.stop() + + def test_backup_purge_deletes_each_rows_prefix_and_row_and_leaves_the_rest(self, sqlite_deployment, tmp_path): + """Rows of every status seeded in the catalog with objects under their + prefixes: purge deletes each row's objects and the row outright; a + prefix the catalog has no row for, the identity file and `exports/` + stay; `backup list` shows the rows before and none after. + """ + try: + # The catalog's schema needs a SQLite newer than some hosts' + # stdlib module links; the wheel carries a current library. + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + + d = sqlite_deployment + # The shortest window the config accepts (three of the 15 s the jobs + # heartbeat at), so a purge that waited on a dead job would fail + # this test in 90 s rather than the default 600. + _append(d.config, "[backup]\nstale_job_after_secs = 45\n") + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + finally: + server.stop() + root = os.path.join(d.dir, "backups") + with open(os.path.join(root, "extenddb-store.json")) as f: + store_id = json.load(f)["store_id"] + source_table = json.dumps({ + "table_name": "Music", "table_id": "t-1", "table_arn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music", + "key_schema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "attribute_definitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "billing_mode": "PAY_PER_REQUEST", "provisioned_throughput": None, "on_demand_throughput": None, + "global_secondary_indexes": [], "local_secondary_indexes": [], "vector_indexes": None, + "stream_specification": None, "time_to_live": None, "sse_description": None, + "item_count": 0, "table_size_bytes": 0, "table_creation_date_time": 0.0, + }) + account = "123456789012" + rows = [ + ("01791976431438-00000001", "AVAILABLE"), + ("01791976431439-00000002", "FAILED"), + ("01791976431440-00000003", "DELETING"), + # A CREATING row whose owner is dead (its heartbeat is old). + ("01791976431441-00000004", "CREATING"), + # A FAILED and a CREATING row whose heartbeats are young (five + # seconds old): the stopped server's jobs wrote them, and under + # the serve lock those jobs are dead, so the command neither + # waits on them nor leaves them. + ("01791976431442-00000005", "FAILED"), + ("01791976431443-00000006", "CREATING"), + ] + young = {"01791976431442-00000005", "01791976431443-00000006"} + + def prefix(backup_id): + return os.path.join(root, account, "Music", backup_id) + + def private_dirs(path): + # Every level 0700: the store refuses a directory under its root + # that another user can reach (`makedirs` leaves the intermediate + # levels at the umask's mode). + parts = [] + while not os.path.isdir(path): + parts.append(path) + path = os.path.dirname(path) + for p in reversed(parts): + os.mkdir(p, 0o700) + + conn = sqlite3.connect(d.db_path, timeout=10) + try: + for backup_id, status in rows: + arn = f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" + heartbeat = ( + "strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-5 seconds')" if backup_id in young + else "'2026-10-01T00:00:00.000Z'" + ) + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', ?, 'filesystem', ?, ?, 1, 1, " + f"'2026-10-01T00:00:00.000Z', ?, 7, {heartbeat})", + (account, backup_id, arn, status, root, store_id, source_table), + ) + # A restore the stopped server left RUNNING (its target CREATING): + # under the serve lock its owner is dead, and the command resolves + # it before the wait instead of waiting on it. + conn.execute( + "INSERT INTO accounts (account_id, account_name) SELECT ?, 'purge-test' WHERE NOT EXISTS " + "(SELECT 1 FROM accounts WHERE account_id = ?)", + (account, account), + ) + conn.execute( + "INSERT INTO tables (account_id, table_name, key_schema, attribute_definitions, table_status, " + "table_arn, table_id) VALUES (?, 'Restored', '[]', '[]', 'CREATING', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Restored', 'tid-restored')", + (account,), + ) + conn.execute( + "INSERT INTO table_restores (table_id, source_backup_arn, restore_date_time, status, kind, " + "owner_generation, heartbeat_at) VALUES ('tid-restored', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01791976431400-00000000', " + "'2026-10-01T00:00:00.000Z', 'RUNNING', 'legacy', 9, '2026-10-01T00:00:00.000Z')", + ) + conn.commit() + finally: + conn.close() + for backup_id, status in rows: + private_dirs(os.path.join(prefix(backup_id), "data")) + # A CREATING backup has its data files and no manifest yet (the + # writer puts the manifest last), so nothing adopts a prefix a + # failure between purge's two steps leaves. + names = ["data/00001.json.gz"] if status == "CREATING" else ["extenddb-manifest.json", "data/00001.json.gz"] + for name in names: + with open(os.path.join(prefix(backup_id), name), "wb") as f: + f.write(b"x") + # A prefix with no row (another catalog's, or a copy) and exports/. + orphan = prefix("01791976431499-0000beef") + private_dirs(orphan) + with open(os.path.join(orphan, "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + private_dirs(os.path.join(root, "exports", "e-1")) + with open(os.path.join(root, "exports", "e-1", "manifest-summary.json"), "wb") as f: + f.write(b"x") + + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + for backup_id, status in rows: + assert backup_id in res.stdout and status in res.stdout, res.stdout + assert f"store_root={root}" in res.stdout and f"store_id={store_id}" in res.stdout, res.stdout + assert "6 row(s)" in res.stdout, res.stdout + res = d.run("backup", "list", "--failed") + assert res.returncode == 0 and "2 row(s)" in res.stdout and "FAILED" in res.stdout, res.stdout + + # The dry run prints the plan, changes nothing and exits 0. + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout and "--confirm" in res.stdout, res.stdout + for backup_id, _ in rows: + assert os.path.isfile(os.path.join(prefix(backup_id), "data", "00001.json.gz")), "nothing deleted without --confirm" + # `--config` after the subcommand is accepted too. + res = subprocess.run( + [EXTENDDB_BINARY, "backup", "purge", "--config", d.config], cwd=d.dir, capture_output=True, text=True, + stdin=subprocess.DEVNULL, timeout=120, + ) + assert res.returncode == 0 and "Dry run" in res.stdout, res.stdout + res.stderr + + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert "6 backup(s) deleted, 0 left" in res.stdout, res.stdout + # The interrupted restore was resolved before the wait, which then + # had nothing to wait for (the default window would be 600 s); the + # young FAILED and CREATING rows were neither waited on (the window + # is zero under the serve lock) nor left. + assert "removed the restore target Restored" in res.stdout, res.stdout + assert "Waiting for running jobs" not in res.stdout, "purge waited on a dead job: " + res.stdout + for backup_id, _ in rows: + assert not os.path.exists(prefix(backup_id)), f"{backup_id}'s prefix is gone" + assert f"deleted arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{backup_id}" in res.stdout + assert os.path.isfile(os.path.join(orphan, "extenddb-manifest.json")), "a prefix with no row stays" + assert os.path.isfile(os.path.join(root, "exports", "e-1", "manifest-summary.json")), "exports/ stays" + assert os.path.isfile(os.path.join(root, "extenddb-store.json")), "the identity file stays" + out = res.stdout + res.stderr + assert "extenddb::audit" in out and out.count("deleted backup objects and catalog row") == 6, out + conn = sqlite3.connect(d.db_path, timeout=10) + try: + assert conn.execute("SELECT count(*) FROM backup_index").fetchone()[0] == 0, "rows gone, no tombstones" + assert conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] == 0, "the claim is released" + assert conn.execute("SELECT count(*) FROM tables WHERE table_name = 'Restored'").fetchone()[0] == 0, \ + "the dead restore's target is gone" + finally: + conn.close() + res = d.run("backup", "list") + assert res.returncode == 0 and "(none)" in res.stdout, res.stdout + + # The wait's limit and its refusal: purge waits the window twice, + # refuses naming the job, releases the claim and deletes nothing. + # The lever is a RUNNING restore of kind `manifest` (a later release's + # job, which this release's reconciler leaves for its resumer): the + # wait lists RUNNING records by status, with no age, so a record no + # heartbeat ages out of holds it. Beside it a FAILED row whose + # heartbeat is stamped 120 s ahead of the clock (a backward clock + # step): under the zero window the serve lock gives every owner is + # dead whatever its heartbeat says, so the row is not waited on, and + # once the restore record is gone it is deleted outright. The rule + # the wait applies on the other backends, a FAILED row with a + # heartbeat within the window held while a job that has not yet seen + # its claim lost may still write under its prefix, is tested on + # PostgreSQL in `crates/storage-postgres/tests/backup_restore.rs` + # (`the_wait_passes_a_failed_record_and_holds_on_a_failed_row_with_a_live_heartbeat`). + def seed_manifest_restore(): + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute( + "INSERT INTO tables (account_id, table_name, key_schema, attribute_definitions, table_status, " + "table_arn, table_id) VALUES (?, 'Resumed', '[]', '[]', 'CREATING', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Resumed', 'tid-resumed')", + (account,), + ) + conn.execute( + "INSERT INTO table_restores (table_id, source_backup_arn, restore_date_time, status, kind, " + "owner_generation, heartbeat_at) VALUES ('tid-resumed', " + "'arn:aws:dynamodb:us-east-1:123456789012:table/Music/backup/01791976431400-00000000', " + "'2026-10-01T00:00:00.000Z', 'RUNNING', 'manifest', 9, strftime('%Y-%m-%dT%H:%M:%fZ', 'now'))", + ) + conn.commit() + finally: + conn.close() + + def remove_manifest_restore(): + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute("DELETE FROM table_restores WHERE table_id = 'tid-resumed'") + conn.execute("DELETE FROM tables WHERE table_id = 'tid-resumed'") + conn.commit() + finally: + conn.close() + + live_failed = "01791976431450-00000005" + seed_manifest_restore() + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', 'FAILED', 'filesystem', ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '+120 seconds'))", + (account, live_failed, f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{live_failed}", root, + store_id, source_table), + ) + conn.commit() + finally: + conn.close() + private_dirs(prefix(live_failed)) + with open(os.path.join(prefix(live_failed), "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + started = time.monotonic() + res = d.run("backup", "purge", "--confirm") + waited = time.monotonic() - started + assert res.returncode != 0, res.stdout + res.stderr + out = res.stdout + res.stderr + assert f"Waiting for running jobs to finish: a restore into Resumed (account {account}) is RUNNING" in out, out + assert "the FAILED backup" not in out, "a heartbeat ahead of the clock is not a live owner under the zero window: " + out + assert "jobs were still running after waiting" in out and "(2 * stale_job_after_secs)" in out, out + assert waited >= 90, f"the wait is twice the window: {waited:.1f} s" + assert os.path.isfile(os.path.join(prefix(live_failed), "extenddb-manifest.json")), "nothing deleted" + conn = sqlite3.connect(d.db_path, timeout=10) + try: + assert conn.execute("SELECT count(*) FROM backup_index").fetchone()[0] == 1 + assert conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] == 0, "the claim is released" + assert conn.execute("SELECT status FROM table_restores WHERE table_id = 'tid-resumed'").fetchone()[0] \ + == "RUNNING", "a manifest restore is left for its resumer" + finally: + conn.close() + # With the restore gone, the FAILED row stamped ahead of the clock is + # deleted outright, with no wait: under the serve lock its owner is + # dead whatever its heartbeat says. + remove_manifest_restore() + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert "Waiting for running jobs" not in res.stdout, res.stdout + assert "1 backup(s) deleted, 0 left" in res.stdout, res.stdout + assert not os.path.exists(prefix(live_failed)) + conn = sqlite3.connect(d.db_path, timeout=10) + try: + assert conn.execute("SELECT count(*) FROM backup_index").fetchone()[0] == 0 + finally: + conn.close() + + # A signal while the command waits releases the claim at once and + # deletes nothing; a SIGKILL leaves the claim's row, which the next + # server start clears, so CreateBackup is not refused for the + # window. The lever is the RUNNING manifest restore again, beside a + # FAILED row that must stay. + waited_on = "01791976431460-00000007" + seed_manifest_restore() + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, source_table_details_json, " + "owner_generation, heartbeat_at) VALUES (?, ?, 'Music', ?, 'nightly', 'FAILED', 'filesystem', ?, ?, 1, 1, " + "'2026-10-01T00:00:00.000Z', ?, 7, '2026-10-01T00:00:00.000Z')", + (account, waited_on, f"arn:aws:dynamodb:us-east-1:{account}:table/Music/backup/{waited_on}", root, + store_id, source_table), + ) + conn.commit() + finally: + conn.close() + private_dirs(prefix(waited_on)) + with open(os.path.join(prefix(waited_on), "extenddb-manifest.json"), "wb") as f: + f.write(b"x") + + def waiting_purge(): + proc = subprocess.Popen( + [EXTENDDB_BINARY, "backup", "--config", d.config, "purge", "--confirm"], cwd=d.dir, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, text=True, + ) + lines = [] + deadline = time.monotonic() + 60 + while time.monotonic() < deadline: + line = proc.stdout.readline() + if not line: + break + lines.append(line) + if "Waiting for running jobs to finish" in line: + return proc, lines + proc.kill() + raise AssertionError("purge did not reach its wait: " + "".join(lines)) + + def claim_rows(): + conn = sqlite3.connect(d.db_path, timeout=10) + try: + return conn.execute("SELECT count(*) FROM backup_maintenance").fetchone()[0] + finally: + conn.close() + + proc, lines = waiting_purge() + assert claim_rows() == 1, "the claim is held during the wait" + signalled = time.monotonic() + proc.send_signal(signal.SIGTERM) + rest = proc.communicate(timeout=30)[0] + out = "".join(lines) + rest + released_after = time.monotonic() - signalled + assert proc.returncode != 0, out + assert "SIGTERM received" in out and "stopped before any deletion: SIGTERM received while waiting" in out, out + assert "The maintenance claim is released" in out, out + assert released_after < 5, f"the signal ended the wait at once, not at the limit: {released_after:.1f} s" + assert claim_rows() == 0, "the claim is released on SIGTERM" + assert os.path.isfile(os.path.join(prefix(waited_on), "extenddb-manifest.json")), "nothing deleted" + + proc, lines = waiting_purge() + proc.kill() + proc.communicate(timeout=30) + assert claim_rows() == 1, "a killed purge leaves its claim's row" + # A second purge right after it proceeds to its wait: under the serve + # lock the killed holder is dead whatever its heartbeat says, so the + # claim is taken, not refused for the stale window. + proc, lines = waiting_purge() + out = "".join(lines) + assert "Took the maintenance claim" in out and "is held by" not in out, out + proc.kill() + proc.communicate(timeout=30) + assert claim_rows() == 1, "the second killed purge leaves its claim's row" + # The server starts on the file and clears the row; the first + # CreateBackup is accepted at once (it would be LimitExceeded for + # the 45 s window otherwise). + server = _Server(d.config, str(tmp_path / "serve-after-kill.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert claim_rows() == 0, "the server start cleared the left-behind claim" + assert "maintenance claim held by extenddb backup purge" in server.log_text() \ + and "was left behind; cleared" in server.log_text(), server.log_text() + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + _create_table(ddb, "Music") + details = ddb.create_backup(TableName="Music", BackupName="after-kill")["BackupDetails"] + assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), details + assert "LimitExceededException" not in server.log_text(), server.log_text() + finally: + server.stop() + conn = sqlite3.connect(d.db_path, timeout=10) + try: + conn.execute("DELETE FROM backup_index") + conn.commit() + finally: + conn.close() + remove_manifest_restore() + # The server's reconciler may already have deleted the stale FAILED + # row's objects (its heartbeat is old): the prefix is gone or ours to + # remove. + shutil.rmtree(prefix(waited_on), ignore_errors=True) + + # The server starts again against the purged root, healthy. + server = _Server(d.config, str(tmp_path / "serve2.log")) + try: + health = _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + assert health["components"]["backup_store"]["status"] == "healthy", health + finally: + server.stop() + + def test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock(self, sqlite_deployment, tmp_path): + d = sqlite_deployment + # Create the default root with a server once, then stop it. + # Before any server: `list` is a read that creates neither the default + # root nor its identity file, and says the root does not exist yet. + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "does not exist yet" in res.stdout and "(none)" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups")), "a read creates nothing" + # The dry run is the same read: it prints an empty plan naming the + # reason and exits 0. + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Store: none (the default root" in res.stdout and "does not exist yet" in res.stdout, res.stdout + assert "(none)" in res.stdout and "Dry run: nothing was deleted" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups")), "a read creates nothing" + # The root created by hand, empty: `list` and the dry run read it and + # leave it without an identity file. + os.mkdir(os.path.join(d.dir, "backups"), 0o700) + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "has no extenddb-store.json yet" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout and "no extenddb-store.json yet" in res.stdout, res.stdout + assert not os.path.exists(os.path.join(d.dir, "backups", "extenddb-store.json")), "a read creates nothing" + server = _Server(d.config, str(tmp_path / "serve.log")) + try: + _wait_health(d.port, lambda h: "backup_store" in h.get("components", {}), server=server) + # While the server runs, `list` and the dry run are reads beside + # it (no serve lock); `purge --confirm` opens the catalog as a + # server does and refuses. + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "backups in the filesystem store" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + # The serve lock held is the catalog refusal `serve` gives for + # it: exit 6, the code `migrate` uses for a locked catalog. + assert res.returncode == 6, res.stdout + res.stderr + assert "another extenddb process" in (res.stdout + res.stderr), res.stdout + res.stderr + assert server.proc.poll() is None, "the server kept running" + finally: + server.stop() + res = d.run("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr + assert "backups in the filesystem store" in res.stdout and "(none)" in res.stdout, res.stdout + res = d.run("backup", "list", "--failed") + assert res.returncode == 0 and "(none)" in res.stdout, res.stdout + res.stderr + res = d.run("backup", "purge") + assert res.returncode == 0, res.stdout + res.stderr + assert "Dry run: nothing was deleted" in res.stdout, res.stdout + res = d.run("backup", "purge", "--confirm") + assert res.returncode == 0, res.stdout + res.stderr + assert "0 backup(s) deleted" in res.stdout, res.stdout + assert "Took the maintenance claim" in res.stdout, res.stdout + assert os.path.isfile(os.path.join(d.dir, "backups", "extenddb-store.json")) + # `destroy` names the store as left untouched. + res = d.run("destroy", "--yes") + assert res.returncode == 0, res.stdout + res.stderr + assert "backup store ([backup.store]) is left untouched" in res.stdout, res.stdout + assert os.path.isfile(os.path.join(d.dir, "backups", "extenddb-store.json")) + + +# --- PostgreSQL ----------------------------------------------------------------------- + + +def _pg_build() -> bool: + if not os.path.isfile(EXTENDDB_BINARY): + return False + res = subprocess.run([EXTENDDB_BINARY, "version"], capture_output=True, text=True, timeout=30) + return "(postgres)" in res.stdout + + +@pytest.fixture() +def pg_env(request): + if not _pg_build(): + pytest.skip("EXTENDDB_BINARY is not a PostgreSQL build") + # `cli_env` requires PostgreSQL and a binary; it is requested only once + # the build is known to be the PostgreSQL one, so the other build skips. + cli_env = request.getfixturevalue("cli_env") + result = lh._run_extenddb( + "init", *lh._init_args(cli_env), config=cli_env["config_path"], + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(cli_env["config_path"], cli_env["port"]) + return cli_env + + +class TestPostgres: + def test_without_a_store_there_is_no_component_and_create_backup_warns(self, pg_env, tmp_path): + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log")) + try: + health = _wait_health(pg_env["port"], lambda h: "components" in h, server=server) + assert health == {"status": "healthy", "components": {}}, health + log = server.log_text() + assert "backup store: none configured" in log, log + creds = _provision(pg_env["port"], os.path.join(pg_env["tls_dir"], "cert.pem"), ADMIN_PASSWORD) + ddb = _client(pg_env["port"], creds) + _create_table(ddb, "Music") + arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] + assert "configure [backup.store] to write backups to a backup store" in server.log_text() + # The in-catalog backup is served. + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupArn"] == arn + listed = ddb.list_backups(TableName="Music")["BackupSummaries"] + assert [b["BackupArn"] for b in listed] == [arn] + finally: + server.stop() + + def test_an_s3_store_without_a_spool_stops_startup(self, pg_env, tmp_path): + _append(pg_env["config_path"], '[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "b"\n') + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log"), env=_s3_env()) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "backup.spool_path is required" in log, log + finally: + server.stop() + + def test_a_filesystem_store_is_healthy_and_a_bad_bucket_name_stops_startup(self, pg_env, tmp_path): + root = tmp_path / "backups" + root.mkdir(mode=0o700) + _append( + pg_env["config_path"], + f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n', + ) + server = _Server(pg_env["config_path"], str(tmp_path / "serve.log")) + try: + health = _wait_health(pg_env["port"], lambda h: "components" in h, server=server) + assert health["components"]["backup_store"] == {"status": "healthy", "kind": "filesystem"} + with open(root / "extenddb-store.json") as f: + assert json.load(f)["backend"] == "postgres" + finally: + server.stop() + # A bucket name the SDK routes to S3 Express is a configuration error + # naming the bucket. + with open(pg_env["config_path"]) as f: + text = f.read() + text = text[: text.index("[backup.store]")] + with open(pg_env["config_path"], "w") as f: + f.write(text) + _append( + pg_env["config_path"], + f'[backup]\nspool_path = "{tmp_path / "spool"}"\n' + '[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "my-bucket--usw2-az1--x-s3"\n', + ) + server = _Server(pg_env["config_path"], str(tmp_path / "serve2.log"), env=_s3_env()) + try: + assert server.wait_exit() != 0 + log = server.log_text() + assert "cannot be opened" in log and "my-bucket--usw2-az1--x-s3" in log, log + finally: + server.stop() + + def test_a_refused_catalog_ends_the_backup_commands_with_the_serve_exit_codes(self, pg_env, tmp_path): + """Every `backup` command opens the catalog through the factory and + holds a serving registration, so a catalog the factory refuses ends + it with the message and exit code `serve` gives: 6 while a migration + holds the serving key (naming the migration's session, without the + server's advice to start the server), 3 on a catalog behind the + binary (with the `extenddb migrate --yes` advice).""" + import psycopg2 + + config = pg_env["config_path"] + + def command(*args): + return lh._run_extenddb(*args, config=config, check=False, timeout=60) + + # A migration's hold of the serving key, as `migrate --yes` takes it: + # the exclusive session lock on (ExtendDB's namespace, serving key). + conn = psycopg2.connect(lh.PG_ADMIN_CONN + "/" + pg_env["db_name"]) + conn.autocommit = True + try: + with conn.cursor() as cur: + cur.execute("SET application_name = 'holds the serving key'") + cur.execute("SELECT pg_advisory_lock(%s, 2)", (0x00454442,)) + for args in (("backup", "list"), ("backup", "purge"), ("backup", "purge", "--confirm")): + res = command(*args) + out = res.stdout + res.stderr + assert res.returncode == 6, f"{args}: {out}" + assert "a migration is running on this catalog" in out, out + assert "holds the serving key" in out, out + assert "start the server" not in out, out + assert "Retry once it has finished" in out, out + finally: + conn.close() + + res = command("migrate", "--forget", "6", *lh._pg_args()) + assert res.returncode == 0, res.stdout + res.stderr + for args in (("backup", "list"), ("backup", "purge", "--confirm")): + res = command(*args) + out = res.stdout + res.stderr + assert res.returncode == 3, f"{args}: {out}" + assert "the catalog is at migration 5" in out, out + assert "extenddb migrate --config" in out and "--yes" in out, out + res = command("migrate", "--yes", *lh._pg_args()) + assert res.returncode == 0, res.stdout + res.stderr + res = command("backup", "list") + assert res.returncode == 0, res.stdout + res.stderr From 33a8e9c75e529f05ed778a3345b49dbf85115e08 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Wed, 4 Nov 2026 10:13:25 +0000 Subject: [PATCH 2/3] feat(backup): CreateBackup job, worker and reconciler duties With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df3: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee Signed-off-by: Scott Robinson --- .github/workflows/clippy.yml | 4 + .github/workflows/integration-cassandra.yml | 8 + .github/workflows/integration-mongodb.yml | 60 +- .github/workflows/integration.yml | 177 +- CONTRIBUTING.md | 5 +- Cargo.lock | 4 + crates/app/Cargo.toml | 1 + crates/app/src/cmd_backup.rs | 18 + crates/app/src/cmd_stop.rs | 51 +- crates/app/src/init_helpers.rs | 9 +- crates/app/src/lib.rs | 7 + crates/bin/src/main.rs | 2 + crates/config/src/backup.rs | 85 +- crates/core/src/backup_format/data_file.rs | 121 +- crates/core/src/backup_format/key_order.rs | 119 +- crates/core/src/backup_format/mod.rs | 4 +- crates/core/src/limits/mod.rs | 8 +- crates/core/src/metrics/collector.rs | 44 + crates/core/src/metrics/collector_tests.rs | 57 + crates/core/src/metrics/mod.rs | 6 +- crates/core/src/metrics/types.rs | 59 + crates/core/src/validation/mod.rs | 5 + crates/core/src/validation/restorable_item.rs | 1514 +++++++ crates/engine/Cargo.toml | 8 + crates/engine/src/backup.rs | 202 +- crates/engine/src/backup/job.rs | 664 +++ crates/engine/src/backup/job/failure.rs | 145 + crates/engine/src/backup/job/heartbeat.rs | 271 ++ crates/engine/src/backup/job/writer.rs | 742 ++++ crates/engine/src/backup/reconciler.rs | 592 ++- crates/engine/src/backup/runtime.rs | 275 +- crates/engine/src/backup/tests.rs | 3910 ++++++++++++++++- crates/engine/src/lib.rs | 7 +- crates/server/Cargo.toml | 3 + crates/server/src/backup_store.rs | 115 +- crates/server/src/lib.rs | 46 +- crates/server/src/metrics_endpoint.rs | 41 + crates/server/src/registration.rs | 6 +- crates/server/src/serve.rs | 263 +- .../storage-cassandra/src/backup_catalog.rs | 205 +- crates/storage-cassandra/src/test_hooks.rs | 29 +- .../tests/direct/backup_engine.rs | 401 ++ crates/storage-mongodb/src/backup_catalog.rs | 121 +- .../storage-mongodb/src/backup_live_tests.rs | 140 + crates/storage-mongodb/src/snapshot_engine.rs | 7 +- crates/storage-postgres/src/backup_catalog.rs | 45 +- crates/storage-postgres/src/backup_engine.rs | 60 + crates/storage-postgres/src/lib.rs | 13 +- .../storage-postgres/src/snapshot_engine.rs | 805 +++- .../storage-postgres/tests/backup_restore.rs | 261 +- .../storage-postgres/tests/table_snapshot.rs | 686 ++- crates/storage-sqlite/Cargo.toml | 5 + crates/storage-sqlite/src/backup.rs | 383 ++ crates/storage-sqlite/src/backup_catalog.rs | 576 ++- crates/storage-sqlite/src/config.rs | 16 +- crates/storage-sqlite/src/lib.rs | 19 +- crates/storage-sqlite/src/snapshot_engine.rs | 247 +- crates/storage-sqlite/src/store.rs | 121 +- .../storage-sqlite/tests/backup_job_panic.rs | 338 ++ crates/storage/src/backup_catalog.rs | 69 +- crates/storage/src/snapshot.rs | 416 +- crates/storage/src/snapshot_conformance.rs | 248 ++ devtools/backup-kill-points.py | 1204 +++++ devtools/backup-kill-points.sh | 284 ++ devtools/run-mongodb-tests | 29 +- devtools/run-tests | 68 +- docker-compose.cassandra.yml | 2 +- docker-compose.mongodb.yml | 2 +- docker-compose.yml | 2 +- docs/design/15-backup-restore-bench-pr8.md | 481 ++ docs/design/15-backup-restore.md | 61 +- docs/design/README.md | 1 + docs/differences-from-dynamodb.md | 13 +- docs/dynamodb-limits.md | 9 +- docs/manuals/01-architecture-guide.md | 2 +- docs/manuals/03-usage-guide.md | 2 +- docs/manuals/05-admin-guide.md | 34 +- docs/manuals/06-developer-test-guide.md | 18 +- docs/manuals/07-upgrade-manual.md | 6 +- docs/manuals/11-deployment-guide.md | 17 +- docs/troubleshooting.md | 4 +- extenddb.sample.toml | 9 +- tests/rust/src/backup_arn_scoping.rs | 54 +- tests/rust/src/backup_restore.rs | 191 +- tests/rust/src/restore_active_completeness.rs | 54 +- tests/test_abac.py | 9 +- tests/test_auth_permissions.py | 16 + tests/test_authz_resource_arn.py | 16 +- tests/test_backup_job.py | 662 +++ tests/test_backup_restore_fidelity.py | 230 +- tests/test_backup_store_config.py | 154 +- tests/test_mongodb_backup_restore.py | 11 + 92 files changed, 17591 insertions(+), 923 deletions(-) create mode 100644 crates/core/src/validation/restorable_item.rs create mode 100644 crates/engine/src/backup/job.rs create mode 100644 crates/engine/src/backup/job/failure.rs create mode 100644 crates/engine/src/backup/job/heartbeat.rs create mode 100644 crates/engine/src/backup/job/writer.rs create mode 100644 crates/storage-sqlite/tests/backup_job_panic.rs create mode 100755 devtools/backup-kill-points.py create mode 100755 devtools/backup-kill-points.sh create mode 100644 docs/design/15-backup-restore-bench-pr8.md create mode 100644 tests/test_backup_job.py diff --git a/.github/workflows/clippy.yml b/.github/workflows/clippy.yml index 8130ef51f..fafd81908 100644 --- a/.github/workflows/clippy.yml +++ b/.github/workflows/clippy.yml @@ -36,3 +36,7 @@ jobs: # hook modules (`src/test_hooks.rs`) and their call sites, which only # the integration test binaries build otherwise, are linted too. - run: cargo clippy --all-targets --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks -- -D warnings + # The backup job's kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), which + # `devtools/backup-kill-points.sh` runs by hand on a build with the + # feature; no other job builds it. + - run: cargo clippy --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings diff --git a/.github/workflows/integration-cassandra.yml b/.github/workflows/integration-cassandra.yml index 3d64445b7..8ec847ce2 100644 --- a/.github/workflows/integration-cassandra.yml +++ b/.github/workflows/integration-cassandra.yml @@ -173,6 +173,10 @@ jobs: - name: Build release (cassandra backend) run: cargo build --release --no-default-features --features cassandra - name: Run Cassandra pytest suite + env: + # run-cassandra-tests initializes the server without [backup.store], + # and Cassandra takes the in-catalog create with one. + EXTENDDB_TEST_BACKUP_STORE: none run: devtools/run-cassandra-tests -- --pytest --comprehensive --parallel cassandra-rust: @@ -228,6 +232,10 @@ jobs: - name: Build release (cassandra backend) run: cargo build --release --no-default-features --features cassandra - name: Run Cassandra rust integration suite + env: + # run-cassandra-tests initializes the server without [backup.store]: + # the Rust backup tests restore the in-catalog create's backups. + EXTENDDB_TEST_BACKUP_STORE: none run: devtools/run-cassandra-tests -- --rust-integration cassandra-production-build: diff --git a/.github/workflows/integration-mongodb.yml b/.github/workflows/integration-mongodb.yml index 3bf940c58..1b5f84c77 100644 --- a/.github/workflows/integration-mongodb.yml +++ b/.github/workflows/integration-mongodb.yml @@ -53,6 +53,10 @@ jobs: - name: Build release (mongodb backend) run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB pytest suite + env: + # run-mongodb-tests initializes the server without [backup.store]: + # CreateBackup is the in-catalog create. + EXTENDDB_TEST_BACKUP_STORE: none run: devtools/run-mongodb-tests -- --pytest --comprehensive --parallel # SnapshotEngine consistency check against a disposable replica set. The @@ -129,8 +133,61 @@ jobs: - name: Build release (mongodb backend) run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB rust integration suite + env: + # run-mongodb-tests initializes the server without [backup.store]: + # the Rust backup tests restore the in-catalog create's backups. + EXTENDDB_TEST_BACKUP_STORE: none run: devtools/run-mongodb-tests --rust-live -- --rust --rust-integration + # The store-backed CreateBackup on MongoDB: a server with a filesystem + # store, the backup selection of the pytest suite with + # EXTENDDB_TEST_BACKUP_STORE=filesystem (the fidelity suite's store branch, + # the ABAC and authorization backup cells), and the job suite + # (tests/test_backup_job.py), which starts servers of its own on a replica + # set of its own: the catalog's database names are fixed per replica set, + # and each of its tests destroys and initializes the catalog. + mongodb-backup-store: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 40 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + # Only pushes to main save, so other runs restore main's cache. + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + run: pip install -r requirements.txt + - name: Build release (mongodb backend) + run: cargo build --release --no-default-features --features mongodb-test-hooks + - name: Run the backup selection against a filesystem store + env: + EXTENDDB_TEST_BACKUP_STORE: filesystem + run: | + devtools/run-mongodb-tests --mongo-port 27022 --port 18453 \ + --backup-store-root "$RUNNER_TEMP/backups" \ + -- --pytest --parallel --filter "backup" + - name: Start the job suite's replica set + # No pipe: a bootstrap failure must fail this step. + run: devtools/run-mongodb-tests --mongo-only --mongo-port 27023 + - name: Run the backup job suite + env: + EXTENDDB_BINARY: ./target/release/extenddb + EXTENDDB_TEST_MONGODB_URI: mongodb://127.0.0.1:27023/?replicaSet=rs0&directConnection=true + run: python3 -m pytest tests/test_backup_job.py -v -rP -p no:cacheprovider + - name: Stop the job suite's replica set + if: always() + run: docker rm -f extenddb-runtests-mongo-27023 + - name: Dump server syslog on failure + if: failure() + run: sudo journalctl -t extenddb --no-pager -n 500 || true + # The catalog migrations against released catalogs and released binaries, # each check on a mongo:7 replica set of its own that the script starts # with its ports on 127.0.0.1 and removes. The schema check uploads the @@ -191,12 +248,13 @@ jobs: mongodb-integration: runs-on: extenddb_ubuntu-2404_4-core timeout-minutes: 40 - needs: [mongodb-pytest, mongodb-rust, mongodb-catalog, mongodb-production-build, mongodb-snapshot] + needs: [mongodb-pytest, mongodb-rust, mongodb-backup-store, mongodb-catalog, mongodb-production-build, mongodb-snapshot] if: always() steps: - run: | if [ "${{ needs.mongodb-pytest.result }}" != "success" ] || \ [ "${{ needs.mongodb-rust.result }}" != "success" ] || \ + [ "${{ needs.mongodb-backup-store.result }}" != "success" ] || \ [ "${{ needs.mongodb-catalog.result }}" != "success" ] || \ [ "${{ needs.mongodb-production-build.result }}" != "success" ] || \ [ "${{ needs.mongodb-snapshot.result }}" != "success" ]; then diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index b9acb4b03..4ea07acce 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -71,19 +71,34 @@ jobs: password=$(echo "$output" | grep -oP 'Password: \K\S+') echo "admin_password=$password" >> "$GITHUB_OUTPUT" + # A filesystem backup store, so this job's CreateBackup is the + # store-backed job (PostgreSQL has no default root). The suite reads + # EXTENDDB_TEST_BACKUP_STORE=filesystem below to expect that path. + - name: Configure a filesystem backup store + run: | + mkdir -m 0700 "$RUNNER_TEMP/backups" + printf '\n[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "%s/backups"\n' \ + "$RUNNER_TEMP" >> extenddb.toml + - name: Start ExtendDB run: | # --write-pid-file so devtools/run-tests can restart the server with # 'extenddb stop' when it needs to apply a config change. + # The wait is for the backup store component to be healthy, not for + # the word "healthy" anywhere in the reply: a server whose store + # failed its check reports "unhealthy" (which that word matches) + # and the suite would then fail on a backup cell with no pointer to + # the cause. ./target/release/extenddb serve --config extenddb.toml --foreground --write-pid-file & for i in $(seq 1 30); do - if curl -sk https://127.0.0.1:18443/health | grep -q healthy; then - echo "Server ready" + if curl -sk https://127.0.0.1:18443/health | python3 -c 'import json, sys; sys.exit(0 if json.load(sys.stdin).get("components", {}).get("backup_store", {}).get("status") == "healthy" else 1)' 2>/dev/null; then + echo "Server ready with a healthy backup store" exit 0 fi sleep 1 done - echo "Server failed to start" + echo "Server failed to start, or its backup store did not become healthy" + curl -sk https://127.0.0.1:18443/health || true exit 1 - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 @@ -154,8 +169,17 @@ jobs: EXTENDDB_ADMIN_USER: admin EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + # The backup store and job suites start servers of their own, so they + # are excluded from run-tests' line and run by name. + - name: Run the backup store and job suites + env: + EXTENDDB_TEST_PG_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 + EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 + run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider + # run-tests restarts the server daemonized to apply the import/export # config, and a daemonized server logs to syslog, not the job log. Without # this dump, the storage error behind a client-visible 500 leaves no @@ -228,15 +252,21 @@ jobs: run: | # --write-pid-file so devtools/run-tests can restart the server with # 'extenddb stop' when it needs to apply a config change. + # The wait is for the backup store component to be healthy, not for + # the word "healthy" anywhere in the reply: a server whose store + # failed its check reports "unhealthy" (which that word matches) + # and the suite would then fail on a backup cell with no pointer to + # the cause. ./target/release/extenddb serve --config extenddb.toml --foreground --write-pid-file & for i in $(seq 1 30); do - if curl -sk https://127.0.0.1:18443/health | grep -q healthy; then - echo "Server ready" + if curl -sk https://127.0.0.1:18443/health | python3 -c 'import json, sys; sys.exit(0 if json.load(sys.stdin).get("components", {}).get("backup_store", {}).get("status") == "healthy" else 1)' 2>/dev/null; then + echo "Server ready with a healthy backup store" exit 0 fi sleep 1 done - echo "Server failed to start" + echo "Server failed to start, or its backup store did not become healthy" + curl -sk https://127.0.0.1:18443/health || true exit 1 - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 @@ -273,8 +303,14 @@ jobs: # The backup store wire tests' S3 cells; the file sets the AWS_* # variables for the servers it starts, so none leaks here. EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9200 + # A file-backed SQLite server without [backup.store] writes to the + # defaulted root beside the database file. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + - name: Run the backup store and job suites + run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider + # Same rationale as the postgres job: the daemonized server logs to # syslog, so this dump is the only server-side evidence on failure. - name: Dump server syslog on failure @@ -285,6 +321,122 @@ jobs: if: failure() run: cat moto.log || true + # The S3 leg: the SQLite build against an S3 backup store on moto. The + # backup tests of the main suite (the tests named "backup") run against it + # with EXTENDDB_TEST_BACKUP_STORE=s3, and the fidelity suite's health check + # confirms the server's store is the S3 one; the store configuration + # suite's S3 cells need the same endpoint. The job suite + # (tests/test_backup_job.py) starts servers of its own with the SQLite + # build's defaulted filesystem root, as on the SQLite job: its deployment + # fixture has no S3 branch, so it covers the job and not this store. + run-integration-sqlite-s3: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 40 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + # Only pushes to main save, so other runs restore main's cache. + save-if: ${{ github.ref == 'refs/heads/main' }} + + - name: Build release (SQLite backend) + run: cargo build --release -p extenddb --no-default-features --features sqlite + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.12" + + - name: Install Python dependencies + run: pip install -r requirements.txt + + # moto is a pip package pinned in requirements-moto.txt (test.yml runs + # the same server for the store crate's conformance suite); its boto3 + # floor differs from requirements.txt's pin, so it gets an environment + # of its own. + - name: Start S3 mock + run: | + python3 -m venv "$RUNNER_TEMP/moto" + "$RUNNER_TEMP/moto/bin/pip" install -r requirements-moto.txt + nohup "$RUNNER_TEMP/moto/bin/moto_server" -p 9000 > moto.log 2>&1 & + for _ in $(seq 1 30); do + if curl -sf http://127.0.0.1:9000/ > /dev/null; then exit 0; fi + sleep 1 + done + echo "moto_server did not start"; cat moto.log; exit 1 + + - name: Initialize ExtendDB + id: init + env: + AWS_ACCESS_KEY_ID: testing + AWS_SECRET_ACCESS_KEY: testing + AWS_REGION: us-east-1 + run: | + output=$(./target/release/extenddb init --backend sqlite --config extenddb.toml 2>&1) + echo "$output" + password=$(echo "$output" | grep -oP 'Password: \K\S+') + echo "admin_password=$password" >> "$GITHUB_OUTPUT" + python3 -c 'import boto3; boto3.client("s3", endpoint_url="http://127.0.0.1:9000", region_name="us-east-1").create_bucket(Bucket="extenddb-ci-backups")' + printf '\n[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "extenddb-ci-backups"\nendpoint = "http://127.0.0.1:9000"\nforce_path_style = true\nregion = "us-east-1"\n' >> extenddb.toml + + - name: Start ExtendDB + env: + AWS_ACCESS_KEY_ID: testing + AWS_SECRET_ACCESS_KEY: testing + AWS_REGION: us-east-1 + AWS_EC2_METADATA_DISABLED: "true" + run: | + # An S3 store is checked after the listener is up and reports + # "checking" until its first check returns; a CreateBackup inside + # that window is refused, so the wait is for the component to be + # healthy, which also names a store that failed its check here + # rather than in a backup cell. + ./target/release/extenddb serve --config extenddb.toml --foreground --write-pid-file & + for i in $(seq 1 30); do + if curl -sk https://127.0.0.1:18443/health | python3 -c 'import json, sys; sys.exit(0 if json.load(sys.stdin).get("components", {}).get("backup_store", {}).get("status") == "healthy" else 1)' 2>/dev/null; then + echo "Server ready with a healthy backup store" + exit 0 + fi + sleep 1 + done + echo "Server failed to start, or its backup store did not become healthy" + curl -sk https://127.0.0.1:18443/health || true + exit 1 + + - name: Run the backup tests against the S3 store + env: + EXTENDDB_TEST_ENDPOINT: https://127.0.0.1:18443 + EXTENDDB_ADMIN_USER: admin + EXTENDDB_ADMIN_PASSWORD: ${{ steps.init.outputs.admin_password }} + EXTENDDB_TEST_BACKUP_STORE: s3 + # run-tests restarts the server daemonized, which needs the store's + # credentials in its environment too. + AWS_ACCESS_KEY_ID: testing + AWS_SECRET_ACCESS_KEY: testing + AWS_REGION: us-east-1 + AWS_EC2_METADATA_DISABLED: "true" + run: devtools/run-tests --extenddb --pytest --parallel --filter "backup" + + # The store configuration suite's S3 cells against moto; the job suite + # against the build's defaulted filesystem root (the job comment above). + - name: Run the store configuration suite's S3 cells and the job suite + env: + EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9000 + run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider + + - name: Dump server syslog on failure + if: failure() + run: sudo journalctl -t extenddb --no-pager -n 500 || true + + - name: Print the S3 mock's log + if: failure() + run: cat moto.log || true + run-integration-dev-mode: # dev-mode was shipped with no CI coverage at all, which is how the batch and # transaction authorization regression reached main: the build compiled, so @@ -426,6 +578,9 @@ jobs: # tests mandatory. The refusal suite moves to the novector job, which is # the only place those assertions still mean anything. EXTENDDB_EXPECT_VECTORS: "1" + # This server has no [backup.store]: CreateBackup is the in-catalog + # create and the backup tests restore its backups. + EXTENDDB_TEST_BACKUP_STORE: none run: devtools/run-tests --extenddb --rust-integration --release # The control plane for vector indexes is not reachable over the wire while @@ -526,6 +681,9 @@ jobs: # the backend reports, so without this it could skip every assertion # and still report green. EXTENDDB_EXPECT_VECTORS: "0" + # No [backup.store] on this server either; the filter runs no backup + # test, and the variable says so for whoever widens it. + EXTENDDB_TEST_BACKUP_STORE: none run: >- devtools/run-tests --extenddb --rust-integration --release --filter vector_index_unsupported @@ -611,6 +769,11 @@ jobs: # which is what would happen if the backend lost the capability. Pinning the # expectation turns that skip into a failure. EXTENDDB_EXPECT_VECTORS: "1" + # A file-backed SQLite server without [backup.store] has the defaulted + # filesystem root beside its database file, so CreateBackup is the + # asynchronous, store-backed one: the backup tests wait for AVAILABLE, + # delete what they create, and expect the restore's refusal. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --rust-integration --release # The daemonized server logs to syslog; dump it so server-side failures @@ -626,6 +789,7 @@ jobs: [ run-integration, run-integration-sqlite, + run-integration-sqlite-s3, run-integration-dev-mode, run-rust-integration, run-rust-integration-postgres-novector, @@ -636,6 +800,7 @@ jobs: - run: | if [ "${{ needs.run-integration.result }}" != "success" ] || \ [ "${{ needs.run-integration-sqlite.result }}" != "success" ] || \ + [ "${{ needs.run-integration-sqlite-s3.result }}" != "success" ] || \ [ "${{ needs.run-integration-dev-mode.result }}" != "success" ] || \ [ "${{ needs.run-rust-integration.result }}" != "success" ] || \ [ "${{ needs.run-rust-integration-postgres-novector.result }}" != "success" ] || \ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index c6b34b7f3..a3f1e2c99 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -47,8 +47,9 @@ cargo build --workspace # Rust unit tests cargo test --workspace -# Integration tests (requires a running extenddb server) -devtools/run-tests --extenddb --pytest +# Integration tests (requires a running extenddb server; the variable says +# where that server writes its backups, see the developer test guide) +EXTENDDB_TEST_BACKUP_STORE=filesystem devtools/run-tests --extenddb --pytest ``` ### Code Style diff --git a/Cargo.lock b/Cargo.lock index b836d650e..6c8178c7d 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1772,6 +1772,7 @@ name = "extenddb-engine" version = "0.1.14" dependencies = [ "base64 0.22.1", + "bytes", "crc32fast", "extenddb-auth", "extenddb-backup-store", @@ -1784,6 +1785,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "tokio", + "tokio-util", "tracing", "uuid", ] @@ -1952,7 +1954,9 @@ dependencies = [ "bigdecimal", "crc32fast", "extenddb-auth", + "extenddb-backup-store", "extenddb-core", + "extenddb-engine", "extenddb-storage", "extenddb-storage-sqlite", "futures", diff --git a/crates/app/Cargo.toml b/crates/app/Cargo.toml index 099ea21cf..fef29af1e 100644 --- a/crates/app/Cargo.toml +++ b/crates/app/Cargo.toml @@ -13,6 +13,7 @@ license.workspace = true # the thin bin enforces that it is paired with a dev/CI backend. dev-mode = ["extenddb-config/dev-mode"] mongodb-test-hooks = ["extenddb-server/mongodb-test-hooks"] +backup-test-hooks = ["extenddb-server/backup-test-hooks"] [dependencies] extenddb-auth = { workspace = true } diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs index 1896cd56e..0f1ae6f4f 100644 --- a/crates/app/src/cmd_backup.rs +++ b/crates/app/src/cmd_backup.rs @@ -851,6 +851,24 @@ async fn resolve_dead_owners(opened: &Opened) { for arn in &report.deletions_finished { println!("finished the deletion of {arn}"); } + for (arn, status) in &report.creating_resolved { + println!( + "resolved the CREATING backup {arn} to {status}: its owner is dead{}", + match status { + extenddb_storage::backup_catalog::BackupStatus::Available => + " and the store holds its complete manifest", + _ => + " and the store holds no complete manifest; its objects are \ + deleted once the row's heartbeat is stale", + } + ); + } + for arn in &report.creating_left_to_owner { + println!( + "left the CREATING backup {arn}: another owner took it over before this \ + command's write" + ); + } } Err(e) => println!("could not resolve the jobs the stopped server left: {e}"), } diff --git a/crates/app/src/cmd_stop.rs b/crates/app/src/cmd_stop.rs index 5fa856d72..a3963ac0c 100755 --- a/crates/app/src/cmd_stop.rs +++ b/crates/app/src/cmd_stop.rs @@ -18,9 +18,19 @@ use crate::util::is_process_alive; #[cfg(unix)] use extenddb_config as config; -/// Maximum time to wait for the process to exit after SIGTERM. +/// Maximum time to wait for the process to exit after SIGTERM: the sum of +/// the server's shutdown bounds, in the order it runs them (the listener's +/// drain of active connections, the backup jobs' failure paths, the +/// registration supervisor's return with the background workers' stop and +/// the serving registration's release, the runtime's blocking threads), so +/// `stop` does not give up on a server that is still inside one of them. #[cfg(unix)] -const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(10); +const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs( + extenddb_server::CONNECTION_DRAIN_TIMEOUT.as_secs() + + extenddb_server::JOB_DRAIN_TIMEOUT.as_secs() + + extenddb_server::WORKER_STOP_TIMEOUT.as_secs() + + crate::RUNTIME_SHUTDOWN_TIMEOUT.as_secs(), +); /// Poll interval when waiting for process exit. #[cfg(unix)] @@ -132,11 +142,46 @@ pub fn run(args: &StopArgs) { } if Instant::now() >= deadline { eprintln!( - "extenddb (pid {pid}) did not exit within {}s after SIGTERM", + "extenddb (pid {pid}) did not exit within {}s after SIGTERM (the server \ + drains connections for up to {}s, waits up to {}s for its backup jobs to end, \ + then up to {}s to stop its background workers and release its serving \ + registration, and {}s for its runtime)", SHUTDOWN_TIMEOUT.as_secs(), + extenddb_server::CONNECTION_DRAIN_TIMEOUT.as_secs(), + extenddb_server::JOB_DRAIN_TIMEOUT.as_secs(), + extenddb_server::WORKER_STOP_TIMEOUT.as_secs(), + crate::RUNTIME_SHUTDOWN_TIMEOUT.as_secs(), ); std::process::exit(1); } thread::sleep(POLL_INTERVAL); } } + +#[cfg(all(test, unix))] +mod tests { + use super::SHUTDOWN_TIMEOUT; + use std::time::Duration; + + /// `stop` waits at least as long as the server's shutdown can take with + /// a backup job in its failure path: 5 s of connection drain, 30 s of + /// job drain, 36 s after it (10 s for the supervisor's return, 16 s for + /// the workers' stop: 5 s of drain, 1 s of abort and 10 s behind a + /// supervisor still stopping them, and 10 s for the registration's + /// release) and 5 s for the runtime. + #[test] + fn stop_waits_for_every_shutdown_bound() { + assert_eq!( + extenddb_server::WORKER_STOP_TIMEOUT, + Duration::from_secs(36) + ); + assert_eq!(SHUTDOWN_TIMEOUT, Duration::from_secs(76)); + assert!( + SHUTDOWN_TIMEOUT + >= extenddb_server::JOB_DRAIN_TIMEOUT + + extenddb_server::WORKER_STOP_TIMEOUT + + crate::RUNTIME_SHUTDOWN_TIMEOUT + + extenddb_server::CONNECTION_DRAIN_TIMEOUT + ); + } +} diff --git a/crates/app/src/init_helpers.rs b/crates/app/src/init_helpers.rs index 8be891e82..1e93d2777 100755 --- a/crates/app/src/init_helpers.rs +++ b/crates/app/src/init_helpers.rs @@ -303,7 +303,7 @@ key_path = "{tls_key}" # [backup] # Backup and restore jobs. stale_job_after_secs must be at least three # heartbeat periods, the larger of heartbeat_interval_secs and the 15 s the -# jobs of this release heartbeat at (45 at the least); pool_size is raised to +# in-catalog jobs heartbeat at (45 at the least); pool_size is raised to # at least 2 * max_concurrent_jobs + 10 on PostgreSQL; max_concurrent_jobs # is at most 1024. # data_file_target_bytes = 67108864 # Uncompressed bytes per data file (minimum 32 MiB) @@ -317,9 +317,10 @@ key_path = "{tls_key}" # # it defaults to backup-spool/ beside the database file # [backup.store] -# Where CreateBackup's backups will live, from a later release (this release's -# CreateBackup writes in-catalog backups with any store, and the store serves -# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem +# Where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB +# (CreateBackup answers CREATING and a job of the server uploads; Cassandra +# writes in-catalog backups with or without a store), and where restores and +# deletes of store-backed backups read. Absent on file-backed SQLite: a filesystem # store at backups/ beside the database file. Absent on PostgreSQL, MongoDB, # Cassandra and an in-memory SQLite database: no store, and CreateBackup # writes in-catalog backups with a deprecation warning. kind = "disabled" diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index d2ea052cc..43151f03e 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -197,6 +197,13 @@ fn print_version(build: BuildInfo) { "disabled" } ); + // The backup jobs' pause points (`EXTENDDB_BACKUP_PAUSE`) are this + // crate's feature, not the binary's, so they are read here. The line is + // printed only on such a build; `test-hooks` above stays the MongoDB + // hooks' line that `devtools/run-mongodb-tests` reads. + if cfg!(feature = "backup-test-hooks") { + println!("backup-test-hooks enabled"); + } } /// Run an async subcommand with a single-threaded tokio runtime and stderr logging. diff --git a/crates/bin/src/main.rs b/crates/bin/src/main.rs index 403986e24..15876a052 100755 --- a/crates/bin/src/main.rs +++ b/crates/bin/src/main.rs @@ -75,6 +75,8 @@ fn main() -> anyhow::Result<()> { version: env!("CARGO_PKG_VERSION"), git_hash: env!("EXTENDDB_GIT_HASH"), build_time: env!("EXTENDDB_BUILD_TIME"), + // The MongoDB hooks; the backup jobs' hooks are `extenddb-app`'s + // `backup-test-hooks` feature, which `version` reports itself. test_hooks_enabled: cfg!(feature = "mongodb-test-hooks"), }) } diff --git a/crates/config/src/backup.rs b/crates/config/src/backup.rs index 83e3a9c83..7ff107f90 100644 --- a/crates/config/src/backup.rs +++ b/crates/config/src/backup.rs @@ -22,17 +22,28 @@ //! //! `stale_job_after_secs` must be at least three heartbeat periods: a job //! checks before each store write that its latest heartbeat confirmed its -//! claim within one period, and the margin is what makes that detection -//! precede a reconciler's deletion of the job's prefix. The period the floor -//! is computed from is the one the jobs run at: the larger of -//! `heartbeat_interval_secs` and [`JOB_HEARTBEAT_INTERVAL`] (15 s), since -//! until a later release wires the knob to the jobs every backend heartbeats -//! at that constant, and a floor on a shorter configured interval would let -//! the reconciler declare a live job dead between two of its heartbeats. The -//! floor is a floor: on SQLite a heartbeat waits for the engine write lock -//! like any writer, so a long write transaction delays it, and a -//! `stale_job_after_secs` near three periods on a busy file can declare a -//! live restore dead. +//! claim recently enough (within two periods: one for the timer, one for the +//! beat to return), and the margin is what makes that detection precede a +//! reconciler's deletion of the job's prefix. The period the floor is +//! computed from is the one the jobs run at: the larger of +//! `heartbeat_interval_secs` and [`JOB_HEARTBEAT_INTERVAL`] (15 s). The +//! store-backed backup job heartbeats at that period; the in-catalog backup +//! and restore paths heartbeat at the constant whatever the knob says, and a +//! floor on a shorter configured interval would let the reconciler declare +//! one of those dead between two of its heartbeats. The floor is a floor: on +//! SQLite a heartbeat waits for the engine write lock like any writer, so a +//! long write transaction delays it, and a `stale_job_after_secs` near three +//! periods on a busy file can declare a live restore dead. +//! +//! `data_file_target_bytes` is read by the store-backed backup job (a data +//! file closes once its uncompressed bytes reach it, so a file exceeds it by +//! less than one line); `failed_row_retention_secs` by the reconciler, which +//! removes a FAILED or DELETED row whose prefix is empty once its +//! `completed_at` is older than that. The retention is at least +//! `stale_job_after_secs`: a job stopped and resumed (SIGSTOP) can land a put +//! after its row was failed, and the reconciler's FAILED-prefix duty deletes +//! that object only while the row exists, so the row must outlive the +//! window in which the stopped owner can still write. //! //! `max_concurrent_jobs` is at most [`MAX_CONCURRENT_JOBS`]: a value over //! the runtime's permit ceiling would stop the server at startup rather than @@ -59,10 +70,11 @@ //! unconfigured and startup logs that. //! - `kind = "disabled"` leaves the store unconfigured on purpose, on any //! backend. -//! - Without a configured store, CreateBackup writes the in-catalog format -//! and logs a deprecation warning naming `[backup.store]` on each call; -//! backups of either format stay readable, listable, restorable and -//! deletable. +//! - With a configured store, CreateBackup on PostgreSQL, SQLite and +//! MongoDB writes the backup to the store and returns CREATING; without +//! one, and on Cassandra always, it writes the in-catalog format and logs +//! a deprecation warning naming `[backup.store]` on each call; backups of +//! either format stay readable, listable, restorable and deletable. //! //! The spool. Restores from an S3 store download to `spool_path`, created //! `0700` with files `0600`. On file-backed SQLite it defaults to @@ -196,9 +208,9 @@ pub struct ResolvedStore { } impl BackupConfig { - /// The period the jobs heartbeat at: the larger of - /// `heartbeat_interval_secs` and [`JOB_HEARTBEAT_INTERVAL`], which is - /// what every backend runs at until the knob reaches the jobs. + /// The period the store-backed backup job heartbeats at, and the floor's + /// unit: the larger of `heartbeat_interval_secs` and + /// [`JOB_HEARTBEAT_INTERVAL`] (the in-catalog paths run at the constant). #[must_use] pub fn effective_heartbeat_interval_secs(&self) -> u64 { self.heartbeat_interval_secs @@ -250,14 +262,22 @@ impl BackupConfig { return Err(format!( "backup.stale_job_after_secs = {} is under {MIN_STALE_HEARTBEATS} * the job \ heartbeat period of {period} s = {floor} (the larger of \ - backup.heartbeat_interval_secs = {} and the {} s the jobs heartbeat at in \ - this release); a reconciler could declare a running job dead between two \ + backup.heartbeat_interval_secs = {} and the {} s the in-catalog jobs \ + heartbeat at); a reconciler could declare a running job dead between two \ of its heartbeats", self.stale_job_after_secs, self.heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL.as_secs() )); } + if self.failed_row_retention_secs < self.stale_job_after_secs { + return Err(format!( + "backup.failed_row_retention_secs = {} is under backup.stale_job_after_secs = \ + {}; a FAILED row removed before its owner's last heartbeat is stale leaves no \ + row for the reconciler to delete that owner's late objects under", + self.failed_row_retention_secs, self.stale_job_after_secs + )); + } if self.data_file_target_bytes < MIN_DATA_FILE_TARGET_BYTES { return Err(format!( "backup.data_file_target_bytes = {} is under the minimum of \ @@ -650,6 +670,31 @@ mod tests { long.validate().unwrap(); } + /// The retention is at least the stale window: a FAILED row removed + /// sooner leaves a stopped owner's late object with no row under which + /// the reconciler deletes it. + #[test] + fn a_retention_under_the_stale_window_is_refused() { + let c = BackupConfig { + stale_job_after_secs: 300, + failed_row_retention_secs: 299, + ..BackupConfig::default() + }; + let e = c.validate().unwrap_err(); + assert!(e.contains("failed_row_retention_secs = 299"), "{e}"); + assert!(e.contains("stale_job_after_secs = 300"), "{e}"); + let ok = BackupConfig { + failed_row_retention_secs: 300, + ..c + }; + ok.validate().unwrap(); + let zero = BackupConfig { + failed_row_retention_secs: 0, + ..BackupConfig::default() + }; + assert!(zero.validate().is_err()); + } + #[test] fn max_concurrent_jobs_has_an_upper_bound() { let at_most = BackupConfig { diff --git a/crates/core/src/backup_format/data_file.rs b/crates/core/src/backup_format/data_file.rs index 10054b8e9..bb5e48033 100644 --- a/crates/core/src/backup_format/data_file.rs +++ b/crates/core/src/backup_format/data_file.rs @@ -137,7 +137,8 @@ pub const MAX_LINE_BYTES: usize = 32 * 1024 * 1024; /// the two paths share one encoding by construction. The line is not checked /// against [`MAX_LINE_BYTES`]; [`DataFileEncoder::write_item`] does that. /// [`encode_item_line_into`] writes the same bytes into a buffer the caller -/// keeps. +/// keeps; [`encode_item_line_len_bounded`] measures them against a cap +/// without a buffer. /// /// # Errors /// @@ -165,6 +166,65 @@ pub fn encode_item_line_into(item: &Item, line: &mut Vec) -> Result<(), serd Ok(()) } +/// The length of [`encode_item_line`]'s line for `item` when it is at most +/// `cap` bytes, measured without a buffer: the line is serialized into a +/// writer that counts and refuses the write that would take the count past +/// `cap`, at which `serde_json::to_writer` stops, so an item whose line is +/// many times the cap costs no memory and no more work than the cap's +/// worth. The backup job measures an item over the default item limit this +/// way before the encoder sees it; for a line within the cap the length is +/// the one `encode_item_line_into` would leave. +/// +/// # Errors +/// +/// Returns [`FormatError::LineTooLong`] once the count would pass `cap` and +/// [`FormatError::Json`] when the item cannot be serialized. +pub fn encode_item_line_len_bounded(item: &Item, cap: usize) -> Result { + let mut counter = LineCounter { + written: 0, + cap, + over: false, + }; + let counted = serde_json::to_writer(&mut counter, &ItemLineRef { item }) + .map_err(FormatError::from) + .and_then(|()| { + counter + .write_all(b"\n") + .map_err(|e| FormatError::Encode(e.to_string())) + }); + if counter.over { + return Err(FormatError::LineTooLong { cap }); + } + counted.map(|()| counter.written) +} + +/// The writer behind [`encode_item_line_len_bounded`]: counts the bytes +/// written and refuses the write that would take the count past `cap`, +/// setting `over`; no byte past the cap is counted and nothing is kept. +struct LineCounter { + written: usize, + cap: usize, + over: bool, +} + +impl Write for LineCounter { + fn write(&mut self, buf: &[u8]) -> std::io::Result { + if buf.len() > self.cap - self.written { + self.over = true; + return Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "the line cap", + )); + } + self.written += buf.len(); + Ok(buf.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + /// Lowercase hex encoding of a byte slice. pub(super) fn hex_encode(bytes: &[u8]) -> String { use std::fmt::Write as _; @@ -1991,6 +2051,65 @@ mod tests { assert!(numbers_line < line, "{numbers_line}"); } + /// The bounded measurement is the line's length up to the cap and + /// `LineTooLong` past it, at the byte (the newline counted, as the + /// encoder counts it), and it holds no line: the counter refuses the + /// write that would pass the cap and counts nothing past it, so a + /// 600 KB item of the worst shape (a 40 MB line) is refused with at most + /// the cap's worth of bytes counted and no buffer. + #[test] + fn the_bounded_line_measurement_stops_at_the_cap_without_a_buffer() { + let small = item(vec![("pk", AttributeValue::S("p".into()))]); + let len = encode_item_line(&small).unwrap().len(); + assert_eq!(encode_item_line_len_bounded(&small, len).unwrap(), len); + assert_eq!( + encode_item_line_len_bounded(&small, MAX_LINE_BYTES).unwrap(), + len + ); + assert!(matches!( + encode_item_line_len_bounded(&small, len - 1), + Err(FormatError::LineTooLong { cap }) if cap == len - 1 + )); + // The counter itself: a write that would pass the cap is refused + // and leaves the count where it was. + let mut counter = LineCounter { + written: 0, + cap: 10, + over: false, + }; + counter.write_all(b"12345").unwrap(); + counter.write_all(b"67890").unwrap(); + assert_eq!((counter.written, counter.over), (10, false)); + assert!(counter.write_all(b"x").is_err()); + assert_eq!((counter.written, counter.over), (10, true)); + // The worst shape over the default limit: 3,700 sets of 81 two-digit + // values at `E-130`, a 40 MB line for a 600 KB item. + let set: BTreeSet = (11..100) + .filter(|m| m % 10 != 0) + .map(|m| format!("0.{}{m}", "0".repeat(129))) + .collect(); + let mut worst = item(vec![("pk", AttributeValue::S("p".into()))]); + worst.insert( + "l".to_owned(), + AttributeValue::L(vec![AttributeValue::NS(set); 3_700]), + ); + assert!(crate::types::item_size_bytes(&worst) > crate::limits::DEFAULT_MAX_ITEM_SIZE_BYTES); + assert!(matches!( + encode_item_line_len_bounded(&worst, MAX_LINE_BYTES), + Err(FormatError::LineTooLong { cap }) if cap == MAX_LINE_BYTES + )); + // A cap of 100 bytes stops the serializer after at most 100 bytes + // of a 40 MB line. + let mut counter = LineCounter { + written: 0, + cap: 100, + over: false, + }; + assert!(serde_json::to_writer(&mut counter, &ItemLineRef { item: &worst }).is_err()); + assert!(counter.over); + assert!(counter.written <= 100, "{}", counter.written); + } + /// Dropping an encoder that did not finish writes nothing more, whether /// it is fresh or has items in its buffer: an abandoned encoder must not /// complete a valid gzip file in the sink. diff --git a/crates/core/src/backup_format/key_order.rs b/crates/core/src/backup_format/key_order.rs index b0060e30c..d8990bdde 100644 --- a/crates/core/src/backup_format/key_order.rs +++ b/crates/core/src/backup_format/key_order.rs @@ -151,6 +151,15 @@ pub fn hash_key_text(value: &AttributeValue) -> Option> { } } +/// An untrusted string bounded for an error message, the format module's +/// rule (`error::truncated`) reached from outside the module: the string +/// when short, a prefix and the total length otherwise. Call sites format +/// the result with `{:?}`. +#[must_use] +pub fn shown(s: &str) -> String { + truncated(s) +} + /// A table's key shape, resolved once from its key schema and attribute /// definitions, able to compare and validate items' keys. #[derive(Debug, Clone, PartialEq, Eq)] @@ -433,9 +442,17 @@ fn parse_number(name: &str, n: &str) -> Result { } /// One key attribute's value checked as [`KeyOrder::validate_key_item`] -/// checks it: the declared scalar type, not empty, and for `N` a valid -/// number in normalized form. The error is the reason alone. -fn check_key_value( +/// checks it: not empty, the declared scalar type, and for `N` a valid +/// number in normalized form, in that order, so a text under a key declared +/// `S` is a value of the wrong type and not a malformed number. The error is +/// the reason alone. Public for the restorable-item validator +/// (`crate::validation::restorable_item`), which checks a multi-part key's +/// attributes one by one with the same rule. +/// +/// # Errors +/// +/// [`FormatError::InvalidKey`] naming the attribute and what is wrong. +pub fn check_key_value( name: &str, ty: ScalarAttributeType, value: &AttributeValue, @@ -454,32 +471,7 @@ fn check_key_value( } let actual = match value { AttributeValue::S(_) => ScalarAttributeType::S, - AttributeValue::N(n) => { - let normalized = validate_and_normalize_number(n).map_err(|e| { - // The reason for a malformed number ends in the input, which - // is shown once, bounded, before it; the other reasons - // (overflow, underflow, too many digits) are fixed texts. - let reason = e.to_string(); - let reason = match reason.strip_suffix(n.as_str()) { - Some(rest) => rest.strip_suffix(": ").unwrap_or(rest).to_owned(), - None => reason, - }; - FormatError::InvalidKey(format!( - "key attribute {:?} value {:?} is not a valid number: {reason}", - shown(), - truncated(n), - )) - })?; - if normalized != *n { - return Err(FormatError::InvalidKey(format!( - "key attribute {:?} value {:?} is not in normalized form ({:?})", - shown(), - truncated(n), - truncated(&normalized) - ))); - } - ScalarAttributeType::N - } + AttributeValue::N(_) => ScalarAttributeType::N, AttributeValue::B(_) => ScalarAttributeType::B, _ => { return Err(FormatError::InvalidKey(format!( @@ -494,6 +486,33 @@ fn check_key_value( shown() ))); } + if let AttributeValue::N(n) = value { + let normalized = validate_and_normalize_number(n).map_err(|e| { + // The reason for a malformed number ends in the input, which is + // shown once, bounded, before it, so the input and the colon + // before it are taken off the end (the wire appends the input + // as it is, trailing whitespace included); the other reasons + // (overflow, underflow, too many digits) are fixed texts. + let reason = e.to_string(); + let reason = match reason.strip_suffix(n.as_str()) { + Some(rest) => rest.strip_suffix(": ").unwrap_or(rest).to_owned(), + None => reason, + }; + FormatError::InvalidKey(format!( + "key attribute {:?} value {:?} is not a valid number: {reason}", + shown(), + truncated(n), + )) + })?; + if normalized != *n { + return Err(FormatError::InvalidKey(format!( + "key attribute {:?} value {:?} is not in normalized form ({:?})", + shown(), + truncated(n), + truncated(&normalized) + ))); + } + } Ok(()) } @@ -973,6 +992,48 @@ mod tests { assert!(err.to_string().len() < 400, "{}", err.to_string().len()); } + /// `check_key_value`'s texts: the declared type is compared before an + /// `N` is parsed, so a text under an `S` key is a value of the wrong + /// type; a malformed `N` under an `N` key gives the wire's reason without + /// the input the wire appends (shown once, bounded, before it) and + /// without the colon that preceded it, whatever whitespace the input + /// ends in; the fixed reasons are carried whole. + #[test] + fn check_key_value_compares_the_type_before_parsing_and_strips_the_echoed_input() { + let reason = |ty, value: AttributeValue| match check_key_value("k", ty, &value) { + Err(FormatError::InvalidKey(reason)) => reason, + other => panic!("{other:?}"), + }; + assert_eq!( + reason(ScalarAttributeType::S, AttributeValue::N("abc".into())), + r#"key attribute "k" is N, the schema declares S"# + ); + for bad in ["abc", "5 ", " 5", "5\t", "1e"] { + assert_eq!( + reason(ScalarAttributeType::N, AttributeValue::N(bad.into())), + format!( + "key attribute \"k\" value {bad:?} is not a valid number: The parameter \ + cannot be converted to a numeric value" + ), + "{bad:?}" + ); + } + assert_eq!( + reason(ScalarAttributeType::N, AttributeValue::N("1E126".into())), + "key attribute \"k\" value \"1E126\" is not a valid number: Number overflow. \ + Attempting to store a number with magnitude larger than supported range" + ); + assert_eq!( + reason(ScalarAttributeType::N, AttributeValue::N(String::new())), + "key attribute \"k\" value \"\" is not a valid number: The parameter cannot be \ + converted to a numeric value" + ); + assert_eq!( + reason(ScalarAttributeType::N, AttributeValue::N("5.0".into())), + r#"key attribute "k" value "5.0" is not in normalized form ("5")"# + ); + } + #[test] fn duplicate_attribute_definitions_are_unorderable() { let ks = vec![KeySchemaElement { diff --git a/crates/core/src/backup_format/mod.rs b/crates/core/src/backup_format/mod.rs index ee88a6c93..ad75ed886 100644 --- a/crates/core/src/backup_format/mod.rs +++ b/crates/core/src/backup_format/mod.rs @@ -73,7 +73,7 @@ mod golden_tests; pub use data_file::{ CompressedDigest, CompressedHasher, DataFileChecksums, DataFileDecoder, DataFileEncoder, DataFileExpectations, GZIP_LEVEL, MAX_LINE_BYTES, decode_data_file, encode_data_file, - encode_item_line, encode_item_line_into, + encode_item_line, encode_item_line_into, encode_item_line_len_bounded, }; pub use ddb_export::{ DYNAMODB_JSON_OUTPUT_FORMAT, EXPORT_SUMMARY_VERSION, ExportFileEntry, ExportSummaryManifest, @@ -81,7 +81,7 @@ pub use ddb_export::{ }; pub use documents::{BackupDocuments, ManifestHeader}; pub use error::{FormatError, JsonError}; -pub use key_order::{KeyOrder, hash_key_text}; +pub use key_order::{KeyOrder, check_key_value, hash_key_text, shown}; pub use layout::{ BACKUP_ID_MILLIS_DIGITS, BackupArn, BackupArnError, BackupArnParts, DATA_PREFIX, EXTENDDB_MANIFEST_FILE, FILES_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE, MAX_BACKUP_ARN_LEN, diff --git a/crates/core/src/limits/mod.rs b/crates/core/src/limits/mod.rs index a894f1b83..5805508bb 100755 --- a/crates/core/src/limits/mod.rs +++ b/crates/core/src/limits/mod.rs @@ -2,6 +2,12 @@ // SPDX-License-Identifier: Apache-2.0 use serde::Deserialize; +/// The default `max_item_size_bytes` (400 KB as `DynamoDB` sizes items), the +/// largest item the backup format's line cap is sized for: the backup job +/// measures the data file line of an item over this size and of no item +/// within it. +pub const DEFAULT_MAX_ITEM_SIZE_BYTES: usize = 409_600; + /// All configurable limits with DynamoDB-compatible defaults. #[derive(Debug, Clone, Deserialize)] #[serde(deny_unknown_fields)] @@ -98,7 +104,7 @@ impl Default for LimitsConfig { } fn default_max_item_size() -> usize { - 409_600 + DEFAULT_MAX_ITEM_SIZE_BYTES } fn default_max_pk_size() -> usize { 2048 diff --git a/crates/core/src/metrics/collector.rs b/crates/core/src/metrics/collector.rs index c5981cdcf..8975e2675 100755 --- a/crates/core/src/metrics/collector.rs +++ b/crates/core/src/metrics/collector.rs @@ -348,6 +348,50 @@ impl MetricsCollector { ); } + /// Record the end of a store-backed backup job: one `BackupJobs` under + /// the outcome and its duration in microseconds under the same + /// dimension. The job measures the duration from the snapshot's open to + /// the end of its last catalog or store call: the AVAILABLE write for a + /// job that succeeded, the FAILED write and the prefix's deletion for + /// one that failed. + pub fn record_backup_job(&self, outcome: super::types::BackupJobOutcome, duration_us: f64) { + let outcome = outcome.to_string(); + self.record(MetricName::BackupJobs, 1.0, None, None, Some(&outcome)); + self.record( + MetricName::BackupJobDuration, + duration_us, + None, + None, + Some(&outcome), + ); + } + + /// Record bytes a backup job wrote to the store; the job records them + /// once it ended AVAILABLE, so a failed job's uploaded and then deleted + /// bytes are not counted. + #[allow(clippy::cast_precision_loss)] // byte counts of one backup + pub fn record_backup_bytes_uploaded(&self, bytes: u64) { + self.record( + MetricName::BackupBytesUploaded, + bytes as f64, + None, + None, + None, + ); + } + + /// Record a running backup job's snapshot age in seconds (gauge). + pub fn record_backup_snapshot_age(&self, age_secs: f64) { + self.record(MetricName::BackupSnapshotAge, age_secs, None, None, None); + } + + /// Record a failed store call of a backup job or of the backup + /// reconciler; `op` names the call (the job's `put` and + /// `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`). + pub fn record_backup_store_error(&self, op: &str) { + self.record(MetricName::BackupStoreErrors, 1.0, None, None, Some(op)); + } + /// Record per-segment latency breakdown for a request. pub fn record_segments( &self, diff --git a/crates/core/src/metrics/collector_tests.rs b/crates/core/src/metrics/collector_tests.rs index 26a50b7f6..385b07e83 100755 --- a/crates/core/src/metrics/collector_tests.rs +++ b/crates/core/src/metrics/collector_tests.rs @@ -175,3 +175,60 @@ fn segments_multiple_operations() { assert_eq!(s.count, 1); } } + +/// The backup job's recorders land under their names and dimensions: one +/// `BackupJobs` and one `BackupJobDuration` (microseconds) per job with the +/// outcome as the operation dimension, the bytes and the snapshot age +/// undimensioned, a store error under the call's name. +#[test] +fn backup_job_recorders_land_under_the_outcome_and_the_store_call() { + use crate::metrics::types::BackupJobOutcome; + + let c = MetricsCollector::new(); + c.record_backup_job(BackupJobOutcome::Succeeded, 1_500.0); + c.record_backup_job(BackupJobOutcome::Failed, 300.0); + c.record_backup_bytes_uploaded(4096); + c.record_backup_snapshot_age(12.5); + c.record_backup_store_error("put"); + + let by = |metric: MetricName| { + let mut snaps = c.query(&MetricsQuery { + metric: Some(metric), + window: Some(TimeWindow::AllTime), + ..Default::default() + }); + snaps.sort_by(|a, b| format!("{:?}", a.dimensions).cmp(&format!("{:?}", b.dimensions))); + snaps + }; + let jobs = by(MetricName::BackupJobs); + let outcomes: Vec = jobs.iter().map(|s| format!("{:?}", s.dimensions)).collect(); + assert_eq!( + outcomes, + [ + "[Operation(\"failed\")]".to_owned(), + "[Operation(\"succeeded\")]".to_owned() + ] + ); + assert!(jobs.iter().all(|s| s.count == 1 && s.sum == 1.0)); + let durations = by(MetricName::BackupJobDuration); + assert_eq!(durations.len(), 2); + assert_eq!(durations.iter().map(|s| s.sum).sum::(), 1_800.0); + let bytes = by(MetricName::BackupBytesUploaded); + assert_eq!(bytes.len(), 1); + assert!(bytes[0].dimensions.is_empty()); + assert_eq!(bytes[0].sum, 4096.0); + let age = by(MetricName::BackupSnapshotAge); + assert_eq!(age[0].max, 12.5); + let errors = by(MetricName::BackupStoreErrors); + assert_eq!( + format!("{:?}", errors[0].dimensions), + "[Operation(\"put\")]" + ); + for (outcome, text) in [ + (BackupJobOutcome::Succeeded, "succeeded"), + (BackupJobOutcome::Failed, "failed"), + (BackupJobOutcome::Cancelled, "cancelled"), + ] { + assert_eq!(outcome.to_string(), text); + } +} diff --git a/crates/core/src/metrics/mod.rs b/crates/core/src/metrics/mod.rs index bf1ff0272..637a97c29 100755 --- a/crates/core/src/metrics/mod.rs +++ b/crates/core/src/metrics/mod.rs @@ -17,7 +17,7 @@ mod types; pub use collector::MetricsCollector; pub use types::{ - Dimension, FlushBucket, LatencySegments, MetricName, MetricSnapshot, MetricsBucket, - MetricsQuery, MetricsResponse, OperationSegments, Percentiles, QueryCategory, QuerySource, - TimeWindow, auto_granularity, parse_granularity, window_duration_secs, + BackupJobOutcome, Dimension, FlushBucket, LatencySegments, MetricName, MetricSnapshot, + MetricsBucket, MetricsQuery, MetricsResponse, OperationSegments, Percentiles, QueryCategory, + QuerySource, TimeWindow, auto_granularity, parse_granularity, window_duration_secs, }; diff --git a/crates/core/src/metrics/types.rs b/crates/core/src/metrics/types.rs index 85811deaa..3d1b747c6 100755 --- a/crates/core/src/metrics/types.rs +++ b/crates/core/src/metrics/types.rs @@ -55,6 +55,56 @@ pub enum MetricName { WorkerCycleLatency, /// P120e: Worker error count. WorkerErrorCount, + /// Store-backed backup jobs ended, one per job; the operation dimension + /// carries the outcome ([`BackupJobOutcome`]: `succeeded`, `failed` or + /// `cancelled`). `/metrics` emits it under this name as JSON, as it does + /// every metric (the endpoint is not Prometheus exposition format); a + /// converter would give it a name like `backup_jobs_total{outcome}`, and + /// nothing in the server renders that. + BackupJobs, + /// Duration of a store-backed backup job in microseconds, from the + /// snapshot's open to the end of its last catalog or store call (the + /// AVAILABLE write, or the FAILED write and the prefix's deletion); + /// recorded for every outcome, with the outcome as the operation + /// dimension. + BackupJobDuration, + /// Bytes a backup job that ended AVAILABLE wrote to the store: every + /// data file and document as the store reported it stored; a failed + /// job's uploaded and then deleted bytes are not counted. + BackupBytesUploaded, + /// Seconds between a running backup job's snapshot instant and now + /// (gauge), recorded at each confirmed heartbeat. + BackupSnapshotAge, + /// Store operations a backup job or the backup reconciler saw fail; the + /// operation dimension names the store call (`put` by the job's + /// uploads, `delete_prefix` by its failure path and the reconciler's + /// deleting duties, `get` and `list` by the reconciler's manifest probe + /// and deleting duties). + BackupStoreErrors, +} + +/// How a store-backed backup job ended, the operation dimension of +/// [`MetricName::BackupJobs`] and [`MetricName::BackupJobDuration`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub enum BackupJobOutcome { + /// The manifest was written and the row is AVAILABLE. + Succeeded, + /// The row is FAILED, or the job stopped without a status write of its + /// own (its claim lost). + Failed, + /// The server's shutdown stopped the job; the row is FAILED with reason + /// `shutdown`. + Cancelled, +} + +impl std::fmt::Display for BackupJobOutcome { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(match self { + Self::Succeeded => "succeeded", + Self::Failed => "failed", + Self::Cancelled => "cancelled", + }) + } } impl std::fmt::Display for MetricName { @@ -86,6 +136,11 @@ impl std::fmt::Display for MetricName { Self::WorkerLastSuccess => f.write_str("WorkerLastSuccess"), Self::WorkerCycleLatency => f.write_str("WorkerCycleLatency"), Self::WorkerErrorCount => f.write_str("WorkerErrorCount"), + Self::BackupJobs => f.write_str("BackupJobs"), + Self::BackupJobDuration => f.write_str("BackupJobDuration"), + Self::BackupBytesUploaded => f.write_str("BackupBytesUploaded"), + Self::BackupSnapshotAge => f.write_str("BackupSnapshotAge"), + Self::BackupStoreErrors => f.write_str("BackupStoreErrors"), } } } @@ -131,6 +186,9 @@ pub enum QuerySource { LoginAttemptCleanup, /// Background: the backup reconciler. BackupReconciler, + /// A store-backed backup job (the worker metrics are recorded once per + /// job, at its end). + BackupJob, /// Management CLI commands. Management, } @@ -152,6 +210,7 @@ impl std::fmt::Display for QuerySource { Self::ControlPlane => f.write_str("ControlPlane"), Self::LoginAttemptCleanup => f.write_str("LoginAttemptCleanup"), Self::BackupReconciler => f.write_str("BackupReconciler"), + Self::BackupJob => f.write_str("BackupJob"), Self::Management => f.write_str("Management"), } } diff --git a/crates/core/src/validation/mod.rs b/crates/core/src/validation/mod.rs index a849f2c95..e4c694209 100755 --- a/crates/core/src/validation/mod.rs +++ b/crates/core/src/validation/mod.rs @@ -1,8 +1,13 @@ // Copyright 2026 ExtendDB contributors // SPDX-License-Identifier: Apache-2.0 pub mod number; +pub mod restorable_item; pub mod vector_item; +pub use restorable_item::{ + RestorableItemError, RestorableItemReason, RestorableKeySchema, RestorableLimits, + SHOWN_VALUE_BYTES, validate_restorable_item, +}; pub use vector_item::{ MAX_HASH_KEY_SIZE, MAX_INLINE_FILTER_SIZE, validate_vector_write, validate_vector_write_changed, vector_components, vector_norm, diff --git a/crates/core/src/validation/restorable_item.rs b/crates/core/src/validation/restorable_item.rs new file mode 100644 index 000000000..5d2b4f671 --- /dev/null +++ b/crates/core/src/validation/restorable_item.rs @@ -0,0 +1,1514 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Whether a stored item can be restored: the check a backup job runs on +//! every item it reads, before the item is written to a data file, and the +//! check PR 9a's restore binds for every item it reads back. +//! +//! The write paths let some items in that a restore's `PutItem`-shaped write +//! would refuse: an `N` value that is not a valid number (`1E126`, `abc`; +//! #618 names the four paths that store one unvalidated: `BatchWriteItem`, +//! `TransactWriteItems`, `UpdateItem` on an absent item, `ImportTable`), an +//! item over the configured size limit stored under a larger one, a key +//! attribute or attribute name over this instance's limits stored under +//! larger ones, and an attribute name over the limit that `UpdateItem` and +//! a `TransactWriteItems` `Update` stored unchecked before every write path +//! checked it (#433). The key-shape rules (a key attribute missing, of the wrong +//! type or empty, an `N` key not in normalized form) cover rows of earlier +//! builds and rows edited outside the wire: every write path of this tree +//! refuses such a key, and the snapshot streams hand every valid `N` over +//! normalized, so the normalized-form branch is unreachable from a +//! snapshot; it stays because it is the format's own rule +//! ([`crate::backup_format::check_key_value`]). A backup that carried such +//! an item could not be restored in full, so the job refuses the backup +//! instead and names the item (its key), the attribute path and the reason. +//! The base key alone is checked: an index key attribute of the wrong type +//! or empty is left to the restore's index build. +//! +//! One walk over the item: every `N` and every `NS` member is parsed once +//! on the accepting path (a key `N` that fails the key rule is parsed a +//! second time to classify the failure), the nesting depth is counted as it +//! goes, every attribute name is measured against `max_attribute_name_bytes` +//! and the key attributes are checked with the backup format's own key rule +//! and against the hash and range key size limits, the rules the wire's +//! `check_key_size` and `check_attribute_name_size` apply, so a key that +//! passes here is one a key-ordered file's range can carry and one the +//! target's write accepts. The item's size is one call on the whole item +//! (`item_size_bytes`). The request path's `validate_put_item` stays what it +//! is: it returns the wire error with no attribute path and is not called +//! here. +//! +//! Every untrusted text an error carries (a key value, an attribute name, a +//! path segment, a number the wire's reason echoes) is bounded on its own +//! ([`SHOWN_VALUE_BYTES`]; a key value at [`SHOWN_KEY_VALUE_BYTES`]) and +//! rendered as a JSON string's content, with the line separators U+0085, +//! U+2028 and U+2029 escaped beside the control characters, so neither a +//! control character nor a Unicode line separator in a name or a number +//! can end the log line it is written in, and a key of any +//! legal size appears whole while only an oversized value is cut. The +//! bound is per text, so the whole message is bounded by the item's own +//! size times a small factor (a byte renders as at most six), not by a +//! constant: a path of thirty nested maps with names at the bound is about +//! 127 KB. + +use std::borrow::Cow; +use std::fmt; + +use crate::backup_format::{FormatError, check_key_value, hash_key_text, shown}; +use crate::limits::LimitsConfig; +use crate::types::{ + AttributeDefinition, AttributeValue, Item, KeySchemaElement, KeyType, ScalarAttributeType, + item_size_bytes, +}; +use crate::validation::number::validate_and_normalize_number; + +/// Longest rendering of one untrusted text in a [`RestorableItemError`] +/// other than a key value (an attribute name, a path segment, a set member, +/// the number a reason echoes), in the bytes the message writes: the text +/// as a JSON string's content, quotes not counted. A round figure that +/// keeps a path segment to four times the longest key attribute name +/// (255 bytes), after which a name is cut. +pub const SHOWN_VALUE_BYTES: usize = 4096; + +/// Longest rendering of a key attribute's value, in the same measure: six +/// times the 2048-byte hash key limit, the most a byte escapes to +/// (`\u00XX`), so a key of any legal size renders whole whatever its +/// bytes, and the 2732-character base64 of a 2048-byte `B` key fits. +pub const SHOWN_KEY_VALUE_BYTES: usize = 6 * 2048; + +/// The prefix kept of a text over its bound, in the same measure, followed +/// by `... ` with the text's length. +const SHOWN_VALUE_PREFIX_BYTES: usize = 1024; + +/// The instance's limits a restore would hold an item to, for +/// [`validate_restorable_item`]: the item size, the hash and range key +/// sizes and the attribute name size of `LimitsConfig`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RestorableLimits { + pub max_item_size_bytes: usize, + pub max_partition_key_size_bytes: usize, + pub max_sort_key_size_bytes: usize, + pub max_attribute_name_bytes: usize, +} + +impl From<&LimitsConfig> for RestorableLimits { + fn from(limits: &LimitsConfig) -> Self { + Self { + max_item_size_bytes: limits.max_item_size_bytes, + max_partition_key_size_bytes: limits.max_partition_key_size_bytes, + max_sort_key_size_bytes: limits.max_sort_key_size_bytes, + max_attribute_name_bytes: limits.max_attribute_name_bytes, + } + } +} + +/// One key attribute of a [`RestorableKeySchema`]. +#[derive(Debug, Clone, PartialEq, Eq)] +struct KeyAttribute { + name: String, + ty: ScalarAttributeType, + key_type: KeyType, +} + +/// The key attributes of a table, resolved once from its key schema and +/// attribute definitions, for [`validate_restorable_item`]. Unlike +/// `KeyOrder`, which only a hash-only or single-range-key table has, every +/// table has one of these: a multi-part key is a list of more than two +/// attributes. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RestorableKeySchema { + attributes: Vec, +} + +impl RestorableKeySchema { + /// Resolve the key attributes, in key-schema order. + /// + /// # Errors + /// + /// The name of a key attribute that has no attribute definition, or more + /// than one; `CreateTable` and the manifest reader both refuse such a + /// schema, so a table's own schema never fails here. + pub fn resolve( + key_schema: &[KeySchemaElement], + attribute_definitions: &[AttributeDefinition], + ) -> Result { + let mut attributes = Vec::with_capacity(key_schema.len()); + for element in key_schema { + let name = element.attribute_name.as_str(); + let mut matching = attribute_definitions + .iter() + .filter(|d| d.attribute_name == name) + .map(|d| d.attribute_type); + let ty = matching.next().ok_or_else(|| { + format!( + "key attribute {:?} has no attribute definition", + shown(name) + ) + })?; + if matching.next().is_some() { + return Err(format!( + "key attribute {:?} has more than one attribute definition", + shown(name) + )); + } + attributes.push(KeyAttribute { + name: name.to_owned(), + ty, + key_type: element.key_type, + }); + } + Ok(Self { attributes }) + } + + /// Names of the key attributes, in key-schema order. + pub fn attribute_names(&self) -> impl Iterator { + self.attributes.iter().map(|a| a.name.as_str()) + } + + /// Whether `name` is a key attribute. + fn is_key(&self, name: &str) -> bool { + self.attributes.iter().any(|a| a.name == name) + } + + /// The key attributes of `item` as compact JSON (`{"pk":{"S":"a"}}`) + /// for an error message or a log line, each name bounded on its own + /// ([`SHOWN_VALUE_BYTES`]) and each value on its own + /// ([`SHOWN_KEY_VALUE_BYTES`]), so every key attribute appears and only + /// an oversized value is cut; a key attribute the item lacks is left + /// out. A `B` value is its base64 text, as the format writes it; a value + /// that is not a scalar (a key attribute of the wrong type) is its own + /// JSON, cut to a JSON string when oversized. + #[must_use] + pub fn shown_key(&self, item: &Item) -> String { + let mut out = String::from("{"); + let mut first = true; + for attribute in &self.attributes { + let Some(value) = item.get(&attribute.name) else { + continue; + }; + if !first { + out.push(','); + } + first = false; + push_json_string(&mut out, &attribute.name, SHOWN_VALUE_BYTES); + out.push(':'); + let tag = match value { + AttributeValue::S(_) => Some("S"), + AttributeValue::N(_) => Some("N"), + AttributeValue::B(_) => Some("B"), + _ => None, + }; + match (tag, hash_key_text(value)) { + (Some(tag), Some(text)) => { + out.push_str("{\""); + out.push_str(tag); + out.push_str("\":"); + push_json_string(&mut out, &text, SHOWN_KEY_VALUE_BYTES); + out.push('}'); + } + _ => { + let text = serde_json::to_string(value).unwrap_or_default(); + if text.len() <= SHOWN_KEY_VALUE_BYTES { + out.push_str(&text); + } else { + push_json_string(&mut out, &text, SHOWN_KEY_VALUE_BYTES); + } + } + } + } + out.push('}'); + out + } +} + +/// Append `s` to `out` as a JSON string: quotes around [`bounded_text`] +/// with `bound`. +fn push_json_string(out: &mut String, s: &str, bound: usize) { + out.push('"'); + out.push_str(&bounded_text(s, bound)); + out.push('"'); +} + +/// Bytes a message writes for `c` inside a JSON string: `\"`, `\\` and +/// the five short escapes take two, the other control characters and the +/// three line separators `\u00XX`-style escapes of six, everything else +/// its UTF-8 length. +fn json_escaped_len(c: char) -> usize { + match c { + '"' | '\\' | '\n' | '\r' | '\t' | '\u{8}' | '\u{c}' => 2, + c if (c as u32) < 0x20 || is_line_separator(c) => 6, + c => c.len_utf8(), + } +} + +/// The Unicode line separators `serde_json` leaves raw and a reader that +/// breaks lines the Unicode way (Python's `str.splitlines`) breaks on: +/// NEXT LINE, LINE SEPARATOR and PARAGRAPH SEPARATOR. +fn is_line_separator(c: char) -> bool { + matches!(c, '\u{85}' | '\u{2028}' | '\u{2029}') +} + +/// `s` as a JSON string's content: the escapes `serde_json` writes, then +/// the three line separators as `\u0085`, `\u2028` and `\u2029` (valid JSON +/// escapes `serde_json` does not write), no quotes, so neither a control +/// character nor a line separator in a tenant's text can stand as a line +/// of its own in a log; borrowed when nothing in it escapes. +fn json_escaped(s: &str) -> Cow<'_, str> { + if s.chars().all(|c| json_escaped_len(c) == c.len_utf8()) { + return Cow::Borrowed(s); + } + let mut quoted = serde_json::to_string(s).unwrap_or_default(); + quoted.pop(); + if !quoted.is_empty() { + quoted.remove(0); + } + if quoted.chars().any(is_line_separator) { + quoted = quoted + .replace('\u{85}', r"\u0085") + .replace('\u{2028}', r"\u2028") + .replace('\u{2029}', r"\u2029"); + } + Cow::Owned(quoted) +} + +/// `s` as a message writes it, a JSON string's content ([`json_escaped`]), +/// bounded: whole when its escaped text is at most `bound` bytes, otherwise +/// a prefix of at most [`SHOWN_VALUE_PREFIX_BYTES`] of escaped text and +/// `... ` with the text's length. +fn bounded_text(s: &str, bound: usize) -> Cow<'_, str> { + let mut rendered = 0; + let mut prefix = 0; + for c in s.chars() { + rendered += json_escaped_len(c); + if rendered > bound { + return Cow::Owned(format!( + "{}... <{} bytes>", + json_escaped(&s[..prefix]), + s.len() + )); + } + if rendered <= SHOWN_VALUE_PREFIX_BYTES { + prefix += c.len_utf8(); + } + } + json_escaped(s) +} + +/// Why an item cannot be restored. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum RestorableItemReason { + /// An `N` value, or a member of an `NS`, that is not a valid `DynamoDB` + /// number: the wire's own reason text, with the member appended for an + /// `NS` when the text does not already end in it (the #618 case; a + /// FAILED backup names the issue when its first refused item has this + /// reason). + InvalidNumber(String), + /// A key attribute missing, of the wrong type, empty, an `N` key not in + /// normalized form (the backup format's key rule's reason text), or a + /// key value over the hash or range key size limit. + KeyShape(String), + /// An attribute name (the path's last segment) longer than the + /// attribute name limit. + AttributeNameTooLong { size: usize, max: usize }, + /// The item is larger than the configured limit. + TooLarge { size: usize, max: usize }, + /// A map or list nests deeper than the item nesting limit. + TooDeep, + /// The item's data file line would exceed the format's line cap + /// (`MAX_LINE_BYTES`): the backup job measures the line of an item over + /// the default item limit, the largest the cap is sized for, and stops + /// measuring once the cap is passed, so the line's own length is not + /// known. + LineTooLong { cap: usize }, +} + +impl fmt::Display for RestorableItemReason { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::InvalidNumber(reason) | Self::KeyShape(reason) => f.write_str(reason), + Self::AttributeNameTooLong { size, max } => { + write!( + f, + "attribute name of {size} bytes exceeds the limit of {max} bytes" + ) + } + Self::TooLarge { size, max } => { + write!(f, "item size {size} bytes exceeds the limit of {max} bytes") + } + Self::TooDeep => write!( + f, + "attributes nest beyond {} levels", + super::MAX_ITEM_NESTING_DEPTH + ), + Self::LineTooLong { cap } => { + write!(f, "data file line exceeds the line cap of {cap} bytes") + } + } + } +} + +/// An item [`validate_restorable_item`] refused: which item (its key +/// attributes as compact JSON, each name and value bounded), where (the +/// attribute path, `a.b[2].c` for a member of a list inside a map, each +/// segment bounded, empty for a rule about the whole item) and why. +/// `Display` writes `key {key}, attribute {path}: {reason}`, without the +/// attribute clause when the path is empty. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RestorableItemError { + pub key: String, + pub path: String, + pub reason: RestorableItemReason, +} + +impl fmt::Display for RestorableItemError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.path.is_empty() { + write!(f, "key {}: {}", self.key, self.reason) + } else { + write!( + f, + "key {}, attribute {}: {}", + self.key, self.path, self.reason + ) + } + } +} + +impl std::error::Error for RestorableItemError {} + +/// Check that `item` is one a restore under `limits` can write: its key +/// attributes present with the declared types, not empty, an `N` key +/// normalized, each within its key size limit; every attribute name within +/// the name limit; every `N` and `NS` member a valid number; no map or list +/// nested beyond the item nesting limit; the item within the item size +/// limit. The first failure, in that order (key attributes in key-schema +/// order, then the other attributes in name order, then the size), is the +/// error. +/// +/// # Errors +/// +/// [`RestorableItemError`] naming the item's key, the attribute path and the +/// reason. +pub fn validate_restorable_item( + item: &Item, + key: &RestorableKeySchema, + limits: &RestorableLimits, +) -> Result<(), RestorableItemError> { + let refuse = |path: String, reason: RestorableItemReason| RestorableItemError { + key: key.shown_key(item), + path, + reason, + }; + for attribute in &key.attributes { + let name = attribute.name.as_str(); + let path = || bounded_text(name, SHOWN_VALUE_BYTES).into_owned(); + if name.len() > limits.max_attribute_name_bytes { + return Err(refuse( + path(), + RestorableItemReason::AttributeNameTooLong { + size: name.len(), + max: limits.max_attribute_name_bytes, + }, + )); + } + let Some(value) = item.get(name) else { + return Err(refuse( + path(), + RestorableItemReason::KeyShape(format!("key attribute {:?} missing", shown(name))), + )); + }; + if let Err(e) = check_key_value(name, attribute.ty, value) { + let reason = match e { + FormatError::InvalidKey(reason) => reason, + other => other.to_string(), + }; + // Classified by the declared type first: a text that is not a + // number under an `N` key is the #618 case; under any other + // declared type it is a key of the wrong type. + let reason = match (value, attribute.ty) { + (AttributeValue::N(n), ScalarAttributeType::N) => { + match validate_and_normalize_number(n) { + Err(wire) => RestorableItemReason::InvalidNumber(number_reason(n, &wire)), + Ok(_) => RestorableItemReason::KeyShape(reason), + } + } + _ => RestorableItemReason::KeyShape(reason), + }; + return Err(refuse(path(), reason)); + } + let (max, kind) = match attribute.key_type { + KeyType::Hash => (limits.max_partition_key_size_bytes, "hash"), + KeyType::Range => (limits.max_sort_key_size_bytes, "range"), + }; + let size = super::key_value_byte_size(value); + if size > max { + return Err(refuse( + path(), + RestorableItemReason::KeyShape(format!( + "key attribute {:?} has {size} bytes; the {kind} key limit is {max} bytes", + shown(name) + )), + )); + } + } + let mut path = String::new(); + for (name, value) in item { + if key.is_key(name) { + continue; + } + path.clear(); + push_segment(&mut path, name); + if let Err(reason) = + check_name(name, limits).and_then(|()| walk(value, &mut path, 0, limits)) + { + return Err(refuse(std::mem::take(&mut path), reason)); + } + } + let size = item_size_bytes(item); + if size > limits.max_item_size_bytes { + return Err(refuse( + String::new(), + RestorableItemReason::TooLarge { + size, + max: limits.max_item_size_bytes, + }, + )); + } + Ok(()) +} + +/// The attribute name rule the wire's `check_attribute_name_size` applies, +/// at the top level and for every map member. +fn check_name(name: &str, limits: &RestorableLimits) -> Result<(), RestorableItemReason> { + if name.len() > limits.max_attribute_name_bytes { + return Err(RestorableItemReason::AttributeNameTooLong { + size: name.len(), + max: limits.max_attribute_name_bytes, + }); + } + Ok(()) +} + +/// Append one attribute name to `path`, bounded and escaped +/// ([`bounded_text`]). +fn push_segment(path: &mut String, name: &str) { + path.push_str(&bounded_text(name, SHOWN_VALUE_BYTES)); +} + +/// Visit `value` and everything under it. On an error `path` names the +/// offending value; otherwise it is left as it was on entry. +fn walk( + value: &AttributeValue, + path: &mut String, + depth: usize, + limits: &RestorableLimits, +) -> Result<(), RestorableItemReason> { + match value { + AttributeValue::N(n) => number(n), + AttributeValue::NS(set) => { + for (i, n) in set.iter().enumerate() { + if let Err(RestorableItemReason::InvalidNumber(text)) = number(n) { + path.push_str(&format!("[{i}]")); + let member = bounded_text(n, SHOWN_VALUE_BYTES); + let text = if text.ends_with(&*member) { + text + } else { + format!("{text} (member {member})") + }; + return Err(RestorableItemReason::InvalidNumber(text)); + } + } + Ok(()) + } + AttributeValue::L(list) => { + let next = depth + 1; + if next >= super::MAX_ITEM_NESTING_DEPTH { + return Err(RestorableItemReason::TooDeep); + } + let len = path.len(); + for (i, v) in list.iter().enumerate() { + path.push_str(&format!("[{i}]")); + walk(v, path, next, limits)?; + path.truncate(len); + } + Ok(()) + } + AttributeValue::M(map) => { + let next = depth + 1; + if next >= super::MAX_ITEM_NESTING_DEPTH { + return Err(RestorableItemReason::TooDeep); + } + let len = path.len(); + for (name, v) in map { + path.push('.'); + push_segment(path, name); + check_name(name, limits)?; + walk(v, path, next, limits)?; + path.truncate(len); + } + Ok(()) + } + AttributeValue::S(_) + | AttributeValue::B(_) + | AttributeValue::SS(_) + | AttributeValue::BS(_) + | AttributeValue::Bool(_) + | AttributeValue::Null => Ok(()), + } +} + +fn number(n: &str) -> Result<(), RestorableItemReason> { + validate_and_normalize_number(n) + .map(drop) + .map_err(|e| RestorableItemReason::InvalidNumber(number_reason(n, &e))) +} + +/// The wire's reason for `n`, which ends in `n` for a malformed number: that +/// echo is bounded and escaped like every other untrusted text +/// ([`bounded_text`]), so the text is the wire's byte for byte up to +/// [`SHOWN_VALUE_BYTES`] for an input with nothing to escape. +fn number_reason(n: &str, e: &crate::error::DynamoDbError) -> String { + let text = e.message(); + match text.strip_suffix(n) { + Some(head) => format!("{head}{}", bounded_text(n, SHOWN_VALUE_BYTES)), + None => text.to_owned(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::types::KeyType; + use std::collections::{BTreeMap, BTreeSet}; + + const MAX: usize = 400 * 1024; + + /// The default limits with `max_item_size_bytes` at `max`. + fn limits(max: usize) -> RestorableLimits { + RestorableLimits { + max_item_size_bytes: max, + ..RestorableLimits::from(&LimitsConfig::default()) + } + } + + fn element(name: &str, key_type: KeyType) -> KeySchemaElement { + KeySchemaElement { + attribute_name: name.to_owned(), + key_type, + } + } + + fn definition(name: &str, ty: ScalarAttributeType) -> AttributeDefinition { + AttributeDefinition { + attribute_name: name.to_owned(), + attribute_type: ty, + } + } + + /// `pk` S alone. + fn hash_only() -> RestorableKeySchema { + RestorableKeySchema::resolve( + &[element("pk", KeyType::Hash)], + &[definition("pk", ScalarAttributeType::S)], + ) + .unwrap() + } + + /// `pk` S, `sk` N. + fn composite() -> RestorableKeySchema { + RestorableKeySchema::resolve( + &[element("pk", KeyType::Hash), element("sk", KeyType::Range)], + &[ + definition("pk", ScalarAttributeType::S), + definition("sk", ScalarAttributeType::N), + ], + ) + .unwrap() + } + + /// `pk` S, `sk` N, `sk2` B: a key `KeyOrder` cannot resolve. + fn multi_part() -> RestorableKeySchema { + RestorableKeySchema::resolve( + &[ + element("pk", KeyType::Hash), + element("sk", KeyType::Range), + element("sk2", KeyType::Range), + ], + &[ + definition("pk", ScalarAttributeType::S), + definition("sk", ScalarAttributeType::N), + definition("sk2", ScalarAttributeType::B), + ], + ) + .unwrap() + } + + fn s(v: &str) -> AttributeValue { + AttributeValue::S(v.to_owned()) + } + + fn n(v: &str) -> AttributeValue { + AttributeValue::N(v.to_owned()) + } + + fn item(pairs: &[(&str, AttributeValue)]) -> Item { + pairs + .iter() + .map(|(k, v)| ((*k).to_owned(), v.clone())) + .collect() + } + + fn refused(item: &Item, key: &RestorableKeySchema, max: usize) -> RestorableItemError { + validate_restorable_item(item, key, &limits(max)).expect_err("refused") + } + + #[test] + fn a_plain_item_passes_on_every_key_shape() { + let it = item(&[ + ("pk", s("a")), + ("sk", n("1")), + ("sk2", AttributeValue::B(vec![1])), + ("x", n("1E125")), + ( + "y", + AttributeValue::NS(BTreeSet::from(["1".to_owned(), "2.5".to_owned()])), + ), + ]); + validate_restorable_item(&it, &hash_only(), &limits(MAX)).unwrap(); + validate_restorable_item(&it, &composite(), &limits(MAX)).unwrap(); + validate_restorable_item(&it, &multi_part(), &limits(MAX)).unwrap(); + } + + #[test] + fn an_overflowing_number_in_a_non_key_attribute_is_refused_with_its_path() { + let it = item(&[("pk", s("a")), ("amount", n("1E126"))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.key, r#"{"pk":{"S":"a"}}"#); + assert_eq!(e.path, "amount"); + let RestorableItemReason::InvalidNumber(reason) = &e.reason else { + panic!("{e:?}"); + }; + assert!(reason.starts_with("Number overflow"), "{reason}"); + assert_eq!( + e.to_string(), + format!(r#"key {{"pk":{{"S":"a"}}}}, attribute amount: {reason}"#) + ); + } + + #[test] + fn a_non_numeric_text_is_refused_with_the_wire_reason() { + let it = item(&[("pk", s("a")), ("amount", n("abc"))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "amount"); + assert_eq!( + e.reason, + RestorableItemReason::InvalidNumber( + "The parameter cannot be converted to a numeric value: abc".to_owned() + ) + ); + } + + #[test] + fn a_nested_number_names_its_path_through_maps_and_lists() { + let inner = AttributeValue::M(BTreeMap::from([("c".to_owned(), n("abc"))])); + let list = AttributeValue::L(vec![n("1"), n("2"), inner]); + let a = AttributeValue::M(BTreeMap::from([("b".to_owned(), list)])); + let it = item(&[("pk", s("a")), ("a", a)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "a.b[2].c"); + assert!(matches!(e.reason, RestorableItemReason::InvalidNumber(_))); + } + + /// A set member's reason carries the member, since the position is the + /// sorted set's and the overflow text alone does not name the value; a + /// text that already ends in the member (the malformed case) is left as + /// it is. + #[test] + fn a_number_set_member_names_its_position_and_the_member() { + let set = AttributeValue::NS(BTreeSet::from([ + "1".to_owned(), + "2".to_owned(), + "x".to_owned(), + ])); + let it = item(&[("pk", s("a")), ("ns", set)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "ns[2]"); + assert_eq!( + e.reason, + RestorableItemReason::InvalidNumber( + "The parameter cannot be converted to a numeric value: x".to_owned() + ) + ); + let set = AttributeValue::NS(BTreeSet::from(["01E126".to_owned(), "1E126".to_owned()])); + let it = item(&[("pk", s("a")), ("ns", set)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "ns[0]"); + let RestorableItemReason::InvalidNumber(reason) = &e.reason else { + panic!("{e:?}"); + }; + assert!(reason.starts_with("Number overflow"), "{reason}"); + assert!(reason.ends_with(" (member 01E126)"), "{reason}"); + } + + /// The path is rebuilt from the attribute for each sibling: a list whose + /// first member is a valid map and whose second member is bad names the + /// second, not a path that still carries the first's members. + #[test] + fn the_path_does_not_carry_a_sibling_that_passed() { + let ok = AttributeValue::M(BTreeMap::from([("deep".to_owned(), n("1"))])); + let list = AttributeValue::L(vec![ok, n("abc")]); + let it = item(&[("pk", s("a")), ("l", list)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "l[1]"); + } + + #[test] + fn validation_does_not_stop_at_the_first_valid_number() { + // Three valid numbers before the bad one, in name order. + let it = item(&[ + ("pk", s("a")), + ("n1", n("1")), + ("n2", n("2")), + ("n3", n("3")), + ("n4", n("1E126")), + ]); + assert_eq!(refused(&it, &hash_only(), MAX).path, "n4"); + } + + #[test] + fn an_item_over_the_size_limit_is_refused_without_a_path() { + let it = item(&[("pk", s("a")), ("blob", s(&"x".repeat(100)))]); + let size = item_size_bytes(&it); + validate_restorable_item(&it, &hash_only(), &limits(size)).unwrap(); + let e = refused(&it, &hash_only(), size - 1); + assert_eq!(e.path, ""); + assert_eq!( + e.reason, + RestorableItemReason::TooLarge { + size, + max: size - 1 + } + ); + assert_eq!( + e.to_string(), + format!( + r#"key {{"pk":{{"S":"a"}}}}: item size {size} bytes exceeds the limit of {} bytes"#, + size - 1 + ) + ); + } + + #[test] + fn nesting_at_the_limit_is_refused_and_one_short_of_it_passes() { + let nest = |levels: usize| { + let mut v = n("1"); + for _ in 0..levels { + v = AttributeValue::M(BTreeMap::from([("m".to_owned(), v)])); + } + item(&[("pk", s("a")), ("deep", v)]) + }; + let limit = super::super::MAX_ITEM_NESTING_DEPTH; + validate_restorable_item(&nest(limit - 1), &hash_only(), &limits(MAX)).unwrap(); + let e = refused(&nest(limit), &hash_only(), MAX); + assert_eq!(e.reason, RestorableItemReason::TooDeep); + assert_eq!(e.path, format!("deep{}", ".m".repeat(limit - 1))); + // The same rule the request path applies. + super::super::validate_item_nesting_depth(&nest(limit - 1)).unwrap(); + super::super::validate_item_nesting_depth(&nest(limit)).unwrap_err(); + } + + #[test] + fn a_missing_key_attribute_is_refused() { + let it = item(&[("pk", s("a"))]); + let e = refused(&it, &composite(), MAX); + assert_eq!(e.path, "sk"); + assert_eq!(e.key, r#"{"pk":{"S":"a"}}"#); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape(r#"key attribute "sk" missing"#.to_owned()) + ); + } + + #[test] + fn a_key_attribute_of_the_wrong_type_is_refused() { + let it = item(&[("pk", s("a")), ("sk", s("1"))]); + let e = refused(&it, &composite(), MAX); + assert_eq!(e.path, "sk"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape( + r#"key attribute "sk" is S, the schema declares N"#.to_owned() + ) + ); + assert_eq!(e.key, r#"{"pk":{"S":"a"},"sk":{"S":"1"}}"#); + } + + /// A text that is not a number under a key declared `S` is a key of the + /// wrong type, not the #618 case: the declared type decides first. + #[test] + fn a_non_numeric_n_under_an_s_key_is_a_key_shape_error() { + let it = item(&[("pk", n("abc"))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "pk"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape( + r#"key attribute "pk" is N, the schema declares S"#.to_owned() + ) + ); + } + + #[test] + fn an_empty_key_value_is_refused() { + let it = item(&[("pk", s(""))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "pk"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape(r#"key attribute "pk" is empty"#.to_owned()) + ); + } + + /// A bad number in the range key is the #618 case, not a key-shape one: + /// the backup's FAILED reason names the issue by this variant. + #[test] + fn an_invalid_number_in_a_range_key_is_an_invalid_number() { + for bad in ["abc", "1E126"] { + let it = item(&[("pk", s("a")), ("sk", n(bad))]); + let e = refused(&it, &composite(), MAX); + assert_eq!(e.path, "sk"); + assert!( + matches!(e.reason, RestorableItemReason::InvalidNumber(_)), + "{bad}: {e:?}" + ); + } + } + + #[test] + fn a_valid_but_unnormalized_number_key_is_a_key_shape_error() { + let it = item(&[("pk", s("a")), ("sk", n("1.0"))]); + let e = refused(&it, &composite(), MAX); + assert_eq!(e.path, "sk"); + let RestorableItemReason::KeyShape(reason) = &e.reason else { + panic!("{e:?}"); + }; + assert!(reason.contains("not in normalized form"), "{reason}"); + } + + #[test] + fn a_multi_part_key_is_checked_attribute_by_attribute() { + let it = item(&[ + ("pk", s("a")), + ("sk", n("1")), + ("sk2", AttributeValue::B(vec![])), + ]); + let e = refused(&it, &multi_part(), MAX); + assert_eq!(e.path, "sk2"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape(r#"key attribute "sk2" is empty"#.to_owned()) + ); + assert_eq!(e.key, r#"{"pk":{"S":"a"},"sk":{"N":"1"},"sk2":{"B":""}}"#); + } + + #[test] + fn key_attributes_are_checked_before_the_other_attributes() { + let it = item(&[("amount", n("abc")), ("pk", s(""))]); + assert_eq!(refused(&it, &hash_only(), MAX).path, "pk"); + } + + #[test] + fn a_schema_whose_key_attribute_lacks_a_definition_does_not_resolve() { + let err = RestorableKeySchema::resolve(&[element("pk", KeyType::Hash)], &[]).unwrap_err(); + assert_eq!(err, r#"key attribute "pk" has no attribute definition"#); + let err = RestorableKeySchema::resolve( + &[element("pk", KeyType::Hash)], + &[ + definition("pk", ScalarAttributeType::S), + definition("pk", ScalarAttributeType::N), + ], + ) + .unwrap_err(); + assert_eq!( + err, + r#"key attribute "pk" has more than one attribute definition"# + ); + } + + /// The key sizes against the instance's limits, the rule the wire's + /// `check_key_size` applies: at the limit the key passes, one byte over + /// it is refused, hash and range each against their own limit; a `B` + /// key's size is its bytes. + #[test] + fn a_key_over_its_size_limit_is_refused_at_the_boundary() { + let limits = RestorableLimits { + max_partition_key_size_bytes: 10, + max_sort_key_size_bytes: 4, + ..limits(MAX) + }; + let ok = item(&[("pk", s(&"p".repeat(10))), ("sk", n("1234"))]); + validate_restorable_item(&ok, &composite(), &limits).unwrap(); + let hash = item(&[("pk", s(&"p".repeat(11))), ("sk", n("1"))]); + let e = validate_restorable_item(&hash, &composite(), &limits).unwrap_err(); + assert_eq!(e.path, "pk"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape( + r#"key attribute "pk" has 11 bytes; the hash key limit is 10 bytes"#.to_owned() + ) + ); + let range = item(&[("pk", s("p")), ("sk", n("12345"))]); + let e = validate_restorable_item(&range, &composite(), &limits).unwrap_err(); + assert_eq!(e.path, "sk"); + assert_eq!( + e.reason, + RestorableItemReason::KeyShape( + r#"key attribute "sk" has 5 bytes; the range key limit is 4 bytes"#.to_owned() + ) + ); + let binary = item(&[ + ("pk", s("p")), + ("sk", n("1")), + ("sk2", AttributeValue::B(vec![0; 5])), + ]); + let e = validate_restorable_item(&binary, &multi_part(), &limits).unwrap_err(); + assert_eq!(e.path, "sk2"); + assert!( + e.to_string().contains("has 5 bytes; the range key limit"), + "{e}" + ); + // The defaults: a 2048-byte hash key and a 1024-byte range key pass. + let largest = item(&[("pk", s(&"p".repeat(2048))), ("sk", n(&"1".repeat(38)))]); + validate_restorable_item(&largest, &composite(), &self::limits(MAX)).unwrap(); + let over = item(&[("pk", s(&"p".repeat(2049))), ("sk", n("1"))]); + assert!(validate_restorable_item(&over, &composite(), &self::limits(MAX)).is_err()); + } + + /// The attribute name rule the wire's `check_attribute_name_size` + /// applies: a name at the limit passes, one byte over is refused, at the + /// top level, inside a map and for a key attribute. + #[test] + fn an_attribute_name_over_its_limit_is_refused_at_the_boundary() { + let limits = RestorableLimits { + max_attribute_name_bytes: 6, + ..limits(MAX) + }; + let ok = item(&[("pk", s("a")), ("abcdef", n("1"))]); + validate_restorable_item(&ok, &hash_only(), &limits).unwrap(); + let top = item(&[("pk", s("a")), ("abcdefg", n("1"))]); + let e = validate_restorable_item(&top, &hash_only(), &limits).unwrap_err(); + assert_eq!(e.path, "abcdefg"); + assert_eq!( + e.reason, + RestorableItemReason::AttributeNameTooLong { size: 7, max: 6 } + ); + assert_eq!( + e.to_string(), + r#"key {"pk":{"S":"a"}}, attribute abcdefg: attribute name of 7 bytes exceeds the limit of 6 bytes"# + ); + let nested = AttributeValue::M(BTreeMap::from([("abcdefg".to_owned(), n("1"))])); + let it = item(&[("pk", s("a")), ("m", nested)]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!(e.path, "m.abcdefg"); + assert_eq!( + e.reason, + RestorableItemReason::AttributeNameTooLong { size: 7, max: 6 } + ); + let key = RestorableKeySchema::resolve( + &[element("abcdefg", KeyType::Hash)], + &[definition("abcdefg", ScalarAttributeType::S)], + ) + .unwrap(); + let it = item(&[("abcdefg", s("a"))]); + let e = validate_restorable_item(&it, &key, &limits).unwrap_err(); + assert_eq!(e.path, "abcdefg"); + assert_eq!( + e.reason, + RestorableItemReason::AttributeNameTooLong { size: 7, max: 6 } + ); + } + + /// The limits are the instance's: `RestorableLimits` carries the four + /// sizes of `LimitsConfig`. + #[test] + fn restorable_limits_are_built_from_the_limits_config() { + let config = LimitsConfig { + max_item_size_bytes: 1, + max_partition_key_size_bytes: 2, + max_sort_key_size_bytes: 3, + max_attribute_name_bytes: 4, + ..LimitsConfig::default() + }; + assert_eq!( + RestorableLimits::from(&config), + RestorableLimits { + max_item_size_bytes: 1, + max_partition_key_size_bytes: 2, + max_sort_key_size_bytes: 3, + max_attribute_name_bytes: 4, + } + ); + } + + /// Keys of ordinary size are shown whole, every attribute present, as + /// `Display` writes them: a UUID hash key with a ten-digit range key, a + /// 58-byte hash key with range `1`, a 2048-byte hash key with a + /// 1024-byte range key (the largest legal key), and a hash key of + /// quotes, whose escaped text is twice its bytes. + #[test] + fn the_shown_key_carries_every_attribute_of_a_legal_key_whole() { + let uuid = "39518456-7a1b-4c2d-9e3f-0a1b2c3d832c"; + let it = item(&[ + ("pk", s(uuid)), + ("sk", n("1761955200")), + ("amount", n("abc")), + ]); + let e = refused(&it, &composite(), MAX); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{uuid}"}},"sk":{{"N":"1761955200"}}}}"#) + ); + assert_eq!( + e.to_string(), + format!( + r#"key {{"pk":{{"S":"{uuid}"}},"sk":{{"N":"1761955200"}}}}, attribute amount: The parameter cannot be converted to a numeric value: abc"# + ) + ); + let tenant = "tenant-42/orders/2026-10-31/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; + assert_eq!(tenant.len(), 58); + let it = item(&[("pk", s(tenant)), ("sk", n("1")), ("amount", n("abc"))]); + let e = refused(&it, &composite(), MAX); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{tenant}"}},"sk":{{"N":"1"}}}}"#) + ); + let hash = "h".repeat(2048); + let range = "r".repeat(1024); + let key = RestorableKeySchema::resolve( + &[element("pk", KeyType::Hash), element("sk", KeyType::Range)], + &[ + definition("pk", ScalarAttributeType::S), + definition("sk", ScalarAttributeType::S), + ], + ) + .unwrap(); + let it = item(&[("pk", s(&hash)), ("sk", s(&range)), ("amount", n("abc"))]); + let e = refused(&it, &key, MAX); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{hash}"}},"sk":{{"S":"{range}"}}}}"#) + ); + let quotes = "\"".repeat(2048); + let it = item(&[("pk", s("es)), ("amount", n("abc"))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!( + e.key, + format!( + r#"{{"pk":{{"S":{}}}}}"#, + serde_json::to_string("es).unwrap() + ) + ); + // A `B` key is its base64 text, the format's rendering. + let it = item(&[ + ("pk", s("a")), + ("sk", n("1")), + ("sk2", AttributeValue::B(vec![0xff; 2048])), + ("amount", n("abc")), + ]); + let e = refused(&it, &multi_part(), MAX); + assert!( + e.key + .ends_with(&format!(r#""sk2":{{"B":"{}"}}}}"#, "/".repeat(2730) + "8=")), + "{}", + e.key + ); + } + + /// Only an oversized value is cut, and on its own: a 13,000-byte hash + /// key (over `SHOWN_KEY_VALUE_BYTES`, under a raised key limit) is cut + /// to its prefix and length while the range key beside it is shown + /// whole; a 3000-byte key, over the service's limit but under the + /// bound, is whole; the bound is on the escaped text, so a key of 6145 + /// quotes (12,290 escaped bytes) is cut and a key of 6144 is not; a cut + /// value is still a JSON string. A key attribute's name is bounded at + /// `SHOWN_VALUE_BYTES` (one of 4097 bytes is cut), and a key attribute + /// holding a list is its JSON, cut to a JSON string when oversized. + #[test] + fn an_oversized_key_value_is_cut_on_its_own() { + let limits = RestorableLimits { + max_partition_key_size_bytes: 20_000, + ..limits(10) + }; + let long = "k".repeat(13_000); + let it = item(&[("pk", s(&long)), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert!(matches!( + e.reason, + RestorableItemReason::TooLarge { max: 10, .. } + )); + assert_eq!( + e.key, + format!( + r#"{{"pk":{{"S":"{}... <13000 bytes>"}},"sk":{{"N":"7"}}}}"#, + "k".repeat(SHOWN_VALUE_PREFIX_BYTES) + ) + ); + let over_the_service_limit = "k".repeat(3000); + let it = item(&[("pk", s(&over_the_service_limit)), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{over_the_service_limit}"}},"sk":{{"N":"7"}}}}"#) + ); + let parsed: serde_json::Value = serde_json::from_str(&e.key).unwrap(); + assert_eq!(parsed["sk"]["N"], "7"); + let plain = "p".repeat(SHOWN_KEY_VALUE_BYTES); + let it = item(&[("pk", s(&plain)), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{plain}"}},"sk":{{"N":"7"}}}}"#) + ); + let quotes = "\"".repeat(SHOWN_KEY_VALUE_BYTES / 2); + let it = item(&[("pk", s("es)), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert_eq!( + e.key, + format!( + r#"{{"pk":{{"S":{}}},"sk":{{"N":"7"}}}}"#, + serde_json::to_string("es).unwrap() + ) + ); + let quotes = "\"".repeat(SHOWN_KEY_VALUE_BYTES / 2 + 1); + let it = item(&[("pk", s("es)), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert!(e.key.contains("... <6145 bytes>"), "{}", e.key); + assert!(e.key.ends_with(r#","sk":{"N":"7"}}"#), "{}", e.key); + let parsed: serde_json::Value = serde_json::from_str(&e.key).unwrap(); + assert_eq!( + parsed["pk"]["S"].as_str().unwrap().len(), + 512 + "... <6145 bytes>".len() + ); + // The whole key stays bounded whatever the key: two 100,000-byte + // values and a 100,000-byte name. + let huge = "x".repeat(100_000); + let key = RestorableKeySchema::resolve( + &[element(&huge, KeyType::Hash), element("sk", KeyType::Range)], + &[ + definition(&huge, ScalarAttributeType::S), + definition("sk", ScalarAttributeType::S), + ], + ) + .unwrap(); + let it = item(&[(&huge, s(&huge)), ("sk", s(&huge))]); + let limits = RestorableLimits { + max_partition_key_size_bytes: 200_000, + max_sort_key_size_bytes: 200_000, + max_attribute_name_bytes: 200_000, + ..self::limits(10) + }; + let e = validate_restorable_item(&it, &key, &limits).unwrap_err(); + assert!( + e.key.len() < 3 * (SHOWN_VALUE_PREFIX_BYTES + 40), + "{}", + e.key.len() + ); + assert_eq!(e.key.matches("... <100000 bytes>").count(), 3); + // A key attribute's name is bounded at `SHOWN_VALUE_BYTES`, the + // bound a name gets, not the key value's: one of 4097 bytes is cut. + let long_name = "k".repeat(SHOWN_VALUE_BYTES + 1); + let key = RestorableKeySchema::resolve( + &[element(&long_name, KeyType::Hash)], + &[definition(&long_name, ScalarAttributeType::S)], + ) + .unwrap(); + let it = item(&[(long_name.as_str(), s("a")), ("amount", n("abc"))]); + let limits = RestorableLimits { + max_attribute_name_bytes: 10_000, + ..self::limits(MAX) + }; + let e = validate_restorable_item(&it, &key, &limits).unwrap_err(); + assert_eq!(e.path, "amount"); + assert_eq!( + e.key, + format!( + r#"{{"{}... <4097 bytes>":{{"S":"a"}}}}"#, + "k".repeat(SHOWN_VALUE_PREFIX_BYTES) + ) + ); + // A key attribute holding a value that is not a scalar (a list of + // 20,000 bytes under `pk`) renders as its JSON cut to a JSON string: + // the key still parses and stays under 1,200 bytes. + let list = AttributeValue::L(vec![s(&"x".repeat(20_000))]); + let list_json_len = serde_json::to_string(&list).unwrap().len(); + assert!(list_json_len > SHOWN_KEY_VALUE_BYTES); + let it = item(&[("pk", list), ("sk", n("7"))]); + let e = validate_restorable_item(&it, &composite(), &self::limits(MAX)).unwrap_err(); + assert!(e.key.len() < 1200, "{}", e.key.len()); + let parsed: serde_json::Value = serde_json::from_str(&e.key).unwrap(); + assert!( + parsed["pk"] + .as_str() + .unwrap() + .ends_with(&format!("... <{list_json_len} bytes>")), + "{}", + e.key + ); + assert_eq!(parsed["sk"]["N"], "7"); + } + + /// The attribute path is bounded segment by segment with the same rule: + /// a 60,000-byte attribute name holding a map with a 60,000-byte member + /// gives a path of two cut segments, each with its length; a name of + /// `SHOWN_VALUE_BYTES` bytes is whole and one byte more is cut. + #[test] + fn the_attribute_path_is_bounded_segment_by_segment() { + let limits = RestorableLimits { + max_attribute_name_bytes: 100_000, + ..limits(MAX) + }; + let long_name = "a".repeat(60_000); + let long_member = "b".repeat(60_000); + let inner = AttributeValue::M(BTreeMap::from([(long_member, n("abc"))])); + let it = item(&[("pk", s("a")), (&long_name, inner)]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!( + e.path, + format!( + "{}... <60000 bytes>.{}... <60000 bytes>", + "a".repeat(SHOWN_VALUE_PREFIX_BYTES), + "b".repeat(SHOWN_VALUE_PREFIX_BYTES) + ) + ); + assert!(e.to_string().len() < 3000, "{}", e.to_string().len()); + let whole = "w".repeat(SHOWN_VALUE_BYTES); + let it = item(&[("pk", s("a")), (&whole, n("abc"))]); + assert_eq!( + validate_restorable_item(&it, &hash_only(), &limits) + .unwrap_err() + .path, + whole + ); + let over = "w".repeat(SHOWN_VALUE_BYTES + 1); + let it = item(&[("pk", s("a")), (&over, n("abc"))]); + assert_eq!( + validate_restorable_item(&it, &hash_only(), &limits) + .unwrap_err() + .path, + format!("{}... <4097 bytes>", "w".repeat(SHOWN_VALUE_PREFIX_BYTES)) + ); + // A set member over the bound is cut in the reason, as a plain `N` + // over it is: the wire's echo of the input is the one text that can + // grow with the item. + let member = format!("1E{}", "9".repeat(SHOWN_VALUE_BYTES)); + let set = AttributeValue::NS(BTreeSet::from([member.clone()])); + let it = item(&[("pk", s("a")), ("ns", set)]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert!(e.to_string().len() < 2000, "{}", e.to_string().len()); + assert!( + e.to_string() + .ends_with(&format!("<{} bytes>", member.len())), + "{e}" + ); + let it = item(&[("pk", s("a")), ("n", n(&member))]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!( + e.reason, + RestorableItemReason::InvalidNumber(format!( + "The parameter cannot be converted to a numeric value: {}... <{} bytes>", + &member[..SHOWN_VALUE_PREFIX_BYTES], + member.len() + )) + ); + // In a key the value is echoed twice: whole in the key (under the + // key bound) and cut in the reason. + let it = item(&[("pk", s("a")), ("sk", n(&member))]); + let e = validate_restorable_item(&it, &composite(), &limits).unwrap_err(); + assert!( + e.to_string().len() < member.len() + SHOWN_VALUE_PREFIX_BYTES + 400, + "{}", + e.to_string().len() + ); + assert!(e.key.contains(&member), "{}", e.key); + assert_eq!(e.to_string().matches("... <4098 bytes>").count(), 1, "{e}"); + } + + /// `LineTooLong`, the reason the backup job's feeder builds for an item + /// whose data file line would exceed the cap, names the cap: the + /// measurement stops at the cap, so the line's own length is unknown. + #[test] + fn a_line_too_long_reason_names_the_cap() { + let e = RestorableItemError { + key: r#"{"pk":{"S":"a"}}"#.to_owned(), + path: String::new(), + reason: RestorableItemReason::LineTooLong { cap: 33_554_432 }, + }; + assert_eq!( + e.to_string(), + r#"key {"pk":{"S":"a"}}: data file line exceeds the line cap of 33554432 bytes"# + ); + } + + /// A control character, or one of the line separators U+0085, U+2028 and + /// U+2029, in an attribute name or in the number the wire's reason + /// echoes is written escaped, as a JSON string's content (`\n`, `\t`, + /// `\"`, `\\`, `\u00XX`, `\u2028`), so a tenant's item cannot add a + /// line of its own to the log: the message holds no raw newline and no + /// separator, and the key beside it is the same escaped rendering. + #[test] + fn a_control_character_in_a_name_or_a_number_is_escaped_in_the_message() { + let it = item(&[ + ("pk", s("a\nb")), + ("amount", n("abc\nFORGED LINE: looks like another record")), + ]); + let e = refused(&it, &hash_only(), MAX); + let message = e.to_string(); + assert!(!message.contains('\n'), "{message:?}"); + assert_eq!( + message, + r#"key {"pk":{"S":"a\nb"}}, attribute amount: The parameter cannot be converted to a numeric value: abc\nFORGED LINE: looks like another record"# + ); + let name = "att\nFORGED LINE TWO\t\"quoted\" \\ \u{1}"; + let it = item(&[("pk", s("a")), (name, n("abc"))]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, r#"att\nFORGED LINE TWO\t\"quoted\" \\ \u0001"#); + assert!(!e.to_string().contains('\n'), "{e}"); + // In a map, in a set member and in a key attribute's name the same. + let set = AttributeValue::NS(BTreeSet::from(["1".to_owned(), "x\ny".to_owned()])); + let it = item(&[("pk", s("a")), ("ns", set)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, "ns[1]"); + assert_eq!( + e.reason, + RestorableItemReason::InvalidNumber( + r"The parameter cannot be converted to a numeric value: x\ny".to_owned() + ) + ); + let inner = AttributeValue::M(BTreeMap::from([("c\rd".to_owned(), n("abc"))])); + let it = item(&[("pk", s("a")), ("m\u{c}", inner)]); + let e = refused(&it, &hash_only(), MAX); + assert_eq!(e.path, r"m\f.c\rd"); + let key = RestorableKeySchema::resolve( + &[element("p\nk", KeyType::Hash)], + &[definition("p\nk", ScalarAttributeType::S)], + ) + .unwrap(); + let it = item(&[("p\nk", s(""))]); + let e = validate_restorable_item(&it, &key, &limits(MAX)).unwrap_err(); + assert_eq!(e.path, r"p\nk"); + assert_eq!(e.key, r#"{"p\nk":{"S":""}}"#); + // The three Unicode line separators, which `serde_json` leaves raw + // and a reader that breaks lines the Unicode way breaks on, in a + // key value, a name and a number: each is written as its `\uXXXX` + // escape and none reaches the message. + let separators = "\u{85}\u{2028}\u{2029}"; + let name = format!("att{separators}"); + let it = item(&[ + ("pk", s(&format!("a{separators}b"))), + (name.as_str(), n(&format!("abc{separators}FORGED LINE"))), + ]); + let e = refused(&it, &hash_only(), MAX); + let message = e.to_string(); + assert!( + !message + .chars() + .any(|c| matches!(c, '\u{85}' | '\u{2028}' | '\u{2029}')), + "{message:?}" + ); + assert_eq!(e.key, r#"{"pk":{"S":"a\u0085\u2028\u2029b"}}"#); + assert_eq!(e.path, r"att\u0085\u2028\u2029"); + assert_eq!( + e.reason, + RestorableItemReason::InvalidNumber( + r"The parameter cannot be converted to a numeric value: abc\u0085\u2028\u2029FORGED LINE" + .to_owned() + ) + ); + } + + /// A key value is bounded at `SHOWN_KEY_VALUE_BYTES`, six times the hash + /// key limit: a 700-byte hash key of U+0001 (4,200 escaped bytes, over + /// the bound a name gets) is shown whole, and a 2049-byte one (12,294 + /// escaped bytes, over the legal key size) is cut. + #[test] + fn a_legal_key_of_control_characters_is_shown_whole() { + let limits = RestorableLimits { + max_partition_key_size_bytes: 4096, + ..limits(MAX) + }; + let legal = "\u{1}".repeat(700); + let it = item(&[("pk", s(&legal)), ("amount", n("abc"))]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!(e.path, "amount"); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{}"}}}}"#, r"\u0001".repeat(700)) + ); + let largest = "\u{1}".repeat(2048); + let it = item(&[("pk", s(&largest)), ("amount", n("abc"))]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!( + e.key, + format!(r#"{{"pk":{{"S":"{}"}}}}"#, r"\u0001".repeat(2048)) + ); + let over = "\u{1}".repeat(2049); + let it = item(&[("pk", s(&over)), ("amount", n("abc"))]); + let e = validate_restorable_item(&it, &hash_only(), &limits).unwrap_err(); + assert_eq!( + e.key, + format!( + r#"{{"pk":{{"S":"{}... <2049 bytes>"}}}}"#, + r"\u0001".repeat(SHOWN_VALUE_PREFIX_BYTES / 6) + ) + ); + let parsed: serde_json::Value = serde_json::from_str(&e.key).unwrap(); + assert_eq!(parsed["pk"]["S"].as_str().unwrap().len(), 170 + 16); + } + + /// The depth rule through lists, and through lists and maps mixed: at + /// the limit refused, one short of it passed, the path naming each + /// level, and the request path's rule agreeing on every shape. + #[test] + fn nesting_through_lists_and_mixed_levels_is_measured_the_same() { + let limit = super::super::MAX_ITEM_NESTING_DEPTH; + let lists = |levels: usize| { + let mut v = n("1"); + for _ in 0..levels { + v = AttributeValue::L(vec![v]); + } + item(&[("pk", s("a")), ("deep", v)]) + }; + let mixed = |levels: usize| { + let mut v = n("1"); + for i in 0..levels { + v = if i % 2 == 0 { + AttributeValue::L(vec![v]) + } else { + AttributeValue::M(BTreeMap::from([("m".to_owned(), v)])) + }; + } + item(&[("pk", s("a")), ("deep", v)]) + }; + for (what, nest) in [ + ("lists", &lists as &dyn Fn(usize) -> Item), + ("mixed", &mixed as &dyn Fn(usize) -> Item), + ] { + validate_restorable_item(&nest(limit - 1), &hash_only(), &limits(MAX)) + .unwrap_or_else(|e| panic!("{what}, one short of the limit: {e}")); + let e = refused(&nest(limit), &hash_only(), MAX); + assert_eq!(e.reason, RestorableItemReason::TooDeep, "{what}"); + super::super::validate_item_nesting_depth(&nest(limit - 1)) + .unwrap_or_else(|e| panic!("{what}: the request path refuses one short: {e}")); + super::super::validate_item_nesting_depth(&nest(limit)).unwrap_err(); + } + let e = refused(&lists(limit), &hash_only(), MAX); + assert_eq!(e.path, format!("deep{}", "[0]".repeat(limit - 1))); + let e = refused(&mixed(limit), &hash_only(), MAX); + // Built inside out: the outermost level is the last one wrapped. + let mut path = "deep".to_owned(); + for i in (0..limit - 1).rev() { + path.push_str(if (i + 1) % 2 == 0 { "[0]" } else { ".m" }); + } + assert_eq!(e.path, path); + } +} diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml index d8e20bc15..6800acfdd 100755 --- a/crates/engine/Cargo.toml +++ b/crates/engine/Cargo.toml @@ -7,6 +7,11 @@ edition.workspace = true rust-version.workspace = true license.workspace = true +[features] +# Kill-point pauses in the backup job (`EXTENDDB_BACKUP_PAUSE`), for the +# binary-level crash tests; never enabled in a release build. +backup-test-hooks = [] + [dependencies] extenddb-auth = { workspace = true } extenddb-core = { workspace = true } @@ -22,6 +27,9 @@ hmac = "0.12" sha2 = "0.10" hex = "0.4" tokio = { workspace = true } +# `TaskTracker` for the backup jobs the server drains at shutdown. +tokio-util = { workspace = true, features = ["rt"] } +bytes = { workspace = true } futures = { workspace = true } [dev-dependencies] diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs index 56445e86b..32d7b4615 100755 --- a/crates/engine/src/backup.rs +++ b/crates/engine/src/backup.rs @@ -16,7 +16,7 @@ use std::sync::Arc; -use extenddb_core::backup_format::{BackupArn, is_region}; +use extenddb_core::backup_format::{BackupArn, check_store_prefix_table_name, is_region}; use extenddb_core::error::{DynamoDbError, ErrorMessageKey, error_message}; use extenddb_core::limits::LimitsConfig; use extenddb_core::types::{ @@ -29,16 +29,17 @@ use extenddb_storage::backup_catalog::{ }; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, LegacyBackupReader, RestoreRequest, RestoreTableOverrides, - TableEngine, + BackupCatalog, BackupRestoreSource, LegacyBackupReader, MetadataEngine, RestoreRequest, + RestoreTableOverrides, SnapshotEngine, TableEngine, }; use futures::TryStreamExt; use serde_json::{Value, json}; +pub use job::KillPoint; pub use maintenance::HeldClaim; pub use runtime::{ - BackupRuntime, ConfiguredStore, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, StoreHealth, - StoreHealthReport, check_configured_store, read_identity_bounded, + BackupRuntime, ConfiguredStore, JobSettings, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, + StoreHealth, StoreHealthReport, check_configured_store, read_identity_bounded, }; pub use store_identity::{STORE_IDENTITY_KEY, StoreIdentityFile, StoreProblem}; @@ -418,10 +419,11 @@ pub(crate) async fn handle_create_backup( ) .await; let details = create_backup( - &*ctx.storage, + &ctx.storage, &ctx.backup, &ctx.account_id, &ctx.region, + &ctx.limits, table_name, backup_name, probe, @@ -435,23 +437,53 @@ pub(crate) async fn handle_create_backup( /// account (`TableNotFoundException`, "Table not found: ", otherwise, /// whether or not a permit is free); the server's region fits a backup ARN /// (`check_backup_arn_fits`: a `server.region` outside `[a-z0-9-]` is a -/// `ValidationException` naming the setting); the table has no vector index -/// (the in-catalog format cannot restore one); then the job permit +/// `ValidationException` naming the setting); on the store-backed path, the +/// table name fits a store prefix (`check_store_prefix_fits`: a name over 255 +/// bytes or made only of dots is a `ValidationException` naming the rule; the +/// in-catalog path writes no prefix and backs such a table up, as the service +/// does); on the in-catalog path, the table has no vector index (the +/// in-catalog format cannot restore one); then the job permit /// (`LimitExceededException` when none is freed within the runtime's wait). /// `probe` is the `DescribeTable` of the table, which the handler ran. /// -/// This is the in-catalog create, the path while no `[backup.store]` is -/// configured; each call logs that the format is deprecated. Last before the -/// permit, the maintenance claim ([`refuse_while_maintenance_held`]). -async fn create_backup( - storage: &S, +/// With a store configured and the backend not bound to the in-catalog +/// create, the create is store-backed: the store must be healthy +/// ([`BackupRuntime::require_healthy_store`], refused before a slot is +/// taken as every operation that touches the store is), the maintenance +/// claim must not be held ([`refuse_while_maintenance_held`], before the +/// permit as on the in-catalog path, so a refusal inserts no row), then the +/// permit (`LimitExceededException` when none is free), then the detached +/// task of [`job::start`] (the snapshot's open, the row under the job's +/// claim, the response CREATING) whose answer the request awaits. A vector +/// table is backed up on this path (the manifest records its indexes). +/// +/// Otherwise the in-catalog create: the table has no vector index (the +/// in-catalog format cannot restore one), the maintenance claim is not held +/// ([`refuse_while_maintenance_held`], last before the permit), the permit, +/// then `LegacyBackupReader::create`; each call logs that the format is +/// deprecated. `probe` is the `DescribeTable` of the table, which the +/// handler ran. +#[allow(clippy::too_many_arguments)] +async fn create_backup( + storage: &Arc, runtime: &BackupRuntime, account_id: &str, region: &str, + limits: &LimitsConfig, table_name: &str, backup_name: &str, probe: Result, -) -> Result { +) -> Result +where + S: BackupCatalog + + LegacyBackupReader + + SnapshotEngine + + MetadataEngine + + ?Sized + + Send + + Sync + + 'static, +{ let desc = match probe { Ok(desc) if desc.table_status == TableStatus::Active => desc, Ok(_) => { @@ -463,20 +495,34 @@ async fn create_backup( Err(e) => return Err(create_backup_storage_err_to_dynamo(e, table_name)), }; check_backup_arn_fits(region)?; - refuse_backup_of_vector_table(&desc)?; - // A create that writes to the configured store needs it healthy, and is - // refused before a slot is taken when it is not, as every operation that - // touches the store is; the in-catalog create touches no store, so the - // store's health does not gate it (a backend bound to the in-catalog - // create, or a server with an unhealthy store, creates as a server - // without a store does). - let store_backed = runtime.configured_store().is_some() - && !runtime.legacy_create_only() - && STORE_BACKED_CREATE; + let store_backed = runtime.configured_store().is_some() && !runtime.legacy_create_only(); if store_backed { - runtime.require_healthy_store()?; + check_store_prefix_fits(region, account_id, table_name)?; + let store = runtime.require_healthy_store()?; + let env = job::JobEnv::new(runtime, store)?; + refuse_while_maintenance_held(&**storage, runtime).await?; + let permit = runtime.take_permit().await?; + let (reply, answer) = tokio::sync::oneshot::channel(); + let request = job::CreateRequest { + account_id: account_id.to_owned(), + region: region.to_owned(), + table_name: table_name.to_owned(), + backup_name: backup_name.to_owned(), + limits: limits.into(), + }; + runtime.spawn_job(job::start(Arc::clone(storage), env, permit, request, reply)); + return answer.await.unwrap_or_else(|_| { + tracing::error!( + "CreateBackup of {table_name} (account {account_id}): the job task ended \ + without answering" + ); + Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )) + }); } - refuse_while_maintenance_held(storage, runtime).await?; + refuse_backup_of_vector_table(&desc)?; + refuse_while_maintenance_held(&**storage, runtime).await?; let permit = runtime.take_permit().await?; match runtime.configured_store() { None => tracing::warn!( @@ -486,8 +532,8 @@ async fn create_backup( ), Some(store) => tracing::info!( "CreateBackup of {table_name} (account {account_id}) is writing the in-catalog \ - backup format; the configured {} backup store serves restores and deletes of the \ - backups it holds, and a later release's CreateBackup writes to it", + backup format: this backend is bound to it; the configured {} backup store serves \ + restores and deletes of the backups it holds", store.kind ), } @@ -498,19 +544,21 @@ async fn create_backup( } /// The jobs' side of the catalog-level maintenance claim: a backup or -/// restore job is refused with `LimitExceededException` while a maintenance -/// command (`extenddb backup purge`) holds the claim, read with -/// `stale_job_after_secs` so a dead command's claim expires. The check runs -/// after every other refusal and before the permit, which on this head is -/// before the job's row exists: the in-catalog create and restore commit -/// their row inside the backend and detach, so the engine has no point -/// between the row and the copy to check at. A claim taken between this -/// check and the row's insert is therefore not seen by the job; the command -/// create that slips through writes no object of the store the command -/// deletes from, and a restore that slips through is a RUNNING record the -/// command's wait lists, so the window costs nothing in this release. The -/// store-backed job of a later release owns its row from the -/// engine and checks after the row exists, as the design's 5.1 step 2 says. +/// restore job is refused with `LimitExceededException` +/// ([`maintenance_held`]) while a maintenance command (`extenddb backup +/// purge`) holds the claim, read with `stale_job_after_secs` so a dead +/// command's claim expires. The check runs after every other refusal and +/// before the permit, which is before the job's row exists: the in-catalog +/// create and restore commit their row inside the backend and detach, so the +/// engine has no point between the row and the copy to check at, and the +/// store-backed create checks here too rather than after its insert, since +/// a row written FAILED for the refusal carried the insert's heartbeat and +/// `backup purge`'s wait counted it as a live job for the stale window. A +/// claim taken between this check and the row's insert is therefore not +/// seen by the job; the in-catalog create that slips through writes no +/// object of the store the command deletes from, and a restore or a +/// store-backed create that slips through is a RUNNING record or a CREATING +/// row the command's wait lists, so the window costs nothing. async fn refuse_while_maintenance_held( storage: &S, runtime: &BackupRuntime, @@ -520,21 +568,19 @@ async fn refuse_while_maintenance_held( .await .map_err(storage_err_to_dynamo)?; if held { - return Err(DynamoDbError::LimitExceededException( - "A backup maintenance command holds this catalog's maintenance claim; backup and \ - restore jobs are refused until it finishes" - .to_owned(), - )); + return Err(maintenance_held()); } Ok(()) } -/// Whether CreateBackup writes to the configured store. No store-backed -/// create exists on this head: every create writes the in-catalog format, -/// whatever the store configuration, and the release that adds the -/// store-backed job turns this on for a server with a store configured and -/// not bound to the in-catalog create. -const STORE_BACKED_CREATE: bool = false; +/// The refusal every job answers while the maintenance claim is held. +fn maintenance_held() -> DynamoDbError { + DynamoDbError::LimitExceededException( + "A backup maintenance command holds this catalog's maintenance claim; backup and \ + restore jobs are refused until it finishes" + .to_owned(), + ) +} /// The one backup ARN rule CreateBackup applies on this path: the server's /// own region (`server.region` outside `[a-z0-9-]` forms no backup ARN @@ -557,6 +603,39 @@ fn check_backup_arn_fits(region: &str) -> Result<(), DynamoDbError> { Ok(()) } +/// The store-backed create's name rules, applied before the job's row exists: +/// the table name has to fit a store prefix +/// (`extenddb_core::backup_format::check_store_prefix_table_name`: over 255 +/// bytes, or made only of dots, fits none) and the backup ARN has to form +/// (checked with a placeholder id, so the message names the limit whatever +/// the length). The in-catalog create applies neither: it writes no prefix, +/// and the service backs up any table it can create. +fn check_store_prefix_fits( + region: &str, + account_id: &str, + table_name: &str, +) -> Result<(), DynamoDbError> { + check_store_prefix_table_name(table_name).map_err(|e| { + DynamoDbError::ValidationException(format!( + "TableName cannot be backed up: a backup ARN cannot be formed for it: {e}" + )) + })?; + let placeholder = extenddb_core::backup_format::format_backup_id(0, 0); + BackupArn::new( + extenddb_storage::util::ARN_PARTITION, + region, + account_id, + table_name, + &placeholder, + ) + .map(drop) + .map_err(|e| { + DynamoDbError::ValidationException(format!( + "TableName cannot be backed up: a backup ARN cannot be formed for it: {e}" + )) + }) +} + /// [`storage_err_to_dynamo`] for CreateBackup, where a table that does not /// exist or is not yet `ACTIVE` is `TableNotFoundException` with the /// service's text (`Table not found: `, measured on DynamoDB for both @@ -571,7 +650,8 @@ fn create_backup_storage_err_to_dynamo(e: StorageError, table_name: &str) -> Dyn } } -/// The CreateBackup refusal for a table with vector indexes. +/// The in-catalog create's refusal for a table with vector indexes (the +/// in-catalog format cannot restore one); the store-backed path has none. fn refuse_backup_of_vector_table(desc: &TableDescription) -> Result<(), DynamoDbError> { let count = desc.vector_indexes.as_ref().map_or(0, Vec::len); if count == 0 { @@ -1451,8 +1531,8 @@ where /// index's row first ([`check_catalog_row_status`]: a CREATING row is /// `BackupInUseException`, a DELETING, DELETED or FAILED row /// `BackupNotFoundException`), else the legacy reader's restore source, a -/// missing backup being `BackupNotFoundException` whatever the target name's -/// state; (5) check the target name under the account through +/// missing backup being `BackupNotFoundException` whatever the target +/// name's state; (5) check the target name under the account through /// `describe_table` ([`check_restore_target_name`]: a table of that name is /// `TableAlreadyExistsException`, a CREATING restore target /// `TableInUseException` naming its source backup), after the backup is @@ -1461,14 +1541,14 @@ where /// of a missing backup into an existing table, into a table CreateTable is /// still creating and into a CREATING restore target all answer /// `BackupNotFoundException`; an in-use backup into an existing table -/// answers `TableAlreadyExistsException`); then, for a row, the rest of its -/// checks ([`check_catalog_row_for_restore`]: a backup in use is +/// answers `TableAlreadyExistsException`, and a restore of a backup into +/// its own CREATING target `TableInUseException`); then, for a row, the +/// rest of its checks ([`check_catalog_row_for_restore`]: a backup in use is /// `BackupInUseException`, a row of another store a `ValidationException`, /// a row of the configured store while it is unhealthy one naming the /// store's kind, and an AVAILABLE row of the healthy store the /// `ValidationException` naming the store, since this release restores -/// in-catalog backups only); -/// (6) refuse what no +/// in-catalog backups only); (6) refuse what no /// backend restores faithfully (multi-part key, vector indexes); (7) check /// the throughput override against the backup's billing mode, build the /// target's `CreateTableInput`, and apply the index and on-demand throughput @@ -1516,7 +1596,8 @@ async fn restore_table_from_backup DynamoDbError { } } +mod job; pub mod maintenance; mod overrides; mod pitr; diff --git a/crates/engine/src/backup/job.rs b/crates/engine/src/backup/job.rs new file mode 100644 index 000000000..c0084d5a7 --- /dev/null +++ b/crates/engine/src/backup/job.rs @@ -0,0 +1,664 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The store-backed CreateBackup job: the detached task the request spawns +//! once its checks have passed and the permit is held. +//! +//! The task ([`start`]) opens the table snapshot, mints the backup id from +//! the millisecond the open returned, reads the table's TTL, builds the +//! manifest's `source_table` from the snapshot's description +//! ([`source_table_schema`]), inserts the CREATING row under the job's +//! claim (`claim_backup`), answers the request through a one-shot channel +//! (`BackupDetails` CREATING, or the refusal) and then goes on as the job +//! ([`run`]) with the same owned values: nothing crosses a thread or a +//! channel between the request and the job but the response (the +//! pipeline's item and chunk channels are the job's own), and a client that +//! disconnects mid-request cannot leave a row whose owner was never +//! spawned. The maintenance claim is checked by the request before the +//! permit, as the in-catalog paths check it, so a held claim refuses the +//! request and inserts no row (a FAILED row written for the refusal was +//! counted by `backup purge`'s wait as a live job for the stale window). +//! The open and the insert are selected against the shutdown token, so a +//! server that is stopping neither waits on an open the table's lock holds +//! nor inserts a row it would fail at once. The `ServiceUnavailable` that +//! refusal answers reaches a client only when the token is cancelled while +//! the listener still serves, which is a registration loss (the restartable +//! worker set stopping calls `BackupRuntime::cancel_jobs`); on SIGTERM the +//! listener's 5 s connection drain closes the connection before +//! `drain_then_release` cancels the token (the maintainer measured a +//! `CreateBackup` waiting in the open: its connection closed 8.2 s after it +//! was sent), so there the refusal's effect is the row not written. +//! +//! The job: the data files through the pipeline ([`writer`]), then the +//! three documents `BackupDocuments::build` returns in its order, a re-check +//! of the claim, the manifest (the commit record), and the AVAILABLE write +//! with the size, the count and the manifest's `source_table` +//! ([`failure::write_status`] gives every status write one meaning; an +//! AVAILABLE write whose outcome is unknown is re-read, one that matches no +//! row ends the job with no store access). Before every put the job checks +//! the store as the runtime holds it: healthy, and its identity the one the +//! row records ([`writer::Pipeline::store_check`]); a store the reconciler +//! has marked unhealthy, or whose identity file now names another store (a +//! root swapped under a running job), fails the row with the store's reason +//! and makes no store call, the put or the prefix's deletion included (the +//! FAILED-prefix duty deletes the prefix once the store is healthy again). +//! The heartbeat runs on a timer for the job's life ([`heartbeat`]); a +//! heartbeat that does not confirm the claim, or does not return within one +//! period, ends the job with no further store or catalog access. Any other +//! failure, and the shutdown token (reason `shutdown`), runs the failure +//! path ([`failure::fail_row`]): FAILED, then the prefix's deletion only +//! when that write matched. +//! +//! Each job records `BackupJobs` and `BackupJobDuration` (microseconds, +//! from the snapshot's open to the end of its last catalog or store call: +//! the AVAILABLE write, or the FAILED write and the prefix's deletion) +//! under its outcome, `BackupBytesUploaded` once it ended AVAILABLE, +//! `BackupSnapshotAge` at each heartbeat, and the worker metrics under +//! `QuerySource::BackupJob` (`WorkerLastSuccess` on success, +//! `WorkerErrorCount` on a failure; a job the shutdown cancelled is neither); +//! one info line at start (ARN, the table's item count as the stats pass +//! last counted it, the order) and at the end (ARN, the items written, +//! bytes, duration), one error line per failure. +//! +//! Kill points ([`kill_point`]) pause the job for the binary-level crash +//! tests under the `backup-test-hooks` feature and are no-ops without it. + +pub(crate) mod failure; +pub(crate) mod heartbeat; +pub(crate) mod writer; + +use std::sync::Arc; + +use extenddb_backup_store::{BackupStore, byte_stream_from}; +use extenddb_core::backup_format::{ + BackupArn, BackupDocuments, ItemOrder, KeyOrder, KeyRangeTracker, ManifestHeader, + SourceTableSchema, backup_creation_date_time, +}; +use extenddb_core::error::DynamoDbError; +use extenddb_core::metrics::{BackupJobOutcome, MetricsCollector, QuerySource}; +use extenddb_core::types::BackupDetails; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits}; +use extenddb_storage::backup_catalog::{ + BACKUP_TYPE_USER, BackupRow, BackupStatus, JobStatus, StatusFields, StoreIdentity, + UNOWNED_GENERATION, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::snapshot::source_table_schema; +use extenddb_storage::{ + BackupCatalog, CancellationToken, JobClaim, JobPermit, MetadataEngine, SnapshotEngine, + TableSnapshot, +}; +use tokio::sync::oneshot; +use tokio::time::Instant; + +use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings}; +use super::{create_backup_storage_err_to_dynamo, storage_err_to_dynamo}; +use failure::{FailurePath, Written}; +use heartbeat::HeartbeatHandle; +use writer::{DataFiles, Pipeline, Stop}; + +/// The reason a row carries when the server's shutdown stopped the job. +pub(crate) const SHUTDOWN: &str = "shutdown"; + +/// What the runtime hands a job: the store as the runtime holds it (its +/// health and identity, read before every put), its identity as the row +/// records it and its manifest names, the settings, the collector, the +/// shutdown token and the source names. +#[derive(Clone)] +pub(crate) struct JobEnv { + pub configured: Arc, + pub store: Arc, + pub identity: StoreIdentity, + pub bucket: String, + pub key_prefix: String, + pub settings: JobSettings, + pub metrics: Option>, + pub shutdown: CancellationToken, + pub source_backend: String, + pub source_extenddb_version: String, +} + +impl JobEnv { + /// From the runtime and its configured store, whose identity must be + /// known (it is once a check has passed, which `require_healthy_store` + /// has established). + pub(crate) fn new( + runtime: &BackupRuntime, + store: &Arc, + ) -> Result { + let identity = store.identity().ok_or_else(|| { + tracing::error!("the configured store is healthy but its id is unknown"); + DynamoDbError::InternalServerError("Internal server error".to_owned()) + })?; + let (source_backend, source_extenddb_version) = runtime.source(); + Ok(Self { + configured: Arc::clone(store), + store: Arc::clone(&store.store), + identity, + bucket: store.bucket.clone(), + key_prefix: store.key_prefix.clone(), + settings: runtime.job_settings(), + metrics: runtime.metrics().cloned(), + shutdown: runtime.shutdown(), + source_backend: source_backend.to_owned(), + source_extenddb_version: source_extenddb_version.to_owned(), + }) + } +} + +/// The request's members the task needs. +pub(crate) struct CreateRequest { + pub account_id: String, + pub region: String, + pub table_name: String, + pub backup_name: String, + /// The instance's limits the validator holds every item to. + pub limits: RestorableLimits, +} + +/// The one-shot the task answers the request through. +pub(crate) type Reply = oneshot::Sender>; + +/// The detached task (module doc): everything from the open to the +/// response, then the job. +pub(crate) async fn start( + storage: Arc, + env: JobEnv, + permit: JobPermit, + request: CreateRequest, + reply: Reply, +) where + S: BackupCatalog + SnapshotEngine + MetadataEngine + ?Sized + Send + Sync + 'static, +{ + let job = match open_and_claim(&*storage, &env, &request).await { + Ok(job) => { + let details = job.details(); + // A request whose client went away still has its job. + let _ = reply.send(Ok(details)); + *job + } + Err(e) => { + drop(permit); + let _ = reply.send(Err(e)); + return; + } + }; + run(job).await; + drop(permit); +} + +/// The job once its row exists. +pub(crate) struct Job { + claim: Arc, + snapshot: TableSnapshot, + arn: BackupArn, + backup_name: String, + source_table: SourceTableSchema, + key_schema: RestorableKeySchema, + /// The table's key order when the stream is key-ordered. + key_order: Option, + limits: RestorableLimits, + env: JobEnv, + /// The id's instant, the row's `created_at`. + created_at: f64, + started: Instant, + /// The instant taken before the claim's insert, the heartbeat's first + /// confirmed beat. + claimed_at: Instant, +} + +impl Job { + /// The response the request answers: the row as inserted, CREATING. + fn details(&self) -> BackupDetails { + BackupDetails { + backup_arn: self.arn.to_arn_string(), + backup_name: self.backup_name.clone(), + backup_status: BackupStatus::Creating.as_str().to_owned(), + backup_type: BACKUP_TYPE_USER.to_owned(), + backup_size_bytes: 0, + backup_creation_date_time: self.created_at, + } + } +} + +/// The open, the id, the TTL read, the row and the claim (module doc). The +/// open and the insert are selected against the shutdown token. +async fn open_and_claim( + storage: &S, + env: &JobEnv, + request: &CreateRequest, +) -> Result, DynamoDbError> +where + S: BackupCatalog + SnapshotEngine + MetadataEngine + ?Sized, +{ + let CreateRequest { + account_id, + region, + table_name, + backup_name, + limits, + } = request; + let started = Instant::now(); + // The open and the claim's insert are selected against the shutdown + // token: an open that waits on a lock the table holds (an `ACCESS + // EXCLUSIVE` on PostgreSQL) does not hold the drain for its whole bound, + // and no row is inserted for a server that is stopping. The answer + // reaches a client only after a registration loss (module doc): on + // SIGTERM the listener's drain has closed the connection by the time the + // token is cancelled. + let snapshot = tokio::select! { + biased; + () = env.shutdown.cancelled() => return Err(shutting_down(table_name)), + opened = storage.open_table_snapshot(account_id, table_name) => { + opened.map_err(|e| create_backup_storage_err_to_dynamo(e, table_name))? + } + }; + // The id carries the millisecond the open returned. + let backup_id = extenddb_storage::util::new_backup_id(); + let created_at = backup_creation_date_time(&backup_id) + .ok_or_else(|| internal("the backup id carries no instant"))?; + let arn = BackupArn::new( + extenddb_storage::util::ARN_PARTITION, + region, + account_id, + table_name, + &backup_id, + ) + .map_err(|e| internal(&format!("no backup ARN for {table_name}: {e}")))?; + let ttl = storage + .describe_ttl(account_id, table_name) + .await + .map_err(storage_err_to_dynamo)?; + let source_table = + source_table_schema(&snapshot.description, Some(ttl)).map_err(storage_err_to_dynamo)?; + let key_schema = RestorableKeySchema::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .map_err(|e| { + internal(&format!( + "the key schema of {table_name} cannot be checked: {e}" + )) + })?; + let key_order = snapshot + .key_ordered + .then(|| { + KeyOrder::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .ok() + }) + .flatten(); + let row = BackupRow { + backup_id: backup_id.clone(), + account_id: account_id.clone(), + table_name: table_name.clone(), + backup_arn: arn.to_arn_string(), + backup_name: backup_name.clone(), + status: BackupStatus::Creating, + store: env.identity.clone(), + size_bytes: 0, + item_count: 0, + created_at, + completed_at: None, + failure_reason: None, + source_table_details_json: serde_json::to_value(&source_table) + .map_err(|e| internal(&format!("the source table does not serialize: {e}")))?, + owner_generation: UNOWNED_GENERATION, + heartbeat_at: 0.0, + }; + let claimed_at = Instant::now(); + // The insert too is selected against the token: a server that is + // stopping inserts no row it would have to fail at once. + let claimed = tokio::select! { + biased; + () = env.shutdown.cancelled() => return Err(shutting_down(table_name)), + claimed = storage.claim_backup(&row) => claimed, + }; + let claim: Arc = match claimed { + Ok(claim) => Arc::from(claim), + Err(StorageError::AlreadyExists(what)) => { + return Err(internal(&format!( + "backup id {backup_id} is already claimed: {what}" + ))); + } + Err(e) => return Err(storage_err_to_dynamo(e)), + }; + Ok(Box::new(Job { + claim, + snapshot, + arn, + backup_name: backup_name.clone(), + source_table, + key_schema, + key_order, + limits: *limits, + env: env.clone(), + created_at, + started, + claimed_at, + })) +} + +fn internal(why: &str) -> DynamoDbError { + tracing::error!("CreateBackup: {why}"); + DynamoDbError::InternalServerError("Internal server error".to_owned()) +} + +/// The refusal of a request whose server is stopping before its row exists +/// (module doc); the client sees it only when the token was cancelled while +/// the listener still served, after a registration loss. +fn shutting_down(table_name: &str) -> DynamoDbError { + tracing::info!("CreateBackup of {table_name}: refused, the server is shutting down"); + DynamoDbError::ServiceUnavailable("The server is shutting down. Retry the request.".to_owned()) +} + +/// The FAILED write of a row whose job makes no store call: nothing of the +/// job's is in the store, or the store is not to be called, so there is +/// nothing to delete here. +async fn fail_without_store_access(claim: &dyn JobClaim, arn: &BackupArn, reason: &str) { + let fields = StatusFields { + failure_reason: Some(reason.to_owned()), + completed: true, + ..StatusFields::default() + }; + match failure::write_status(claim, JobStatus::Failed, &fields).await { + Written::Matched => {} + Written::NotMatched(why) | Written::Error(why) => tracing::error!( + "backup {}: the row could not be marked FAILED ({reason}): {why}", + arn.to_arn_string() + ), + } +} + +/// The job (module doc), from the data files to the final status write. +pub(crate) async fn run(job: Job) -> BackupJobOutcome { + let Job { + claim, + snapshot, + arn, + backup_name, + mut source_table, + key_schema, + key_order, + limits, + env, + created_at, + started, + claimed_at, + } = job; + let arn_text = arn.to_arn_string(); + let prefix = arn.store_prefix(); + let metrics = env.metrics.as_deref(); + let heartbeat = HeartbeatHandle::start( + Arc::clone(&claim), + env.settings.heartbeat_interval, + env.metrics.clone(), + created_at, + claimed_at, + ); + tracing::info!( + "backup {arn_text}: started, {} items in the table as last counted by the stats pass, {}", + snapshot.description.item_count, + if key_order.is_some() { + "key-ordered" + } else { + "unordered" + } + ); + let TableSnapshot { info, items, .. } = snapshot; + let pipeline = Pipeline { + store: &*env.store, + configured: &env.configured, + identity: &env.identity, + prefix: &prefix, + arn: &arn_text, + heartbeat: &heartbeat, + shutdown: &env.shutdown, + metrics, + target_bytes: env.settings.data_file_target_bytes, + key_prefix: &env.key_prefix, + bucket: &env.bucket, + }; + let tracker = key_order.as_ref().map(KeyRangeTracker::new); + let mut item_count = 0; + let end = match writer::write_data_files(&pipeline, items, key_schema, limits, tracker).await { + Ok(files) => { + item_count = files.item_count; + // Both figures are the stream's: the items written and their + // DynamoDB sizes summed, not the statistics pass's (which can + // be stale, or 0 on a fresh table). + source_table.item_count = i64::try_from(files.item_count).unwrap_or(i64::MAX); + source_table.table_size_bytes = i64::try_from(files.item_bytes).unwrap_or(i64::MAX); + let header = ManifestHeader { + backup_arn: arn.clone(), + backup_name, + source_backend: env.source_backend.clone(), + source_extenddb_version: env.source_extenddb_version.clone(), + snapshot: info, + item_order: if key_order.is_some() { + ItemOrder::KeyOrdered + } else { + ItemOrder::Unordered + }, + source_table, + }; + write_documents(&pipeline, &claim, header, files).await + } + Err(stop) => Err(stop), + }; + let (outcome, bytes) = match end { + Ok(bytes) => { + tracing::info!( + "backup {arn_text}: AVAILABLE, {item_count} items, {bytes} bytes written in {:.1} s", + started.elapsed().as_secs_f64() + ); + (BackupJobOutcome::Succeeded, bytes) + } + Err(Stop::ClaimLost(why)) => { + tracing::error!( + "backup {arn_text}: stopped with no further store or catalog access: {why}; the \ + row is left to its owner or to the reconciler" + ); + (BackupJobOutcome::Failed, 0) + } + Err(Stop::StoreUnhealthy(reason)) => { + // The store is not to be called: FAILED with the store's reason, + // the prefix left to the FAILED-prefix duty once the store is + // healthy again. + fail_without_store_access(&*claim, &arn, &reason).await; + tracing::error!( + "backup {arn_text}: FAILED ({reason}); no store call was made and its objects \ + are deleted by the reconciler once the store's check passes and the row's \ + heartbeat is stale" + ); + (BackupJobOutcome::Failed, 0) + } + Err(stop) => { + let (reason, outcome) = match stop { + Stop::Cancelled => (SHUTDOWN.to_owned(), BackupJobOutcome::Cancelled), + Stop::Fail(reason) => (reason, BackupJobOutcome::Failed), + Stop::ClaimLost(_) | Stop::StoreUnhealthy(_) => unreachable!("matched above"), + }; + match failure::fail_row(&*claim, &*env.store, &prefix, &reason, metrics).await { + FailurePath::Deleted => tracing::error!( + "backup {arn_text}: FAILED ({reason}); its objects were deleted" + ), + FailurePath::DeleteFailed(e) => tracing::error!( + "backup {arn_text}: FAILED ({reason}); its objects could not be deleted and \ + the reconciler deletes them once the row's heartbeat is stale: {e}" + ), + FailurePath::NoStoreAccess(why) => tracing::error!( + "backup {arn_text}: failed ({reason}) and the row is not this job's to \ + write ({why}); no store access was made" + ), + } + (outcome, 0) + } + }; + // After the failure path too, so a FAILED job's duration covers its + // FAILED write and the prefix's deletion. + let duration = started.elapsed(); + drop(heartbeat); + if let Some(metrics) = metrics { + #[allow(clippy::cast_precision_loss)] + let duration_us = duration.as_micros() as f64; + metrics.record_backup_job(outcome, duration_us); + if bytes > 0 { + metrics.record_backup_bytes_uploaded(bytes); + } + match outcome { + BackupJobOutcome::Succeeded => { + metrics.record_worker_success(QuerySource::BackupJob, duration_us); + } + BackupJobOutcome::Failed => { + metrics.record_worker_error(QuerySource::BackupJob); + } + // A clean shutdown is not a worker error. + BackupJobOutcome::Cancelled => {} + } + } + outcome +} + +/// After the data files: the three documents, the claim re-check, the +/// manifest, the AVAILABLE write. Returns the bytes written over the whole +/// backup. +async fn write_documents( + p: &Pipeline<'_>, + claim: &Arc, + header: ManifestHeader, + files: DataFiles, +) -> Result { + let DataFiles { + entries, + item_count, + item_bytes: _, + mut bytes_uploaded, + } = files; + let docs = BackupDocuments::build(header, entries, p.key_prefix, p.bucket) + .map_err(|e| Stop::Fail(format!("building the manifest: {e}")))?; + for (name, bytes) in docs.documents_before_manifest() { + bytes_uploaded += put_document(p, name, bytes).await?; + } + // The re-check narrows the window in which a fenced-out job can still + // write a manifest; the bound is the backend's (`heartbeat_within`). + p.heartbeat + .recheck() + .await + .map_err(|why| Stop::ClaimLost(format!("the claim re-check before the manifest: {why}")))?; + kill_point(KillPoint::BeforeManifest, p.shutdown).await; + let (name, bytes) = docs.manifest_document(); + bytes_uploaded += put_document(p, name, bytes).await?; + kill_point(KillPoint::AfterManifest, p.shutdown).await; + let manifest = docs.manifest(); + let fields = StatusFields { + size_bytes: Some(manifest.backup_size_bytes), + item_count: Some(i64::try_from(item_count).unwrap_or(i64::MAX)), + failure_reason: None, + source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), + completed: true, + }; + match failure::write_status(&**claim, JobStatus::Available, &fields).await { + Written::Matched => Ok(bytes_uploaded), + Written::NotMatched(why) | Written::Error(why) => Err(Stop::ClaimLost(format!( + "the AVAILABLE write did not settle: {why}" + ))), + } +} + +/// One document's put under the backup prefix, after the pre-put check; +/// returns the bytes the store reported. +pub(crate) async fn put_document(p: &Pipeline<'_>, name: &str, bytes: &[u8]) -> Result { + if p.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + p.store_check(name)?; + if !p.heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost(format!( + "no heartbeat confirmed the claim recently enough for {name}" + ))); + } + let key = format!("{}{name}", p.prefix); + tokio::select! { + biased; + () = p.shutdown.cancelled() => Err(Stop::Cancelled), + () = p.heartbeat.lost().cancelled() => Err(Stop::ClaimLost(p.heartbeat.reason())), + result = p.store.put(&key, byte_stream_from(bytes.to_vec())) => match result { + Ok(meta) => Ok(meta.size), + Err(e) => { + if let Some(metrics) = p.metrics { + metrics.record_backup_store_error("put"); + } + Err(Stop::Fail(format!("writing {name}: {e}"))) + } + }, + } +} + +/// The points at which a job pauses under `backup-test-hooks` when +/// `EXTENDDB_BACKUP_PAUSE` names one, until the file +/// `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists (or for good, for a test +/// that kills the server there). Each is logged as `backup job reached kill +/// point `. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum KillPoint { + /// After the put of the first data file returned, before the next file + /// or the documents. + AfterDataFile, + /// After the three documents, before `extenddb-manifest.json`. + BeforeManifest, + /// After the manifest's put returned, before the AVAILABLE write. + AfterManifest, +} + +impl KillPoint { + /// The value of `EXTENDDB_BACKUP_PAUSE` that names this point. + #[must_use] + pub fn name(self) -> &'static str { + match self { + Self::AfterDataFile => "after_data_file", + Self::BeforeManifest => "before_manifest", + Self::AfterManifest => "after_manifest", + } + } +} + +/// Pause at `point` if the environment names it (type doc); a ready future +/// without `backup-test-hooks`. +#[cfg_attr(not(feature = "backup-test-hooks"), allow(clippy::unused_async))] +pub(crate) async fn kill_point(point: KillPoint, shutdown: &CancellationToken) { + #[cfg(feature = "backup-test-hooks")] + { + if std::env::var("EXTENDDB_BACKUP_PAUSE").ok().as_deref() != Some(point.name()) { + return; + } + let release = std::env::var("EXTENDDB_BACKUP_PAUSE_RELEASE").ok(); + tracing::warn!( + "backup job reached kill point {}; paused until {} exists", + point.name(), + release + .as_deref() + .unwrap_or("no release file (the pause does not end)") + ); + loop { + if let Some(release) = &release + && std::path::Path::new(release).exists() + { + tracing::warn!("backup job released from kill point {}", point.name()); + return; + } + if shutdown.is_cancelled() { + tracing::warn!( + "backup job at kill point {} released by the shutdown", + point.name() + ); + return; + } + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + } + } + #[cfg(not(feature = "backup-test-hooks"))] + let _ = (point, shutdown); +} diff --git a/crates/engine/src/backup/job/failure.rs b/crates/engine/src/backup/job/failure.rs new file mode 100644 index 000000000..398ab7880 --- /dev/null +++ b/crates/engine/src/backup/job/failure.rs @@ -0,0 +1,145 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The conditional status writes of a backup row and the job's failure +//! path. +//! +//! Every status write goes through [`write_status`]: the backend writes +//! conditioned on the claim's generation and on the row being CREATING, and +//! this module gives the three outcomes one meaning. `Matched` is the +//! write. `NotMatched` means the row is no longer this claim's (gone, +//! another generation, or not CREATING) and the caller makes no further +//! store or table access of any kind. `Unknown` (the connection failed +//! after the statement was sent) is followed by a re-read of the row, not +//! treated as a failure: a row that shows the write landed is `Matched`; a +//! row still CREATING under this generation did not take the write, which +//! is sent once more, at most; anything else is `NotMatched`. Each unknown +//! outcome has its re-read, the second write's included, so a second +//! unknown outcome whose write landed is `Matched` (the row is in the +//! target status under this generation) and one whose write was lost is +//! `NotMatched`. An error on the claim connection, on the write or the +//! re-read, ends the caller's work the way `NotMatched` does. +//! +//! The second write is for a backend whose claim connection survives the +//! unknown outcome (MongoDB and Cassandra, whose claim writes go through +//! the shared client). On PostgreSQL an unknown outcome is an error after +//! which the claim session's one connection is dead, so the second write +//! fails with the session and its re-read settles it as `NotMatched`: the +//! job stops, the session's end frees the advisory lock, and the row is the +//! reconciler's (AVAILABLE from a complete manifest, FAILED otherwise). +//! +//! The failure path ([`fail_row`]) is the FAILED write followed, only when +//! it matched, by `delete_prefix` of the backup's prefix; the holder of the +//! claim owns the prefix, so a job whose FAILED write did not match never +//! deletes. A deletion that fails leaves the prefix to the reconciler, +//! which deletes a FAILED row's prefix once its heartbeat is stale. + +use extenddb_backup_store::BackupStore; +use extenddb_core::metrics::MetricsCollector; +use extenddb_storage::JobClaim; +use extenddb_storage::backup_catalog::{JobStatus, StatusFields, WriteOutcome}; + +/// What a conditional status write settled to (module doc). +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Written { + /// The row was moved. + Matched, + /// The row is not this claim's to write; the text says what the row + /// showed. + NotMatched(String), + /// The claim connection failed on the write or the re-read. + Error(String), +} + +/// Move the row from CREATING to `to` through `claim` (module doc). +pub(crate) async fn write_status( + claim: &dyn JobClaim, + to: JobStatus, + fields: &StatusFields, +) -> Written { + for attempt in 0..2 { + let outcome = match claim.set_status(JobStatus::Creating, to, fields).await { + Ok(outcome) => outcome, + Err(e) => return Written::Error(format!("the {to} write failed: {e}")), + }; + match outcome { + WriteOutcome::Matched => return Written::Matched, + WriteOutcome::NotMatched => { + return Written::NotMatched( + "the row is no longer CREATING under this claim's generation".to_owned(), + ); + } + WriteOutcome::Unknown => {} + } + let row = match claim.read().await { + Ok(row) => row, + Err(e) => { + return Written::Error(format!( + "the {to} write's outcome is unknown and the row could not be re-read: {e}" + )); + } + }; + if row.owner_generation != claim.generation() { + return Written::NotMatched(format!( + "the row carries generation {} (this claim's is {})", + row.owner_generation, + claim.generation() + )); + } + if row.status == to { + return Written::Matched; + } + if row.status != JobStatus::Creating || attempt == 1 { + return Written::NotMatched(format!( + "the row is {} after a {to} write whose outcome was unknown", + row.status + )); + } + tracing::warn!( + "the {to} write's outcome was unknown and the row is still CREATING under this \ + claim; writing once more" + ); + } + unreachable!("the loop returns on its second attempt") +} + +/// The end of the failure path. +#[derive(Debug)] +pub(crate) enum FailurePath { + /// FAILED written and the prefix deleted. + Deleted, + /// FAILED written; the deletion failed and is the reconciler's. + DeleteFailed(String), + /// FAILED not written by this claim (the row is another's, or the claim + /// connection failed): no store access was made. + NoStoreAccess(String), +} + +/// The failure path (module doc): FAILED with `reason`, then the deletion +/// of `prefix` when the write matched. +pub(crate) async fn fail_row( + claim: &dyn JobClaim, + store: &dyn BackupStore, + prefix: &str, + reason: &str, + metrics: Option<&MetricsCollector>, +) -> FailurePath { + let fields = StatusFields { + failure_reason: Some(reason.to_owned()), + completed: true, + ..StatusFields::default() + }; + match write_status(claim, JobStatus::Failed, &fields).await { + Written::Matched => {} + Written::NotMatched(why) | Written::Error(why) => return FailurePath::NoStoreAccess(why), + } + match store.delete_prefix(prefix).await { + Ok(_) => FailurePath::Deleted, + Err(e) => { + if let Some(metrics) = metrics { + metrics.record_backup_store_error("delete_prefix"); + } + FailurePath::DeleteFailed(e.to_string()) + } + } +} diff --git a/crates/engine/src/backup/job/heartbeat.rs b/crates/engine/src/backup/job/heartbeat.rs new file mode 100644 index 000000000..6395f707d --- /dev/null +++ b/crates/engine/src/backup/job/heartbeat.rs @@ -0,0 +1,271 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The job's heartbeat: a timer task that refreshes the claim every period +//! for as long as the job runs, independent of upload progress, and the +//! two things the job reads from it: the instant of the last confirmed beat +//! (checked before every put) and the signal that the claim is lost. +//! +//! One rule decides both: a beat that does not confirm the claim +//! ([`Heartbeat::Lost`] or an error) or that has not returned within one +//! period ends the timer and raises the lost signal, after which the job +//! makes no further store or catalog access. The pre-put check is the same +//! rule read the other way: the last confirmed beat is at most two periods +//! old (one for the timer to fire, one for the beat to return, the bound +//! the timer itself enforces), or a beat is in flight (the last paragraph), +//! and no loss has been raised. The claim's +//! insert stands for the first confirmed beat: the check measures from the +//! instant the job took before `claim_backup`, so the maintenance check +//! that runs between the insert and the handle's start (one catalog query +//! with no bound of its own) counts against the two periods, and a job +//! whose first put comes within two periods of the insert needs no beat of +//! its own; `stale_job_after_secs` is at least three periods, so a row that +//! passes the check is never one a reconciler may take over. +//! +//! A process stopped and resumed (SIGSTOP, a paused VM) meets the same +//! rule. On PostgreSQL its claim session keeps the advisory lock while it +//! is stopped, so no takeover happens while the session lives. On resume a +//! job with no beat in flight at the stop runs on under its own generation +//! once its next beat succeeds, however long the stop was (a job frozen +//! 160 s finished AVAILABLE: the timer's next tick beats, the beat confirms, +//! and the puts that follow find a fresh confirmation); it stops as a lost +//! claim with no catalog write only when a beat in flight at the stop +//! outlives its one-period bound, or when a put comes before the next beat +//! and finds the last confirmed beat more than two periods old. In that +//! case its claim closes, the lock is freed and the reconciler resolves the +//! row (AVAILABLE from a complete manifest, FAILED otherwise). On MongoDB +//! and Cassandra the row may have been taken over during the stop: a put in +//! flight at the stop may still land (the reconciler's FAILED-prefix retry +//! deletes it), and the job's next beat and status write match no row. +//! +//! The one-period bound is the backend's to place +//! ([`JobClaim::heartbeat_within`]): on SQLite the beat is an `UPDATE` +//! under the engine write lock, and the bound starts once the beat holds +//! that lock, so a writer holding it past a period (a GSI backfill) delays +//! the beat and does not end the job. The re-check before the manifest +//! ([`HeartbeatHandle::recheck`]) is the same call, and a confirmation it +//! gets counts as a beat for the pre-put check. +//! +//! The pre-put check has a third clause for the same case: a beat that is +//! in flight and has not failed confirms the claim, however old the last +//! confirmed beat is. Without it a backup large enough to reach a put while +//! a writer holds the lock past two periods stopped at that put (`no +//! heartbeat confirmed the claim recently enough`), its beat parked on the +//! lock and the claim intact (the maintainer measured it twice at the third +//! or fourth data file of a 180,000-item table beside a GSI build). The +//! clause is safe because a beat that is in flight is one no backend has +//! answered: on SQLite the beat parks only on this process's own write +//! lock, one process serves the file (the serve lock; `backup purge` +//! refuses while a server runs on it) and the reconciler's liveness test is +//! the in-process slot the job holds (design 4(e)), so a parked beat cannot +//! coincide with a takeover, whatever `heartbeat_at` reads meanwhile; the +//! backend says so through [`JobClaim::heartbeat_parks_before_its_bound`], +//! and there the clause holds for as long as the beat waits. On PostgreSQL, +//! MongoDB and Cassandra no lock of this process sits in front of the beat +//! and `heartbeat_within` bounds the whole call, and there the clause holds +//! only within one period of the beat's start, the bound itself: a process +//! stopped mid-beat (SIGSTOP, a paused VM) and resumed keeps the beat in +//! flight until its timer runs again and raises the loss, a window of tens +//! of microseconds in which another owner may already hold the row, and a +//! clause that read the flight alone passed the checks in that window (the +//! maintainer measured 64 and 85 µs). A beat that returns Lost, fails or +//! outlives its bound raises the loss first, and the clause reads the loss +//! before the flight. A timer task that stops mid-beat (a panic in the +//! backend's beat) clears the flight on the way out, so the two-period +//! clause is what remains for it. + +use std::sync::{Arc, Mutex, PoisonError}; +use std::time::Duration; + +use extenddb_core::metrics::MetricsCollector; +use extenddb_storage::CancellationToken; +use extenddb_storage::JobClaim; +use extenddb_storage::backup_catalog::Heartbeat; +use extenddb_storage::error::StorageError; +use tokio::sync::watch; +use tokio::time::Instant; + +/// The job's side of the heartbeat timer. Dropping it stops the timer. +pub(crate) struct HeartbeatHandle { + claim: Arc, + confirmed: Arc>, + last_confirmed: watch::Receiver, + /// The instant the beat in flight (the timer's or the re-check's) + /// started, while one is; the pre-put check's third clause (module doc). + in_flight: Arc>>, + /// Whether the claim's beat can wait outside its bound + /// ([`JobClaim::heartbeat_parks_before_its_bound`]): the third clause + /// holds for as long as the beat is in flight where it can, and for one + /// period from the beat's start where it cannot. + parks: bool, + lost: CancellationToken, + reason: Arc>>, + period: Duration, + timer: tokio::task::JoinHandle<()>, +} + +/// Marks a beat in flight, with its start, for as long as it lives, and +/// clears the mark when dropped, by the beat's return or by its task's end. +struct InFlight<'a>(&'a Mutex>); + +impl<'a> InFlight<'a> { + fn mark(flag: &'a Mutex>) -> Self { + *flag.lock().unwrap_or_else(PoisonError::into_inner) = Some(Instant::now()); + Self(flag) + } +} + +impl Drop for InFlight<'_> { + fn drop(&mut self) { + *self.0.lock().unwrap_or_else(PoisonError::into_inner) = None; + } +} + +/// The third clause of the pre-put check (module doc): a beat in flight +/// since `since` confirms the claim for as long as it waits where the beat +/// parks before its bound, and within one `period` of its start otherwise. +fn in_flight_confirms(in_flight: &Mutex>, parks: bool, period: Duration) -> bool { + match *in_flight.lock().unwrap_or_else(PoisonError::into_inner) { + Some(since) => parks || since.elapsed() <= period, + None => false, + } +} + +/// Why a beat did not confirm the claim. +fn beat_failure(outcome: Option>, bound: Duration) -> String { + match outcome { + Some(Ok(Heartbeat::Confirmed)) => unreachable!("a confirmed beat is not a failure"), + Some(Ok(Heartbeat::Lost)) => { + "the claim was lost: the row is gone or carries another generation".to_owned() + } + Some(Err(e)) => format!("the heartbeat failed: {e}"), + None => format!("the heartbeat did not return within {} s", bound.as_secs()), + } +} + +impl HeartbeatHandle { + /// Start the timer on `claim`, beating every `period` from one period + /// after now; `claimed_at`, the instant of the claim's insert, stands + /// for the first confirmed beat. With `metrics`, each confirmed beat + /// records the snapshot's age (`BackupSnapshotAge`), seconds since + /// `created_at`. + pub(crate) fn start( + claim: Arc, + period: Duration, + metrics: Option>, + created_at: f64, + claimed_at: Instant, + ) -> Self { + let (tx, last_confirmed) = watch::channel(claimed_at); + let confirmed = Arc::new(tx); + let in_flight = Arc::new(Mutex::new(None)); + let parks = claim.heartbeat_parks_before_its_bound(); + let lost = CancellationToken::new(); + let reason = Arc::new(Mutex::new(None)); + let timer = tokio::spawn({ + let claim = Arc::clone(&claim); + let confirmed = Arc::clone(&confirmed); + let in_flight = Arc::clone(&in_flight); + let lost = lost.clone(); + let reason = Arc::clone(&reason); + async move { + let mut ticks = tokio::time::interval(period); + ticks.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay); + ticks.tick().await; + loop { + ticks.tick().await; + let outcome = { + let _beat = InFlight::mark(&in_flight); + claim.heartbeat_within(period).await + }; + if let Some(Ok(Heartbeat::Confirmed)) = outcome { + let _ = confirmed.send(Instant::now()); + if let Some(metrics) = &metrics { + metrics.record_backup_snapshot_age(seconds_since(created_at)); + } + continue; + } + *reason.lock().unwrap_or_else(PoisonError::into_inner) = + Some(beat_failure(outcome, period)); + lost.cancel(); + return; + } + } + }); + Self { + claim, + confirmed, + last_confirmed, + in_flight, + parks, + lost, + reason, + period, + timer, + } + } + + /// The signal that the claim is lost; select on `lost().cancelled()`. + pub(crate) fn lost(&self) -> &CancellationToken { + &self.lost + } + + /// Why the claim is lost, once it is. + pub(crate) fn reason(&self) -> String { + self.reason + .lock() + .unwrap_or_else(PoisonError::into_inner) + .clone() + .unwrap_or_else(|| "the claim is lost".to_owned()) + } + + /// The pre-put check (module doc): no loss raised, and either the last + /// confirmed beat at most two periods old or a beat in flight (parked on + /// SQLite's write lock behind a long writer; within one period of its + /// start on the other backends). The second clause is what catches a + /// timer task that stopped running without raising the loss; the third + /// never outlives its beat. + pub(crate) fn confirms_the_claim(&self) -> bool { + !self.lost.is_cancelled() + && (self.last_confirmed.borrow().elapsed() <= 2 * self.period + || in_flight_confirms(&self.in_flight, self.parks, self.period)) + } + + /// One beat outside the timer, under the period's bound (the claim + /// re-check before the manifest): a confirmation counts as a beat for + /// [`Self::confirms_the_claim`]; anything else is the reason the claim + /// is lost, with the timer's own text. + pub(crate) async fn recheck(&self) -> Result<(), String> { + let outcome = { + let _beat = InFlight::mark(&self.in_flight); + self.claim.heartbeat_within(self.period).await + }; + if let Some(Ok(Heartbeat::Confirmed)) = outcome { + let _ = self.confirmed.send(Instant::now()); + return Ok(()); + } + Err(beat_failure(outcome, self.period)) + } + + /// End the timer without raising the loss: a timer task that stopped + /// running, for the test of the pre-put check's second clause. + #[cfg(test)] + pub(crate) fn stop_timer(&self) { + self.timer.abort(); + } +} + +impl Drop for HeartbeatHandle { + fn drop(&mut self) { + self.timer.abort(); + } +} + +/// Seconds between `epoch_secs` and now, on the wall clock. +pub(crate) fn seconds_since(epoch_secs: f64) -> f64 { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs_f64()) + .unwrap_or_default(); + (now - epoch_secs).max(0.0) +} diff --git a/crates/engine/src/backup/job/writer.rs b/crates/engine/src/backup/job/writer.rs new file mode 100644 index 000000000..ba479a168 --- /dev/null +++ b/crates/engine/src/backup/job/writer.rs @@ -0,0 +1,742 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-file pipeline of a backup job: items from the snapshot stream, +//! through the validator, into gzip data files, into the store. +//! +//! Three parts run at once. The feeder (a task of its own) reads the +//! snapshot stream, checks every item with +//! [`validate_restorable_item`] and sends the ones that pass over a bounded +//! item channel ([`ITEM_CHANNEL`] items). One encoder per data file runs on +//! the blocking pool: it takes items off that channel, writes them through +//! `DataFileEncoder` into a [`ChunkSink`] that hands [`CHUNK_BYTES`]-sized +//! chunks over a bounded chunk channel ([`CHUNK_CHANNEL`] chunks), and +//! closes the file once its uncompressed bytes reach +//! `data_file_target_bytes`, so a file exceeds the knob by less than one +//! line. The uploader (the job's own task) starts the store `put` of each +//! file when the file's first chunk arrives, with the chunk channel as the +//! body, so a file is never buffered whole; a file that gets no item (the +//! stream ended or stopped at a file boundary) is never started. The +//! memory the pipeline holds is the item channel's items, the chunk +//! channel's chunks and the encoder's own state, none of which grows with +//! the table or the file size; the feeder's measurement of an oversized +//! item's line (below) holds no line. +//! +//! The validator's rule: after the first refused item the job encodes and +//! uploads nothing more but keeps reading and checking the rest of the +//! stream, logs each refused item (key, attribute path, reason, backup ARN) +//! up to [`REFUSAL_LOG_CAP`] and counts the rest, and then fails with a +//! reason that gives the count and the first item, naming #618 only when +//! the first refusal is an invalid `N` value. An item whose data file line +//! would exceed the format's cap (`MAX_LINE_BYTES`) is refused the same way +//! (`RestorableItemReason::LineTooLong`): the feeder measures the line of +//! every item over the default item limit (`DEFAULT_MAX_ITEM_SIZE_BYTES`), +//! the largest item the cap is sized for, on the blocking pool and through +//! a writer that counts and stops at the cap, so the measurement holds no +//! line and an instance with a raised `max_item_size_bytes` names the item +//! instead of failing in the encoder with no item named, and an instance at +//! the default measures none. Such an item's validation runs on the +//! blocking pool with the measurement (its cost grows with the item: the +//! number rule over every member); an item within the default is validated +//! on the feeder's own task. +//! +//! Before every `put` the uploader checks the heartbeat +//! ([`HeartbeatHandle::confirms_the_claim`]) and starts none otherwise; a +//! lost claim or the shutdown token ends the pipeline at its next await. +//! When the uploader fails on its own (a put refused, an encoder error) it +//! cancels a token the feeder selects on, so the feeder stops reading at +//! once rather than at the stream's next item and the FAILED write does not +//! wait on a slow or dead snapshot stream; the token is cancelled before +//! the uploader awaits the encoder, since an encoder waiting on the item +//! channel returns only once the feeder closes it; the feeder's own refusal +//! leaves that token alone, since the rest of the stream is still to be +//! checked. +//! When the pipeline returns, no `put` is in flight: a stop lets the +//! current put end (its body ends when the encoder drops the sink) or drops +//! it, and the store's own rule for a dropped put applies (the filesystem +//! store removes its temporary, the S3 store aborts the upload, and at +//! shutdown the failure path's `delete_prefix` sweeps what is left). + +use std::io::Write; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; + +use bytes::{Bytes, BytesMut}; +use extenddb_backup_store::{BackupStore, ByteStream}; +use extenddb_core::backup_format::{ + DataFileChecksums, DataFileEncoder, DataFileEntry, FormatError, KeyRange, KeyRangeTracker, + MAX_LINE_BYTES, data_file_key, encode_item_line_len_bounded, +}; +use extenddb_core::limits::DEFAULT_MAX_ITEM_SIZE_BYTES; +use extenddb_core::metrics::MetricsCollector; +use extenddb_core::types::{Item, item_size_bytes}; +use extenddb_core::validation::{ + RestorableItemError, RestorableItemReason, RestorableKeySchema, RestorableLimits, + validate_restorable_item, +}; +use extenddb_storage::CancellationToken; +use extenddb_storage::backup_catalog::StoreIdentity; +use extenddb_storage::error::StorageError; +use futures::StreamExt; +use futures::stream::BoxStream; +use tokio::sync::mpsc; + +use super::super::runtime::{ConfiguredStore, StoreHealth}; +use super::heartbeat::HeartbeatHandle; +use super::{KillPoint, kill_point}; + +/// Bytes per chunk handed from the encoder to the upload. +pub(crate) const CHUNK_BYTES: usize = 1024 * 1024; +/// Chunks the chunk channel holds before the encoder waits. +pub(crate) const CHUNK_CHANNEL: usize = 4; +/// Items the item channel holds before the feeder waits. Four, not one: a +/// one-item channel made a 1M-row backup on SQLite take 75% longer (12.4 s +/// to 21.6 s, measured in both orders under one host state) for about +/// 50 MiB less at the peak of the worst-shape run (design section 7). +pub(crate) const ITEM_CHANNEL: usize = 4; +/// Refused items logged one by one; the rest are counted. +pub(crate) const REFUSAL_LOG_CAP: u64 = 1000; + +/// Why the job stops before the manifest. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Stop { + /// The row is this job's and the failure path runs with this reason. + Fail(String), + /// The shutdown token fired: the failure path runs with reason + /// `shutdown`. + Cancelled, + /// The claim is lost or its heartbeat did not return: no further store + /// or catalog access. + ClaimLost(String), + /// The store, as the runtime holds it, is unhealthy or is no longer the + /// store the row records ([`Pipeline::store_check`]): FAILED with this + /// reason and no store call, the prefix's deletion included. + StoreUnhealthy(String), +} + +/// What the pipeline wrote. +#[derive(Debug, Default)] +pub(crate) struct DataFiles { + /// Each file's manifest entry with the etag the store reported. + pub entries: Vec<(DataFileEntry, Option)>, + /// Items in every file. + pub item_count: u64, + /// The DynamoDB sizes of those items, summed (the manifest's + /// `source_table.table_size_bytes`). + pub item_bytes: u64, + /// Bytes the store reported stored, over every file. + pub bytes_uploaded: u64, +} + +/// What the uploader needs from the job. +pub(crate) struct Pipeline<'a> { + pub store: &'a dyn BackupStore, + /// The store as the runtime holds it: its health and identity, read + /// before every put. + pub configured: &'a ConfiguredStore, + /// The store's identity as the row records it. + pub identity: &'a StoreIdentity, + /// The backup's store prefix (`BackupArn::store_prefix`). + pub prefix: &'a str, + /// The ARN, for the log lines. + pub arn: &'a str, + pub heartbeat: &'a HeartbeatHandle, + pub shutdown: &'a CancellationToken, + pub metrics: Option<&'a MetricsCollector>, + /// `data_file_target_bytes`. + pub target_bytes: u64, + /// The store's configured key prefix and the summary's bucket name + /// (`BackupDocuments::build`). + pub key_prefix: &'a str, + pub bucket: &'a str, +} + +impl Pipeline<'_> { + /// The check before every put: the store is healthy as the runtime holds + /// it (the reconciler's check sets it) and its identity is the one the + /// row records. A store marked unhealthy mid-job, or whose identity file + /// now names another store (a root swapped under the job), stops the + /// job before the put of `name` with the store's reason + /// ([`Stop::StoreUnhealthy`]): the objects a job would write into such a + /// store are never the backup the row describes. + pub(crate) fn store_check(&self, name: &str) -> Result<(), Stop> { + match self.configured.health() { + StoreHealth::Healthy => {} + StoreHealth::Checking => { + return Err(Stop::StoreUnhealthy(format!( + "the {} backup store has no passed check; {name} was not written", + self.configured.kind + ))); + } + StoreHealth::Unhealthy(problem) => { + return Err(Stop::StoreUnhealthy(format!( + "the {} backup store is unhealthy ({problem}); {name} was not written", + self.configured.kind + ))); + } + } + match self.configured.identity() { + Some(identity) if identity == *self.identity => Ok(()), + Some(identity) => Err(Stop::StoreUnhealthy(format!( + "the {} backup store's id is {} and the row's is {}; {name} was not written", + self.configured.kind, identity.store_id, self.identity.store_id + ))), + None => Err(Stop::StoreUnhealthy(format!( + "the {} backup store's id is unknown; {name} was not written", + self.configured.kind + ))), + } + } +} + +/// The reason the FAILED row carries for refused items (module doc). +fn refusal_reason(count: u64, first: &RestorableItemError) -> String { + let issue = if matches!(first.reason, RestorableItemReason::InvalidNumber(_)) { + " (#618)" + } else { + "" + }; + format!("{count} item(s) cannot be restored; first: {first}{issue}") +} + +/// What the feeder found by the end of the stream. +#[derive(Default)] +struct FeedEnd { + refusals: Option<(u64, RestorableItemError)>, + stream_error: Option, + interrupted: Option, +} + +/// The feeder (module doc). `stop` is shared with the encoders: the feeder +/// sets it at the first refusal, and stops sending when another part set +/// it. `failed` is the uploader's: cancelled when the upload failed, so the +/// feeder returns at once. +struct Feeder { + tx: mpsc::Sender, + key: RestorableKeySchema, + limits: RestorableLimits, + stop: Arc, + failed: CancellationToken, + lost: CancellationToken, + shutdown: CancellationToken, + arn: String, +} + +impl Feeder { + async fn run(self, mut items: BoxStream<'static, Result>) -> FeedEnd { + let Self { + tx, + mut key, + limits, + stop, + failed, + lost, + shutdown, + arn, + } = self; + let mut tx = Some(tx); + let mut end = FeedEnd::default(); + loop { + let next = tokio::select! { + biased; + () = shutdown.cancelled() => { + end.interrupted = Some(Stop::Cancelled); + break; + } + () = lost.cancelled() => { + end.interrupted = Some(Stop::ClaimLost(String::new())); + break; + } + () = failed.cancelled() => break, + next = items.next() => next, + }; + let mut item = match next { + None => break, + Some(Ok(item)) => item, + Some(Err(e)) => { + end.stream_error = Some(e); + break; + } + }; + let refusal = if item_size_bytes(&item) > DEFAULT_MAX_ITEM_SIZE_BYTES { + // An item over the default item limit: the validator's walk + // (the number rule over every member, CPU work proportional + // to the item's size) and then the line's measurement (up + // to the cap's bytes) run on the blocking pool as the + // encoder does; the item and the key come back with the + // result. + let measured = tokio::task::spawn_blocking(move || { + let refusal = validate_restorable_item(&item, &key, &limits) + .err() + .or_else(|| line_too_long(&item, &key)); + (item, key, refusal) + }) + .await; + match measured { + Ok((back, key_back, refusal)) => { + item = back; + key = key_back; + refusal + } + Err(e) => { + end.stream_error = Some(StorageError::Internal(format!( + "checking an item over the default item limit: {e}" + ))); + break; + } + } + } else { + validate_restorable_item(&item, &key, &limits).err() + }; + if let Some(e) = refusal { + match &mut end.refusals { + None => { + tracing::error!("backup {arn}: item cannot be restored: {e}"); + end.refusals = Some((1, e)); + stop.store(true, Ordering::Release); + tx = None; + } + Some((count, _)) => { + *count += 1; + if *count <= REFUSAL_LOG_CAP { + tracing::error!("backup {arn}: item cannot be restored: {e}"); + } + } + } + continue; + } + if end.refusals.is_some() { + continue; + } + if stop.load(Ordering::Acquire) { + break; + } + let Some(sender) = &tx else { break }; + if sender.send(item).await.is_err() { + break; + } + } + if let Some((count, _)) = &end.refusals + && *count > REFUSAL_LOG_CAP + { + tracing::error!( + "backup {arn}: {} more item(s) cannot be restored (not listed)", + count - REFUSAL_LOG_CAP + ); + } + drop(items); + end + } +} + +/// The line-cap rule (module doc) for an item over the default item limit: +/// its line is measured against `MAX_LINE_BYTES` through a counting writer +/// that holds no line ([`encode_item_line_len_bounded`]), and one whose +/// line would exceed the cap is refused with `LineTooLong`. The feeder +/// measures no item within the default limit: the cap is sized to hold +/// every line such an item can expand to (`MAX_LINE_BYTES`'s doc, pinned by +/// the format's own test). An item the serializer refuses is left to the +/// encoder, which reports it. +fn line_too_long(item: &Item, key: &RestorableKeySchema) -> Option { + match encode_item_line_len_bounded(item, MAX_LINE_BYTES) { + Err(FormatError::LineTooLong { cap }) => Some(RestorableItemError { + key: key.shown_key(item), + path: String::new(), + reason: RestorableItemReason::LineTooLong { cap }, + }), + Ok(_) | Err(_) => None, + } +} + +const STOPPED: &str = "the pipeline stopped"; + +/// The encoder's sink: chunks of [`CHUNK_BYTES`] over the chunk channel, the +/// remainder on `flush`. A write after the pipeline stopped, or after the +/// upload ended, fails and poisons the encoder. Both failures are +/// `BrokenPipe`, never `Interrupted`: the encoder retries `Interrupted` on +/// every path (it is std's "try again" kind), and a stopped pipeline does +/// not come back, so an `Interrupted` here would spin the compressor's +/// trailer write for as long as the drain lasts. +struct ChunkSink { + tx: mpsc::Sender, + buf: BytesMut, + stop: Arc, +} + +impl ChunkSink { + fn send(&self, chunk: Bytes) -> std::io::Result<()> { + self.tx + .blocking_send(chunk) + .map_err(|_| std::io::Error::new(std::io::ErrorKind::BrokenPipe, "the upload ended")) + } +} + +impl Write for ChunkSink { + fn write(&mut self, data: &[u8]) -> std::io::Result { + if self.stop.load(Ordering::Acquire) { + return Err(std::io::Error::new(std::io::ErrorKind::BrokenPipe, STOPPED)); + } + self.buf.extend_from_slice(data); + while self.buf.len() >= CHUNK_BYTES { + let chunk = self.buf.split_to(CHUNK_BYTES).freeze(); + self.send(chunk)?; + } + Ok(data.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + if !self.buf.is_empty() { + let chunk = self.buf.split().freeze(); + self.send(chunk)?; + } + Ok(()) + } +} + +/// How one encoder ended. +enum FileEnd { + /// No item reached the encoder: nothing was written. + Empty, + /// The file reached `data_file_target_bytes` and closed; the stream goes + /// on. + Closed { + checksums: DataFileChecksums, + range: Option, + /// The DynamoDB sizes of the file's items, summed. + item_bytes: u64, + }, + /// The stream ended inside this file, which closed. + Ended { + checksums: DataFileChecksums, + range: Option, + item_bytes: u64, + }, +} + +/// What an encoder hands back: the item channel and the tracker for the +/// next file, and how it ended. +type Encoded = ( + mpsc::Receiver, + Option, + Result, +); + +/// One data file's encoder (module doc), on the blocking pool. +fn encode_file( + mut items: mpsc::Receiver, + sink: ChunkSink, + mut tracker: Option, + target_bytes: u64, +) -> Encoded { + let stop = Arc::clone(&sink.stop); + let mut encoder = DataFileEncoder::new(sink); + let mut item_bytes: u64 = 0; + let result = loop { + if stop.load(Ordering::Acquire) { + break Err(FormatError::Encode(STOPPED.to_owned())); + } + let Some(item) = items.blocking_recv() else { + if encoder.item_count() == 0 { + break Ok(FileEnd::Empty); + } + break close(&mut encoder, &mut tracker).map(|(checksums, range)| FileEnd::Ended { + checksums, + range, + item_bytes, + }); + }; + if let Some(tracker) = &mut tracker + && let Err(e) = tracker.push(&item) + { + break Err(e); + } + if let Err(e) = encoder.write_item(&item) { + break Err(e); + } + item_bytes += item_size_bytes(&item) as u64; + if encoder.uncompressed_bytes() >= target_bytes { + break close(&mut encoder, &mut tracker).map(|(checksums, range)| FileEnd::Closed { + checksums, + range, + item_bytes, + }); + } + }; + // A finished encoder's drop writes nothing; an open one's drop takes + // the sink out first, so no trailer reaches the upload. + drop(encoder); + (items, tracker, result) +} + +/// Close the file: the trailer and the remainder go to the sink, the +/// tracker starts the next file. +fn close( + encoder: &mut DataFileEncoder, + tracker: &mut Option, +) -> Result<(DataFileChecksums, Option), FormatError> { + let checksums = encoder.try_finish()?; + Ok(( + checksums, + tracker.as_mut().and_then(KeyRangeTracker::next_file), + )) +} + +/// The pipeline (module doc). `tracker` is `Some` for a key-ordered backup. +pub(crate) async fn write_data_files( + p: &Pipeline<'_>, + items: BoxStream<'static, Result>, + key: RestorableKeySchema, + limits: RestorableLimits, + mut tracker: Option, +) -> Result { + let stop = Arc::new(AtomicBool::new(false)); + let failed = CancellationToken::new(); + let (item_tx, item_rx) = mpsc::channel(ITEM_CHANNEL); + let feeder = tokio::spawn( + Feeder { + tx: item_tx, + key, + limits, + stop: Arc::clone(&stop), + failed: failed.clone(), + lost: p.heartbeat.lost().clone(), + shutdown: p.shutdown.clone(), + arn: p.arn.to_owned(), + } + .run(items), + ); + let mut item_rx = Some(item_rx); + let mut files = DataFiles::default(); + let mut index: u32 = 1; + let uploaded: Result<(), Stop> = loop { + let (chunk_tx, mut chunk_rx) = mpsc::channel::(CHUNK_CHANNEL); + let sink = ChunkSink { + tx: chunk_tx, + buf: BytesMut::with_capacity(CHUNK_BYTES), + stop: Arc::clone(&stop), + }; + let rx = item_rx.take().expect("the previous encoder returned it"); + let file_tracker = tracker.take(); + let target = p.target_bytes; + let encoder = + tokio::task::spawn_blocking(move || encode_file(rx, sink, file_tracker, target)); + + // The file's first chunk, or the channel closing with none. + let first = tokio::select! { + biased; + () = p.shutdown.cancelled() => Err(Stop::Cancelled), + () = p.heartbeat.lost().cancelled() => Err(Stop::ClaimLost(p.heartbeat.reason())), + chunk = chunk_rx.recv() => Ok(chunk), + }; + let first = match first { + Ok(Some(first)) => first, + Ok(None) => { + let (_, _, end) = match encoder.await { + Ok(encoded) => encoded, + Err(e) => break Err(Stop::Fail(format!("the encoder task failed: {e}"))), + }; + break match end { + Ok(FileEnd::Empty) => Ok(()), + Ok(_) => Err(Stop::Fail(format!( + "data file {index} closed without a chunk" + ))), + Err(_) if stop.load(Ordering::Acquire) => Ok(()), + Err(e) => Err(Stop::Fail(format!("encoding data file {index}: {e}"))), + }; + } + Err(why) => { + stop.store(true, Ordering::Release); + drop(chunk_rx); + // The feeder first (module doc): an encoder waiting on the + // item channel returns only when the feeder closes it, and + // the feeder waits on the stream's next item until told. + failed.cancel(); + let _ = encoder.await; + break Err(why); + } + }; + + let name = data_file_key(index); + if let Err(why) = p.store_check(&name) { + stop.store(true, Ordering::Release); + drop(chunk_rx); + failed.cancel(); + let _ = encoder.await; + break Err(why); + } + if !p.heartbeat.confirms_the_claim() { + stop.store(true, Ordering::Release); + drop(chunk_rx); + failed.cancel(); + let _ = encoder.await; + break Err(Stop::ClaimLost(format!( + "no heartbeat confirmed the claim recently enough for data file {index}" + ))); + } + let key = format!("{}{name}", p.prefix); + let body: ByteStream = Box::pin(futures::stream::once(async move { Ok(first) }).chain( + futures::stream::unfold(chunk_rx, |mut rx| async move { + rx.recv().await.map(|chunk| (Ok(chunk), rx)) + }), + )); + let put = tokio::select! { + biased; + () = p.shutdown.cancelled() => Err(Stop::Cancelled), + () = p.heartbeat.lost().cancelled() => Err(Stop::ClaimLost(p.heartbeat.reason())), + result = p.store.put(&key, body) => result.map_err(|e| { + if let Some(metrics) = p.metrics { + metrics.record_backup_store_error("put"); + } + Stop::Fail(format!("writing {name}: {e}")) + }), + }; + let meta = match put { + Ok(meta) => meta, + Err(why) => { + stop.store(true, Ordering::Release); + // Before the encoder (module doc): a put that fails mid-file + // leaves the encoder in `blocking_recv`, which returns only + // once the feeder, told here, closes the channel. + failed.cancel(); + let _ = encoder.await; + break Err(why); + } + }; + // The put ended because the encoder dropped the sink, so the encoder + // has returned or is about to. + let (rx, file_tracker, end) = match encoder.await { + Ok(encoded) => encoded, + Err(e) => break Err(Stop::Fail(format!("the encoder task failed: {e}"))), + }; + item_rx = Some(rx); + tracker = file_tracker; + let (checksums, range, item_bytes, last) = match end { + Ok(FileEnd::Closed { + checksums, + range, + item_bytes, + }) => (checksums, range, item_bytes, false), + Ok(FileEnd::Ended { + checksums, + range, + item_bytes, + }) => (checksums, range, item_bytes, true), + Ok(FileEnd::Empty) => { + break Err(Stop::Fail(format!( + "data file {index} sent a chunk and then reported no item" + ))); + } + Err(_) if stop.load(Ordering::Acquire) => break Ok(()), + Err(e) => break Err(Stop::Fail(format!("encoding data file {index}: {e}"))), + }; + // The store's size is the manifest's: the reconciler's dead-owner + // probe refuses a data file whose stored size differs from the + // recorded one, so a live job must not commit a manifest that probe + // would call incomplete. + if meta.size != checksums.compressed_bytes { + break Err(Stop::Fail(format!( + "the store reports {} bytes for {name} and the encoder wrote {}", + meta.size, checksums.compressed_bytes + ))); + } + files.item_count += checksums.item_count; + files.item_bytes += item_bytes; + files.bytes_uploaded += meta.size; + files.entries.push(( + DataFileEntry::from_checksums(name, &checksums, range), + meta.etag, + )); + if index == 1 { + kill_point(KillPoint::AfterDataFile, p.shutdown).await; + } + if last { + break Ok(()); + } + index += 1; + }; + stop.store(true, Ordering::Release); + if uploaded.is_err() { + // The uploader's own failure: the feeder has nothing left to check. + failed.cancel(); + } + drop(item_rx); + let end = match feeder.await { + Ok(end) => end, + Err(e) => return Err(Stop::Fail(format!("the snapshot reader task failed: {e}"))), + }; + if let Err(why @ (Stop::Cancelled | Stop::ClaimLost(_) | Stop::StoreUnhealthy(_))) = uploaded { + return Err(why); + } + match end.interrupted { + Some(Stop::Cancelled) => return Err(Stop::Cancelled), + Some(Stop::ClaimLost(_)) => return Err(Stop::ClaimLost(p.heartbeat.reason())), + Some(Stop::Fail(_) | Stop::StoreUnhealthy(_)) | None => {} + } + match (end.refusals, end.stream_error) { + // A stream that fails after a refusal (the pass that checks the rest + // reads the whole table) keeps the refusals in the reason, counted + // over the items read before the failure, so `backup list --failed` + // still names the first refused item. + (Some((count, first)), Some(e)) => { + return Err(Stop::Fail(format!( + "{}; the snapshot then failed: {e}", + refusal_reason(count, &first) + ))); + } + (None, Some(e)) => return Err(Stop::Fail(format!("reading the snapshot: {e}"))), + (Some((count, first)), None) => { + return Err(Stop::Fail(refusal_reason(count, &first))); + } + (None, None) => {} + } + uploaded.map(|()| files) +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use extenddb_core::types::AttributeValue; + + use super::*; + + fn stopped_sink() -> ChunkSink { + let (tx, _rx) = mpsc::channel(CHUNK_CHANNEL); + ChunkSink { + tx, + buf: BytesMut::with_capacity(CHUNK_BYTES), + stop: Arc::new(AtomicBool::new(true)), + } + } + + /// A write after the stop is `BrokenPipe`, the kind nothing retries. + #[test] + fn a_write_after_the_stop_is_a_broken_pipe() { + let e = stopped_sink().write(b"x").unwrap_err(); + assert_eq!(e.kind(), std::io::ErrorKind::BrokenPipe); + assert_eq!(e.to_string(), STOPPED); + } + + /// An encoder holding items when the pipeline stops fails its close at + /// once instead of retrying the trailer write for the whole drain: the + /// shutdown of a job in the middle of a large table ends within the + /// server's drain window. + #[test] + fn closing_an_encoder_after_the_stop_fails_at_once() { + let (done_tx, done_rx) = std::sync::mpsc::channel(); + std::thread::spawn(move || { + let mut encoder = DataFileEncoder::new(stopped_sink()); + let item = Item::from([("pk".to_owned(), AttributeValue::S("a".to_owned()))]); + // The gzip header is the first write to reach the sink. + let written = encoder.write_item(&item); + let closed = encoder.try_finish().map(|_| ()); + drop(encoder); + let _ = done_tx.send((written, closed)); + }); + let (written, closed) = done_rx + .recv_timeout(Duration::from_secs(5)) + .expect("the close returned within the window"); + let e = written.and(closed).unwrap_err(); + assert!(e.to_string().contains(STOPPED), "{e}"); + } +} diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs index 10c0068e6..9c1b8faef 100644 --- a/crates/engine/src/backup/reconciler.rs +++ b/crates/engine/src/backup/reconciler.rs @@ -2,15 +2,16 @@ // SPDX-License-Identifier: Apache-2.0 //! The reconciler: the periodic worker that resolves catalog rows whose -//! owner is dead, and keeps the configured store's health current. +//! owner is dead, deletes what FAILED and DELETED rows leave in the store, +//! removes tombstones past their retention, and keeps the configured +//! store's health current. //! //! One cycle runs every `interval`. It first checks the catalog-level //! maintenance claim; while a maintenance command holds it the reconciler -//! runs only the duties below, which resolve the states the command waits -//! on or finishes itself (the dead-owner duties, the store's check and the -//! DELETING rows); the duties a later release adds that delete or move a -//! row the command does not wait on (a FAILED row's prefix, tombstone -//! retention) go in the branch the flag skips. +//! runs only the duties the command waits on or finishes itself (the +//! dead-owner duties, the store's check and the DELETING rows) and skips +//! the deleting duties it must not race (a FAILED row's prefix, a DELETED +//! row's objects, tombstone retention, the registrations sweep). //! //! The claim is read through `check_maintenance_claim(stale_job_after)`, //! except in a cycle a maintenance command runs itself while it holds the @@ -21,7 +22,7 @@ //! when its heartbeat and the check land in the same instant, so the //! branch would be decided by scheduler timing. //! -//! The duties, over the records `list_dead_restore_jobs` returns: +//! The duties, in the order a cycle runs them: //! //! - A RUNNING restore job record whose `heartbeat_at` is older than //! `stale_job_after` (on the catalog's clock) has a dead owner. One of kind @@ -51,43 +52,89 @@ //! then the identity check and manifest probe) and sets the store's //! health, so a store whose checks run after the listener is up, or that //! failed at startup or later, is healthy once a check passes, and one -//! that fails is refused from the next cycle on. The check runs after the -//! duties above and under the store's bound -//! ([`super::ConfiguredStore::check_timeout`]: 5 s for a filesystem root, -//! 30 s for a bucket), so a store that has stopped answering costs each -//! cycle that bound and nothing more: the dead-owner duties run every -//! `interval` plus the bound, the check is recorded as a transport -//! failure, and a filesystem check leaves one blocking thread parked on -//! the mount until it answers, after which no check starts until that -//! call returns (the runtime module's in-flight rule), so a mount that -//! stays stalled parks one thread and not one per cycle. The check is -//! also selected against the shutdown token, so a cycle that is checking -//! when the server stops returns at once and the drain finishes. -//! While the store is healthy, each DELETING row of the configured store -//! (a DeleteBackup whose deletion did not finish) is finished -//! (`finish_backup_delete` in the engine's backup module): the manifest, then the prefix, then -//! the row to DELETED. While it is unhealthy the reconciler acts on no row -//! of the store. +//! that fails is refused from the next cycle on. The check runs under the +//! store's bound ([`super::ConfiguredStore::check_timeout`]: 5 s for a +//! filesystem root, 30 s for a bucket), so a store that has stopped +//! answering costs each cycle that bound and nothing more: the duties +//! above run every `interval` plus the bound, the check is recorded as a +//! transport failure, and a filesystem check leaves one blocking thread +//! parked on the mount until it answers, after which no check starts +//! until that call returns (the runtime module's in-flight rule), so a +//! mount that stays stalled parks one thread and not one per cycle. The +//! check is also selected against the shutdown token, so a cycle that is +//! checking when the server stops returns at once and the drain finishes. +//! While the store is unhealthy the reconciler acts on no row of the +//! store. While it is healthy, on the rows of the configured store: +//! - Each DELETING row (a DeleteBackup whose deletion did not finish) is +//! finished (`finish_backup_delete` in the engine's backup module): the +//! manifest, then the prefix, then the row to DELETED. +//! - Each CREATING row whose owner is dead is resolved. The backend's +//! `claim_dead_backup` is the test and the takeover (a fresh generation +//! fences the dead job's writes out; `heartbeat_at` is left as it was); +//! a row it does not hand over has a live owner or is another's. Under +//! the claim the store is probed for a complete manifest: +//! `extenddb-manifest.json` and its sidecar present and agreeing, the +//! manifest parsing with the row's ARN, and every data file it lists +//! present under the prefix with the recorded size. A complete manifest +//! makes the row AVAILABLE with the manifest's size, item count and +//! `source_table`; anything else makes it FAILED with reason +//! [`OWNER_DEAD_NO_MANIFEST`], and nothing is deleted here: the +//! FAILED-prefix duty, which runs later in this cycle, deletes it once +//! the row's heartbeat is older than `stale_job_after` on the database's +//! clock: in this cycle on PostgreSQL, MongoDB and Cassandra, where a +//! heartbeat that old is what made the owner dead (on PostgreSQL +//! together with the owner's claim session gone, its advisory lock +//! free); on SQLite, where the slot did, once the dead owner's last +//! beat is that old. A store failure during the probe (not an absent +//! object) leaves the row for the next cycle. The two writes are +//! conditioned on the row being CREATING under the new generation +//! (`job::failure::write_status`); one that matches no row (another +//! reconciler took the row over in the gap, item 31) leaves the row to +//! that owner. +//! - Not while the maintenance claim is held: a FAILED row whose +//! `heartbeat_at` is older than `stale_job_after` (the backend's +//! `list_backup_rows_older_than` on the heartbeat clock) and whose +//! prefix lists objects has its prefix deleted, retried each cycle while +//! the listing shows objects; a DELETED row whose prefix lists objects +//! has its prefix deleted; a FAILED or DELETED row whose prefix lists +//! nothing and whose `completed_at` is older than `failed_row_retention` +//! (the completed clock) is removed. Every age is the database's +//! comparison; the engine holds no clock. +//! - Not while the maintenance claim is held, on every backend: the +//! backend's `sweep_restore_registrations` (a no-op but on Cassandra). //! -//! A cycle that fails logs and waits for the next; each cycle records the -//! worker metrics the other workers record (`WorkerLastSuccess` and -//! `WorkerCycleLatency` on success, `WorkerErrorCount` on failure, under -//! `QuerySource::BackupReconciler`); the worker stops at the shutdown token. +//! Every duty acts only on rows of the configured store (kind, root and +//! id), the legacy duty excepted. A cycle that fails logs and waits for the +//! next; each cycle records the worker metrics the other workers record +//! (`WorkerLastSuccess` and `WorkerCycleLatency` on success, +//! `WorkerErrorCount` on failure, under `QuerySource::BackupReconciler`); +//! a duty's store call that fails (a `get` or `list` of the probe or the +//! deleting duties, a `delete_prefix`; not an absent object, and not the +//! store's check, which sets the store's health instead) records +//! `BackupStoreErrors` under the call's name, as the job's calls do, so a +//! store that refuses one row's calls is counted though the cycle +//! completes; the worker stops at the shutdown token. use std::collections::HashSet; use std::sync::Arc; use std::time::Duration; -use extenddb_core::backup_format::BackupArn; +use extenddb_backup_store::{BackupStore, StoreError}; +use extenddb_core::backup_format::{ + BackupArn, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, MANIFEST_CHECKSUM_FILE, + verify_manifest_checksum, +}; use extenddb_core::metrics::{MetricsCollector, QuerySource}; use extenddb_storage::backup_catalog::{ - BackupStatus, JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, + BackupRow, BackupStatus, JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, + StatusFields, StoreIdentity, }; use extenddb_storage::error::StorageError; use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader, sleep_or_shutdown}; use futures::TryStreamExt; use super::BackupRuntime; +use super::job::failure::{Written, write_status}; /// The two durations a reconciler reads from `[backup]`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -101,6 +148,11 @@ pub struct ReconcilerSettings { /// The reason written on a restore job record whose owner died. pub const OWNER_DEAD: &str = "owner dead: no heartbeat within stale_job_after_secs"; +/// The reason written on a CREATING backup row whose owner died without a +/// complete manifest in the store. +pub const OWNER_DEAD_NO_MANIFEST: &str = + "owner dead: no heartbeat within stale_job_after_secs; no complete manifest"; + /// What one cycle did. #[derive(Debug, Default, Clone, PartialEq, Eq)] pub struct CycleReport { @@ -127,6 +179,38 @@ pub struct CycleReport { pub store_healthy: Option, /// DELETING rows of the configured store moved to DELETED, by ARN. pub deletions_finished: Vec, + /// CREATING rows whose owner was dead, resolved to the status given, by + /// ARN. + pub creating_resolved: Vec<(String, BackupStatus)>, + /// CREATING rows this cycle took over whose status write matched no row + /// (another owner took them over first), left to that owner, by ARN. + pub creating_left_to_owner: Vec, + /// FAILED rows past the stale window whose prefix was deleted, by ARN. + pub failed_prefixes_deleted: Vec, + /// DELETED rows whose prefix still listed objects and was deleted, by + /// ARN. + pub deleted_prefixes_swept: Vec, + /// FAILED and DELETED rows past their retention with an empty prefix, + /// removed, by ARN. + pub tombstones_removed: Vec, + /// Backups whose restore registrations the backend swept, by ARN. + pub registrations_swept: Vec, +} + +impl CycleReport { + /// Whether the cycle changed a row or an object. + #[must_use] + pub fn acted(&self) -> bool { + !(self.legacy_restores_removed.is_empty() + && self.failed_targets_removed.is_empty() + && self.legacy_backups_removed.is_empty() + && self.deletions_finished.is_empty() + && self.creating_resolved.is_empty() + && self.failed_prefixes_deleted.is_empty() + && self.deleted_prefixes_swept.is_empty() + && self.tombstones_removed.is_empty() + && self.registrations_swept.is_empty()) + } } /// What a reconciler carries from one cycle to the next: the manifest @@ -208,17 +292,22 @@ pub async fn run( #[allow(clippy::cast_precision_loss)] let cycle_us = started.elapsed().as_micros() as f64; metrics.record_worker_success(QuerySource::BackupReconciler, cycle_us); - if !report.legacy_restores_removed.is_empty() - || !report.failed_targets_removed.is_empty() - || !report.legacy_backups_removed.is_empty() - || !report.deletions_finished.is_empty() - { + if report.acted() { tracing::info!( - "backup reconciler: removed {} abandoned restore target(s) and {} \ - abandoned in-catalog backup(s), finished {} deletion(s)", - report.legacy_restores_removed.len() + report.failed_targets_removed.len(), + "backup reconciler: failed and removed {} dead legacy restore(s), removed \ + the target(s) of {} FAILED restore record(s) and {} abandoned in-catalog \ + backup(s), finished {} deletion(s), resolved {} dead CREATING row(s), \ + deleted {} FAILED prefix(es), swept {} DELETED prefix(es), removed {} \ + tombstone(s), swept the registrations of {} backup(s)", + report.legacy_restores_removed.len(), + report.failed_targets_removed.len(), report.legacy_backups_removed.len(), - report.deletions_finished.len() + report.deletions_finished.len(), + report.creating_resolved.len(), + report.failed_prefixes_deleted.len(), + report.deleted_prefixes_swept.len(), + report.tombstones_removed.len(), + report.registrations_swept.len() ); } } @@ -399,18 +488,437 @@ where } report.store_healthy = Some(store.is_healthy()); if let Some(identity) = store.identity().filter(|_| store.is_healthy()) { + let calls = StoreCalls { + store: &*store.store, + metrics: runtime.metrics().map(|m| &**m), + }; report.deletions_finished = finish_deleting_rows(storage, store, &identity).await?; + (report.creating_resolved, report.creating_left_to_owner) = + resolve_dead_creating_rows(storage, calls, &identity, stale_job_after).await?; + if !report.maintenance_held { + let retention = runtime.job_settings().failed_row_retention; + report.failed_prefixes_deleted = + delete_stale_failed_prefixes(storage, calls, &identity, stale_job_after) + .await?; + report.deleted_prefixes_swept = + sweep_deleted_prefixes(storage, calls, &identity).await?; + report.tombstones_removed = + remove_tombstones(storage, calls, &identity, retention).await?; + } } } if !report.maintenance_held { - // The duties a maintenance command does not wait on and must not - // race: a later release's deletion of a FAILED row's prefix and its - // removal of tombstones past their retention run here. + report.registrations_swept = storage.sweep_restore_registrations().await?; } Ok(report) } +/// The duty for CREATING rows of the configured store whose owner is dead +/// (module doc). Returns the rows resolved with their new status and the +/// rows whose write matched no row, left to their new owner; a row the +/// backend does not hand over is skipped, a row whose probe or write fails +/// is logged and left for the next cycle. +async fn resolve_dead_creating_rows( + storage: &S, + store: StoreCalls<'_>, + identity: &StoreIdentity, + stale_job_after: Duration, +) -> Result<(Vec<(String, BackupStatus)>, Vec), StorageError> { + let rows: Vec = storage + .list_backup_rows(Some(BackupStatus::Creating), identity) + .try_collect() + .await?; + let mut resolved = Vec::new(); + let mut left_to_owner = Vec::new(); + for row in rows { + let claim = match storage.claim_dead_backup(&row.key(), stale_job_after).await { + Ok(Some(claim)) => claim, + Ok(None) => continue, + Err(e) => { + tracing::error!( + "could not test whether the owner of CREATING backup {} (account {}) is \ + dead; the next cycle retries: {e}", + row.backup_arn, + row.account_id + ); + continue; + } + }; + let arn = match BackupArn::parse(&row.backup_arn) { + Ok(arn) => arn, + Err(e) => { + tracing::error!( + "CREATING row {} carries an ARN this build cannot parse; left as it is: {e}", + row.backup_arn + ); + continue; + } + }; + let (to, fields) = match probe_manifest(store, &arn).await { + Ok(Probe::Complete(fields)) => (JobStatus::Available, fields), + Ok(Probe::Incomplete(why)) => { + tracing::warn!( + "backup {} (account {}): its owner is dead and the store holds no complete \ + manifest ({why}); the row is FAILED and its prefix is deleted once the row's \ + heartbeat is stale", + row.backup_arn, + row.account_id + ); + ( + JobStatus::Failed, + StatusFields { + failure_reason: Some(OWNER_DEAD_NO_MANIFEST.to_owned()), + completed: true, + ..StatusFields::default() + }, + ) + } + Err(e) => { + tracing::error!( + "backup {} (account {}): its owner is dead and the store could not be \ + probed for its manifest; the next cycle retries: {e}", + row.backup_arn, + row.account_id + ); + continue; + } + }; + match write_status(&*claim, to, &fields).await { + Written::Matched => { + tracing::info!( + "backup {} (account {}): its owner was dead; the row is now {to}", + row.backup_arn, + row.account_id + ); + let status = match to { + JobStatus::Available => BackupStatus::Available, + _ => BackupStatus::Failed, + }; + resolved.push((row.backup_arn, status)); + } + Written::NotMatched(why) => { + tracing::info!( + "backup {} was no longer this cycle's to resolve ({why}); left to its owner", + row.backup_arn + ); + left_to_owner.push(row.backup_arn); + } + Written::Error(e) => tracing::error!( + "backup {} (account {}): the {to} write failed; the next cycle retries: {e}", + row.backup_arn, + row.account_id + ), + } + drop(claim); + } + Ok((resolved, left_to_owner)) +} + +/// The configured store as the reconciler's duties call it: each failed +/// call (not an absent object) records `BackupStoreErrors` under the call's +/// name (`get`, `list` or `delete_prefix`), as the job records its own, so +/// a store that refuses the reconciler's calls moves the metric even when +/// every cycle completes. +#[derive(Clone, Copy)] +struct StoreCalls<'a> { + store: &'a dyn BackupStore, + metrics: Option<&'a MetricsCollector>, +} + +impl StoreCalls<'_> { + fn failed(self, op: &str, e: StoreError) -> StoreError { + if let Some(metrics) = self.metrics { + metrics.record_backup_store_error(op); + } + e + } + + /// One object's bytes, `None` when absent. + async fn read_object(self, key: &str) -> Result>, StoreError> { + let body = match self.store.get(key).await { + Ok(body) => body, + Err(StoreError::NotFound) => return Ok(None), + Err(e) => return Err(self.failed("get", e)), + }; + let bytes = body + .try_fold(Vec::new(), |mut acc, chunk| async move { + acc.extend_from_slice(&chunk); + Ok(acc) + }) + .await + .map_err(|e| self.failed("get", e))?; + Ok(Some(bytes)) + } + + /// The prefix's objects, key to size. + async fn list_sizes( + self, + prefix: &str, + ) -> Result, StoreError> { + self.store + .list(prefix) + .map_ok(|meta| (meta.key, meta.size)) + .try_collect() + .await + .map_err(|e| self.failed("list", e)) + } + + /// Whether the prefix lists at least one object. + async fn prefix_has_objects(self, prefix: &str) -> Result { + match self.store.list(prefix).try_next().await { + Ok(first) => Ok(first.is_some()), + Err(e) => Err(self.failed("list", e)), + } + } + + async fn delete_prefix(self, prefix: &str) -> Result { + self.store + .delete_prefix(prefix) + .await + .map_err(|e| self.failed("delete_prefix", e)) + } +} + +/// What the manifest probe found under a dead owner's prefix. +enum Probe { + /// A complete, verified manifest: the AVAILABLE write's fields. + Complete(StatusFields), + /// No manifest, or one the reader refuses, or a data file missing or of + /// another size; the text says which. + Incomplete(String), +} + +/// The manifest probe of the dead-owner duty (module doc). A store error +/// other than an absent object is the `Err`. +async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result { + let prefix = arn.store_prefix(); + let manifest = match store + .read_object(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")) + .await? + { + Some(bytes) => bytes, + None => return Ok(Probe::Incomplete("no manifest".to_owned())), + }; + let sidecar = match store + .read_object(&format!("{prefix}{MANIFEST_CHECKSUM_FILE}")) + .await? + { + Some(bytes) => bytes, + None => { + return Ok(Probe::Incomplete( + "a manifest without its sidecar".to_owned(), + )); + } + }; + if let Err(e) = verify_manifest_checksum(&manifest, &sidecar) { + return Ok(Probe::Incomplete(format!( + "the sidecar does not match: {e}" + ))); + } + let manifest = match ExtenddbManifest::from_json_bytes(&manifest) { + Ok(manifest) => manifest, + Err(e) => return Ok(Probe::Incomplete(format!("the manifest is refused: {e}"))), + }; + match manifest.backup_arn() { + Ok(found) if found == *arn => {} + Ok(found) => { + return Ok(Probe::Incomplete(format!( + "the manifest names {}, not the row's backup", + found.to_arn_string() + ))); + } + Err(e) => { + return Ok(Probe::Incomplete(format!( + "the manifest's ARN is refused: {e}" + ))); + } + } + let listed = store.list_sizes(&prefix).await?; + for file in &manifest.data_files { + match listed.get(&format!("{prefix}{}", file.key)) { + Some(size) if *size == file.bytes => {} + Some(size) => { + return Ok(Probe::Incomplete(format!( + "data file {} has {size} bytes in the store and {} in the manifest", + file.key, file.bytes + ))); + } + None => { + return Ok(Probe::Incomplete(format!( + "data file {} is not in the store", + file.key + ))); + } + } + } + let item_count = manifest + .data_files + .iter() + .map(|f| f.item_count) + .sum::(); + Ok(Probe::Complete(StatusFields { + size_bytes: Some(manifest.backup_size_bytes), + item_count: Some(i64::try_from(item_count).unwrap_or(i64::MAX)), + failure_reason: None, + source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), + completed: true, + })) +} + +/// The FAILED-prefix duty (module doc): FAILED rows whose heartbeat is older +/// than `stale_job_after`, their prefix deleted while it lists objects. +async fn delete_stale_failed_prefixes( + storage: &S, + store: StoreCalls<'_>, + identity: &StoreIdentity, + stale_job_after: Duration, +) -> Result, StorageError> { + let rows: Vec = storage + .list_backup_rows_older_than( + BackupStatus::Failed, + identity, + StaleClock::Heartbeat, + stale_job_after, + ) + .try_collect() + .await?; + let mut deleted = Vec::new(); + for row in rows { + if delete_prefix_if_listed(store, &row, "FAILED").await { + deleted.push(row.backup_arn); + } + } + Ok(deleted) +} + +/// The DELETED-prefix duty (module doc): DELETED rows whose prefix still +/// lists objects, the prefix deleted. +async fn sweep_deleted_prefixes( + storage: &S, + store: StoreCalls<'_>, + identity: &StoreIdentity, +) -> Result, StorageError> { + let rows: Vec = storage + .list_backup_rows(Some(BackupStatus::Deleted), identity) + .try_collect() + .await?; + let mut swept = Vec::new(); + for row in rows { + if delete_prefix_if_listed(store, &row, "DELETED").await { + swept.push(row.backup_arn); + } + } + Ok(swept) +} + +/// Delete `row`'s prefix when it lists objects; returns whether a deletion +/// ran. A store failure is logged and left for the next cycle. +async fn delete_prefix_if_listed(store: StoreCalls<'_>, row: &BackupRow, status: &str) -> bool { + let prefix = match BackupArn::parse(&row.backup_arn) { + Ok(arn) => arn.store_prefix(), + Err(e) => { + tracing::error!( + "{status} row {} carries an ARN this build cannot parse; left as it is: {e}", + row.backup_arn + ); + return false; + } + }; + match store.prefix_has_objects(&prefix).await { + Ok(true) => {} + Ok(false) => return false, + Err(e) => { + tracing::error!( + "could not list the prefix of {status} backup {}; the next cycle retries: {e}", + row.backup_arn + ); + return false; + } + } + match store.delete_prefix(&prefix).await { + Ok(n) => { + tracing::info!( + "deleted {n} object(s) left under the prefix of {status} backup {} (account {})", + row.backup_arn, + row.account_id + ); + true + } + Err(e) => { + tracing::error!( + "could not delete the prefix of {status} backup {} (account {}); the next cycle \ + retries: {e}", + row.backup_arn, + row.account_id + ); + false + } + } +} + +/// The tombstone duty (module doc): FAILED and DELETED rows whose +/// `completed_at` is older than `retention` and whose prefix lists nothing +/// are removed. +async fn remove_tombstones( + storage: &S, + store: StoreCalls<'_>, + identity: &StoreIdentity, + retention: Duration, +) -> Result, StorageError> { + let mut removed = Vec::new(); + for status in [BackupStatus::Failed, BackupStatus::Deleted] { + let rows: Vec = storage + .list_backup_rows_older_than(status, identity, StaleClock::Completed, retention) + .try_collect() + .await?; + for row in rows { + let prefix = match BackupArn::parse(&row.backup_arn) { + Ok(arn) => arn.store_prefix(), + Err(e) => { + tracing::error!( + "{status} row {} carries an ARN this build cannot parse; left as it \ + is: {e}", + row.backup_arn + ); + continue; + } + }; + match store.prefix_has_objects(&prefix).await { + Ok(false) => {} + Ok(true) => continue, + Err(e) => { + tracing::error!( + "could not list the prefix of {status} backup {} before removing its \ + row; the next cycle retries: {e}", + row.backup_arn + ); + continue; + } + } + match storage.remove_backup_row(&row.key()).await { + Ok(true) => { + tracing::info!( + "removed the {status} row of backup {} (account {}): its prefix is empty \ + and its retention has passed", + row.backup_arn, + row.account_id + ); + removed.push(row.backup_arn); + } + Ok(false) => {} + Err(e) => tracing::error!( + "could not remove the {status} row of backup {} (account {}); the next cycle \ + retries: {e}", + row.backup_arn, + row.account_id + ), + } + } + } + Ok(removed) +} + /// The duty for DELETING rows of the configured store: finish each deletion. /// Returns the ARNs moved to DELETED; a row whose deletion fails is logged /// and left for the next cycle. diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs index e57ada84f..7d660355a 100644 --- a/crates/engine/src/backup/runtime.rs +++ b/crates/engine/src/backup/runtime.rs @@ -48,14 +48,16 @@ use std::time::Duration; use extenddb_backup_store::{BackupStore, ByteStream, ObjectMeta, StoreError}; use extenddb_core::error::DynamoDbError; +use extenddb_core::metrics::MetricsCollector; use extenddb_storage::backup_catalog::{ - JobBound, StoreIdentity, StoreKind, limit_exceeded_message, + JOB_HEARTBEAT_INTERVAL, JobBound, StoreIdentity, StoreKind, limit_exceeded_message, }; -use extenddb_storage::{BackupCatalog, JobPermit}; +use extenddb_storage::{BackupCatalog, CancellationToken, JobPermit}; use futures::StreamExt; use futures::future::BoxFuture; use futures::stream::BoxStream; use tokio::sync::Semaphore; +use tokio_util::task::TaskTracker; use super::store_identity::{self, StoreProblem}; @@ -71,11 +73,43 @@ const MAX_CONCURRENT_JOBS_KEY: &str = "[backup] max_concurrent_jobs"; /// over the bound is served in turn rather than refused. pub const PERMIT_WAIT: Duration = Duration::from_secs(30); +/// The `[backup]` durations and sizes the store-backed backup job and the +/// reconciler read, as the server resolved them from the configuration. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct JobSettings { + /// `data_file_target_bytes`: a data file closes once its uncompressed + /// bytes reach this. + pub data_file_target_bytes: u64, + /// The job's heartbeat period: `max(heartbeat_interval_secs, + /// JOB_HEARTBEAT_INTERVAL)`, which the server computes; a heartbeat that + /// has not confirmed the claim within one period aborts the job. + pub heartbeat_interval: Duration, + /// `stale_job_after_secs`: the window the maintenance-claim check reads + /// with, and the heartbeat age after which a job's owner is dead. + pub stale_job_after: Duration, + /// `failed_row_retention_secs`: how long a FAILED or DELETED row whose + /// prefix is empty is kept. + pub failed_row_retention: Duration, +} + +impl Default for JobSettings { + /// The configuration's defaults: 64 MiB files, the 15 s heartbeat, + /// 300 s staleness, 7 days of retention. + fn default() -> Self { + Self { + data_file_target_bytes: 64 * 1024 * 1024, + heartbeat_interval: JOB_HEARTBEAT_INTERVAL, + stale_job_after: Duration::from_secs(300), + failed_row_retention: Duration::from_secs(7 * 24 * 3600), + } + } +} + /// How long a filesystem store's check may run, at startup and in a /// reconciler cycle, before the store is treated as not answering: a local /// disk answers in milliseconds, and a network mount that takes longer is /// one that has stopped answering. -pub const LOCAL_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5); +pub const LOCAL_CHECK_TIMEOUT: Duration = Duration::from_secs(5); /// How long an S3 store's check may run in a reconciler cycle. The check /// sends a few small requests (`validate`'s listing, the identity file's @@ -85,14 +119,18 @@ pub const LOCAL_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_s /// so a timed-out check is dropped before the SDK starts a retry of its /// stalled request, and no retry outlives the check. A stalled endpoint thus /// costs a cycle this bound, not the SDK's whole retry budget. -pub const S3_CHECK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30); +pub const S3_CHECK_TIMEOUT: Duration = Duration::from_secs(30); const _: () = assert!( S3_CHECK_TIMEOUT.as_secs() < extenddb_backup_store::S3_ATTEMPT_TIMEOUT.as_secs(), "the S3 check bound must fire before the SDK's attempt timeout" ); /// One per server process, shared by every request through the operation -/// context. +/// context. Besides the slots and the store it carries what a store-backed +/// backup job needs and no request hands it: the job settings, the metrics +/// collector, the shutdown token the jobs are cancelled on, the tracker the +/// server drains them through ([`BackupRuntime::drain_jobs`]), and the +/// backend and version the manifest names as its source. pub struct BackupRuntime { slots: Arc, max_concurrent_jobs: usize, @@ -101,15 +139,31 @@ pub struct BackupRuntime { permit_wait: Duration, /// The store this instance writes new backups to and deletes from; /// `None` while no store is configured, when every catalog row is a row - /// of another store. - configured_store: Option, + /// of another store. Shared with the jobs, which read its health and + /// identity before every put. + configured_store: Option>, /// Whether CreateBackup takes the in-catalog path whatever the store /// configuration (a backend that cannot pin a table snapshot); such a /// create is not refused by an unhealthy store. legacy_create_only: bool, - /// `[backup] stale_job_after_secs`: the window a job's claim check reads - /// the maintenance claim with. - stale_job_after: std::time::Duration, + job_settings: JobSettings, + /// The collector the jobs record into; `None` in commands and tests + /// without one. + metrics: Option>, + /// The token the server's shutdown and a registration loss cancel the + /// jobs on ([`Self::with_shutdown`] makes it a child of the server's + /// token): a running job then runs its failure path with reason + /// `shutdown`. A loss cancels and replaces it ([`Self::cancel_jobs`]), + /// so the jobs started under a new registration run on a fresh token. + shutdown: RwLock, + /// The server's token the jobs' token is a child of. + shutdown_parent: CancellationToken, + /// Every job task spawned through [`BackupRuntime::spawn_job`]. + jobs: TaskTracker, + /// The storage backend's name, the manifest's `source_backend`. + source_backend: String, + /// The build's version, the manifest's `source_extenddb_version`. + source_extenddb_version: String, } impl std::fmt::Debug for BackupRuntime { @@ -118,19 +172,18 @@ impl std::fmt::Debug for BackupRuntime { .field("max_concurrent_jobs", &self.max_concurrent_jobs) .field( "configured_store", - &self - .configured_store - .as_ref() - .map(ConfiguredStore::identity), + &self.configured_store.as_ref().map(|store| store.identity()), ) .field("legacy_create_only", &self.legacy_create_only) + .field("job_settings", &self.job_settings) .finish_non_exhaustive() } } /// The store this instance is configured with: its kind and root (the /// configuration's), its id once the identity check has read it, the handle, -/// the spool for restores from it, and its health. +/// the spool for restores from it, the names the DynamoDB-shaped manifests +/// carry for it, and its health. /// /// The id is `None` until a check passes: a store whose startup check failed /// (a transport failure, say) has an unknown id, and a row of its kind and @@ -146,6 +199,15 @@ pub struct ConfiguredStore { /// The restore spool for an S3 store: the directory and the /// `:` label a restore job record names. pub spool: Option, + /// `manifest-summary.json`'s `s3Bucket`: the bucket name on S3, the root + /// path on a filesystem store ([`Self::with_manifest_names`]; the root + /// label until set). + pub bucket: String, + /// The key prefix the store was opened with (`[backup.store.s3] prefix`; + /// a filesystem store has none), which every key of the DynamoDB-shaped + /// manifests starts with ([`Self::with_manifest_names`]; empty until + /// set). + pub key_prefix: String, store_id: RwLock>, health: RwLock, /// Whether the store's `validate` has passed; the checks run it first @@ -245,6 +307,8 @@ impl ConfiguredStore { }; Self { kind, + bucket: root.clone(), + key_prefix: String::new(), root, store, spool, @@ -271,6 +335,8 @@ impl ConfiguredStore { ) -> Self { Self { kind, + bucket: root.clone(), + key_prefix: String::new(), root, store, spool, @@ -315,6 +381,17 @@ impl ConfiguredStore { self.set_aside.load(Ordering::Acquire) } + /// The same store with the names the DynamoDB-shaped manifests carry + /// for it: `bucket` is `manifest-summary.json`'s `s3Bucket` (the bucket + /// on S3, the root path on a filesystem store) and `key_prefix` the + /// store's configured key prefix (empty when it has none). + #[must_use] + pub fn with_manifest_names(mut self, bucket: String, key_prefix: String) -> Self { + self.bucket = bucket; + self.key_prefix = key_prefix; + self + } + /// The store's identity, once a check has read its id. #[must_use] pub fn identity(&self) -> Option { @@ -433,7 +510,7 @@ impl ConfiguredStore { /// ([`LOCAL_CHECK_TIMEOUT`] for a filesystem root, [`S3_CHECK_TIMEOUT`] /// for a bucket). #[must_use] - pub fn check_timeout(&self) -> std::time::Duration { + pub fn check_timeout(&self) -> Duration { match self.kind { StoreKind::Filesystem => LOCAL_CHECK_TIMEOUT, StoreKind::S3 => S3_CHECK_TIMEOUT, @@ -781,7 +858,9 @@ impl BackupStore for BoundedCalls<'_> { } impl BackupRuntime { - /// `max_concurrent_jobs` slots and no configured store. + /// `max_concurrent_jobs` slots, no configured store, the default job + /// settings, no metrics, a shutdown token nothing cancels, and `test` as + /// the source backend with this crate's version. #[must_use] pub fn new(max_concurrent_jobs: usize) -> Self { Self { @@ -790,7 +869,13 @@ impl BackupRuntime { permit_wait: PERMIT_WAIT, configured_store: None, legacy_create_only: false, - stale_job_after: std::time::Duration::from_secs(300), + job_settings: JobSettings::default(), + metrics: None, + shutdown: RwLock::new(CancellationToken::new()), + shutdown_parent: CancellationToken::new(), + jobs: TaskTracker::new(), + source_backend: "test".to_owned(), + source_extenddb_version: env!("CARGO_PKG_VERSION").to_owned(), } } @@ -798,21 +883,155 @@ impl BackupRuntime { /// the jobs' maintenance-claim check reads with (the default is the /// configuration's default, 300 s). #[must_use] - pub fn with_stale_job_after(mut self, stale_job_after: std::time::Duration) -> Self { - self.stale_job_after = stale_job_after; + pub fn with_stale_job_after(mut self, stale_job_after: Duration) -> Self { + self.job_settings.stale_job_after = stale_job_after; self } /// `[backup] stale_job_after_secs`. #[must_use] - pub fn stale_job_after(&self) -> std::time::Duration { - self.stale_job_after + pub fn stale_job_after(&self) -> Duration { + self.job_settings.stale_job_after + } + + /// The same runtime with every job setting set. + #[must_use] + pub fn with_job_settings(mut self, settings: JobSettings) -> Self { + self.job_settings = settings; + self + } + + /// The settings the jobs and the reconciler read. + #[must_use] + pub fn job_settings(&self) -> JobSettings { + self.job_settings + } + + /// The same runtime with the collector the jobs record into. + #[must_use] + pub fn with_metrics(mut self, metrics: Arc) -> Self { + self.metrics = Some(metrics); + self + } + + /// The collector the jobs record into, if any. + #[must_use] + pub fn metrics(&self) -> Option<&Arc> { + self.metrics.as_ref() + } + + /// The same runtime with the server's shutdown token: the jobs run on a + /// child of it, which [`Self::drain_jobs`] and [`Self::cancel_jobs`] + /// cancel. + #[must_use] + pub fn with_shutdown(mut self, shutdown: CancellationToken) -> Self { + self.shutdown = RwLock::new(shutdown.child_token()); + self.shutdown_parent = shutdown; + self + } + + /// The token the jobs started now are cancelled on. + #[must_use] + pub fn shutdown(&self) -> CancellationToken { + self.shutdown + .read() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone() + } + + /// Cancel the jobs running now (each runs its failure path with reason + /// `shutdown`: FAILED, then its prefix deleted) and give the jobs that + /// start afterwards a fresh token. The server calls this when its + /// serving registration is lost: a migration can take the catalog once + /// the registration is gone, and a job's heartbeats and status writes + /// would otherwise go on beside it. Returns how many jobs were running. + pub fn cancel_jobs(&self) -> usize { + let fresh = self.shutdown_parent.child_token(); + let old = std::mem::replace( + &mut *self + .shutdown + .write() + .unwrap_or_else(std::sync::PoisonError::into_inner), + fresh, + ); + old.cancel(); + self.jobs.len() + } + + /// The same runtime with the backend name and build version the + /// manifests name as their source. + #[must_use] + pub fn with_source(mut self, backend: &str, version: &str) -> Self { + self.source_backend = backend.to_owned(); + self.source_extenddb_version = version.to_owned(); + self + } + + /// The manifest's `source_backend` and `source_extenddb_version`. + #[must_use] + pub fn source(&self) -> (&str, &str) { + (&self.source_backend, &self.source_extenddb_version) + } + + /// Spawn a job task under the tracker [`Self::drain_jobs`] waits on. + /// Public for the server's test of its shutdown sequence, which spawns a + /// task that outlives the drain; the engine's own jobs come through + /// CreateBackup. + pub fn spawn_job(&self, job: F) -> tokio::task::JoinHandle + where + F: std::future::Future + Send + 'static, + F::Output: Send + 'static, + { + self.jobs.spawn(job) + } + + /// Jobs spawned and not yet ended. + #[must_use] + pub fn running_jobs(&self) -> usize { + self.jobs.len() + } + + /// Cancel the jobs' shutdown token and wait up to `bound` for every job + /// to end: a cancelled job runs its failure path (the FAILED write and + /// the deletion of its prefix), which is what sweeps an upload still in + /// flight on S3, where the runtime's own shutdown sends no abort. Job + /// tasks still running at the bound are logged by their count (the + /// tracker counts tasks, from the request's spawn to the task's end, + /// whether or not a row exists for one yet) and left to the runtime's + /// shutdown; a row such a task leaves is CREATING (a task still in its + /// pipeline) or FAILED (one still in its `delete_prefix`), and the + /// reconciler resolves or sweeps it once its heartbeat is stale. Returns + /// the number still running, for the caller's information: the server + /// reads the warn line and nothing else. + pub async fn drain_jobs(&self, bound: Duration) -> usize { + self.shutdown().cancel(); + self.jobs.close(); + let running = self.jobs.len(); + if running == 0 { + return 0; + } + tracing::info!( + "waiting up to {} s for {running} backup job(s) to end", + bound.as_secs() + ); + if tokio::time::timeout(bound, self.jobs.wait()).await.is_err() { + let left = self.jobs.len(); + tracing::warn!( + "{left} backup job task(s) did not end within {} s (tasks spawned by \ + CreateBackup, counted whether or not a row exists for them yet); a row one \ + left CREATING or FAILED is resolved or swept by the reconciler once its \ + heartbeat is stale", + bound.as_secs() + ); + return left; + } + 0 } /// The same runtime with `store` configured. #[must_use] pub fn with_store(mut self, store: ConfiguredStore) -> Self { - self.configured_store = Some(store); + self.configured_store = Some(Arc::new(store)); self } @@ -849,14 +1068,14 @@ impl BackupRuntime { #[must_use] pub fn configured_store(&self) -> Option<&ConfiguredStore> { self.configured_store - .as_ref() + .as_deref() .filter(|store| !store.is_set_aside()) } /// The configured store, taken out of a runtime a command built only to /// reach it. #[must_use] - pub fn into_configured_store(self) -> Option { + pub fn into_configured_store(self) -> Option> { self.configured_store } @@ -922,9 +1141,11 @@ impl BackupRuntime { /// and the problem's category and not its root (no API error names a /// root). Every caller has checked that a store is configured /// (`is_configured_store`, or the store-backed create's own condition). - pub(crate) fn require_healthy_store(&self) -> Result<&ConfiguredStore, DynamoDbError> { + pub(crate) fn require_healthy_store(&self) -> Result<&Arc, DynamoDbError> { let store = self - .configured_store() + .configured_store + .as_ref() + .filter(|store| !store.is_set_aside()) .expect("every caller checks that a store is configured first"); match store.health() { StoreHealth::Healthy => Ok(store), diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index f3c72b532..3277407c7 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -14,21 +14,22 @@ use extenddb_core::limits::LimitsConfig; use extenddb_core::types::{ AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, BillingMode, BillingModeSummary, CreateTableInput, DeleteTableInput, DescribeTableInput, GsiDescription, - GsiInput, IndexInfo, KeySchemaElement, KeyType, ListTablesInput, ListTablesOutput, LsiInput, - OnDemandThroughput, Projection, ProjectionType, ProvisionedThroughput, + GsiInput, IndexInfo, Item, KeySchemaElement, KeyType, ListTablesInput, ListTablesOutput, + LsiInput, OnDemandThroughput, Projection, ProjectionType, ProvisionedThroughput, ProvisionedThroughputDescription, RestoreSummary, ScalarAttributeType, SourceTableDetails, StreamSpecification, StreamViewType, TableDescription, TableKeyInfo, TableStatus, TimeToLiveDescription, TimeToLiveStatus, UpdateTableInput, }; use extenddb_storage::backup_catalog::{ - AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, JobClaim, - RestoreJobRow, RestoreTargetKey, StaleClock, StoreIdentity, StoreKind, TimeBounds, + AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, + JobRow, JobStatus, RestoreJobRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, + StoreKind, TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, RestoreRequest, - TableEngine, + BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, MetadataEngine, + RestoreRequest, SnapshotEngine, TableEngine, TableSnapshot, }; use futures::TryStreamExt; use futures::stream::BoxStream; @@ -67,8 +68,8 @@ struct Fake { listed_for: Mutex>>, /// When set, `restore_source` waits for a permit before answering. source_gate: Option>, - /// The catalog index: rows by ARN. - rows: Mutex>, + /// The catalog index: rows by ARN, shared with the claims handed out. + rows: Arc>>, /// `backup_in_use` answers: the CREATING target's name, by backup ARN. in_use: Mutex>, /// What `start_backup_delete` answers after the row check, by ARN. @@ -106,6 +107,9 @@ struct Fake { /// behind), in seconds; `list_backup_rows_older_than` measures ages on /// the catalog's clock. catalog_clock_offset_secs: Mutex, + /// When set, `check_maintenance_claim` takes this long to answer (a + /// catalog query held behind the pool). + maintenance_delay: Mutex>, /// The `TimeBounds` `list_account_backup_rows` received. bounds_seen: Mutex>, /// The account's tables, by name, as `describe_table` reports them; a @@ -121,6 +125,8 @@ struct Fake { set_deleted: Mutex>, /// Rows `remove_backup_row` removed, by ARN. removed_rows: Mutex>, + /// Every `remove_backup_row` call, by ARN, refused ones included. + remove_attempts: Mutex>, /// When set, `list_backup_rows` fails with it. list_error: Mutex>, /// The stores `newest_available_backup_row` was asked for. @@ -129,6 +135,300 @@ struct Fake { /// which `list_running_restore_jobs` lists and `list_dead_restore_jobs` /// does not. running_jobs: Mutex>, + /// The table snapshots `open_table_snapshot` serves, by table name; a + /// name absent is `TableNotFound`. + snapshots: Mutex>, + /// When set, `open_table_snapshot` fails with it (once). + snapshot_error: Mutex>, + /// Snapshots opened, by table name. + snapshots_opened: Mutex>, + /// What `describe_ttl` answers; disabled when unset. + ttl: Mutex>, + /// The rows `claim_backup` received, as given. + claims: Mutex>, + /// The ARNs `claim_dead_backup` hands over (the backend's liveness test + /// modelled as a set): a CREATING row named here has a dead owner. + dead: Mutex>, + /// The ARNs `claim_dead_backup` took over. + takeovers: Mutex>, + /// Every `set_status` a claim received, in order. + status_writes: Arc>>, + /// What a claim's `heartbeat` does. + heartbeat_answer: Arc>, + /// Releases a beat parked under `HeartbeatAnswer::Park`. + park_release: Arc, + /// What the fake's claims declare about their beat + /// (`JobClaim::heartbeat_parks_before_its_bound`): `true` models SQLite, + /// `false` (the default) the other backends. + parks_before_bound: Arc, + /// Heartbeats the claims received. + heartbeats: Arc, + /// When set, the next `set_status` reports `Unknown` (applied or not as + /// the flag says) instead of its outcome; taken by that write unless + /// `unknown_write_sticky` is set (a connection that fails on every + /// write). + unknown_write: Arc>>, + unknown_write_sticky: Arc, + /// When set, moved into `unknown_write` once a (non-sticky) unknown + /// write has been taken: the write sent after an unknown outcome is + /// itself unknown, in this way. + unknown_write_then: Arc>>, + /// When set, every `set_status` fails as a claim connection that was + /// reset. + status_write_error: Arc, + /// When set, `claim_dead_backup` hands the claim over and then gives + /// the row a newer generation: another reconciler took the row over in + /// the gap before this one's write (item 31's two-reconciler race). + takeover_lost_to_another: std::sync::atomic::AtomicBool, + /// What `sweep_restore_registrations` answers. + swept_registrations: Mutex>, +} + +/// A table snapshot the `Fake` serves. +struct SnapshotFixture { + description: TableDescription, + key_ordered: bool, + items: Vec>, + /// When set, the stream takes one permit per item before yielding it, + /// so a test can hold the job inside the stream. + gate: Option>, +} + +/// One `set_status` a claim received. +#[derive(Debug, Clone)] +struct StatusWrite { + arn: String, + generation: u64, + from: JobStatus, + to: JobStatus, + fields: StatusFields, +} + +/// What a `Fake` claim's heartbeat does. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +enum HeartbeatAnswer { + /// Confirm while the row carries the claim's generation, whatever its + /// status (the backends' `UPDATE ... WHERE owner_generation = ?` has no + /// status clause, so a job's beats during its own `delete_prefix` + /// refresh the FAILED row), refreshing `heartbeat_at`; `Lost` otherwise. + #[default] + Confirm, + /// Confirm whatever the row carries (a connection whose session still + /// holds its lock while the row was rewritten). + AlwaysConfirm, + /// `Lost`, whatever the row. + Lost, + /// Never return. + Stall, + /// An error. + Error, + /// Park outside the bound, as SQLite's beat does behind the engine + /// write lock, until `Fake::park_release` is notified; then answer as + /// `heartbeat_answer` reads at that moment. + Park, +} + +/// How the next `set_status` reports `Unknown`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum UnknownWrite { + /// The write landed and the connection failed after it. + Applied, + /// The write was lost with the connection. + Lost, +} + +/// A `Fake` claim on one row. +struct FakeClaim { + rows: Arc>>, + arn: String, + generation: u64, + status_writes: Arc>>, + heartbeat_answer: Arc>, + park_release: Arc, + parks_before_bound: Arc, + heartbeats: Arc, + unknown_write: Arc>>, + unknown_write_sticky: Arc, + unknown_write_then: Arc>>, + status_write_error: Arc, +} + +impl Fake { + fn claim_on(&self, arn: &str, generation: u64) -> Box { + Box::new(FakeClaim { + rows: Arc::clone(&self.rows), + arn: arn.to_owned(), + generation, + status_writes: Arc::clone(&self.status_writes), + heartbeat_answer: Arc::clone(&self.heartbeat_answer), + park_release: Arc::clone(&self.park_release), + parks_before_bound: Arc::clone(&self.parks_before_bound), + heartbeats: Arc::clone(&self.heartbeats), + unknown_write: Arc::clone(&self.unknown_write), + unknown_write_sticky: Arc::clone(&self.unknown_write_sticky), + unknown_write_then: Arc::clone(&self.unknown_write_then), + status_write_error: Arc::clone(&self.status_write_error), + }) + } +} + +/// `JobStatus` as a backup row carries it. +fn row_status(status: JobStatus) -> Option { + match status { + JobStatus::Creating => Some(BackupStatus::Creating), + JobStatus::Available => Some(BackupStatus::Available), + JobStatus::Failed => Some(BackupStatus::Failed), + JobStatus::Running | JobStatus::Done => None, + } +} + +/// `BackupStatus` as a job record reports it. +fn job_status(status: BackupStatus) -> JobStatus { + match status { + BackupStatus::Creating => JobStatus::Creating, + BackupStatus::Available => JobStatus::Available, + BackupStatus::Failed => JobStatus::Failed, + BackupStatus::Deleting | BackupStatus::Deleted => JobStatus::Done, + } +} + +impl JobClaim for FakeClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxedFuture<'_, Result> { + self.heartbeats + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let answer = *self.heartbeat_answer.lock().unwrap(); + let confirmed = match self.rows.lock().unwrap().get_mut(&self.arn) { + Some(row) + if answer == HeartbeatAnswer::Confirm + && row.owner_generation == self.generation => + { + row.heartbeat_at = seconds_since_epoch(); + true + } + _ => false, + }; + Box::pin(async move { + match answer { + HeartbeatAnswer::Confirm if confirmed => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::AlwaysConfirm => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::Confirm | HeartbeatAnswer::Lost => Ok(Heartbeat::Lost), + HeartbeatAnswer::Stall => futures::future::pending().await, + HeartbeatAnswer::Error => Err(StorageError::Internal( + "the claim connection failed".to_owned(), + )), + HeartbeatAnswer::Park => futures::future::pending().await, + } + }) + } + + /// SQLite's shape: under `Park` the beat waits outside the bound (the + /// engine write lock's queue) and the bound covers the answer alone. + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxedFuture<'_, Option>> { + Box::pin(async move { + if *self.heartbeat_answer.lock().unwrap() == HeartbeatAnswer::Park { + self.park_release.notified().await; + } + tokio::time::timeout(bound, self.heartbeat()).await.ok() + }) + } + + /// What the fake declares about its beat: `true` models SQLite (the + /// park is the write lock's queue), `false` the other backends, where a + /// beat still in flight past its bound is one the process was stopped + /// in (`Park` then stands for the stop). + fn heartbeat_parks_before_its_bound(&self) -> bool { + self.parks_before_bound + .load(std::sync::atomic::Ordering::Acquire) + } + + /// The backends' write: conditioned on the claim's generation and on + /// the row being in `from`; `completed` stamps `completed_at`. + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxedFuture<'a, Result> { + self.status_writes.lock().unwrap().push(StatusWrite { + arn: self.arn.clone(), + generation: self.generation, + from, + to, + fields: fields.clone(), + }); + if self + .status_write_error + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(async move { + Err(StorageError::Internal( + "claim session: connection reset".to_owned(), + )) + }); + } + let unknown = if self + .unknown_write_sticky + .load(std::sync::atomic::Ordering::Acquire) + { + *self.unknown_write.lock().unwrap() + } else { + let taken = self.unknown_write.lock().unwrap().take(); + if taken.is_some() { + *self.unknown_write.lock().unwrap() = + self.unknown_write_then.lock().unwrap().take(); + } + taken + }; + let apply = unknown != Some(UnknownWrite::Lost); + let mut outcome = WriteOutcome::NotMatched; + if apply + && let Some(row) = self.rows.lock().unwrap().get_mut(&self.arn) + && row.owner_generation == self.generation + && Some(row.status) == row_status(from) + && let Some(to_status) = row_status(to) + { + row.status = to_status; + if let Some(size) = fields.size_bytes { + row.size_bytes = size; + } + if let Some(count) = fields.item_count { + row.item_count = count; + } + if let Some(reason) = &fields.failure_reason { + row.failure_reason = Some(reason.clone()); + } + if let Some(details) = &fields.source_table_details_json { + row.source_table_details_json = details.clone(); + } + if fields.completed { + row.completed_at = Some(seconds_since_epoch()); + } + outcome = WriteOutcome::Matched; + } + if unknown.is_some() { + outcome = WriteOutcome::Unknown; + } + Box::pin(async move { Ok(outcome) }) + } + + fn read(&self) -> BoxedFuture<'_, Result> { + let found = self.rows.lock().unwrap().get(&self.arn).map(|row| JobRow { + status: job_status(row.status), + owner_generation: row.owner_generation, + heartbeat_at: row.heartbeat_at, + failure_reason: row.failure_reason.clone(), + }); + Box::pin(async move { + found.ok_or_else(|| StorageError::Internal("the row is gone".to_owned())) + }) + } } impl Fake { @@ -186,6 +486,26 @@ struct MemStore { delay: Mutex>, deletes: Mutex>, puts: Mutex>, + /// When set, `put` fails as a transport error after reading its body; + /// every other call answers. + fail_puts: std::sync::atomic::AtomicBool, + /// When set, `put` fails as a transport error 200 ms after reading its + /// body's first chunk, while the encoder waits for the file's next item + /// (a connection reset mid-upload). + fail_puts_after_first_chunk: std::sync::atomic::AtomicBool, + /// When set, `put` reports one byte more than it stored (a store whose + /// size accounting differs from the bytes written). + misreport_put_size: std::sync::atomic::AtomicBool, + /// When set, `get` of a key under this prefix fails as a transport + /// error; every other call answers (the identity file's read among + /// them, so the store stays healthy). + fail_gets_under: Mutex>, + /// Called with the key of every `put` that stored its object, so a test + /// can act at a point of the job (after the sidecar's put, before the + /// claim re-check). + on_put: Mutex>, + /// Bytes `put` stored, by key, in the order the puts ended. + put_sizes: Mutex>, /// When set, the bytes `put_if_absent` leaves under the key right after /// it stored its own: a server that ignores `If-None-Match` and a second /// writer that won the plain-put race. @@ -202,8 +522,18 @@ struct MemStore { /// followed by a put of this key under the prefix: a writer whose put /// overlapped the deletion. late_put_after_delete: Mutex>, + /// When set, `delete_prefix` takes this long before it answers (a slow + /// store's deletion). + delete_delay: Mutex>, + /// When set, `list` of this prefix fails as a transport error. + fail_lists_of: Mutex>, + /// When set, `delete_prefix` of this prefix fails as a transport error. + fail_deletes_of: Mutex>, } +/// What `MemStore::on_put` runs after a stored `put`, with the object's key. +type OnPut = Box; + /// A transport error with the SDK's shape: a top link naming only the class /// and the cause one link below it. #[derive(Debug)] @@ -303,13 +633,43 @@ impl extenddb_backup_store::BackupStore for MemStore { Box::pin(async move { self.stall().await; self.check()?; + let mut body = body; + if self + .fail_puts_after_first_chunk + .load(std::sync::atomic::Ordering::Acquire) + { + let _first = body.try_next().await?; + tokio::time::sleep(Duration::from_millis(200)).await; + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "connection reset mid-body".to_owned(), + ))), + }); + } let bytes = collect_body(body).await?; + if self.fail_puts.load(std::sync::atomic::Ordering::Acquire) { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("put refused".to_owned()))), + }); + } self.puts.lock().unwrap().push(key.to_owned()); let size = bytes.len() as u64; + self.put_sizes.lock().unwrap().push((key.to_owned(), size)); self.insert(key, &bytes); + if let Some(on_put) = self.on_put.lock().unwrap().as_ref() { + on_put(key); + } + let reported = if self + .misreport_put_size + .load(std::sync::atomic::Ordering::Acquire) + { + size + 1 + } else { + size + }; Ok(extenddb_backup_store::ObjectMeta { key: key.to_owned(), - size, + size: reported, last_modified: None, etag: None, }) @@ -361,6 +721,17 @@ impl extenddb_backup_store::BackupStore for MemStore { Box::pin(async move { self.stall().await; self.check()?; + if self + .fail_gets_under + .lock() + .unwrap() + .as_deref() + .is_some_and(|prefix| key.starts_with(prefix)) + { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("get refused".to_owned()))), + }); + } let bytes = self .objects .lock() @@ -401,6 +772,13 @@ impl extenddb_backup_store::BackupStore for MemStore { if let Err(e) = self.check() { return Box::pin(futures::stream::once(async move { Err(e) })); } + if self.fail_lists_of.lock().unwrap().as_deref() == Some(prefix) { + return Box::pin(futures::stream::once(async move { + Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("list refused".to_owned()))), + }) + })); + } let metas: Vec<_> = self .objects .lock() @@ -426,6 +804,15 @@ impl extenddb_backup_store::BackupStore for MemStore { Box::pin(async move { self.stall().await; self.check()?; + let delay = *self.delete_delay.lock().unwrap(); + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + if self.fail_deletes_of.lock().unwrap().as_deref() == Some(prefix) { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError("delete refused".to_owned()))), + }); + } self.deletes.lock().unwrap().push(prefix.to_owned()); let mut objects = self.objects.lock().unwrap(); let before = objects.len(); @@ -528,11 +915,61 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(id) }) } + /// The backends' insert: the row as given, CREATING under a fresh + /// generation with `heartbeat_at` now; `AlreadyExists` when the ARN has a + /// row. fn claim_backup<'a>( &'a self, - _row: &'a BackupRow, + row: &'a BackupRow, ) -> BoxedFuture<'a, Result, StorageError>> { - unused() + self.claims.lock().unwrap().push(row.clone()); + let mut rows = self.rows.lock().unwrap(); + if rows.contains_key(&row.backup_arn) { + let arn = row.backup_arn.clone(); + return Box::pin(async move { Err(StorageError::AlreadyExists(arn)) }); + } + let generation = new_generation(); + let mut inserted = row.clone(); + inserted.status = BackupStatus::Creating; + inserted.owner_generation = generation; + inserted.heartbeat_at = seconds_since_epoch(); + rows.insert(row.backup_arn.clone(), inserted); + drop(rows); + let claim = self.claim_on(&row.backup_arn, generation); + Box::pin(async move { Ok(claim) }) + } + + /// The takeover: a CREATING row whose ARN is in the `dead` set gets a + /// fresh generation, `heartbeat_at` untouched; anything else is `None`. + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + _stale_after: Duration, + ) -> BoxedFuture<'a, Result>, StorageError>> { + let dead = self.dead.lock().unwrap().contains(&key.backup_arn); + let mut rows = self.rows.lock().unwrap(); + let claim = match rows.get_mut(&key.backup_arn) { + Some(row) if dead && row.status == BackupStatus::Creating => { + row.owner_generation = new_generation(); + let generation = row.owner_generation; + if self + .takeover_lost_to_another + .load(std::sync::atomic::Ordering::Acquire) + { + row.owner_generation = new_generation(); + } + drop(rows); + self.takeovers.lock().unwrap().push(key.backup_arn.clone()); + Some(self.claim_on(&key.backup_arn, generation)) + } + _ => None, + }; + Box::pin(async move { Ok(claim) }) + } + + fn sweep_restore_registrations(&self) -> BoxedFuture<'_, Result, StorageError>> { + let swept = self.swept_registrations.lock().unwrap().clone(); + Box::pin(async move { Ok(swept) }) } /// The backend's listing: rows of the store (kind, root and id), of the @@ -685,6 +1122,10 @@ impl BackupCatalog for Fake { &'a self, key: &'a BackupKey, ) -> BoxedFuture<'a, Result> { + self.remove_attempts + .lock() + .unwrap() + .push(key.backup_arn.clone()); let mut rows = self.rows.lock().unwrap(); let removed = match rows.get(&key.backup_arn) { Some(row) if matches!(row.status, BackupStatus::Deleted | BackupStatus::Failed) => { @@ -868,7 +1309,11 @@ impl BackupCatalog for Fake { None => *self.maintenance_held.lock().unwrap(), }; let error = self.maintenance_error.lock().unwrap().clone(); + let delay = *self.maintenance_delay.lock().unwrap(); Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } match error { Some(e) => Err(StorageError::Internal(e)), None => Ok(held), @@ -877,6 +1322,168 @@ impl BackupCatalog for Fake { } } +impl SnapshotEngine for Fake { + /// The fixture for the table, or `TableNotFound`; the description and + /// the items are the fixture's, in its order, through the gate when it + /// has one. + fn open_table_snapshot( + &self, + _account_id: &str, + table_name: &str, + ) -> BoxedFuture<'_, Result> { + self.snapshots_opened + .lock() + .unwrap() + .push(table_name.to_owned()); + if let Some(e) = self.snapshot_error.lock().unwrap().take() { + return Box::pin(async move { Err(e) }); + } + let mut snapshots = self.snapshots.lock().unwrap(); + let Some(fixture) = snapshots.get_mut(table_name) else { + let name = table_name.to_owned(); + return Box::pin(async move { + Err(StorageError::TableNotFound(format!( + "Table not found: {name}" + ))) + }); + }; + let items = std::mem::take(&mut fixture.items); + let gate = fixture.gate.clone(); + let snapshot = TableSnapshot { + description: fixture.description.clone(), + info: extenddb_core::backup_format::SnapshotDescriptor { + kind: "test".to_owned(), + marker: String::new(), + }, + key_ordered: fixture.key_ordered, + items: Box::pin(futures::stream::unfold( + (items.into_iter(), gate), + |(mut items, gate)| async move { + let next = items.next()?; + if let Some(gate) = &gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + Some((next, (items, gate))) + }, + )), + }; + Box::pin(async move { Ok(snapshot) }) + } +} + +impl MetadataEngine for Fake { + fn describe_ttl( + &self, + _account_id: &str, + _table_name: &str, + ) -> BoxedFuture<'_, Result> { + let ttl = self + .ttl + .lock() + .unwrap() + .clone() + .unwrap_or(TimeToLiveDescription { + time_to_live_status: TimeToLiveStatus::Disabled, + attribute_name: None, + }); + Box::pin(async move { Ok(ttl) }) + } + + fn update_ttl( + &self, + _: &str, + _: &str, + _: &str, + _: bool, + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn tag_resource( + &self, + _: &str, + _: &[extenddb_core::types::Tag], + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn untag_resource(&self, _: &str, _: &[String]) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn list_tags( + &self, + _: &str, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn tables_with_ttl( + &self, + _: &str, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn all_tables_with_ttl( + &self, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn all_tables_with_ttl_index_ready( + &self, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn create_ttl_index( + &self, + _: &str, + _: &str, + _: &str, + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn drop_ttl_index( + &self, + _: &str, + _: &str, + _: &str, + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn find_expired_items_indexed( + &self, + _: &str, + _: &str, + _: &str, + _: usize, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn refresh_table_size(&self, _: &str, _: &str) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn list_active_table_names( + &self, + _: &str, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn all_active_tables(&self) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } +} + impl TableEngine for Fake { fn create_table( &self, @@ -5215,13 +5822,14 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { }; let full = BackupRuntime::new(1).with_permit_wait(Duration::ZERO); let _held = full.take_permit().await.unwrap(); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let err = create_backup( &fake, &full, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Err(StorageError::TableNotFound("Music".to_owned())), @@ -5241,6 +5849,7 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { &full, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(creating), @@ -5283,6 +5892,7 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { &full, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(with_vectors), @@ -5297,6 +5907,7 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { &full, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active.clone()), @@ -5318,6 +5929,7 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { &free, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active), @@ -5341,7 +5953,7 @@ async fn create_backup_checks_the_table_then_takes_the_permit_last() { /// no store prefix. The store-prefix rules belong to the store-backed writer. #[tokio::test] async fn create_backup_accepts_a_name_the_store_prefix_refuses() { - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); for name in ["x".repeat(256), "x".repeat(1024), "...".to_owned()] { let runtime = BackupRuntime::new(1); let desc = TableDescription { @@ -5349,9 +5961,18 @@ async fn create_backup_accepts_a_name_the_store_prefix_refuses() { table_status: TableStatus::Active, ..Default::default() }; - create_backup(&fake, &runtime, ACCOUNT, REGION, &name, "nightly", Ok(desc)) - .await - .unwrap_or_else(|e| panic!("{name:?}: {e:?}")); + create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &LimitsConfig::default(), + &name, + "nightly", + Ok(desc), + ) + .await + .unwrap_or_else(|e| panic!("{name:?}: {e:?}")); assert_eq!(runtime.free_slots(), 0, "the backend holds the slot"); } let created: Vec = fake @@ -5367,6 +5988,51 @@ async fn create_backup_accepts_a_name_the_store_prefix_refuses() { ); } +/// The store-backed create applies the store-prefix rules the in-catalog one +/// does not (`check_store_prefix_fits`): a name over 255 bytes or made only +/// of dots fits no prefix, so it is refused before a snapshot is opened or a +/// row is written, naming the rule (the length before the ARN is formed, so +/// the limit is named whatever the length). +#[tokio::test] +async fn a_store_backed_create_refuses_a_name_no_store_prefix_can_carry() { + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + let runtime = job_runtime(&mem); + for (name, needle) in [ + ("x".repeat(256), "256 bytes"), + ("x".repeat(1024), "1024 bytes"), + ("...".to_owned(), "only dots"), + ] { + let desc = TableDescription { + table_name: name.clone(), + table_status: TableStatus::Active, + ..Default::default() + }; + let msg = validation_message( + create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &LimitsConfig::default(), + &name, + "nightly", + Ok(desc), + ) + .await + .unwrap_err(), + ); + assert!( + msg.contains("cannot be backed up") && msg.contains(needle), + "{name:?}: {msg}" + ); + assert!(!msg.contains("not a backup ARN"), "{msg}"); + } + assert!(fake.rows.lock().unwrap().is_empty(), "no row was claimed"); + assert!(mem.keys().is_empty(), "nothing was written to the store"); + assert_eq!(runtime.free_slots(), 4, "no slot was taken"); +} + /// Under a `server.region` outside `[a-z0-9-]` no backup ARN can be formed /// for any table, so CreateBackup is refused naming `server.region`, not the /// table name, before the permit (the runtime's one permit is held). The @@ -5379,7 +6045,7 @@ async fn create_backup_accepts_a_name_the_store_prefix_refuses() { async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting() { let runtime = BackupRuntime::new(1).with_permit_wait(Duration::ZERO); let _held = runtime.take_permit().await.unwrap(); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let desc = TableDescription { table_name: "Music".to_owned(), table_status: TableStatus::Active, @@ -5391,6 +6057,7 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting &runtime, ACCOUNT, "LOCAL", + &LimitsConfig::default(), "Music", "nightly", Ok(desc), @@ -5417,7 +6084,7 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting .unwrap() .insert(stored.clone(), row(&stored, BackupStatus::Available)); let spellings = backup_arn_field(&json!({ "BackupArn": stored }), ACCOUNT, "LOCAL").unwrap(); - let desc = describe_backup(&fake, ACCOUNT, "LOCAL", &spellings) + let desc = describe_backup(&*fake, ACCOUNT, "LOCAL", &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_arn, stored); @@ -5446,7 +6113,7 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting /// the claim is gone both run. #[tokio::test] async fn jobs_are_refused_while_a_maintenance_claim_is_held() { - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); + let fake = Arc::new(Fake::with_source(source(Some(definition("PROVISIONED"))))); *fake.maintenance_held.lock().unwrap() = true; let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7)); let err = create_backup( @@ -5454,6 +6121,7 @@ async fn jobs_are_refused_while_a_maintenance_claim_is_held() { &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), @@ -5472,6 +6140,7 @@ async fn jobs_are_refused_while_a_maintenance_claim_is_held() { &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Err(StorageError::TableNotFound("Music".to_owned())), @@ -5505,6 +6174,7 @@ async fn jobs_are_refused_while_a_maintenance_claim_is_held() { &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), @@ -5530,6 +6200,7 @@ async fn jobs_are_refused_while_a_maintenance_claim_is_held() { &full, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), @@ -6140,9 +6811,11 @@ async fn the_wait_reads_a_live_owner_on_the_catalogs_clock() { /// window (`backup purge` on SQLite) takes the held branch without reading /// the claim: the command's claim was heartbeated a millisecond ago, which /// a zero window reads as stale, so a cycle that read it would take the -/// branch for a free claim. The same cycle without the holder's state reads -/// the claim through the window and finds it free; that is the read the -/// holder's state replaces. +/// branch for a free claim. The held branch skips the deleting duties the +/// command must not race: a stale FAILED row keeps its prefix and the +/// registrations sweep does not run. The same cycle without the holder's +/// state reads the claim through the window, finds it free and runs them; +/// that is the read the holder's state replaces. #[tokio::test] async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_window() { use extenddb_storage::backup_catalog::RestoreKind; @@ -6150,6 +6823,14 @@ async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_wind mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}"); let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]); + let stale_failed = arn_n(1); + let mut r = row(&stale_failed, BackupStatus::Failed); + r.heartbeat_at = seconds_since_epoch() - 1000.0; + r.completed_at = Some(seconds_since_epoch()); + fake.rows.lock().unwrap().insert(stale_failed.clone(), r); + let prefix = BackupArn::parse(&stale_failed).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + fake.swept_registrations.lock().unwrap().push(arn_n(9)); *fake.maintenance_beat_age.lock().unwrap() = Some(Duration::from_millis(1)); *fake.target_still_creating.lock().unwrap() = true; *fake.record_still_running.lock().unwrap() = true; @@ -6166,6 +6847,15 @@ async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_wind ) .await .unwrap(); + assert!( + report.failed_prefixes_deleted.is_empty() && objects_under(&mem, &stale_failed).len() == 1, + "the holder's cycle deleted a FAILED row's prefix: {:?}", + report.failed_prefixes_deleted + ); + assert!( + report.registrations_swept.is_empty(), + "the holder's cycle swept the registrations" + ); assert!( report.maintenance_held, "the holder's cycle took the branch for a free claim" @@ -6196,6 +6886,12 @@ async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_wind [Duration::ZERO], "a default state reads the claim through the window it is given" ); + assert_eq!( + report.failed_prefixes_deleted, + std::slice::from_ref(&stale_failed) + ); + assert!(objects_under(&mem, &stale_failed).is_empty()); + assert_eq!(report.registrations_swept, [arn_n(9)]); } /// The worker runs a cycle at once (observed through the maintenance check, @@ -6498,14 +7194,97 @@ fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake { fake } -/// An empty root and a catalog with no rows for it: startup creates the file -/// with this catalog's id and a fresh store id, re-reads it and uses that id. +/// A restore of a store-backed row checks the target name once the row is +/// known to exist and AVAILABLE, before the backup's in-use rule and before +/// this release's refusal of the store-backed restore (the service's order: +/// a restore of a backup into its own CREATING target is +/// `TableInUseException`, measured): an existing table of the name is +/// `TableAlreadyExistsException`, a CREATING restore target +/// `TableInUseException` naming its source backup, and a free name reaches +/// the refusal naming the store. A row that is not AVAILABLE answers for +/// itself before the name is read. #[tokio::test] -async fn establish_creates_the_identity_file_on_an_empty_root() { - let mem = MemStore::default(); - let fake = Fake::default(); - let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") - .await +async fn a_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusal() { + let mem = Arc::new(MemStore::default()); + let runtime = runtime_on(&mem, store()); + let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Available)]); + + fake.tables.lock().unwrap().insert( + "MusicRestored".to_owned(), + TableDescription { + table_name: "MusicRestored".to_owned(), + table_status: TableStatus::Active, + ..Default::default() + }, + ); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableAlreadyExistsException(_)), + "{err:?}" + ); + + fake.tables.lock().unwrap().insert( + "MusicRestored".to_owned(), + TableDescription { + table_name: "MusicRestored".to_owned(), + table_status: TableStatus::Creating, + restore_summary: Some(RestoreSummary { + source_backup_arn: Some(arn(ACCOUNT)), + source_table_arn: None, + restore_date_time: 0.0, + restore_in_progress: true, + }), + ..Default::default() + }, + ); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableInUseException(ref m) if m.contains(&arn(ACCOUNT))), + "{err:?}" + ); + + fake.tables.lock().unwrap().remove("MusicRestored"); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::ValidationException(ref m) + if m.contains("is in a backup store; this release restores in-catalog backups only")), + "{err:?}" + ); + + // A row that is not AVAILABLE: its own answer, the name unread. + let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleted)]); + fake.tables.lock().unwrap().insert( + "MusicRestored".to_owned(), + TableDescription { + table_name: "MusicRestored".to_owned(), + table_status: TableStatus::Active, + ..Default::default() + }, + ); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupNotFoundException(_)), + "{err:?}" + ); + assert!(fake.table_reads.lock().unwrap().is_empty()); +} + +/// An empty root and a catalog with no rows for it: startup creates the file +/// with this catalog's id and a fresh store id, re-reads it and uses that id. +#[tokio::test] +async fn establish_creates_the_identity_file_on_an_empty_root() { + let mem = MemStore::default(); + let fake = Fake::default(); + let identity = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + .await .unwrap(); assert_eq!(identity.store_kind, StoreKind::Filesystem); assert_eq!(identity.store_root, "/r"); @@ -7352,18 +8131,17 @@ async fn a_store_checked_after_startup_is_checking_until_the_first_check() { assert_eq!(runtime.store_health().unwrap().status, "healthy"); } -/// The refusals while the configured store is unhealthy: DeleteBackup of a -/// row of the store, and the restore of such a row, each a -/// ValidationException naming the store's kind and the problem's category -/// and never the root; the legacy reader is not reached, and the restore's -/// refusal comes before the permit (with no slot free it is still the -/// store's, not LimitExceeded). CreateBackup is not refused: the in-catalog -/// create, the one this release runs, touches no store, so it runs with an -/// unhealthy store as it does with none (a server that never configured -/// backups keeps CreateBackup through a failed check of the default root), -/// and on a backend bound to the in-catalog create. +/// An unhealthy store refuses the operations that touch it, before a slot is +/// taken and naming the store's kind and the problem's category and never +/// its root or the cause: CreateBackup (store-backed on this runtime), +/// DeleteBackup of a row of the store, and the restore of one (with no slot +/// free it is still the store's refusal, not LimitExceeded). A backend +/// bound to the in-catalog create is not refused: its create touches no +/// store. A row of another store is refused as another store's, whatever +/// the configured one's health. #[tokio::test] -async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_create() { +async fn an_unhealthy_store_refuses_create_delete_and_restore_unless_bound_to_the_in_catalog_create() + { let mem = Arc::new(MemStore::default()); let unhealthy = || { runtime_unhealthy( @@ -7381,53 +8159,47 @@ async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_cr assert!(!m.contains("connect"), "{m}"); }; - // CreateBackup runs the in-catalog create, taking its slot. - let fake = Fake::default(); - for runtime in [unhealthy(), unhealthy().with_legacy_create_only(true)] { - let before = fake.creates.lock().unwrap().len(); - create_backup( - &fake, - &runtime, - ACCOUNT, - REGION, - "Music", - "nightly", - Ok(active_table("Music")), - ) - .await - .unwrap(); - assert_eq!(fake.creates.lock().unwrap().len(), before + 1); - assert_eq!(runtime.free_slots(), 3, "the create holds its slot"); - fake.permits.lock().unwrap().clear(); - } - // With no slot free the create is LimitExceeded, the store's health - // notwithstanding. - let full = BackupRuntime::new(0).with_store(ConfiguredStore::new( - store().store_kind, - store().store_root, - Arc::clone(&mem) as Arc, - None, - Err(StoreProblem::Transport("refused".to_owned())), - )); + // The store-backed create is refused before the permit and the open. + let fake = Arc::new(Fake::default()); + let runtime = unhealthy(); let err = create_backup( &fake, - &full, + &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), ) .await .unwrap_err(); - assert!( - matches!(err, DynamoDbError::LimitExceededException(_)), - "{err:?}" - ); + check(err); + assert_eq!(runtime.free_slots(), 4, "no slot was taken"); + assert!(fake.snapshots_opened.lock().unwrap().is_empty()); + assert!(fake.creates.lock().unwrap().is_empty()); + + // A backend bound to the in-catalog create runs it, taking its slot. + let runtime = unhealthy().with_legacy_create_only(true); + create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &LimitsConfig::default(), + "Music", + "nightly", + Ok(active_table("Music")), + ) + .await + .unwrap(); + assert_eq!(fake.creates.lock().unwrap().len(), 1); + assert_eq!(runtime.free_slots(), 3, "the create holds its slot"); + fake.permits.lock().unwrap().clear(); // DeleteBackup of a row of the store. - let fake = fake_with_row(BackupStatus::Available); - let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let fake = Arc::new(fake_with_row(BackupStatus::Available)); + let err = delete_backup(&*fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); check(err); @@ -7457,9 +8229,9 @@ async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_cr // of the configured one. let mut other = row(&arn(ACCOUNT), BackupStatus::Available); other.store.store_root = "/elsewhere".to_owned(); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); fake.rows.lock().unwrap().insert(arn(ACCOUNT), other); - let err = delete_backup(&fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = delete_backup(&*fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -7468,25 +8240,43 @@ async fn an_unhealthy_store_refuses_delete_and_restore_but_not_the_in_catalog_cr ); } -/// With a healthy store configured, CreateBackup takes the in-catalog path -/// (this release has no store-backed create) and logs that the store is -/// configured; the slot is taken and the reader's create runs. +/// With a healthy store configured, CreateBackup is store-backed: the +/// legacy reader is not asked, the table's snapshot is opened and the answer +/// is CREATING; a table with vector indexes is backed up on this path (the +/// refusal belongs to the in-catalog create alone). #[tokio::test] -async fn a_healthy_store_leaves_create_backup_on_the_in_catalog_path() { - let fake = Fake::default(); - let runtime = runtime_with_store(); - create_backup( +async fn a_healthy_store_makes_create_backup_store_backed() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut table = music_table(); + table.vector_indexes = Some(vec![vector_index()]); + snapshot_fixture(&fake, &table, Vec::new(), true, None); + let runtime = job_runtime(&mem); + let details = create_backup( &fake, &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", - Ok(active_table("Music")), + Ok(table), ) .await .unwrap(); - assert_eq!(fake.creates.lock().unwrap().len(), 1); + assert_eq!(details.backup_status, "CREATING"); + assert!( + fake.creates.lock().unwrap().is_empty(), + "the legacy reader was not asked" + ); + assert_eq!(fake.snapshots_opened.lock().unwrap().as_slice(), ["Music"]); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!( + row.source_table_details_json["vector_indexes"][0]["IndexName"], + "v" + ); } /// DeleteBackup of a row of the configured store deletes the manifest first @@ -7929,7 +8719,9 @@ mod review_round_2 { /// their own ValidationException, not LimitExceededException. #[tokio::test] async fn the_maintenance_refusal_comes_after_the_requests_own_checks() { - let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); + let fake = Arc::new(Fake::with_source(source(Some(definition( + "PAY_PER_REQUEST", + ))))); *fake.maintenance_held.lock().unwrap() = true; let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7)); @@ -7938,6 +8730,7 @@ mod review_round_2 { &runtime, ACCOUNT, "US_EAST_1", + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), @@ -7973,6 +8766,7 @@ mod review_round_2 { &runtime, ACCOUNT, REGION, + &LimitsConfig::default(), "Music", "nightly", Ok(active_table("Music")), @@ -8415,6 +9209,2948 @@ mod review_round_2 { } } +// --- the store-backed create and its job ------------------------------------ + +/// The `Music` table as a snapshot describes it: composite key, one GSI, +/// provisioned. +fn music_table() -> TableDescription { + TableDescription { + table_name: "Music".to_owned(), + key_schema: vec![ + key("Artist", KeyType::Hash), + key("SongTitle", KeyType::Range), + ], + attribute_definitions: vec![attr("Artist"), attr("SongTitle"), attr("Genre")], + table_status: TableStatus::Active, + creation_date_time: 1_757_400_000.0, + table_size_bytes: 123_456, + item_count: 100_000, + table_arn: source_arn(), + table_id: "12345a12-abcd-123a-ab12-1234abc12345".to_owned(), + provisioned_throughput: ProvisionedThroughputDescription { + read_capacity_units: 5, + write_capacity_units: 5, + ..Default::default() + }, + billing_mode_summary: Some(BillingModeSummary { + billing_mode: BillingMode::Provisioned, + last_update_to_pay_per_request_date_time: None, + }), + global_secondary_indexes: Some(vec![GsiDescription { + index_name: "GenreIndex".to_owned(), + key_schema: vec![key("Genre", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + index_status: "ACTIVE".to_owned(), + provisioned_throughput: Some(ProvisionedThroughputDescription { + read_capacity_units: 1, + write_capacity_units: 1, + ..Default::default() + }), + index_size_bytes: 0, + item_count: 0, + index_arn: String::new(), + }]), + ..Default::default() + } +} + +/// A vector index description for a table with vector support. +fn vector_index() -> extenddb_core::types::VectorIndexDescription { + serde_json::from_value(json!({ + "IndexName": "v", + "VectorAttribute": { "AttributeName": "emb" }, + "Dimensions": 3, + "DistanceFunction": "COSINE", + "IndexStatus": "ACTIVE", + "IndexSizeBytes": 0, + "ItemCount": 0, + "IndexArn": "" + })) + .unwrap() +} + +/// An item of the `Music` table with a `Year` number. +fn song(artist: &str, title: &str) -> Item { + use extenddb_core::types::AttributeValue; + Item::from([ + ("Artist".to_owned(), AttributeValue::S(artist.to_owned())), + ("SongTitle".to_owned(), AttributeValue::S(title.to_owned())), + ("Year".to_owned(), AttributeValue::N("1999".to_owned())), + ]) +} + +/// `song` with a `Year` the wire refuses (#618's shape). +fn song_with_bad_number(artist: &str, title: &str, year: &str) -> Item { + use extenddb_core::types::AttributeValue; + let mut item = song(artist, title); + item.insert("Year".to_owned(), AttributeValue::N(year.to_owned())); + item +} + +/// `n` songs in key order. +fn songs(n: usize) -> Vec { + (0..n) + .map(|i| song(&format!("artist-{i:05}"), &format!("title-{i:05}"))) + .collect() +} + +/// Make `fake` serve a snapshot of `table` yielding `items` (through `gate` +/// when given). +fn snapshot_fixture( + fake: &Fake, + table: &TableDescription, + items: Vec, + key_ordered: bool, + gate: Option>, +) { + fake.snapshots.lock().unwrap().insert( + table.table_name.clone(), + SnapshotFixture { + description: table.clone(), + key_ordered, + items: items.into_iter().map(Ok).collect(), + gate, + }, + ); +} + +/// A gate with no permit: the stream yields nothing until the test adds +/// some. +fn closed_gate() -> Arc { + Arc::new(tokio::sync::Semaphore::new(0)) +} + +/// The settings the job tests run with: small files, a short heartbeat. +fn job_settings() -> JobSettings { + JobSettings { + data_file_target_bytes: 64 * 1024 * 1024, + heartbeat_interval: Duration::from_millis(50), + stale_job_after: Duration::from_secs(300), + failed_row_retention: Duration::from_secs(7 * 24 * 3600), + } +} + +/// A healthy runtime on `mem` with the job settings above and a collector. +fn job_runtime(mem: &Arc) -> BackupRuntime { + runtime_on(mem, store()) + .with_job_settings(job_settings()) + .with_metrics(Arc::new(extenddb_core::metrics::MetricsCollector::new())) +} + +/// CreateBackup of `Music` through `runtime`. +async fn create_music_backup( + fake: &Arc, + runtime: &BackupRuntime, +) -> Result { + create_backup( + fake, + runtime, + ACCOUNT, + REGION, + &LimitsConfig::default(), + "Music", + "nightly", + Ok(music_table()), + ) + .await +} + +/// Wait until no job task of `runtime` runs, failing after 30 s: a hang +/// guard for jobs that take well under a second alone. A test whose job +/// takes seconds calls [`wait_for_jobs_within`] with a bound at least ten +/// times its time alone on an idle core, so a loaded host cannot reach it. +async fn wait_for_jobs(runtime: &BackupRuntime) { + wait_for_jobs_within(runtime, Duration::from_secs(30)).await; +} + +/// Wait until no job task of `runtime` runs, failing after `bound`. +async fn wait_for_jobs_within(runtime: &BackupRuntime, bound: Duration) { + let deadline = tokio::time::Instant::now() + bound; + while runtime.running_jobs() > 0 { + assert!( + tokio::time::Instant::now() < deadline, + "a job did not end within {bound:?}" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// The objects under the backup's prefix, by key. +fn objects_under(mem: &MemStore, arn: &str) -> Vec { + let prefix = BackupArn::parse(arn).unwrap().store_prefix(); + mem.keys() + .into_iter() + .filter(|k| k.starts_with(&prefix)) + .collect() +} + +/// The stored manifest of `arn`, parsed. +fn stored_manifest(mem: &MemStore, arn: &str) -> extenddb_core::backup_format::ExtenddbManifest { + let bytes = mem.objects.lock().unwrap()[&manifest_key(arn)].clone(); + extenddb_core::backup_format::ExtenddbManifest::from_json_bytes(&bytes).unwrap() +} + +/// The status writes the fake's claims received, as `(to, failure_reason)`. +fn writes(fake: &Fake) -> Vec<(JobStatus, Option)> { + fake.status_writes + .lock() + .unwrap() + .iter() + .map(|w| (w.to, w.fields.failure_reason.clone())) + .collect() +} + +/// The `BackupJobs` outcomes the collector drained, with their count. +fn job_outcomes(runtime: &BackupRuntime) -> Vec<(String, f64)> { + use extenddb_core::metrics::MetricName; + let mut outcomes: Vec<(String, f64)> = runtime + .metrics() + .unwrap() + .drain(Duration::ZERO) + .into_iter() + .filter(|b| b.metric == MetricName::BackupJobs) + .map(|b| (b.operation, b.sum)) + .collect(); + outcomes.sort_by(|a, b| a.0.cmp(&b.0)); + outcomes +} + +/// The request's order and the job's happy path. The checks before the +/// permit: a missing table is TableNotFound with every slot held and the +/// store untouched. Then the detached task: the snapshot is opened, the row +/// is inserted CREATING under the claim with the manifest's `source_table` +/// (the TTL read beside the open), the maintenance claim is read, and the +/// request is answered CREATING with the id's instant. The job then writes +/// the data file, `manifest-files.json`, `manifest-summary.json`, the +/// sidecar and the manifest, in that order and nothing else, and moves the +/// row to AVAILABLE with the manifest's size, the stream's count and the +/// manifest's `source_table`; the ARN's id, the row's `created_at`, the +/// response's time and the manifest's `backup_creation_date_time` are one +/// instant; the data file decodes back to the items; the slot is freed; the +/// metrics name the outcome, and `BackupSnapshotAge` was recorded by the +/// beats the job's three periods inside the stream gave it. +#[tokio::test] +async fn a_store_backed_create_answers_creating_and_its_job_writes_the_backup() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let items = songs(5); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + items.clone(), + true, + Some(Arc::clone(&gate)), + ); + *fake.ttl.lock().unwrap() = Some(TimeToLiveDescription { + time_to_live_status: TimeToLiveStatus::Enabled, + attribute_name: Some("expires_at".to_owned()), + }); + let full = job_runtime(&mem); + let mut _held = Vec::new(); + for _ in 0..4 { + _held.push(full.take_permit().await.unwrap()); + } + let err = create_backup( + &fake, + &full, + ACCOUNT, + REGION, + &LimitsConfig::default(), + "Music", + "nightly", + Err(StorageError::TableNotFound("Music".to_owned())), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableNotFoundException(_)), + "{err:?}" + ); + let err = create_music_backup(&fake, &full).await.unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(_)), + "{err:?}" + ); + assert!( + fake.snapshots_opened.lock().unwrap().is_empty(), + "nothing was opened" + ); + drop(_held); + + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + assert_eq!(details.backup_status, "CREATING"); + assert_eq!(details.backup_type, "USER"); + assert_eq!(details.backup_name, "nightly"); + assert_eq!(details.backup_size_bytes, 0); + let arn = BackupArn::parse(&details.backup_arn).unwrap(); + assert_eq!( + extenddb_storage::util::backup_creation_date_time(&details.backup_arn), + Some(details.backup_creation_date_time), + "the response's time is the id's instant" + ); + let claimed = fake.claims.lock().unwrap().clone(); + assert_eq!(claimed.len(), 1); + assert_eq!(claimed[0].status, BackupStatus::Creating); + assert_eq!(claimed[0].store, store()); + assert_eq!(claimed[0].created_at, details.backup_creation_date_time); + assert_eq!(claimed[0].item_count, 0); + assert_eq!(claimed[0].source_table_details_json["item_count"], 100_000); + assert_eq!( + claimed[0].source_table_details_json["time_to_live"]["AttributeName"], + "expires_at" + ); + assert_eq!( + fake.maintenance_checks.lock().unwrap().as_slice(), + [Duration::from_secs(300), Duration::from_secs(300)], + "the maintenance claim is read before the permit, by the refused request too" + ); + + // Three heartbeat periods inside the stream, so the timer beats before + // the job ends. + tokio::time::sleep(3 * job_settings().heartbeat_interval).await; + gate.add_permits(items.len()); + wait_for_jobs(&runtime).await; + assert_eq!(runtime.free_slots(), 4, "the job freed its slot"); + let prefix = arn.store_prefix(); + assert_eq!( + mem.puts.lock().unwrap().as_slice(), + [ + format!("{prefix}data/000001.json.gz"), + format!("{prefix}manifest-files.json"), + format!("{prefix}manifest-summary.json"), + format!("{prefix}extenddb-manifest.sha256"), + format!("{prefix}extenddb-manifest.json"), + ], + "the write order" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + let manifest = stored_manifest(&mem, &details.backup_arn); + assert_eq!(manifest.backup_arn, details.backup_arn); + assert_eq!( + manifest.backup_creation_date_time, + details.backup_creation_date_time + ); + assert_eq!(manifest.source_table.item_count, 5, "the stream's count"); + assert_eq!( + manifest.source_table.table_size_bytes, + items + .iter() + .map(|item| extenddb_core::types::item_size_bytes(item) as i64) + .sum::(), + "the stream's size, not the stats pass's 123,456" + ); + assert_eq!(manifest.source_backend, "test"); + assert_eq!( + manifest.item_order, + extenddb_core::backup_format::ItemOrder::KeyOrdered + ); + assert_eq!(manifest.data_files.len(), 1); + assert!(manifest.data_files[0].key_range.is_some()); + let file = mem.objects.lock().unwrap()[&format!("{prefix}data/000001.json.gz")].clone(); + assert_eq!(manifest.data_files[0].bytes, file.len() as u64); + let mut reader = manifest.reader(&manifest.data_files[0], &file[..]).unwrap(); + let mut decoded = Vec::new(); + while let Some(item) = reader.next_item().unwrap() { + decoded.push(item); + } + assert_eq!(decoded, items); + let manifest_bytes = + mem.objects.lock().unwrap()[&format!("{prefix}extenddb-manifest.json")].clone(); + let sidecar = mem.objects.lock().unwrap()[&format!("{prefix}extenddb-manifest.sha256")].clone(); + extenddb_core::backup_format::verify_manifest_checksum(&manifest_bytes, &sidecar).unwrap(); + + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!(row.size_bytes, manifest.backup_size_bytes); + assert_eq!(row.item_count, 5); + assert_eq!( + row.source_table_details_json, + serde_json::to_value(&manifest.source_table).unwrap() + ); + assert!(row.completed_at.is_some()); + assert_eq!(writes(&fake), [(JobStatus::Available, None)]); + let write = fake.status_writes.lock().unwrap()[0].clone(); + assert_eq!( + (write.arn.as_str(), write.from, write.generation), + ( + details.backup_arn.as_str(), + JobStatus::Creating, + row.owner_generation + ), + "every status write is conditioned on CREATING under the claim's generation" + ); + assert!(fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) >= 1); + + use extenddb_core::metrics::MetricName; + let drained = runtime.metrics().unwrap().drain(Duration::ZERO); + let find = |metric: MetricName| { + drained + .iter() + .filter(|b| b.metric == metric) + .map(|b| (b.operation.clone(), b.sum)) + .collect::>() + }; + assert_eq!( + find(MetricName::BackupJobs), + [("succeeded".to_owned(), 1.0)] + ); + assert_eq!(find(MetricName::BackupJobDuration).len(), 1); + let total: u64 = mem.put_sizes.lock().unwrap().iter().map(|(_, n)| n).sum(); + assert_eq!( + find(MetricName::BackupBytesUploaded), + [(String::new(), total as f64)] + ); + assert_eq!( + find(MetricName::WorkerLastSuccess) + .iter() + .map(|(op, _)| op.as_str()) + .collect::>(), + ["BackupJob"] + ); + assert!(find(MetricName::WorkerErrorCount).is_empty()); + let beats = fake.heartbeats.load(std::sync::atomic::Ordering::Acquire); + let ages: usize = drained + .iter() + .filter(|b| b.metric == MetricName::BackupSnapshotAge) + .map(|b| b.count as usize) + .sum(); + assert!( + ages >= 1 && ages < beats, + "the snapshot's age is recorded at each confirmed timer beat and not at the re-check \ + before the manifest: {ages} recorded, {beats} beats" + ); +} + +/// The maintenance claim is checked before the permit, as on the +/// in-catalog path: held, the request is LimitExceeded naming a backup +/// maintenance command and the claim, no snapshot is opened, no row is +/// inserted (a FAILED `maintenance` row carried the insert's heartbeat and +/// `backup purge`'s wait counted it as a live job), nothing reaches the +/// store and every slot is free. A claim that cannot be read answers the +/// read's error, with no row either. +#[tokio::test] +async fn a_store_backed_create_under_a_maintenance_claim_is_refused_before_its_row() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + *fake.maintenance_held.lock().unwrap() = true; + let runtime = job_runtime(&mem); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + matches!(&err, DynamoDbError::LimitExceededException(m) + if m.contains("A backup maintenance command holds") && !m.contains("purge")), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert!(fake.snapshots_opened.lock().unwrap().is_empty()); + assert!(fake.rows.lock().unwrap().is_empty(), "no row is inserted"); + assert!(fake.claims.lock().unwrap().is_empty()); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(runtime.free_slots(), 4); + assert_eq!(job_outcomes(&runtime), []); + + *fake.maintenance_held.lock().unwrap() = false; + *fake.maintenance_error.lock().unwrap() = Some("the claim table is locked".to_owned()); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + matches!(&err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + assert!(fake.rows.lock().unwrap().is_empty(), "no row is inserted"); + assert_eq!(runtime.free_slots(), 4); +} + +/// A snapshot the backend refuses to open answers the request with the +/// open's error (here the backend's bound, LimitExceeded) and leaves no row +/// and no object; the slot is freed. +#[tokio::test] +async fn an_open_that_fails_answers_the_request_and_leaves_nothing() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + *fake.snapshot_error.lock().unwrap() = + Some(StorageError::LimitExceeded("too many snapshots".to_owned())); + let runtime = job_runtime(&mem); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + matches!(&err, DynamoDbError::LimitExceededException(m) if m == "too many snapshots"), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert!(fake.rows.lock().unwrap().is_empty()); + assert!(fake.claims.lock().unwrap().is_empty()); + assert!(mem.keys().is_empty()); + assert_eq!(runtime.free_slots(), 4); +} + +/// The file close rule: a file closes at the first item that takes its +/// uncompressed bytes to or past `data_file_target_bytes` (a knob of exactly +/// three lines closes at the third), so every file but the last is at least +/// the knob and under the knob plus one line, and the items are split across +/// files in order. At the knob's minimum (32 MiB) with 5 MiB items, each +/// file takes the items until the sum passes 32 MiB. +#[tokio::test] +async fn a_data_file_closes_at_the_first_item_that_reaches_the_knob() { + use extenddb_core::types::AttributeValue; + let line_bytes = |item: &Item| { + extenddb_core::backup_format::encode_item_line(item) + .unwrap() + .len() as u64 + }; + + // A low knob and small items: the exact rule. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let items = songs(10); + let line = line_bytes(&items[0]); + let target = 3 * line; + snapshot_fixture(&fake, &music_table(), items.clone(), true, None); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + data_file_target_bytes: target, + ..job_settings() + }); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + { + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!( + (row.status, row.failure_reason.clone()), + (BackupStatus::Available, None), + "{:?}", + mem.puts.lock().unwrap() + ); + } + let manifest = stored_manifest(&mem, &details.backup_arn); + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + assert_eq!( + counts, + [3, 3, 3, 1], + "the third line takes the file to the knob, so it closes there" + ); + for file in &manifest.data_files[..3] { + assert!(file.uncompressed_bytes >= target); + assert!(file.uncompressed_bytes - line < target); + } + assert_eq!( + manifest + .data_files + .iter() + .map(|f| f.key.as_str()) + .collect::>(), + [ + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + "data/000004.json.gz" + ] + ); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + let mut decoded = Vec::new(); + for file in &manifest.data_files { + let bytes = mem.objects.lock().unwrap()[&format!("{prefix}{}", file.key)].clone(); + let mut reader = manifest.reader(file, &bytes[..]).unwrap(); + while let Some(item) = reader.next_item().unwrap() { + decoded.push(item); + } + } + assert_eq!(decoded, items); + assert_eq!( + fake.rows.lock().unwrap()[&details.backup_arn].item_count, + 10 + ); + + // The knob's minimum, 32 MiB, with 5 MiB binary items (a 6.7 MiB + // line each): five lines pass 32 MiB, then three. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let items: Vec = (0..8) + .map(|i| { + let mut item = song("a", &format!("t{i}")); + item.insert( + "blob".to_owned(), + AttributeValue::B(vec![0u8; 5 * 1024 * 1024]), + ); + item + }) + .collect(); + let line = line_bytes(&items[0]); + snapshot_fixture(&fake, &music_table(), items, true, None); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + data_file_target_bytes: 32 * 1024 * 1024, + ..job_settings() + }); + let limits = LimitsConfig { + max_item_size_bytes: 8 * 1024 * 1024, + ..LimitsConfig::default() + }; + let details = create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &limits, + "Music", + "nightly", + Ok(music_table()), + ) + .await + .unwrap(); + // The whole test takes about 5 s alone on an idle core (eight 6.7 MiB + // lines encoded and compressed into two files), so the guard is 120 s. + wait_for_jobs_within(&runtime, Duration::from_secs(120)).await; + assert_eq!( + fake.rows.lock().unwrap()[&details.backup_arn].failure_reason, + None + ); + let manifest = stored_manifest(&mem, &details.backup_arn); + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + assert_eq!(counts, [5, 3]); + assert!(manifest.data_files[0].uncompressed_bytes >= 32 * 1024 * 1024); + assert!(manifest.data_files[0].uncompressed_bytes - line < 32 * 1024 * 1024); +} + +/// An unordered snapshot writes `unordered` with no key ranges; an ordered +/// one `key_ordered` with a range per file. +#[tokio::test] +async fn the_manifest_records_the_snapshots_order() { + for key_ordered in [false, true] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), key_ordered, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let manifest = stored_manifest(&mem, &details.backup_arn); + let expected = if key_ordered { + extenddb_core::backup_format::ItemOrder::KeyOrdered + } else { + extenddb_core::backup_format::ItemOrder::Unordered + }; + assert_eq!(manifest.item_order, expected); + assert_eq!(manifest.data_files[0].key_range.is_some(), key_ordered); + } +} + +/// A snapshot stream that fails after a refused item: the FAILED reason +/// carries both, the refusals' count and first item and then the stream's +/// error, so the item the operator is told to find is not lost behind the +/// read failure. Nothing is uploaded and the prefix is deleted. +#[tokio::test] +async fn a_stream_error_after_a_refusal_keeps_the_refusal_in_the_reason() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let items = vec![ + song("a", "1"), + song_with_bad_number("a", "2", "1E126"), + song_with_bad_number("a", "3", "abc"), + ]; + snapshot_fixture(&fake, &music_table(), items, true, None); + fake.snapshots + .lock() + .unwrap() + .get_mut("Music") + .unwrap() + .items + .push(Err(StorageError::Internal( + "the page read failed".to_owned(), + ))); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap(); + assert!( + reason.starts_with( + "2 item(s) cannot be restored; first: key {\"Artist\":{\"S\":\"a\"},\"SongTitle\":{\"S\":\"2\"}}, \ + attribute Year: " + ), + "{reason}" + ); + assert!( + reason.ends_with(" (#618); the snapshot then failed: Internal error: the page read failed"), + "{reason}" + ); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(objects_under(&mem, &details.backup_arn).is_empty()); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + + // The stream's error alone keeps its own reason. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(1), true, None); + fake.snapshots + .lock() + .unwrap() + .get_mut("Music") + .unwrap() + .items + .push(Err(StorageError::Internal( + "the page read failed".to_owned(), + ))); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.rows.lock().unwrap()[&details.backup_arn] + .failure_reason + .as_deref(), + Some("reading the snapshot: Internal error: the page read failed") + ); +} + +/// The validator's rule: with three items the wire let in, the job reads +/// and checks the whole stream past the first refusal (every gate permit is +/// taken), encodes nothing more, and fails the row with the count and the +/// first item, naming #618 for an invalid number; the FAILED write comes +/// first and the prefix is deleted after it. The stream is fed one item +/// every 20 ms, slower than the pipeline, so the encoder's stop lands +/// before the stream's end and an uploader that read that stop as its own +/// failure would end the feeder with the rest unchecked. An item refused +/// for its size gives a reason that does not name #618. +#[tokio::test] +async fn the_validator_fails_the_backup_after_the_rest_is_validated() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let items = vec![ + song("a", "1"), + song_with_bad_number("a", "2", "1E126"), + song("a", "3"), + song_with_bad_number("a", "4", "abc"), + song_with_bad_number("a", "5", "1E126"), + song("a", "6"), + ]; + let count = items.len(); + let gate = closed_gate(); + snapshot_fixture(&fake, &music_table(), items, true, Some(Arc::clone(&gate))); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + let feed = tokio::spawn({ + let gate = Arc::clone(&gate); + async move { + for _ in 0..count { + tokio::time::sleep(Duration::from_millis(20)).await; + gate.add_permits(1); + } + } + }); + wait_for_jobs(&runtime).await; + feed.await.unwrap(); + assert_eq!(gate.available_permits(), 0, "every item was read"); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap(); + assert!( + reason.starts_with( + "3 item(s) cannot be restored; first: key {\"Artist\":{\"S\":\"a\"},\"SongTitle\":{\"S\":\"2\"}}, \ + attribute Year: " + ), + "{reason}" + ); + assert!(reason.ends_with(" (#618)"), "{reason}"); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!(writes(&fake).len(), 1); + assert_eq!(writes(&fake)[0].0, JobStatus::Failed); + assert_eq!( + mem.deletes.lock().unwrap().as_slice(), + std::slice::from_ref(&prefix) + ); + assert!(objects_under(&mem, &details.backup_arn).is_empty()); + assert!( + mem.puts.lock().unwrap().is_empty(), + "after the first refused item nothing is uploaded, the first file with its one valid \ + item included: {:?}", + mem.puts.lock().unwrap() + ); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + + // Over `max_item_size_bytes`: the reason names the size rule, not #618. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + let runtime = job_runtime(&mem); + let limits = LimitsConfig { + max_item_size_bytes: 20, + ..LimitsConfig::default() + }; + let details = create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &limits, + "Music", + "nightly", + Ok(music_table()), + ) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap(); + assert!( + reason.starts_with("2 item(s) cannot be restored; first: key "), + "{reason}" + ); + assert!(reason.contains("20 bytes"), "{reason}"); + assert!(!reason.contains("#618"), "{reason}"); +} + +/// A put that fails fails the row with the file named, and only then +/// deletes the prefix; the store error is counted. +#[tokio::test] +async fn a_put_failure_fails_the_row_then_deletes_the_prefix() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(3), true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert!( + row.failure_reason + .as_deref() + .unwrap() + .starts_with("writing data/000001.json.gz: "), + "{:?}", + row.failure_reason + ); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!( + writes(&fake), + [(JobStatus::Failed, row.failure_reason.clone())] + ); + assert_eq!(mem.deletes.lock().unwrap().as_slice(), [prefix]); + use extenddb_core::metrics::MetricName; + let drained = runtime.metrics().unwrap().drain(Duration::ZERO); + assert!(drained.iter().any(|b| { + b.metric == MetricName::BackupStoreErrors && b.operation == "put" && b.sum == 1.0 + })); + assert!( + drained + .iter() + .any(|b| { b.metric == MetricName::BackupJobs && b.operation == "failed" }) + ); + assert!( + drained + .iter() + .any(|b| { b.metric == MetricName::WorkerErrorCount && b.operation == "BackupJob" }) + ); +} + +/// A FAILED write that matches no row (the row's generation changed under +/// a heartbeat that kept confirming) ends the job with no store access: no +/// `delete_prefix`, the row untouched. +#[tokio::test] +async fn a_failed_write_that_does_not_match_makes_no_store_call() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + // Another generation takes the row while the job waits on its stream; + // the put then fails. + fake.rows + .lock() + .unwrap() + .get_mut(&details.backup_arn) + .unwrap() + .owner_generation = 999; + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + gate.add_permits(3); + wait_for_jobs(&runtime).await; + assert_eq!(writes(&fake).len(), 1); + assert_eq!(writes(&fake)[0].0, JobStatus::Failed); + assert!( + mem.deletes.lock().unwrap().is_empty(), + "no store access after NotMatched" + ); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert_eq!(row.owner_generation, 999); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A status write whose outcome is unknown is re-read, not treated as a +/// failure: an AVAILABLE write that landed is AVAILABLE with no store +/// access after it; one that was lost is written once more. The same for +/// the FAILED write: landed, the deletion follows; lost, it is written again +/// and the deletion follows. +#[tokio::test] +async fn an_unknown_write_is_re_read() { + for (unknown, writes_expected) in [(UnknownWrite::Applied, 1), (UnknownWrite::Lost, 2)] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + *fake.unknown_write.lock().unwrap() = Some(unknown); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Available, "{unknown:?}"); + assert_eq!(writes(&fake).len(), writes_expected, "{unknown:?}"); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(objects_under(&mem, &details.backup_arn).len(), 5); + assert_eq!(job_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); + } + for (unknown, writes_expected) in [(UnknownWrite::Applied, 1), (UnknownWrite::Lost, 2)] { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + *fake.unknown_write.lock().unwrap() = Some(unknown); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed, "{unknown:?}"); + assert_eq!(writes(&fake).len(), writes_expected, "{unknown:?}"); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!(mem.deletes.lock().unwrap().as_slice(), [prefix]); + } +} + +/// A heartbeat that reports the claim lost, or that does not return within +/// a period, ends the job with no put and no catalog write: the row is left +/// as it is for its owner or the reconciler. +#[tokio::test] +async fn a_lost_or_stalled_heartbeat_stops_the_job_with_no_put() { + for answer in [ + HeartbeatAnswer::Lost, + HeartbeatAnswer::Stall, + HeartbeatAnswer::Error, + ] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture(&fake, &music_table(), songs(3), true, Some(gate)); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + // The job waits on its stream (the encoder is a blocking task, so + // the clock runs: a paused clock would not advance behind it); the + // first beat, a period later, finds the claim lost (or never + // returns: a period more). + *fake.heartbeat_answer.lock().unwrap() = answer; + wait_for_jobs(&runtime).await; + assert!(mem.puts.lock().unwrap().is_empty(), "{answer:?}: no put"); + assert!( + mem.deletes.lock().unwrap().is_empty(), + "{answer:?}: no delete" + ); + assert!(writes(&fake).is_empty(), "{answer:?}: no catalog write"); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating, "{answer:?}"); + assert_eq!(runtime.free_slots(), 4); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + } +} + +/// The pre-put check's numbers: with the claim confirming, the last +/// confirmed beat stays recent however long the job runs; a timer that +/// stopped running leaves the check true for two periods after the last +/// confirmed beat and false after; with a beat that never returns, the +/// check still passes until the timer has waited one period for the beat +/// (two periods from the last confirmation), and the claim is lost from +/// then on. +#[tokio::test(start_paused = true)] +async fn the_heartbeat_handle_confirms_the_claim_within_two_periods() { + let fake = Arc::new(Fake::default()); + fake.rows + .lock() + .unwrap() + .insert(arn(ACCOUNT), row(&arn(ACCOUNT), BackupStatus::Creating)); + let claim: Arc = Arc::from(fake.claim_on(&arn(ACCOUNT), 7)); + let period = Duration::from_secs(10); + let handle = job::heartbeat::HeartbeatHandle::start( + Arc::clone(&claim), + period, + None, + 0.0, + tokio::time::Instant::now(), + ); + // Half a period past the tenth tick, so that tick's beat has confirmed. + tokio::time::sleep(10 * period + period / 2).await; + assert!(handle.confirms_the_claim()); + assert!(fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) >= 10); + // A timer that stopped running (no loss raised): the check holds for two + // periods after the last confirmed beat, the second period's end + // included, and then refuses. + handle.stop_timer(); + tokio::time::sleep(period).await; + assert!(handle.confirms_the_claim(), "a period and a half"); + tokio::time::sleep(period / 2).await; + assert!( + handle.confirms_the_claim(), + "exactly two periods: the bound is inclusive" + ); + tokio::time::sleep(period / 2).await; + assert!(!handle.lost().is_cancelled()); + assert!( + !handle.confirms_the_claim(), + "two and a half periods without a confirmed beat" + ); + drop(handle); + + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Stall; + let handle = job::heartbeat::HeartbeatHandle::start( + claim, + period, + None, + 0.0, + tokio::time::Instant::now(), + ); + tokio::time::sleep(period + period / 2).await; + assert!(handle.confirms_the_claim(), "the beat may still return"); + assert!(!handle.lost().is_cancelled()); + tokio::time::sleep(period).await; + assert!(!handle.confirms_the_claim()); + assert!(handle.lost().is_cancelled()); + assert!(handle.reason().contains("did not return")); +} + +/// The shutdown token runs the failure path: FAILED with reason `shutdown`, +/// then the prefix deleted; the outcome is `cancelled`. `drain_jobs` is what +/// the server calls: it cancels and waits for the job. +#[tokio::test] +async fn cancellation_runs_the_failure_path() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + // The job holds its first item and waits for the second. + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while gate.available_permits() > 0 { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert_eq!(runtime.running_jobs(), 1); + let left = runtime.drain_jobs(Duration::from_secs(10)).await; + assert_eq!(left, 0); + assert_eq!(runtime.running_jobs(), 0); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert_eq!(row.failure_reason.as_deref(), Some("shutdown")); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!( + writes(&fake), + [(JobStatus::Failed, Some("shutdown".to_owned()))] + ); + assert_eq!(mem.deletes.lock().unwrap().as_slice(), [prefix]); + assert!(objects_under(&mem, &details.backup_arn).is_empty()); + use extenddb_core::metrics::MetricName; + let drained = runtime.metrics().unwrap().drain(Duration::ZERO); + assert_eq!( + drained + .iter() + .filter(|b| b.metric == MetricName::BackupJobs) + .map(|b| (b.operation.clone(), b.sum)) + .collect::>(), + [("cancelled".to_owned(), 1.0)] + ); + assert!( + !drained + .iter() + .any(|b| b.metric == MetricName::WorkerErrorCount), + "a clean shutdown is not a worker error" + ); + assert!( + !drained + .iter() + .any(|b| b.metric == MetricName::WorkerLastSuccess), + "nor a success" + ); +} + +/// With `max_concurrent_jobs = 1`, a second CreateBackup while a job runs +/// is LimitExceeded naming the bound; once the job ends the slot is free. +#[tokio::test] +async fn a_second_create_over_the_job_bound_is_limit_exceeded() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(2), + true, + Some(Arc::clone(&gate)), + ); + let runtime = BackupRuntime::new(1) + .with_store(ConfiguredStore::new( + store().store_kind, + store().store_root, + Arc::clone(&mem) as Arc, + None, + Ok(store()), + )) + .with_job_settings(job_settings()); + create_music_backup(&fake, &runtime).await.unwrap(); + let err = create_music_backup(&fake, &runtime).await.unwrap_err(); + assert!( + matches!(&err, DynamoDbError::LimitExceededException(m) if m.contains("max_concurrent_jobs")), + "{err:?}" + ); + assert_eq!( + fake.snapshots_opened.lock().unwrap().len(), + 1, + "the second opened nothing" + ); + gate.add_permits(2); + wait_for_jobs(&runtime).await; + assert_eq!(runtime.free_slots(), 1); + snapshot_fixture(&fake, &music_table(), songs(1), true, None); + create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(fake.rows.lock().unwrap().len(), 2); +} + +/// A `tracing` subscriber that keeps every event's level and message, for +/// the tests of the job's log lines. Set as the thread's default for the +/// test's life through [`LogLines::capture`]; a `#[tokio::test]` runtime +/// runs the job's tasks on the test's thread, so their events reach it (the +/// encoders, on the blocking pool, log nothing). +#[derive(Clone, Default)] +struct LogLines(Arc>>); + +/// The process's global `tracing` subscriber while these tests run: +/// interested in every callsite and enabled for none. `tracing` caches each +/// callsite's interest the first time any thread reaches it; while exactly +/// one scoped subscriber is registered in the process, that cache is +/// computed from the default of the thread that got there first. A test +/// without a subscriber running beside a `LogLines` test on another thread +/// would so cache the job's log callsites as "never", and the `LogLines` +/// test would then see no line from them (tokio-rs/tracing#2743). With this +/// global default the cache reads "sometimes" instead, and each event asks +/// the thread's own default, the `LogLines` where one is set. +struct InterestedInEveryCallsite; + +impl tracing::Subscriber for InterestedInEveryCallsite { + fn register_callsite( + &self, + _: &'static tracing::Metadata<'static>, + ) -> tracing::subscriber::Interest { + tracing::subscriber::Interest::sometimes() + } + fn enabled(&self, _: &tracing::Metadata<'_>) -> bool { + false + } + fn new_span(&self, _: &tracing::span::Attributes<'_>) -> tracing::span::Id { + tracing::span::Id::from_u64(1) + } + fn record(&self, _: &tracing::span::Id, _: &tracing::span::Record<'_>) {} + fn record_follows_from(&self, _: &tracing::span::Id, _: &tracing::span::Id) {} + fn event(&self, _: &tracing::Event<'_>) {} + fn enter(&self, _: &tracing::span::Id) {} + fn exit(&self, _: &tracing::span::Id) {} +} + +impl LogLines { + /// A fresh `LogLines` set as this thread's `tracing` default until the + /// guard drops, with the process's global default installed once. + fn capture() -> (Self, tracing::subscriber::DefaultGuard) { + static GLOBAL: std::sync::Once = std::sync::Once::new(); + GLOBAL.call_once(|| { + // Another global may already be set by a test outside this + // module; then its interest rule applies and this one is moot. + let _ = tracing::subscriber::set_global_default(InterestedInEveryCallsite); + }); + let lines = Self::default(); + let guard = tracing::subscriber::set_default(lines.clone()); + (lines, guard) + } + + /// The messages containing `needle`, in order. + fn containing(&self, needle: &str) -> Vec { + self.0 + .lock() + .unwrap() + .iter() + .filter(|(_, m)| m.contains(needle)) + .map(|(_, m)| m.clone()) + .collect() + } +} + +struct MessageVisitor<'a>(&'a mut String); + +/// A callsite met first by a thread without a scoped subscriber is still +/// seen by a `LogLines` on another thread (the global default's interest +/// rule; the doc of `InterestedInEveryCallsite`). Without that default the +/// callsite's interest is cached as "never" whenever this is the only +/// `LogLines` alive in the process, and nothing is captured. +#[test] +fn a_callsite_first_met_on_another_thread_still_reaches_the_lines() { + fn shared_callsite(n: u32) { + tracing::error!("probe {n}: a callsite shared by two threads"); + } + let (lines, _guard) = LogLines::capture(); + std::thread::spawn(|| shared_callsite(1)).join().unwrap(); + shared_callsite(2); + assert_eq!( + lines.containing("probe "), + ["probe 2: a callsite shared by two threads"] + ); +} + +impl tracing::field::Visit for MessageVisitor<'_> { + fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) { + if field.name() == "message" { + use std::fmt::Write as _; + let _ = write!(self.0, "{value:?}"); + } + } +} + +impl tracing::Subscriber for LogLines { + fn enabled(&self, _: &tracing::Metadata<'_>) -> bool { + true + } + fn new_span(&self, _: &tracing::span::Attributes<'_>) -> tracing::span::Id { + tracing::span::Id::from_u64(1) + } + fn record(&self, _: &tracing::span::Id, _: &tracing::span::Record<'_>) {} + fn record_follows_from(&self, _: &tracing::span::Id, _: &tracing::span::Id) {} + fn event(&self, event: &tracing::Event<'_>) { + let mut message = String::new(); + event.record(&mut MessageVisitor(&mut message)); + self.0 + .lock() + .unwrap() + .push((*event.metadata().level(), message)); + } + fn enter(&self, _: &tracing::span::Id) {} + fn exit(&self, _: &tracing::span::Id) {} +} + +/// The bytes of one item's data file line. +fn line_bytes(item: &Item) -> u64 { + extenddb_core::backup_format::encode_item_line(item) + .unwrap() + .len() as u64 +} + +/// A put that fails while the snapshot stream is between pages ends the job +/// at once: the stream holds one permit, so it yields the first item and +/// then waits; the first file closes at that item (a one-line knob), its put +/// fails, and the FAILED write lands with the stream still held, no permit +/// released. Without the uploader's stop reaching the feeder, the feeder +/// would wait for the stream's next item and the row would stay CREATING +/// with a live heartbeat for as long as the stream did. +#[tokio::test] +async fn a_put_failure_ends_the_job_without_waiting_for_the_streams_next_item() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + let items = songs(3); + let target = line_bytes(&items[0]); + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + snapshot_fixture(&fake, &music_table(), items, true, Some(Arc::clone(&gate))); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + data_file_target_bytes: target, + ..job_settings() + }); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(gate.available_permits(), 0, "the stream was never released"); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert!( + row.failure_reason + .as_deref() + .unwrap() + .starts_with("writing data/000001.json.gz: "), + "{:?}", + row.failure_reason + ); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!(mem.deletes.lock().unwrap().as_slice(), [prefix]); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// `BackupJobDuration` of a FAILED job covers its failure path: with the +/// store's deletion taking 300 ms, the duration recorded under `failed` is +/// at least that, so the metric measures the job to the end of its last +/// store call for either outcome. +#[tokio::test] +async fn a_failed_jobs_duration_covers_its_failed_write_and_deletion() { + use extenddb_core::metrics::MetricName; + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + *mem.delete_delay.lock().unwrap() = Some(Duration::from_millis(300)); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert_eq!(mem.deletes.lock().unwrap().len(), 1); + let drained = runtime.metrics().unwrap().drain(Duration::ZERO); + let durations: Vec<(String, f64)> = drained + .iter() + .filter(|b| b.metric == MetricName::BackupJobDuration) + .map(|b| (b.operation.clone(), b.sum)) + .collect(); + assert_eq!(durations.len(), 1, "{durations:?}"); + assert_eq!(durations[0].0, "failed"); + assert!( + durations[0].1 >= 300_000.0, + "{} µs: the duration stops before the deletion", + durations[0].1 + ); +} + +/// The two info lines: the start line gives the table's count as the stats +/// pass last counted it (the description's figure, here 100,000), the +/// completion line the stream's count (the items written) and the bytes. +#[tokio::test] +async fn the_start_and_completion_lines_name_the_two_item_counts() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(5), true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let arn = &details.backup_arn; + let started = lines.containing(": started, "); + assert_eq!( + started, + [format!( + "backup {arn}: started, 100000 items in the table as last counted by the stats pass, \ + key-ordered" + )] + ); + let total: u64 = mem.put_sizes.lock().unwrap().iter().map(|(_, n)| n).sum(); + let available = lines.containing(": AVAILABLE, "); + assert_eq!(available.len(), 1, "{available:?}"); + assert!( + available[0].starts_with(&format!( + "backup {arn}: AVAILABLE, 5 items, {total} bytes written in " + )), + "{}", + available[0] + ); +} + +/// The refusal log cap: with 1,001 refused items the first 1,000 are +/// logged one by one, one line counts the remainder, the reason carries the +/// full count and the first item, the stream is read to its end and nothing +/// is uploaded. +#[tokio::test] +async fn refusals_past_the_log_cap_are_counted_in_one_line() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut items = vec![song("a", "0")]; + items.extend((1..=1001).map(|i| song_with_bad_number("a", &format!("{i:05}"), "1E126"))); + items.push(song("a", "zzz")); + let gate = Arc::new(tokio::sync::Semaphore::new(items.len())); + snapshot_fixture(&fake, &music_table(), items, true, Some(Arc::clone(&gate))); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(gate.available_permits(), 0, "every item was read"); + let logged = lines.containing(": item cannot be restored: key "); + assert_eq!(logged.len(), 1000); + assert!( + logged[0].contains(r#""SongTitle":{"S":"00001"}"#) + && logged[0].contains(", attribute Year: "), + "{}", + logged[0] + ); + assert!( + logged[999].contains(r#""SongTitle":{"S":"01000"}"#), + "{}", + logged[999] + ); + assert_eq!( + lines.containing(" more item(s) cannot be restored (not listed)"), + [format!( + "backup {}: 1 more item(s) cannot be restored (not listed)", + details.backup_arn + )] + ); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap(); + assert!( + reason.starts_with("1001 item(s) cannot be restored; first: key "), + "{reason}" + ); + assert!(reason.contains(r#""SongTitle":{"S":"00001"}"#), "{reason}"); + assert!(mem.puts.lock().unwrap().is_empty()); +} + +/// An item whose data file line would exceed the format's cap is refused +/// through the validator's path, not the encoder's: under a raised item +/// limit, a 600 KB item of the worst-expansion shape (a list of number +/// sets, a 40 MB line) is counted, logged with its key and the cap, the +/// rest of the stream is checked, nothing is uploaded and the FAILED +/// reason names the item without #618. The item passes the validator +/// itself (its numbers are valid, its size within the limit). The line is +/// measured through a counting writer that stops at the cap (the format's +/// own test pins that it holds no line), so the reason carries the cap and +/// not the line's length. +#[tokio::test] +async fn an_item_whose_line_exceeds_the_cap_is_refused_and_named() { + use extenddb_core::backup_format::MAX_LINE_BYTES; + use extenddb_core::types::AttributeValue; + use std::collections::BTreeSet; + let (lines, _guard) = LogLines::capture(); + // 81 two-digit values at `E-130`, 133 characters each in normalized + // form, 2 bytes of item size each (the `MAX_LINE_BYTES` doc's shape). + let set: BTreeSet = (11..100) + .filter(|m| m % 10 != 0) + .map(|m| format!("0.{}{m}", "0".repeat(129))) + .collect(); + assert_eq!(set.len(), 81); + let mut worst = song("a", "2"); + worst.insert( + "l".to_owned(), + AttributeValue::L(vec![AttributeValue::NS(set); 3_700]), + ); + let size = extenddb_core::types::item_size_bytes(&worst); + assert!(size > 409_600 && size < 2 * 1024 * 1024, "{size}"); + let line = line_bytes(&worst); + assert!(line > MAX_LINE_BYTES as u64, "{line}"); + let items = vec![song("a", "1"), worst, song("a", "3")]; + let gate = Arc::new(tokio::sync::Semaphore::new(items.len())); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), items, true, Some(Arc::clone(&gate))); + let runtime = job_runtime(&mem); + let limits = LimitsConfig { + max_item_size_bytes: 2 * 1024 * 1024, + ..LimitsConfig::default() + }; + let details = create_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &limits, + "Music", + "nightly", + Ok(music_table()), + ) + .await + .unwrap(); + // The test takes about 20 s alone on an idle core, so the guard is + // 300 s. + wait_for_jobs_within(&runtime, Duration::from_secs(300)).await; + assert_eq!( + gate.available_permits(), + 0, + "the rest of the stream was checked" + ); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.unwrap(); + assert_eq!( + reason, + format!( + "1 item(s) cannot be restored; first: key {{\"Artist\":{{\"S\":\"a\"}},\"SongTitle\":\ + {{\"S\":\"2\"}}}}: data file line exceeds the line cap of {MAX_LINE_BYTES} bytes" + ) + ); + assert_eq!(lines.containing(": item cannot be restored: key ").len(), 1); + assert!( + mem.puts.lock().unwrap().is_empty(), + "{:?}", + mem.puts.lock().unwrap() + ); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A status write that errors on the claim connection makes no store call, +/// whichever write it is. The FAILED write: no `delete_prefix`, the row +/// CREATING for the reconciler. The AVAILABLE write: the manifest was +/// already put, so the five objects stay for the reconciler's probe, which +/// is the order the design rests on (the manifest is the commit record, the +/// AVAILABLE write follows it). +#[tokio::test] +async fn a_status_write_error_makes_no_store_call_and_leaves_the_manifest() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + fake.status_write_error + .store(true, std::sync::atomic::Ordering::Release); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert_eq!(writes(&fake).len(), 1); + assert!( + mem.deletes.lock().unwrap().is_empty(), + "no delete after a claim error on the FAILED write" + ); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + fake.status_write_error + .store(true, std::sync::atomic::Ordering::Release); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert_eq!(writes(&fake), [(JobStatus::Available, None)]); + assert!(mem.deletes.lock().unwrap().is_empty()); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!( + mem.puts.lock().unwrap().as_slice(), + [ + format!("{prefix}data/000001.json.gz"), + format!("{prefix}manifest-files.json"), + format!("{prefix}manifest-summary.json"), + format!("{prefix}extenddb-manifest.sha256"), + format!("{prefix}extenddb-manifest.json"), + ], + "the manifest was put before the AVAILABLE write" + ); + assert_eq!(objects_under(&mem, &details.backup_arn).len(), 5); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A second unknown outcome, on the write sent once more, counts as +/// matching no row: two writes and no third, the row left as it is, no +/// delete, the outcome `failed` (the job ended by its own path, not a +/// panic). +#[tokio::test] +async fn a_second_unknown_outcome_ends_the_job_without_a_third_write() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::Lost); + fake.unknown_write_sticky + .store(true, std::sync::atomic::Ordering::Release); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(writes(&fake).len(), 2); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A second unknown outcome whose write landed counts as matched: the first +/// FAILED write is lost with its connection (the re-read shows CREATING, so +/// the write is sent once more), the second lands and its outcome is +/// unknown too, and the second re-read shows the row FAILED under this +/// claim, so the failure path goes on to the one `delete_prefix`. The row +/// is FAILED under this generation and the prefix is the job's, so the +/// deletion is right; a job that treated the second unknown outcome as +/// matching no row would leave the prefix to the reconciler for nothing. +#[tokio::test] +async fn a_second_unknown_outcome_whose_write_landed_is_matched() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::Lost); + *fake.unknown_write_then.lock().unwrap() = Some(UnknownWrite::Applied); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + writes(&fake).iter().map(|w| w.0).collect::>(), + [JobStatus::Failed, JobStatus::Failed] + ); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!( + mem.deletes.lock().unwrap().as_slice(), + [prefix], + "the second re-read shows FAILED under this claim: the deletion follows" + ); + assert!(objects_under(&mem, &details.backup_arn).is_empty()); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// The pre-put check in its two places: with the heartbeat timer stopped +/// without raising the loss (a timer task that died) and two periods gone, +/// neither a data file's put nor a document's put starts, each stopping the +/// job as a lost claim naming the object. +#[tokio::test] +async fn a_stopped_heartbeat_timer_starts_no_put() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = arn(ACCOUNT); + fake.rows + .lock() + .unwrap() + .insert(arn.clone(), row(&arn, BackupStatus::Creating)); + let claim: Arc = Arc::from(fake.claim_on(&arn, 7)); + let period = Duration::from_millis(50); + let heartbeat = job::heartbeat::HeartbeatHandle::start( + claim, + period, + None, + 0.0, + tokio::time::Instant::now(), + ); + heartbeat.stop_timer(); + tokio::time::sleep(3 * period).await; + assert!(!heartbeat.lost().is_cancelled()); + let shutdown = extenddb_storage::CancellationToken::new(); + let parsed = BackupArn::parse(&arn).unwrap(); + let prefix = parsed.store_prefix(); + let runtime = runtime_on(&mem, store()); + let identity = store(); + let pipeline = job::writer::Pipeline { + store: &*mem, + configured: runtime.configured_store().unwrap(), + identity: &identity, + prefix: &prefix, + arn: &arn, + heartbeat: &heartbeat, + shutdown: &shutdown, + metrics: None, + target_bytes: 64 * 1024 * 1024, + key_prefix: "", + bucket: &store().store_root, + }; + let table = music_table(); + let key = extenddb_core::validation::RestorableKeySchema::resolve( + &table.key_schema, + &table.attribute_definitions, + ) + .unwrap(); + let stream: BoxStream<'static, Result> = + Box::pin(futures::stream::iter(songs(1).into_iter().map(Ok))); + let limits = extenddb_core::validation::RestorableLimits::from(&LimitsConfig::default()); + let stop = job::writer::write_data_files(&pipeline, stream, key, limits, None) + .await + .unwrap_err(); + assert_eq!( + stop, + job::writer::Stop::ClaimLost( + "no heartbeat confirmed the claim recently enough for data file 1".to_owned() + ) + ); + assert!(mem.puts.lock().unwrap().is_empty()); + let stop = job::put_document(&pipeline, "manifest-files.json", b"{}") + .await + .unwrap_err(); + assert_eq!( + stop, + job::writer::Stop::ClaimLost( + "no heartbeat confirmed the claim recently enough for manifest-files.json".to_owned() + ) + ); + assert!(mem.puts.lock().unwrap().is_empty()); + assert!(mem.keys().is_empty()); +} + +/// The maintenance check runs before the permit and the row, so a slow +/// check delays the request and not the job's first put: with a period of +/// 100 ms, `stale_job_after` of 300 ms (the floor's ratio) and a check of +/// 400 ms, the request takes at least the check, the row is inserted after +/// it (its heartbeat the insert's, within two periods of the first put), and +/// the backup completes AVAILABLE. +#[tokio::test] +async fn a_slow_maintenance_check_delays_the_request_and_not_the_first_put() { + let period = Duration::from_millis(100); + let settings = JobSettings { + heartbeat_interval: period, + stale_job_after: 3 * period, + ..job_settings() + }; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + *fake.maintenance_delay.lock().unwrap() = Some(4 * period); + let runtime = job_runtime(&mem).with_job_settings(settings); + let asked = tokio::time::Instant::now(); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + assert!( + asked.elapsed() >= 4 * period, + "the request waited for the check" + ); + assert_eq!(fake.maintenance_checks.lock().unwrap().len(), 1); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!( + (row.status, row.failure_reason), + (BackupStatus::Available, None) + ); + assert_eq!(objects_under(&mem, &details.backup_arn).len(), 5); + assert_eq!(job_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); + assert_eq!(runtime.free_slots(), 4); +} + +/// The claim re-check before the manifest: the heartbeat answers `Lost` +/// from the sidecar's put on, with a period long enough that no timer beat +/// runs during the job, so the re-check is the one beat the job sends; it +/// finds the claim lost, the manifest is not written, the row is left +/// CREATING with its four objects and no catalog write. +#[tokio::test] +async fn the_claim_re_check_before_the_manifest_stops_a_lost_claim() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + let answer = Arc::clone(&fake.heartbeat_answer); + *mem.on_put.lock().unwrap() = Some(Box::new(move |key: &str| { + if key.ends_with("extenddb-manifest.sha256") { + *answer.lock().unwrap() = HeartbeatAnswer::Lost; + } + })); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + heartbeat_interval: Duration::from_secs(10), + ..job_settings() + }); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire), + 1, + "the re-check is the one beat" + ); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert!( + writes(&fake).is_empty(), + "no catalog write after a lost claim" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + let objects = objects_under(&mem, &details.backup_arn); + assert_eq!(objects.len(), 4, "{objects:?}"); + assert!( + !objects + .iter() + .any(|k| k.ends_with("extenddb-manifest.json")), + "the manifest was not written" + ); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +// --- the reconciler's duties over the store's rows ---------------------------- + +/// A CREATING row whose owner is dead, with a complete, verified backup +/// under its prefix (the golden manifest's documents and data files of the +/// recorded sizes), in `mem` and `fake`; returns its ARN. +fn seed_dead_creating_with_manifest(fake: &Fake, mem: &MemStore, complete: bool) -> String { + use extenddb_core::backup_format::{BackupDocuments, ExtenddbManifest, ManifestHeader}; + let golden = ExtenddbManifest::from_json_bytes(include_bytes!( + "../../../core/testdata/extenddb-manifest/v1.json" + )) + .unwrap(); + let docs = BackupDocuments::build( + ManifestHeader { + backup_arn: golden.backup_arn().unwrap(), + backup_name: golden.backup_name.clone(), + source_backend: golden.source_backend.clone(), + source_extenddb_version: golden.source_extenddb_version.clone(), + snapshot: golden.snapshot.clone(), + item_order: golden.item_order, + source_table: golden.source_table.clone(), + }, + golden + .data_files + .iter() + .cloned() + .map(|entry| (entry, None)) + .collect(), + "", + &store().store_root, + ) + .unwrap(); + let arn = golden.backup_arn.clone(); + let prefix = golden.backup_arn().unwrap().store_prefix(); + for (i, file) in golden.data_files.iter().enumerate() { + let size = if complete || i == 0 { + file.bytes + } else { + file.bytes - 1 + }; + mem.insert( + &format!("{prefix}{}", file.key), + &vec![0u8; usize::try_from(size).unwrap()], + ); + } + for (name, bytes) in docs.documents() { + mem.insert(&format!("{prefix}{name}"), bytes); + } + let mut dead = row(&arn, BackupStatus::Creating); + dead.heartbeat_at = 1234.5; + dead.size_bytes = 0; + dead.item_count = 0; + fake.rows.lock().unwrap().insert(arn.clone(), dead); + fake.dead.lock().unwrap().insert(arn.clone()); + arn +} + +/// One cycle on `fake` and `mem`, the store healthy. +async fn cycle_on(fake: &Fake, runtime: &BackupRuntime) -> reconciler::CycleReport { + reconciler::cycle( + fake, + runtime, + Duration::from_secs(300), + "test", + &mut reconciler::ReconcilerState::default(), + ) + .await + .unwrap() +} + +/// The reconciler's store calls that fail (not an absent object) record +/// `BackupStoreErrors` under the call's name, as the job's do: a `get` of a +/// dead owner's manifest, a `list` of a DELETED row's prefix, a +/// `delete_prefix` of a stale FAILED row's prefix. Each row is left for the +/// next cycle and the cycle itself completes, so `WorkerErrorCount` would +/// not move; the store's health is untouched. The next cycle, with the +/// store answering, finishes each duty and records nothing more. +#[tokio::test] +async fn the_reconcilers_failed_store_calls_are_counted_by_call() { + use extenddb_core::metrics::{MetricName, MetricsCollector}; + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let now = seconds_since_epoch(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + let dead_prefix = BackupArn::parse(&dead).unwrap().store_prefix(); + let stale_failed = arn_n(1); + let deleted = arn_n(2); + for (a, status) in [ + (&stale_failed, BackupStatus::Failed), + (&deleted, BackupStatus::Deleted), + ] { + let mut r = row(a, status); + r.heartbeat_at = now - 1000.0; + r.completed_at = Some(now); + fake.rows.lock().unwrap().insert(a.clone(), r); + let prefix = BackupArn::parse(a).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + } + let failed_prefix = BackupArn::parse(&stale_failed).unwrap().store_prefix(); + let deleted_prefix = BackupArn::parse(&deleted).unwrap().store_prefix(); + *mem.fail_gets_under.lock().unwrap() = Some(dead_prefix.clone()); + *mem.fail_deletes_of.lock().unwrap() = Some(failed_prefix.clone()); + *mem.fail_lists_of.lock().unwrap() = Some(deleted_prefix.clone()); + let metrics = Arc::new(MetricsCollector::new()); + let runtime = runtime_on(&mem, store()).with_metrics(Arc::clone(&metrics)); + + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert!(report.creating_resolved.is_empty()); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + let errors = |drained: &[extenddb_core::metrics::FlushBucket]| { + let mut ops: Vec<(String, f64)> = drained + .iter() + .filter(|b| b.metric == MetricName::BackupStoreErrors) + .map(|b| (b.operation.clone(), b.sum)) + .collect(); + ops.sort_by(|a, b| a.0.cmp(&b.0)); + ops + }; + assert_eq!( + errors(&metrics.drain(Duration::ZERO)), + [ + ("delete_prefix".to_owned(), 1.0), + ("get".to_owned(), 1.0), + ("list".to_owned(), 1.0), + ] + ); + + *mem.fail_gets_under.lock().unwrap() = None; + *mem.fail_deletes_of.lock().unwrap() = None; + *mem.fail_lists_of.lock().unwrap() = None; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); + assert_eq!(report.failed_prefixes_deleted, [stale_failed]); + assert_eq!(report.deleted_prefixes_swept, [deleted]); + assert!(errors(&metrics.drain(Duration::ZERO)).is_empty()); +} + +/// A dead CREATING row whose prefix holds a complete, verified manifest is +/// AVAILABLE with the manifest's size, count and `source_table` (the row's +/// own `source_table_details_json`, the stats pass's, is replaced: it is +/// seeded here with another object so the write is seen to carry the +/// manifest's); the takeover leaves `heartbeat_at` as the dead owner wrote +/// it and heartbeats nothing; the objects stay. A row whose owner is alive +/// is left alone. +#[tokio::test] +async fn the_reconciler_resolves_a_dead_creating_row_with_a_manifest_to_available() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + let stats_pass = serde_json::json!({"TableName": "Music", "ItemCount": 0}); + fake.rows + .lock() + .unwrap() + .get_mut(&dead) + .unwrap() + .source_table_details_json = stats_pass.clone(); + let live = arn(ACCOUNT); + fake.rows + .lock() + .unwrap() + .insert(live.clone(), row(&live, BackupStatus::Creating)); + let objects_before = mem.keys().len(); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.creating_resolved, + [(dead.clone(), BackupStatus::Available)] + ); + assert_eq!( + fake.takeovers.lock().unwrap().as_slice(), + std::slice::from_ref(&dead) + ); + let manifest = stored_manifest(&mem, &dead); + let row = fake.rows.lock().unwrap()[&dead].clone(); + assert_eq!(row.status, BackupStatus::Available); + assert_eq!(row.size_bytes, manifest.backup_size_bytes); + assert_eq!(row.item_count, 100_000); + assert_ne!( + stats_pass, + serde_json::to_value(&manifest.source_table).unwrap() + ); + assert_eq!( + row.source_table_details_json, + serde_json::to_value(&manifest.source_table).unwrap(), + "the AVAILABLE write carries the manifest's source_table" + ); + assert_eq!(row.heartbeat_at, 1234.5, "the takeover left the heartbeat"); + assert_ne!(row.owner_generation, 7, "a fresh generation"); + assert!(row.completed_at.is_some()); + assert_eq!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire), + 0, + "the reconciler heartbeats no takeover claim" + ); + assert_eq!(mem.keys().len(), objects_before, "nothing deleted"); + assert_eq!( + fake.rows.lock().unwrap()[&live].status, + BackupStatus::Creating + ); + assert!(mem.deletes.lock().unwrap().is_empty()); +} + +/// A dead CREATING row without a complete manifest is FAILED with the +/// owner-dead reason and `completed_at`, and the resolution itself deletes +/// nothing; the FAILED-prefix duty deletes the prefix once it runs, the +/// heartbeat the owner left being stale. Every guard of the probe has its +/// shape: no manifest; a manifest without its sidecar; a sidecar that does +/// not match; a manifest the reader refuses under a sidecar computed over +/// its bytes (a sidecar matches what was stored, not what parses); a data +/// file missing; a data file shorter than recorded; one larger than +/// recorded; a manifest naming another backup under the row's prefix. A +/// store error during the probe (not an absent object) is none of these: +/// the row stays CREATING for the next cycle, nothing written. +#[tokio::test] +async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() { + for shape in [ + "none", + "no sidecar", + "sidecar mismatch", + "unparseable manifest", + "missing data file", + "short file", + "oversized file", + "foreign ARN", + ] { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let mut dead = seed_dead_creating_with_manifest(&fake, &mem, shape != "short file"); + let prefix = BackupArn::parse(&dead).unwrap().store_prefix(); + let golden = stored_manifest(&mem, &dead); + match shape { + "none" => { + mem.objects + .lock() + .unwrap() + .remove(&format!("{prefix}extenddb-manifest.json")); + } + "no sidecar" => { + mem.objects + .lock() + .unwrap() + .remove(&format!("{prefix}extenddb-manifest.sha256")); + } + "sidecar mismatch" => { + mem.insert(&format!("{prefix}extenddb-manifest.sha256"), b"0000"); + } + "unparseable manifest" => { + use sha2::Digest as _; + let garbage = b"{\"format_version\":1,\"backup_arn\":"; + mem.insert(&format!("{prefix}extenddb-manifest.json"), garbage); + let sidecar = format!( + "{} extenddb-manifest.json\n", + hex::encode(sha2::Sha256::digest(garbage)) + ); + mem.insert( + &format!("{prefix}extenddb-manifest.sha256"), + sidecar.as_bytes(), + ); + } + "missing data file" => { + mem.objects + .lock() + .unwrap() + .remove(&format!("{prefix}{}", golden.data_files[1].key)); + } + "oversized file" => { + let key = format!("{prefix}{}", golden.data_files[1].key); + let bytes = usize::try_from(golden.data_files[1].bytes).unwrap() + 1; + mem.insert(&key, &vec![0u8; bytes]); + } + "foreign ARN" => { + // The golden backup's objects, whole, under another row's + // prefix: that row's manifest names the golden backup. + let other = arn_n(3); + let other_prefix = BackupArn::parse(&other).unwrap().store_prefix(); + let objects: Vec<(String, Vec)> = mem + .objects + .lock() + .unwrap() + .iter() + .filter(|(k, _)| k.starts_with(&prefix)) + .map(|(k, v)| (k.clone(), v.clone())) + .collect(); + for (k, v) in objects { + mem.objects.lock().unwrap().remove(&k); + mem.insert(&format!("{other_prefix}{}", &k[prefix.len()..]), &v); + } + fake.rows.lock().unwrap().remove(&dead); + fake.dead.lock().unwrap().remove(&dead); + let mut r = row(&other, BackupStatus::Creating); + r.heartbeat_at = 1234.5; + fake.rows.lock().unwrap().insert(other.clone(), r); + fake.dead.lock().unwrap().insert(other.clone()); + dead = other; + } + _ => {} + } + let objects_before = mem.keys().len(); + // Under the maintenance claim, so the FAILED-prefix duty of the same + // cycle does not run and the resolution's own effect is visible. + *fake.maintenance_held.lock().unwrap() = true; + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.creating_resolved, + [(dead.clone(), BackupStatus::Failed)], + "{shape}" + ); + assert!(report.failed_prefixes_deleted.is_empty()); + let row = fake.rows.lock().unwrap()[&dead].clone(); + assert_eq!(row.status, BackupStatus::Failed, "{shape}"); + assert_eq!( + row.failure_reason.as_deref(), + Some(reconciler::OWNER_DEAD_NO_MANIFEST) + ); + assert!(row.completed_at.is_some()); + assert_eq!(row.heartbeat_at, 1234.5); + assert_eq!(mem.keys().len(), objects_before, "{shape}: nothing deleted"); + assert!(mem.deletes.lock().unwrap().is_empty(), "{shape}"); + + // The next cycle without the claim: the FAILED row's heartbeat is + // stale (the dead owner's), the prefix lists objects, so it is + // deleted. + *fake.maintenance_held.lock().unwrap() = false; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.failed_prefixes_deleted, + std::slice::from_ref(&dead), + "{shape}" + ); + assert!(objects_under(&mem, &dead).is_empty(), "{shape}"); + assert_eq!(mem.keys(), [STORE_IDENTITY_KEY]); + } + + // A store error during the probe: the row is taken over and left + // CREATING, no status write, nothing deleted; the next cycle, with the + // store answering, resolves it. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + *mem.fail_gets_under.lock().unwrap() = Some(BackupArn::parse(&dead).unwrap().store_prefix()); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert!(report.creating_resolved.is_empty()); + assert!( + report.creating_left_to_owner.is_empty(), + "a probe that failed is neither resolved nor left to an owner" + ); + assert_eq!( + fake.takeovers.lock().unwrap().as_slice(), + std::slice::from_ref(&dead) + ); + let row = fake.rows.lock().unwrap()[&dead].clone(); + assert_eq!( + row.status, + BackupStatus::Creating, + "left for the next cycle" + ); + assert!(fake.status_writes.lock().unwrap().is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); + *mem.fail_gets_under.lock().unwrap() = None; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); +} + +/// A dead CREATING row whose status write fails on the claim connection is +/// counted neither as resolved nor as left to its owner: the row stays +/// CREATING for the next cycle (its takeover done, one write attempted), +/// nothing deleted; with the connection back the next cycle resolves it. +#[tokio::test] +async fn a_dead_creating_row_whose_status_write_errors_is_left_for_the_next_cycle() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + fake.status_write_error + .store(true, std::sync::atomic::Ordering::Release); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert!(report.creating_resolved.is_empty(), "{report:?}"); + assert!(report.creating_left_to_owner.is_empty(), "{report:?}"); + assert!(!report.acted(), "nothing changed"); + assert_eq!( + fake.takeovers.lock().unwrap().as_slice(), + std::slice::from_ref(&dead) + ); + assert_eq!(writes(&fake), [(JobStatus::Available, None)]); + let row = fake.rows.lock().unwrap()[&dead].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert!(mem.deletes.lock().unwrap().is_empty()); + fake.status_write_error + .store(false, std::sync::atomic::Ordering::Release); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); +} + +/// Within one cycle the dead-owner duty runs before the FAILED-prefix duty, +/// so a row the cycle just failed, whose heartbeat the dead owner left +/// stale, loses its prefix in that same cycle (the design states this as a +/// property of the cycle, not a convenience). +#[tokio::test] +async fn a_row_failed_by_the_cycle_loses_its_prefix_in_that_cycle() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, false); + assert!(!objects_under(&mem, &dead).is_empty()); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.creating_resolved, + [(dead.clone(), BackupStatus::Failed)] + ); + assert_eq!( + report.failed_prefixes_deleted, + std::slice::from_ref(&dead), + "the same cycle" + ); + assert!(objects_under(&mem, &dead).is_empty()); + assert_eq!( + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Failed + ); +} + +/// A dead CREATING row whose status write matches no row (another +/// reconciler took the row over in the gap before this one's write, item +/// 29's race on MongoDB and Cassandra) is left to that owner: counted as +/// such and not as resolved, the row still CREATING under the other +/// generation, nothing deleted. +#[tokio::test] +async fn a_dead_creating_row_taken_over_by_another_is_left_to_that_owner() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + fake.takeover_lost_to_another + .store(true, std::sync::atomic::Ordering::Release); + let runtime = runtime_on(&mem, store()); + let report = cycle_on(&fake, &runtime).await; + assert!(report.creating_resolved.is_empty()); + assert_eq!(report.creating_left_to_owner, std::slice::from_ref(&dead)); + assert!(!report.acted(), "nothing changed"); + let writes = fake.status_writes.lock().unwrap().clone(); + assert_eq!(writes.len(), 1); + assert_eq!(writes[0].to, JobStatus::Available); + let row = fake.rows.lock().unwrap()[&dead].clone(); + assert_eq!(row.status, BackupStatus::Creating); + assert_ne!(row.owner_generation, writes[0].generation); + assert!(mem.deletes.lock().unwrap().is_empty()); +} + +/// The deleting duties and their gates. A FAILED row's prefix is deleted +/// only once its heartbeat is older than `stale_job_after` and only while +/// no maintenance claim is held; a DELETED row's objects are swept; a FAILED +/// or DELETED row whose prefix is empty is removed once its `completed_at` +/// is older than `failed_row_retention` (a prefix emptied earlier in the +/// cycle counts), and kept while the retention has not passed or while its +/// prefix still lists objects (a FAILED row past its retention whose +/// heartbeat is fresh: no duty empties its prefix, so the tombstone duty +/// keeps the row); an AVAILABLE row is no tombstone, whatever its clock and +/// prefix. The dead-CREATING duty runs under the maintenance claim; +/// the deleting duties and the registrations sweep do not. With the store +/// unhealthy none of them runs. +#[tokio::test] +async fn the_reconciler_deletes_failed_and_deleted_prefixes_and_removes_tombstones_on_their_clocks() +{ + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let now = seconds_since_epoch(); + let stale_failed = arn_n(1); + let fresh_failed = arn_n(2); + let deleted_with_objects = arn_n(3); + let old_failed_tombstone = arn_n(4); + let old_deleted_tombstone = arn_n(5); + let young_tombstone = arn_n(6); + let old_tombstone_with_objects = arn_n(7); + let old_failed_with_objects_and_a_live_heartbeat = arn_n(8); + for (a, status, heartbeat_at, completed_at, objects) in [ + ( + &stale_failed, + BackupStatus::Failed, + now - 1000.0, + Some(now - 1000.0), + true, + ), + (&fresh_failed, BackupStatus::Failed, now, Some(now), true), + ( + &deleted_with_objects, + BackupStatus::Deleted, + now - 1000.0, + Some(now - 1000.0), + true, + ), + ( + &old_failed_tombstone, + BackupStatus::Failed, + now - 1000.0, + Some(now - 1000.0), + false, + ), + ( + &old_deleted_tombstone, + BackupStatus::Deleted, + now - 1000.0, + Some(now - 1000.0), + false, + ), + ( + &young_tombstone, + BackupStatus::Failed, + now - 1000.0, + Some(now - 10.0), + false, + ), + ( + &old_tombstone_with_objects, + BackupStatus::Deleted, + now - 1000.0, + Some(now - 1000.0), + true, + ), + ( + &old_failed_with_objects_and_a_live_heartbeat, + BackupStatus::Failed, + now, + Some(now - 1000.0), + true, + ), + ] { + let mut r = row(a, status); + r.heartbeat_at = heartbeat_at; + r.completed_at = completed_at; + fake.rows.lock().unwrap().insert(a.clone(), r); + if objects { + let prefix = BackupArn::parse(a).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + } + } + let retention = Duration::from_secs(600); + let settings = JobSettings { + failed_row_retention: retention, + ..job_settings() + }; + + // Unhealthy (the root's identity file names another store while the + // store answers): no duty over the store's rows runs, the dead-CREATING + // one included. + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + mem.insert( + STORE_IDENTITY_KEY, + &identity_file("another-store", "catalog-1"), + ); + let runtime = runtime_on(&mem, store()).with_job_settings(settings); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(false)); + assert!(report.creating_resolved.is_empty()); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + assert!(report.tombstones_removed.is_empty()); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(fake.rows.lock().unwrap().len(), 9); + assert_eq!( + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Creating + ); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + + // Under the maintenance claim, the store healthy again: the + // dead-CREATING duty runs, the deleting duties and the registrations + // sweep (which the backend has something for) do not. + fake.swept_registrations.lock().unwrap().push(arn_n(9)); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert!(report.maintenance_held); + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + assert!(report.tombstones_removed.is_empty()); + assert!( + report.registrations_swept.is_empty(), + "the sweep does not run under the claim" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(fake.rows.lock().unwrap().len(), 9); + + // An AVAILABLE row past the retention with an empty prefix: a tombstone + // is a FAILED or DELETED row and nothing else. It enters the catalog + // here, older than the row the cycle above resolved, because the + // store's check heads the newest AVAILABLE row's manifest and this row + // has none. + let old_available_with_an_empty_prefix = + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/01700000000000-00000000"); + let mut r = row(&old_available_with_an_empty_prefix, BackupStatus::Available); + r.heartbeat_at = now - 1000.0; + r.completed_at = Some(now - 1000.0); + fake.rows + .lock() + .unwrap() + .insert(old_available_with_an_empty_prefix.clone(), r); + + // The claim released: each duty on its clock. + *fake.maintenance_held.lock().unwrap() = false; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.failed_prefixes_deleted, + std::slice::from_ref(&stale_failed) + ); + assert_eq!( + report.deleted_prefixes_swept, + [ + deleted_with_objects.clone(), + old_tombstone_with_objects.clone() + ] + ); + // The tombstone duty runs after the two deleting duties, so a prefix + // they emptied this cycle counts as empty here. + let mut removed = report.tombstones_removed.clone(); + removed.sort(); + assert_eq!( + removed, + [ + stale_failed.clone(), + deleted_with_objects.clone(), + old_failed_tombstone.clone(), + old_deleted_tombstone.clone(), + old_tombstone_with_objects.clone(), + ] + ); + assert_eq!(report.registrations_swept, [arn_n(9)]); + assert!(report.acted()); + assert_eq!( + objects_under(&mem, &fresh_failed).len(), + 1, + "a FAILED row with a live heartbeat keeps its prefix" + ); + assert!(objects_under(&mem, &stale_failed).is_empty()); + assert!(objects_under(&mem, &deleted_with_objects).is_empty()); + assert_eq!( + objects_under(&mem, &old_failed_with_objects_and_a_live_heartbeat).len(), + 1, + "no duty empties the prefix of a FAILED row with a live heartbeat" + ); + { + let rows = fake.rows.lock().unwrap(); + assert!(rows.contains_key(&fresh_failed), "its heartbeat is live"); + assert!( + rows.contains_key(&young_tombstone), + "its retention has not passed" + ); + assert!( + rows.contains_key(&old_failed_with_objects_and_a_live_heartbeat), + "its prefix still lists an object" + ); + assert!( + rows.contains_key(&old_available_with_an_empty_prefix), + "AVAILABLE is not a tombstone status" + ); + assert_eq!( + rows[&old_available_with_an_empty_prefix].status, + BackupStatus::Available + ); + assert_eq!(rows.len(), 5, "the four kept and the resolved CREATING row"); + } + assert!( + !fake + .remove_attempts + .lock() + .unwrap() + .contains(&old_available_with_an_empty_prefix), + "the duty asks the catalog to remove FAILED and DELETED rows only; the backends would \ + refuse an AVAILABLE one, and the duty does not lean on that" + ); + + // The next cycle has nothing left to do. + let report = cycle_on(&fake, &runtime).await; + assert!(report.tombstones_removed.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.creating_resolved.is_empty()); +} + +// --- the review's cells (PR 8, Lee's items) ------------------------------- + +/// An item of the `Music` table carrying `n` pseudo-random bytes (a `B` +/// attribute gzip cannot shrink), under the default item limit at 300,000. +fn song_with_noise(i: usize, n: usize) -> Item { + use extenddb_core::types::AttributeValue; + let mut state: u64 = 0x9E37_79B9_7F4A_7C15 ^ (i as u64 + 1); + let noise: Vec = (0..n) + .map(|_| { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + (state >> 24) as u8 + }) + .collect(); + let mut item = song(&format!("artist-{i:05}"), &format!("title-{i:05}")); + item.insert("Noise".to_owned(), AttributeValue::B(noise)); + item +} + +/// The mid-file case of +/// `a_put_failure_ends_the_job_without_waiting_for_the_streams_next_item`: +/// the data file's first chunk is out during the third item (three +/// incompressible items of 380,000 bytes, 1.15 MB once base64-encoded and +/// gzipped), the stream is held before the fourth, and the put fails 200 ms +/// after reading that chunk, by when the encoder has finished the third +/// item and waits on the empty item channel while the feeder waits on the +/// stream, which the test never releases. The uploader cancels the feeder's +/// token before it awaits the encoder, so the feeder stops, closes the item +/// channel, the encoder returns, and the row is FAILED naming the file +/// within the test's bound; with the encoder awaited first the job runs for +/// as long as the stream is held. +#[tokio::test] +async fn a_mid_file_put_failure_ends_the_job_while_the_stream_is_held() { + let mem = Arc::new(MemStore::default()); + mem.fail_puts_after_first_chunk + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + let items: Vec = (0..5).map(|i| song_with_noise(i, 380_000)).collect(); + let gate = Arc::new(tokio::sync::Semaphore::new(3)); + snapshot_fixture(&fake, &music_table(), items, true, Some(Arc::clone(&gate))); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs_within(&runtime, Duration::from_secs(10)).await; + assert_eq!(gate.available_permits(), 0, "the stream was never released"); + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert!( + row.failure_reason + .as_deref() + .unwrap() + .starts_with("writing data/000001.json.gz: "), + "{:?}", + row.failure_reason + ); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + assert_eq!(mem.deletes.lock().unwrap().as_slice(), [prefix]); + assert!(mem.puts.lock().unwrap().is_empty(), "the put never stored"); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + assert_eq!(runtime.free_slots(), 4); +} + +/// The check before every put reads the store as the runtime holds it. A +/// job held in its stream while the reconciler marks the store unhealthy +/// (here `record_check` with the problem a check would report) fails its +/// row with the store's reason and makes no store call: no put, no +/// `delete_prefix` (the FAILED-prefix duty deletes the prefix once the store +/// is healthy again). A store whose check passed with another id (the +/// identity file names another store: a root swapped under the job) fails +/// the row the same way, naming both ids. +#[tokio::test] +async fn a_store_marked_unhealthy_or_renamed_mid_job_fails_the_row_with_no_store_call() { + // Unhealthy. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + runtime + .configured_store() + .unwrap() + .record_check(Err(StoreProblem::FileUnreadable( + "not an identity file".to_owned(), + ))); + assert!(!runtime.configured_store().unwrap().is_healthy()); + gate.add_permits(3); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let reason = row.failure_reason.clone().unwrap(); + assert!( + reason.starts_with("the filesystem backup store is unhealthy (") + && reason.contains("extenddb-store.json could not be read: not an identity file") + && reason.ends_with("); data/000001.json.gz was not written"), + "{reason}" + ); + assert!(row.completed_at.is_some()); + assert!(mem.puts.lock().unwrap().is_empty(), "no put"); + assert!(mem.deletes.lock().unwrap().is_empty(), "no delete_prefix"); + assert!(mem.keys().is_empty()); + assert_eq!( + writes(&fake), + [(JobStatus::Failed, Some(reason))], + "the FAILED write and nothing else" + ); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + assert_eq!(runtime.free_slots(), 4); + + // Another id. + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let gate = closed_gate(); + snapshot_fixture( + &fake, + &music_table(), + songs(3), + true, + Some(Arc::clone(&gate)), + ); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + let swapped = StoreIdentity { + store_id: "store-2".to_owned(), + ..store() + }; + runtime + .configured_store() + .unwrap() + .record_check(Ok(swapped)); + assert!(runtime.configured_store().unwrap().is_healthy()); + gate.add_permits(3); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert_eq!( + row.failure_reason.as_deref(), + Some( + "the filesystem backup store's id is store-2 and the row's is store-1; \ + data/000001.json.gz was not written" + ) + ); + assert!(mem.puts.lock().unwrap().is_empty(), "no put"); + assert!(mem.deletes.lock().unwrap().is_empty(), "no delete_prefix"); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A key-ordered stream that is not in key order: the manifest would say +/// `KeyOrdered` over items a restore's key-range seek cannot trust, so the +/// tracker refuses the item and the job fails `encoding data file 1` with +/// an empty prefix (no chunk was out, so no put started). One cell with a +/// lower key after a higher one, one with a key repeated. +#[tokio::test] +async fn a_key_ordered_stream_out_of_order_fails_the_first_data_file() { + let lower = vec![ + song("artist-00002", "title-00002"), + song("artist-00001", "title-00001"), + ]; + let repeated = vec![ + song("artist-00001", "title-00001"), + song("artist-00001", "title-00001"), + ]; + for (what, items) in [("a lower key", lower), ("a repeated key", repeated)] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), items, true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed, "{what}"); + assert!( + row.failure_reason + .as_deref() + .unwrap() + .starts_with("encoding data file 1: "), + "{what}: {:?}", + row.failure_reason + ); + assert!(mem.puts.lock().unwrap().is_empty(), "{what}: no put"); + assert!( + objects_under(&mem, &details.backup_arn).is_empty(), + "{what}: an empty prefix" + ); + assert_eq!( + job_outcomes(&runtime), + [("failed".to_owned(), 1.0)], + "{what}" + ); + } +} + +/// `BackupStoreErrors` is recorded for the job's `delete_prefix` when its +/// failure path's deletion fails (the row FAILED, the prefix left to the +/// reconciler), and for the probe's `list` when the listing of a dead +/// owner's prefix fails after its manifest read (the row left for the next +/// cycle); each once, under the call's name. +#[tokio::test] +async fn the_jobs_failed_deletion_and_the_probes_failed_listing_are_counted() { + use extenddb_core::metrics::MetricName; + let errors = |drained: &[extenddb_core::metrics::FlushBucket]| { + let mut ops: Vec<(String, f64)> = drained + .iter() + .filter(|b| b.metric == MetricName::BackupStoreErrors) + .map(|b| (b.operation.clone(), b.sum)) + .collect(); + ops.sort_by(|a, b| a.0.cmp(&b.0)); + ops + }; + + // The job's deletion. + let mem = Arc::new(MemStore::default()); + mem.fail_puts + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + let prefix = BackupArn::parse(&details.backup_arn) + .unwrap() + .store_prefix(); + // Set before the job reaches its deletion: the put fails first and the + // FAILED write comes between; the prefix is known from the response. + *mem.fail_deletes_of.lock().unwrap() = Some(prefix.clone()); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + assert!( + mem.deletes.lock().unwrap().is_empty(), + "the deletion failed" + ); + assert_eq!( + errors(&runtime.metrics().unwrap().drain(Duration::ZERO)), + [("delete_prefix".to_owned(), 1.0), ("put".to_owned(), 1.0)] + ); + + // The probe's listing. + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Fake::default(); + let dead = seed_dead_creating_with_manifest(&fake, &mem, true); + let dead_prefix = BackupArn::parse(&dead).unwrap().store_prefix(); + *mem.fail_lists_of.lock().unwrap() = Some(dead_prefix); + let metrics = Arc::new(extenddb_core::metrics::MetricsCollector::new()); + let runtime = runtime_on(&mem, store()).with_metrics(Arc::clone(&metrics)); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert!(report.creating_resolved.is_empty(), "{report:?}"); + assert_eq!( + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Creating, + "left for the next cycle" + ); + assert_eq!( + errors(&metrics.drain(Duration::ZERO)), + [("list".to_owned(), 1.0)] + ); + *mem.fail_lists_of.lock().unwrap() = None; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); + assert!(errors(&metrics.drain(Duration::ZERO)).is_empty()); +} + +/// `write_status`'s generation check: a FAILED write whose outcome is +/// unknown is re-read, and a row that shows FAILED under another generation +/// is `NotMatched` (the reconciler resolved the same backup; the row is not +/// this claim's), so the failure path makes no `delete_prefix`. Without the +/// check the status alone would read as `Matched`. +#[tokio::test] +async fn a_re_read_showing_the_target_status_under_another_generation_is_not_matched() { + let mem = Arc::new(MemStore::default()); + let fake = Fake::default(); + let arn = arn(ACCOUNT); + let prefix = BackupArn::parse(&arn).unwrap().store_prefix(); + mem.insert(&format!("{prefix}data/000001.json.gz"), b"x"); + let mut r = row(&arn, BackupStatus::Failed); + r.owner_generation = 9; + fake.rows.lock().unwrap().insert(arn.clone(), r); + let claim = fake.claim_on(&arn, 7); + *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::Lost); + let path = job::failure::fail_row(&*claim, &*mem, &prefix, "a reason", None).await; + match path { + job::failure::FailurePath::NoStoreAccess(why) => { + assert_eq!(why, "the row carries generation 9 (this claim's is 7)"); + } + other => panic!("{other:?}"), + } + assert!(mem.deletes.lock().unwrap().is_empty(), "no delete_prefix"); + assert_eq!(objects_under(&mem, &arn).len(), 1); + assert_eq!(fake.status_writes.lock().unwrap().len(), 1, "one write"); +} + +/// A data file whose stored size, as the store reports it, differs from the +/// encoder's compressed count fails the job: the manifest records the +/// encoder's count, the reconciler's dead-owner probe refuses a data file +/// whose stored size differs from the recorded one, and a live job must not +/// commit a manifest that probe would call incomplete. FAILED names both +/// figures; the prefix is deleted. +#[tokio::test] +async fn a_data_file_whose_stored_size_differs_from_the_encoders_fails_the_job() { + let mem = Arc::new(MemStore::default()); + mem.misreport_put_size + .store(true, std::sync::atomic::Ordering::Release); + let fake = Arc::new(Fake::default()); + snapshot_fixture(&fake, &music_table(), songs(2), true, None); + let runtime = job_runtime(&mem); + let details = create_music_backup(&fake, &runtime).await.unwrap(); + wait_for_jobs(&runtime).await; + let row = fake.rows.lock().unwrap()[&details.backup_arn].clone(); + assert_eq!(row.status, BackupStatus::Failed); + let stored = mem.put_sizes.lock().unwrap()[0].1; + assert_eq!( + row.failure_reason.as_deref(), + Some( + format!( + "the store reports {} bytes for data/000001.json.gz and the encoder wrote {stored}", + stored + 1 + ) + .as_str() + ) + ); + assert_eq!(mem.puts.lock().unwrap().len(), 1, "no document followed"); + assert!(objects_under(&mem, &details.backup_arn).is_empty()); + assert_eq!(job_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +// --- the maintainer's second round on PR 8: a beat parked on the write lock -- + +mod review_round_3 { + use super::*; + + /// The pre-put check's third clause: a beat parked outside its bound + /// (SQLite's engine write lock, held by a GSI build) confirms the claim + /// however old the last confirmed beat is, and stops confirming the + /// moment the beat returns `Lost`. A parked beat that confirms on its + /// release refreshes the last confirmed instant and clears the flight, + /// so a timer that then stops leaves the check to the two-period rule. + #[tokio::test(start_paused = true)] + async fn a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost() { + let fake = Arc::new(Fake::default()); + fake.rows + .lock() + .unwrap() + .insert(arn(ACCOUNT), row(&arn(ACCOUNT), BackupStatus::Creating)); + // SQLite's shape: the park is the write lock's queue. + fake.parks_before_bound + .store(true, std::sync::atomic::Ordering::Release); + let claim: Arc = Arc::from(fake.claim_on(&arn(ACCOUNT), 7)); + let period = Duration::from_secs(10); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Park; + let handle = job::heartbeat::HeartbeatHandle::start( + Arc::clone(&claim), + period, + None, + 0.0, + tokio::time::Instant::now(), + ); + // The first tick's beat parks at one period; at two and a half the + // claim's insert is too old for the two-period clause alone. + tokio::time::sleep(2 * period + period / 2).await; + assert_eq!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire), + 0 + ); + assert!(!handle.lost().is_cancelled()); + assert!( + handle.confirms_the_claim(), + "a beat parked on the lock confirms the claim" + ); + tokio::time::sleep(10 * period).await; + assert!(handle.confirms_the_claim(), "however long it is parked"); + assert!(!handle.lost().is_cancelled()); + // The lock is released and the beat finds the claim lost. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + fake.park_release.notify_one(); + tokio::time::sleep(Duration::from_millis(1)).await; + assert_eq!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire), + 1 + ); + assert!(handle.lost().is_cancelled()); + assert!( + !handle.confirms_the_claim(), + "the loss is read before the flight" + ); + assert!(handle.reason().contains("the claim was lost")); + drop(handle); + + // The same park, released with a confirmation: the check passes + // from the fresh beat, and once the timer stops the flight is gone + // and the two-period rule refuses two and a half periods later. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Park; + let handle = job::heartbeat::HeartbeatHandle::start( + claim, + period, + None, + 0.0, + tokio::time::Instant::now(), + ); + tokio::time::sleep(5 * period).await; + assert!(handle.confirms_the_claim()); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Confirm; + fake.park_release.notify_one(); + tokio::time::sleep(Duration::from_millis(1)).await; + assert!(!handle.lost().is_cancelled()); + assert!(handle.confirms_the_claim()); + handle.stop_timer(); + tokio::time::sleep(2 * period + period / 2).await; + assert!( + !handle.confirms_the_claim(), + "no beat in flight and the last confirmed beat too old" + ); + assert!(!handle.lost().is_cancelled()); + } + + /// On a backend whose beat does not park before its bound (PostgreSQL, + /// MongoDB, Cassandra), a beat in flight confirms the claim only within + /// one period of its start: a process stopped mid-beat (SIGSTOP, a + /// paused VM) and resumed keeps the beat in flight until its timer runs + /// again and raises the loss, and in that window another owner may hold + /// the row, so the check refuses it; the same beat on a backend whose + /// beat parks (SQLite) confirms the claim for as long as it waits. + #[tokio::test(start_paused = true)] + async fn a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks() { + let fake = Arc::new(Fake::default()); + fake.rows + .lock() + .unwrap() + .insert(arn(ACCOUNT), row(&arn(ACCOUNT), BackupStatus::Creating)); + let period = Duration::from_secs(10); + for parks in [false, true] { + fake.parks_before_bound + .store(parks, std::sync::atomic::Ordering::Release); + let claim: Arc = Arc::from(fake.claim_on(&arn(ACCOUNT), 7)); + // `Park` holds the beat outside the bound: the stop, on a backend + // that does not park, or the lock's queue on one that does. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Park; + // The claim's insert is old enough that the two-period clause + // alone refuses from the first tick on. + let handle = job::heartbeat::HeartbeatHandle::start( + claim, + period, + None, + 0.0, + tokio::time::Instant::now() - 2 * period, + ); + // The first tick's beat starts at one period; half a period in it + // is within its bound on every backend. + tokio::time::sleep(period + period / 2).await; + assert!(!handle.lost().is_cancelled()); + assert!( + handle.confirms_the_claim(), + "parks={parks}: a beat within its bound confirms the claim" + ); + // One and a half periods into the beat: past its bound. + tokio::time::sleep(period).await; + assert!(!handle.lost().is_cancelled(), "the timer has not run again"); + assert_eq!( + handle.confirms_the_claim(), + parks, + "parks={parks}: a beat past its bound confirms the claim only where it parks" + ); + // The beat returns Lost: the loss is read before the flight. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + fake.park_release.notify_one(); + tokio::time::sleep(Duration::from_millis(1)).await; + assert!(handle.lost().is_cancelled()); + assert!(!handle.confirms_the_claim()); + drop(handle); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Confirm; + } + } +} + /// A restore of a backup the catalog index has a row for checks the target /// name once the row is known to exist and AVAILABLE, before the backup's /// in-use rule and before the store checks, the service's order (measured on diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index 3f8828d0b..b6c0e25b9 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -9,9 +9,10 @@ mod backup; pub use backup::{ - BackupRuntime, ConfiguredStore, HeldClaim, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, - STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, - check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity, + BackupRuntime, ConfiguredStore, HeldClaim, JobSettings, KillPoint, LOCAL_CHECK_TIMEOUT, + S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, + StoreProblem, check_configured_store, maintenance, read_identity_bounded, reconciler, + store_identity, }; mod batch_get_item; mod batch_write_item; diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml index e674e5c58..ed72f54fb 100755 --- a/crates/server/Cargo.toml +++ b/crates/server/Cargo.toml @@ -9,6 +9,9 @@ license.workspace = true [features] mongodb-test-hooks = [] +# The backup job's kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), for the +# binary-level crash tests. +backup-test-hooks = ["extenddb-engine/backup-test-hooks"] [dependencies] extenddb-cache = { workspace = true } diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs index 14dc6949a..8fc5f3518 100644 --- a/crates/server/src/backup_store.rs +++ b/crates/server/src/backup_store.rs @@ -79,7 +79,9 @@ use std::sync::Arc; use extenddb_backup_store::{BackupStore, BackupStoreKind, StoreError}; use extenddb_config::backup::{BackupConfig, EffectiveStore, Unconfigured}; pub use extenddb_engine::LOCAL_CHECK_TIMEOUT; -use extenddb_engine::{BackupRuntime, ConfiguredStore, Spool, StoreProblem, store_identity}; +use extenddb_engine::{ + BackupRuntime, ConfiguredStore, JobSettings, Spool, StoreProblem, store_identity, +}; use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind}; use extenddb_storage::{BackupCatalog, StorageEngine}; @@ -200,6 +202,26 @@ impl StoreSetup { } } +/// Why CreateBackup is bound to the in-catalog path on `backend`, whatever +/// the store configuration, or `None` where the store-backed job runs. +/// Cassandra cannot pin a table snapshot. SQLite's job claim is an +/// in-process slot, sound only with one process per database file, and the +/// serve lock that enforces that is `flock`, taken on Unix only +/// (`serve_lock.rs`): two servers on one file on another platform would +/// each take the other's rows for dead, so there the create stays +/// in-catalog and the store serves restores and deletes. +#[must_use] +pub fn in_catalog_create_only(backend: &str) -> Option<&'static str> { + match backend { + "cassandra" => Some("this backend cannot pin a table snapshot"), + "sqlite" if !cfg!(unix) => Some( + "the serve lock that keeps one process per SQLite database file is taken on Unix \ + only, and the job's claim is sound only with one process per file", + ), + _ => None, + } +} + /// Build the backup runtime for this process: the job slots and, when the /// configuration and the backend select one, the store with its health /// ([`configure_described`] without the reason for an absent store). @@ -218,10 +240,28 @@ pub async fn configure( Ok((setup.runtime, setup.spool_lock)) } -/// Build the backup runtime for this process: the job slots and, when the -/// configuration and the backend select one, the store with its health; -/// otherwise why there is none. `catalog` is the catalog the identity check -/// reads and the rows rule of the module doc consults. +/// The `[backup]` settings the jobs and the reconciler read, as the +/// configuration resolves them: the heartbeat period is +/// [`BackupConfig::effective_heartbeat_interval_secs`]. +#[must_use] +pub fn job_settings(backup: &BackupConfig) -> JobSettings { + JobSettings { + data_file_target_bytes: backup.data_file_target_bytes, + heartbeat_interval: std::time::Duration::from_secs( + backup.effective_heartbeat_interval_secs(), + ), + stale_job_after: std::time::Duration::from_secs(backup.stale_job_after_secs), + failed_row_retention: std::time::Duration::from_secs(backup.failed_row_retention_secs), + } +} + +/// Build the backup runtime for this process: the job slots, the job +/// settings and, when the configuration and the backend select one, the +/// store with its health and the names the DynamoDB-shaped manifests carry +/// for it; otherwise why there is none. `catalog` is the catalog the +/// identity check reads and the rows rule of the module doc consults. The +/// server adds the metrics collector, the shutdown token and the source +/// names to the runtime it gets back. /// /// # Errors /// @@ -233,9 +273,16 @@ pub async fn configure_described( catalog: &C, caller: Caller, ) -> anyhow::Result { + let in_catalog_only = in_catalog_create_only(backend); + if let Some(why) = in_catalog_only { + tracing::info!( + "CreateBackup on this server writes the in-catalog format whatever [backup.store] \ + says: {why}" + ); + } let runtime = BackupRuntime::new(backup.max_concurrent_jobs) - .with_legacy_create_only(backend == "cassandra") - .with_stale_job_after(std::time::Duration::from_secs(backup.stale_job_after_secs)); + .with_legacy_create_only(in_catalog_only.is_some()) + .with_job_settings(job_settings(backup)); let resolved = backup .resolve_store(database_file) .map_err(|e| anyhow::anyhow!("[backup] configuration: {e}"))?; @@ -328,6 +375,15 @@ pub async fn configure_described( BackupStoreKind::S3 => StoreKind::S3, BackupStoreKind::Disabled => unreachable!("open_described never returns Disabled"), }; + // `manifest-summary.json`'s `s3Bucket` and the key prefix every key of + // the DynamoDB-shaped manifests starts with: the bucket and the + // configured prefix on S3, the root path and no prefix on a filesystem + // store. + let (bucket, key_prefix) = match (kind, &config.s3) { + (StoreKind::S3, Some(s3)) => (s3.bucket.clone(), s3.prefix.clone()), + _ => (opened.root.clone(), String::new()), + }; + let configured = |store: ConfiguredStore| store.with_manifest_names(bucket, key_prefix); // The spool and its lock: configuration errors on every backend. let (spool, spool_lock) = match (resolved.spool_path, caller) { @@ -346,19 +402,28 @@ pub async fn configure_described( // listener is up, in the reconciler's first cycle. A read-only command // runs no check at all. if caller == Caller::ReadOnlyCommand { - let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool); + let configured = configured(ConfiguredStore::checking( + kind, + opened.root, + opened.store, + spool, + )); return Ok(StoreSetup::with_store(runtime, configured, spool_lock)); } if kind == StoreKind::S3 { if caller == Caller::Server { tracing::info!( "backup store: {kind} {}{spool_line}; its validation and identity check run \ - after the listener is up, and CreateBackup writes in-catalog backups in this \ - release", + after the listener is up, and CreateBackup writes to it once a check passes", opened.root ); } - let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool); + let configured = configured(ConfiguredStore::checking( + kind, + opened.root, + opened.store, + spool, + )); return Ok(StoreSetup::with_store(runtime, configured, spool_lock)); } @@ -404,16 +469,17 @@ pub async fn configure_described( ); // `defaulted` travels with the store, so the reconciler applies // the rows rule to the first check's outcome. - let configured = ConfiguredStore::checking(kind, opened.root, opened.store, spool) - .defaulted(defaulted); + let configured = configured( + ConfiguredStore::checking(kind, opened.root, opened.store, spool) + .defaulted(defaulted), + ); return Ok(StoreSetup::with_store(runtime, configured, spool_lock)); } }; match &outcome { Ok(identity) => tracing::info!( - "backup store: {kind} {} (store id {}){spool_line}; CreateBackup writes in-catalog \ - backups in this release, and the store serves restores and deletes of the backups \ - it holds", + "backup store: {kind} {} (store id {}){spool_line}; CreateBackup writes to it, and \ + it serves restores and deletes of the backups it holds", opened.root, identity.store_id ), @@ -436,7 +502,13 @@ pub async fn configure_described( rows_note(defaulted) ), } - let configured = ConfiguredStore::new(kind, opened.root, opened.store, spool, outcome); + let configured = configured(ConfiguredStore::new( + kind, + opened.root, + opened.store, + spool, + outcome, + )); Ok(StoreSetup::with_store(runtime, configured, spool_lock)) } @@ -863,7 +935,7 @@ mod tests { use extenddb_storage::BoxedFuture; use extenddb_storage::backup_catalog::{ BackupKey, BackupRow, BackupStatus, DeleteStart, JobClaim, MaintenanceClaim, RestoreJobRow, - RestoreTargetKey, TimeBounds, + RestoreTargetKey, StoreIdentity, TimeBounds, }; use extenddb_storage::error::StorageError; use futures::stream::BoxStream; @@ -895,6 +967,13 @@ mod tests { ) -> BoxedFuture<'a, Result, StorageError>> { unused() } + fn claim_dead_backup<'a>( + &'a self, + _key: &'a BackupKey, + _stale_after: std::time::Duration, + ) -> BoxedFuture<'a, Result>, StorageError>> { + unused() + } fn list_backup_rows<'a>( &'a self, _status: Option, diff --git a/crates/server/src/lib.rs b/crates/server/src/lib.rs index dd4cb480b..7df7aba50 100755 --- a/crates/server/src/lib.rs +++ b/crates/server/src/lib.rs @@ -30,8 +30,8 @@ mod throttle_helpers; mod workers; pub use serve::{ - BuildInfo, CatalogRefusal, LogTarget, RegistrationLost, ServeParams, catalog_refusal, - log_to_syslog_raw, serve, + BuildInfo, CatalogRefusal, JOB_DRAIN_TIMEOUT, LogTarget, RegistrationLost, ServeParams, + WORKER_STOP_TIMEOUT, catalog_refusal, log_to_syslog_raw, serve, }; use std::path::PathBuf; @@ -115,6 +115,11 @@ pub struct ServerTlsConfig { pub key_path: PathBuf, } +/// How long the listener drains active connections after a stop: the bound +/// on HTTPS, and on plain HTTP the time after which the PID file is removed. +/// `extenddb stop` waits for the sum of the shutdown bounds. +pub const CONNECTION_DRAIN_TIMEOUT: Duration = Duration::from_secs(5); + /// Build and start the HTTP server on a pre-bound listener. /// /// The caller is responsible for binding the `TcpListener` before passing it in. @@ -266,7 +271,7 @@ pub async fn start_server( let stop = admission.clone(); tokio::spawn(async move { signal_or_stop(&stop).await; - shutdown_handle.graceful_shutdown(Some(Duration::from_secs(5))); + shutdown_handle.graceful_shutdown(Some(CONNECTION_DRAIN_TIMEOUT)); // P50: Log timeout but don't call std::process::exit — let the // runtime shut down normally so destructors (including ZeroizeOnDrop) run. tokio::time::sleep(Duration::from_secs(10)).await; @@ -289,12 +294,34 @@ pub async fn start_server( .await?; } else { tracing::info!("extenddb listening on {local_addr}"); - axum::serve( + // The plain-HTTP listener's drain is bounded as the TLS listener's + // is: axum's graceful shutdown waits for every open connection with + // no bound of its own, and `extenddb stop` waits for the sum of the + // server's bounds, which counts this drain as `CONNECTION_DRAIN_TIMEOUT`. + let stopping = extenddb_storage::CancellationToken::new(); + let serve = axum::serve( nodelay_listener(listener), app.into_make_service_with_connect_info::(), ) - .with_graceful_shutdown(graceful_shutdown(pid_file.clone(), admission)) - .await?; + .with_graceful_shutdown({ + let stopping = stopping.clone(); + let pid_file = pid_file.clone(); + async move { + graceful_shutdown(pid_file, admission).await; + stopping.cancel(); + } + }); + let drain_bound = async { + stopping.cancelled().await; + tokio::time::sleep(CONNECTION_DRAIN_TIMEOUT).await; + }; + tokio::select! { + served = serve => served?, + () = drain_bound => tracing::warn!( + "connections still open {} s after the stop; the listener is closed with them", + CONNECTION_DRAIN_TIMEOUT.as_secs() + ), + } } // Normal shutdown path — clean up PID file after connections drain. @@ -344,8 +371,11 @@ async fn graceful_shutdown(pid_file: Option, admission: registration::A // don't drain. Does NOT call std::process::exit — the runtime shuts down // normally so destructors (including ZeroizeOnDrop) run. tokio::spawn(async move { - tokio::time::sleep(Duration::from_secs(5)).await; - tracing::warn!("Graceful shutdown timed out after 5s, forcing PID file cleanup"); + tokio::time::sleep(CONNECTION_DRAIN_TIMEOUT).await; + tracing::warn!( + "Graceful shutdown timed out after {}s, forcing PID file cleanup", + CONNECTION_DRAIN_TIMEOUT.as_secs() + ); cleanup_pid_file(pid_file.as_deref()); }); } diff --git a/crates/server/src/metrics_endpoint.rs b/crates/server/src/metrics_endpoint.rs index 0ec7b06fd..9803f3b23 100755 --- a/crates/server/src/metrics_endpoint.rs +++ b/crates/server/src/metrics_endpoint.rs @@ -364,4 +364,45 @@ mod tests { "GSI-name dimension must be retained, got {dims:?}" ); } + + /// The backup job's metric names round-trip through the store's string + /// column: a row written under each name renders as that name and not + /// as the fallback `SystemErrors`, with its outcome or store-call + /// dimension kept. + #[test] + fn backup_job_metric_names_render_from_the_store() { + use extenddb_core::metrics::MetricName; + let names = [ + MetricName::BackupJobs, + MetricName::BackupJobDuration, + MetricName::BackupBytesUploaded, + MetricName::BackupSnapshotAge, + MetricName::BackupStoreErrors, + ]; + let rows: Vec = names + .iter() + .map(|name| MetricsRow { + bucket: time::OffsetDateTime::UNIX_EPOCH, + metric: name.to_string(), + table_name: None, + index_name: None, + operation: Some("succeeded".to_owned()), + sum: 1.0, + count: 1, + min: 1.0, + max: 1.0, + }) + .collect(); + let (snapshots, _buckets) = aggregate_rows(&rows, 60, TimeWindow::Last5Minutes); + let mut rendered: Vec = snapshots.iter().map(|s| s.metric).collect(); + rendered.sort_by_key(|m| m.to_string()); + let mut expected = names.to_vec(); + expected.sort_by_key(|m| m.to_string()); + assert_eq!(rendered, expected); + assert!(snapshots.iter().all(|s| { + s.dimensions + .iter() + .any(|d| matches!(d, Dimension::Operation(o) if o == "succeeded")) + })); + } } diff --git a/crates/server/src/registration.rs b/crates/server/src/registration.rs index a763f774b..7bcc563ec 100644 --- a/crates/server/src/registration.rs +++ b/crates/server/src/registration.rs @@ -9,7 +9,11 @@ //! `ServiceUnavailable` (HTTP 503), which the AWS SDKs retry, and `/health` //! answers the same. It also stops its background workers and the tasks //! they started: their token fires, and the ones still running after 5 s are -//! aborted. What requests started goes on until it ends or the process +//! aborted; and it cancels the store-backed backup jobs running then, each +//! of which fails its row with reason `shutdown` and deletes its objects, +//! since a job's heartbeats and status writes would otherwise go on beside +//! the migration (the jobs started under a new registration run on a fresh +//! token). What requests started goes on until it ends or the process //! exits: the requests already running, and the work they left running in //! the background, which is a `RestoreTableFromBackup`'s item copy on every //! backend, an `UpdateTimeToLive` backfill on Cassandra and PostgreSQL, and diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs index dc4acc2f4..b4d9c1b67 100644 --- a/crates/server/src/serve.rs +++ b/crates/server/src/serve.rs @@ -665,6 +665,11 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { // checked. A configuration error stops startup here; a transport or // identity failure starts the server with the store unhealthy // (`backup_store` module doc). The spool lock lives until shutdown. + // Workers and backup jobs run until `shutdown` is cancelled, which + // happens after the HTTP server stops accepting; the jobs are cancelled + // on a child token first, so their failure paths run and record their + // metrics before the workers (the metrics flush among them) are drained. + let shutdown = CancellationToken::new(); let backend_name = app_config.storage.backend.clone(); let (backup_runtime, _spool_lock) = crate::backup_store::configure( &app_config.backup, @@ -674,7 +679,12 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { crate::backup_store::Caller::Server, ) .await?; - let backup_runtime = Arc::new(backup_runtime); + let backup_runtime = Arc::new( + backup_runtime + .with_metrics(metrics.clone()) + .with_shutdown(shutdown.child_token()) + .with_source(&backend_name, build.version), + ); let backup_runtime_for_reconciler = Arc::clone(&backup_runtime); let state = AppState { storage: storage.clone(), @@ -706,13 +716,11 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { }; // The server's own workers and the backend's run as one set, each set on - // a child token of `shutdown`, which is cancelled after the HTTP server - // stops accepting. The registration supervisor stops the set while the - // server holds no registration and starts a new one once it holds one. - // Handles are kept so a stop can drain them (the metrics flush worker - // persists its final bucket on the way out) instead of leaving the work - // to a runtime drop. - let shutdown = CancellationToken::new(); + // a child token of `shutdown`. The registration supervisor stops the set + // while the server holds no registration and starts a new one once it + // holds one. Handles are kept so a stop can drain them (the metrics flush + // worker persists its final bucket on the way out) instead of leaving the + // work to a runtime drop. let hooks: Option> = runtime_hooks.map(Arc::from); let reconciler_settings = extenddb_engine::reconciler::ReconcilerSettings { interval: std::time::Duration::from_secs(app_config.backup.reconcile_interval_secs), @@ -730,7 +738,7 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { let config_log_level = app_config.logging.level.clone(); let metrics = metrics.clone(); let storage = storage.clone(); - let backup_runtime = backup_runtime_for_reconciler; + let backup_runtime = Arc::clone(&backup_runtime_for_reconciler); Box::new(move |token: CancellationToken| { let catalog_store = catalog_store.clone(); let reload_handle = reload_handle.clone(); @@ -804,7 +812,10 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { }) }) }; - let background = Arc::new(BackgroundWorkers::new(spawn, &shutdown)); + let background = Arc::new( + BackgroundWorkers::new(spawn, &shutdown) + .with_jobs(Arc::clone(&backup_runtime_for_reconciler)), + ); background.start().await; // A backend whose registration the database can end: the supervisor // refuses requests and stops the workers while it is lost, and asks for @@ -831,22 +842,16 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { let server_result = crate::start_server(listener, state, pid_file, tls_config).await; - // The HTTP server has stopped accepting. The supervisor returns once the - // token fires; then the workers stop, so in-flight cycles finish and the - // metrics flush worker writes its final bucket, and the serving - // registration is given up once every one of them has ended. - shutdown.cancel(); - if let Some(supervisor) = supervisor - && tokio::time::timeout(SUPERVISOR_TIMEOUT, supervisor) - .await - .is_err() - { - tracing::warn!( - "The registration supervisor did not return within {}s", - SUPERVISOR_TIMEOUT.as_secs() - ); - } - stop_then_release(&background, held.as_deref()).await; + // The HTTP server has stopped accepting. + drain_then_release( + &backup_runtime_for_reconciler, + JOB_DRAIN_TIMEOUT, + &shutdown, + supervisor, + &background, + held.as_deref(), + ) + .await; server_result?; @@ -862,6 +867,44 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { Ok(()) } +/// The shutdown sequence once the listener has stopped. The backup jobs are +/// cancelled and awaited first, within `job_drain` (a cancelled job's +/// failure path deletes its prefix, which on S3 is what sweeps an upload +/// still in flight), so their failure paths record their metrics before the +/// workers, the metrics flush among them, stop. Then the supervisor returns +/// once the token fires, the workers stop, so in-flight cycles finish and +/// the metrics flush worker writes its final bucket, and the serving +/// registration is given up once every job and worker has ended: a job +/// still running at the drain's bound keeps the registration, which expires +/// by itself ([`stop_then_release`]). +async fn drain_then_release( + jobs: &extenddb_engine::BackupRuntime, + job_drain: Duration, + shutdown: &CancellationToken, + supervisor: Option>, + workers: &BackgroundWorkers, + registrar: Option<&dyn extenddb_storage::hooks::ServingRegistrar>, +) { + let jobs_left = jobs.drain_jobs(job_drain).await; + shutdown.cancel(); + if let Some(supervisor) = supervisor + && tokio::time::timeout(SUPERVISOR_TIMEOUT, supervisor) + .await + .is_err() + { + tracing::warn!( + "The registration supervisor did not return within {}s", + SUPERVISOR_TIMEOUT.as_secs() + ); + } + stop_then_release(workers, registrar, jobs_left == 0).await; +} + +/// How long shutdown waits for the backup jobs' failure paths, before the +/// supervisor and the workers: a `delete_prefix` of a large S3 prefix takes +/// longer than the workers' drain allows. +pub const JOB_DRAIN_TIMEOUT: Duration = Duration::from_secs(30); + /// How long `serve` waits for the registration supervisor once the shutdown /// token fires: longer than a stop of the workers, which the supervisor may /// be running then. @@ -874,6 +917,23 @@ const DRAIN_TIMEOUT: Duration = Duration::from_secs(5); /// How long a stop waits for the aborted workers to end. const ABORT_TIMEOUT: Duration = Duration::from_secs(1); +/// How long shutdown's stop of the workers waits: the drain and the abort, +/// plus a supervisor's own stop it may queue behind (one that did not +/// return within [`SUPERVISOR_TIMEOUT`]). +const STOP_TIMEOUT: Duration = Duration::from_secs( + DRAIN_TIMEOUT.as_secs() + ABORT_TIMEOUT.as_secs() + SUPERVISOR_TIMEOUT.as_secs(), +); + +/// How long shutdown waits for the serving registration's release. +const RELEASE_TIMEOUT: Duration = Duration::from_secs(10); + +/// The longest shutdown takes after the job drain: the supervisor's return, +/// the workers' stop and the registration's release, each under its own +/// bound. `extenddb stop` waits for the sum of the shutdown bounds. +pub const WORKER_STOP_TIMEOUT: Duration = Duration::from_secs( + SUPERVISOR_TIMEOUT.as_secs() + STOP_TIMEOUT.as_secs() + RELEASE_TIMEOUT.as_secs(), +); + /// Starts one set of background workers on the token it is given, and /// returns their handles. type SpawnWorkers = Box< @@ -894,7 +954,11 @@ struct WorkerSet { /// and stop as one set. `serve` starts the first set and stops the one that /// runs at shutdown; the registration supervisor stops the set while the /// server holds no registration and starts a new one once it holds one. -/// Each set runs on a child token of the server's shutdown token. +/// Each set runs on a child token of the server's shutdown token. A stop +/// also cancels the backup jobs running then ([`Self::with_jobs`]): a +/// migration can take the catalog once a lost registration expires, and a +/// job's heartbeats and status writes would otherwise go on beside it; the +/// jobs started under a new registration run on a fresh token. struct BackgroundWorkers { spawn: SpawnWorkers, shutdown: CancellationToken, @@ -903,6 +967,8 @@ struct BackgroundWorkers { left_running: AtomicBool, drain: Duration, abort: Duration, + /// The runtime whose jobs a stop cancels, when the server has one. + jobs: Option>, } impl BackgroundWorkers { @@ -923,8 +989,15 @@ impl BackgroundWorkers { left_running: AtomicBool::new(false), drain, abort, + jobs: None, } } + + /// The same workers with the backup jobs a stop cancels (type doc). + fn with_jobs(mut self, jobs: Arc) -> Self { + self.jobs = Some(jobs); + self + } } #[async_trait::async_trait] @@ -935,6 +1008,15 @@ impl WorkerControl for BackgroundWorkers { /// outlives its abort (one that blocks its thread) is reported once and /// for the rest of the process. async fn stop(&self) -> bool { + if let Some(jobs) = &self.jobs { + let cancelled = jobs.cancel_jobs(); + if cancelled > 0 { + tracing::warn!( + "{cancelled} backup job(s) cancelled with the workers' stop; each fails its \ + row with reason shutdown and deletes its objects" + ); + } + } let mut running = self.running.lock().await; if let Some(set) = running.take() && !stop_set(set, self.drain, self.abort).await @@ -997,16 +1079,16 @@ async fn stop_set(set: WorkerSet, drain: Duration, abort: Duration) -> bool { /// Stop the workers, then release the serving registration: a migration /// that waits for the servers starts once the registration is gone, so -/// releasing it while a worker runs would let the migration run beside a -/// worker that still writes to the catalog. When a worker could not be -/// stopped, the registration is not released and expires by itself. The +/// releasing it while a worker or a backup job runs would let the migration +/// run beside work that still writes to the catalog. When a worker could +/// not be stopped, or `jobs_ended` is false (a backup job outlived its +/// drain), the registration is not released and expires by itself. The /// stop and the release are bounded. async fn stop_then_release( workers: &BackgroundWorkers, registrar: Option<&dyn extenddb_storage::hooks::ServingRegistrar>, + jobs_ended: bool, ) { - const RELEASE_TIMEOUT: Duration = Duration::from_secs(10); - let bound = workers.drain + workers.abort + SUPERVISOR_TIMEOUT; let ended = tokio::time::timeout(bound, workers.stop()) .await @@ -1021,6 +1103,13 @@ async fn stop_then_release( ); return; } + if !jobs_ended { + tracing::warn!( + "A backup job may still run, so the serving registration is not released; it expires \ + by itself" + ); + return; + } if tokio::time::timeout(RELEASE_TIMEOUT, registrar.release()) .await .is_err() @@ -1132,7 +1221,7 @@ mod tests { workers.start().await; let registrar = Recorder(events.clone()); shutdown.cancel(); - super::stop_then_release(&workers, Some(®istrar)).await; + super::stop_then_release(&workers, Some(®istrar), true).await; assert_eq!(*events.lock().unwrap(), ["worker stopped", "released"]); } @@ -1158,7 +1247,7 @@ mod tests { workers.start().await; let registrar = Recorder(events.clone()); shutdown.cancel(); - super::stop_then_release(&workers, Some(®istrar)).await; + super::stop_then_release(&workers, Some(®istrar), true).await; assert_eq!(*events.lock().unwrap(), ["released"]); let at_release = ticks.load(Ordering::SeqCst); tokio::time::sleep(Duration::from_millis(50)).await; @@ -1184,11 +1273,115 @@ mod tests { tokio::time::sleep(Duration::from_millis(50)).await; let registrar = Recorder(events.clone()); shutdown.cancel(); - super::stop_then_release(&workers, Some(®istrar)).await; + super::stop_then_release(&workers, Some(®istrar), true).await; assert!(events.lock().unwrap().is_empty(), "released"); assert!(!workers.stop().await, "a later stop forgot the worker"); } + /// A backup job still running after its drain keeps the registration + /// as a worker that could not be stopped does: the workers stop, and + /// the registration is left to expire. + #[tokio::test] + async fn a_job_left_running_keeps_the_registration() { + let events: Events = Arc::new(Mutex::new(Vec::new())); + let shutdown = CancellationToken::new(); + let workers = { + let events = events.clone(); + workers(&shutdown, move |token| { + let events = events.clone(); + async move { + token.cancelled().await; + events.lock().unwrap().push("worker stopped"); + } + }) + }; + workers.start().await; + let registrar = Recorder(events.clone()); + shutdown.cancel(); + super::stop_then_release(&workers, Some(®istrar), false).await; + assert_eq!(*events.lock().unwrap(), ["worker stopped"]); + } + + /// The shutdown sequence `serve` runs after its listener stops, with a + /// job that outlives a short drain bound: the job's token is cancelled, + /// the drain gives up at its bound, the server's token fires, the + /// workers stop, and the registration is not released. The same + /// sequence with the job ending at its cancellation releases it, after + /// the worker stopped. + #[tokio::test] + async fn the_shutdown_sequence_keeps_the_registration_while_a_job_outlives_the_drain() { + for (ends_on_cancel, expected) in [ + (false, vec!["worker stopped"]), + (true, vec!["job ended", "worker stopped", "released"]), + ] { + let events: Events = Arc::new(Mutex::new(Vec::new())); + let shutdown = CancellationToken::new(); + let runtime = + extenddb_engine::BackupRuntime::new(4).with_shutdown(shutdown.child_token()); + let token = runtime.shutdown(); + runtime.spawn_job({ + let events = events.clone(); + async move { + if ends_on_cancel { + token.cancelled().await; + events.lock().unwrap().push("job ended"); + } else { + // A job in a store call that does not return. + std::future::pending::<()>().await; + } + } + }); + let workers = { + let events = events.clone(); + workers(&shutdown, move |token| { + let events = events.clone(); + async move { + token.cancelled().await; + events.lock().unwrap().push("worker stopped"); + } + }) + }; + workers.start().await; + let registrar = Recorder(events.clone()); + super::drain_then_release( + &runtime, + Duration::from_millis(100), + &shutdown, + None, + &workers, + Some(®istrar), + ) + .await; + assert!(shutdown.is_cancelled()); + assert_eq!(*events.lock().unwrap(), expected, "{ends_on_cancel}"); + assert_eq!(runtime.running_jobs(), usize::from(!ends_on_cancel)); + } + } + + /// A stop of the workers (the supervisor's, at a registration loss) + /// cancels the jobs running then and leaves a fresh token for the jobs + /// that start under the new registration. + #[tokio::test] + async fn a_stop_of_the_workers_cancels_the_jobs_and_renews_their_token() { + let shutdown = CancellationToken::new(); + let runtime = + Arc::new(extenddb_engine::BackupRuntime::new(4).with_shutdown(shutdown.child_token())); + let before = runtime.shutdown(); + let workers = workers(&shutdown, |token| async move { token.cancelled().await }) + .with_jobs(Arc::clone(&runtime)); + workers.start().await; + assert!(workers.stop().await); + assert!(before.is_cancelled(), "the jobs' token is cancelled"); + let after = runtime.shutdown(); + assert!(!after.is_cancelled(), "a fresh token for the next jobs"); + assert!(!shutdown.is_cancelled(), "the server's token is untouched"); + shutdown.cancel(); + assert!( + after.is_cancelled(), + "the fresh token is the server's child" + ); + } + /// A stopped set starts again on a new token: the supervisor stops the /// workers at a loss and starts them after a new registration. #[tokio::test] diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs index 68bc77c50..f24e1f0a7 100644 --- a/crates/storage-cassandra/src/backup_catalog.rs +++ b/crates/storage-cassandra/src/backup_catalog.rs @@ -22,7 +22,12 @@ //! later write by the job is a lightweight transaction `IF owner_generation = //! `, and a heartbeat that does not apply means another owner has //! the row. Times are the coordinator's clock; staleness is measured against -//! it (`coordinator_now_millis`), never an application host's. +//! it (`coordinator_now_millis`), never an application host's. A dead +//! owner's row (`claim_dead_backup`) is taken over by an LWT on the +//! generation, the status and the heartbeat as read at `LOCAL_SERIAL`, the +//! heartbeat left as the owner wrote it. `sweep_restore_registrations` +//! removes, entry by entry under an LWT, the `active_restores` entries whose +//! target is no longer a CREATING restore target of the backup. //! //! DeleteBackup's lock is the lightweight transaction on the `backups_by_arn` //! row, `IF backup_status = 'AVAILABLE' AND active_restores = null`, on the @@ -339,6 +344,10 @@ impl JobClaim for CassandraBackupClaim { if fields.completed { assignments.push("completed_at = toTimestamp(now())".to_owned()); } + if let Some(source_table) = fields.source_table_details_json.as_ref() { + assignments.push("source_table_details_json = ?".to_owned()); + values.push(Value::from(source_table.to_string())); + } values.extend([ Value::from(self.key.account_id.as_str()), Value::from(self.key.backup_arn.as_str()), @@ -1109,6 +1118,114 @@ impl BackupCatalog for CassandraEngine { }) } + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + stale_after: Duration, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The owner is dead when the row's heartbeat is older than the + // window on the coordinator's clock. The row is read at + // LOCAL_SERIAL, so a heartbeat an LWT just wrote is seen, and the + // takeover is conditioned on the generation, the status and the + // heartbeat as read: a beat landing between the read and the + // write (the owner alive after all), or another taker's + // generation, leaves the row and answers `None`. The heartbeat is + // not refreshed: the FAILED-prefix window counts from the dead + // owner's last beat. + let catalog = self.catalog_keyspace(); + let query = format!( + "SELECT backup_status, owner_generation, heartbeat_at, store_kind, store_root, \ + store_id FROM {catalog}.backups_by_arn WHERE account_id = ? AND backup_arn = ?" + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.backup_arn.as_str()), + "dead backup row", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(None); + }; + let status: Option = row.get_by_name("backup_status").ok().flatten(); + if status.as_deref() != Some(BackupStatus::Creating.as_str()) { + return Ok(None); + } + // An in-catalog row (no store) is the legacy duty's. + let store_kind: Option = row.get_by_name("store_kind").ok().flatten(); + let Some(store_kind) = store_kind.as_deref().and_then(StoreKind::parse) else { + return Ok(None); + }; + let store = StoreIdentity { + store_kind, + store_root: crate::cassandra_util::get_column( + row, + "store_root", + "dead backup row", + )?, + store_id: crate::cassandra_util::get_column(row, "store_id", "dead backup row")?, + }; + let owner_generation: Option = row.get_by_name("owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("heartbeat_at").ok().flatten(); + let now = coordinator_now_millis(self).await?; + let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX); + if heartbeat_at.is_some_and(|at| at >= now.saturating_sub(stale_ms)) { + return Ok(None); + } + crate::test_hooks::before_takeover_write(&key.backup_arn).await; + let generation = new_generation(); + let take_over = format!( + "UPDATE {catalog}.backups_by_arn SET owner_generation = ? \ + WHERE account_id = ? AND backup_arn = ? \ + IF owner_generation = ? AND backup_status = 'CREATING' AND heartbeat_at = ?" + ); + let taken = conditional_write( + &self.session, + &take_over, + cdrs_tokio::query_values!( + generation_column(generation), + key.account_id.as_str(), + key.backup_arn.as_str(), + owner_generation, + heartbeat_at + ), + ) + .await?; + let claimed = match taken { + WriteOutcome::Matched => true, + WriteOutcome::NotMatched => false, + // A takeover whose answer was lost may have committed: a row + // carrying this call's generation is this call's claim. + WriteOutcome::Unknown => { + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &format!( + "SELECT owner_generation FROM {catalog}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(key.account_id.as_str(), key.backup_arn.as_str()), + "dead backup row", + ) + .await?; + let current: Option = rows + .first() + .and_then(|r| r.get_by_name("owner_generation").ok().flatten()); + current == Some(generation_column(generation)) + } + }; + if !claimed { + return Ok(None); + } + Ok(Some(Box::new(CassandraBackupClaim { + engine: self.clone(), + key: key.clone(), + store, + generation, + }) as Box)) + }) + } + fn list_backup_rows_older_than<'a>( &'a self, status: BackupStatus, @@ -1147,6 +1264,92 @@ impl BackupCatalog for CassandraEngine { }) } + fn sweep_restore_registrations(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async move { + // A registration is live while its target is a CREATING restore + // target with the registered table id restoring from this backup + // (the rule the in-use check applies); any other entry is one a + // restore left behind, its removal lost with its owner, and is + // removed by a lightweight transaction on the entry as read. The + // map has no index, so the scan is a range read of `backups_by_arn` + // at quorum; rows carrying an entry are few. + use cdrs_tokio::types::AsRust; + use cdrs_tokio::types::map::Map; + let catalog = self.catalog_keyspace(); + let scan = format!( + "SELECT account_id, backup_arn, active_restores FROM {catalog}.backups_by_arn" + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &scan, + cdrs_tokio::query_values!(), + "restore registrations", + ) + .await?; + let remove = format!( + "DELETE active_restores[?] FROM {catalog}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ? IF active_restores[?] = ?" + ); + let mut swept = Vec::new(); + for row in rows { + let map: Option = row.get_by_name("active_restores").ok().flatten(); + let Some(map) = map else { + continue; + }; + let entries: std::collections::HashMap = map + .as_rust() + .map_err(|e| StorageError::Internal(format!("Parse active_restores: {e}")))? + .unwrap_or_default(); + if entries.is_empty() { + continue; + } + let account_id: String = + crate::cassandra_util::get_column(&row, "account_id", "restore registrations")?; + let backup_arn: String = + crate::cassandra_util::get_column(&row, "backup_arn", "restore registrations")?; + let mut touched = false; + for (table_id, target_name) in entries { + let live = self + .restore_target_row(&account_id, &target_name) + .await? + .is_some_and(|target| { + target.table_id == table_id + && target.is_creating_target() + && target.source_backup_arn.as_deref() == Some(backup_arn.as_str()) + }); + if live { + continue; + } + let removed = conditional_write( + &self.session, + &remove, + cdrs_tokio::query_values!( + table_id.as_str(), + account_id.as_str(), + backup_arn.as_str(), + table_id.as_str(), + target_name.as_str() + ), + ) + .await?; + if removed == WriteOutcome::Matched { + tracing::info!( + "backup {backup_arn}: removed the restore registration of table \ + {target_name} ({table_id}); the target is no longer being restored \ + from it" + ); + touched = true; + } + } + if touched { + swept.push(backup_arn); + } + } + swept.sort(); + Ok(swept) + }) + } + fn backup_in_use<'a>( &'a self, key: &'a BackupKey, diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs index 961226110..0e70abcb8 100644 --- a/crates/storage-cassandra/src/test_hooks.rs +++ b/crates/storage-cassandra/src/test_hooks.rs @@ -27,7 +27,10 @@ //! * a lost answer on the maintenance claim's insert, so a test can see the //! re-read settle a claim whose `INSERT ... IF NOT EXISTS` committed; //! * a pause between a maintenance claim takeover's read and its write, so -//! a test can hold two takers of one stale row there and see one win. +//! a test can hold two takers of one stale row there and see one win; +//! * a pause in `claim_dead_backup` between its `LOCAL_SERIAL` read and its +//! takeover LWT, so a test can land the owner's beat in that gap and see +//! the LWT's `heartbeat_at = ` condition refuse the takeover. use std::time::Duration; @@ -80,6 +83,7 @@ mod armed { pub(super) before_abandoned_fence: HashMap>, pub(super) lose_maintenance_insert_answer: bool, pub(super) before_maintenance_takeover: Option>, + pub(super) before_takeover_write: HashMap>, } pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default); @@ -150,6 +154,20 @@ pub fn pause_before_abandoned_fence(backup_arn: &str) -> std::sync::Arc { pause } +/// Arm a pause in the next `claim_dead_backup` of `backup_arn`, between its +/// read of the row and its takeover write. The pause is taken once and then +/// disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_before_takeover_write(backup_arn: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .before_takeover_write + .insert(backup_arn.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + /// The period the jobs of `account_id` heartbeat at. pub(crate) fn heartbeat_interval(account_id: &str) -> Duration { #[cfg(feature = "test-hooks")] @@ -205,6 +223,15 @@ pub(crate) async fn before_abandoned_fence(backup_arn: &str) { let _ = backup_arn; } +/// Hold here if a test armed a pause for the takeover of `backup_arn`. +pub(crate) async fn before_takeover_write(backup_arn: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = armed::with_hooks(|hooks| hooks.before_takeover_write.remove(backup_arn)) { + pause.hold().await; + } + let _ = backup_arn; +} + /// Make the next `maintenance_claim`'s `INSERT ... IF NOT EXISTS` lose its /// answer: the statement runs and commits, and the caller sees /// `WriteOutcome::Unknown`, as after a write timeout on the Paxos round. The diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index a63f70ba7..067d49b61 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -2204,6 +2204,7 @@ async fn test_claim_backup_excludes_a_second_owner_and_fences_its_writes() { size_bytes: Some(1234), item_count: Some(5), failure_reason: None, + source_table_details_json: None, completed: true, }; assert_eq!( @@ -5467,3 +5468,403 @@ async fn test_the_newest_available_row_and_the_running_restores_are_read_by_stat "the age-based listing misses a heartbeat ahead of the clock" ); } +/// The index row of `row`, read back through the catalog. +async fn stored_index_row(engine: &CassandraEngine, row: &BackupRow) -> BackupRow { + engine + .read_backup_row(&arn_of(row)) + .await + .expect("read") + .expect("the row exists") +} + +/// `claim_dead_backup`: refused while the owner's heartbeat is within the +/// window, on the coordinator's clock; once the window has passed (waited +/// out, not hand-dated) the row is taken over under a fresh generation with +/// its heartbeat left as the dead owner wrote it, the old claim finds its +/// heartbeat lost and its writes unmatched, and the successor's AVAILABLE +/// write carries the source table and a completion time and moves the status +/// index. A successor that heartbeats is a live owner to a third taker; a row +/// that is not CREATING, and a missing row, answer `None`. +#[tokio::test] +async fn test_claim_dead_backup_takes_over_a_stale_row_and_refuses_a_live_one() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = index_row(&account_id, "Music", "dead-owner", &store); + let key = row.key(); + let window = Duration::from_secs(1); + assert!( + engine + .claim_dead_backup(&key, window) + .await + .expect("claim") + .is_none(), + "a missing row" + ); + let owner = engine.claim_backup(&row).await.expect("claim"); + let before = stored_index_row(&engine, &row).await; + assert!( + engine + .claim_dead_backup(&key, Duration::from_secs(300)) + .await + .expect("claim") + .is_none(), + "a live owner within a long window" + ); + assert!( + engine + .claim_dead_backup(&key, window) + .await + .expect("claim") + .is_none(), + "a heartbeat written an instant ago is within a one-second window" + ); + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_dead_backup(&key, window) + .await + .expect("claim") + .expect("the window passed on the coordinator's clock: taken over"); + assert_ne!(successor.generation(), owner.generation()); + assert_ne!(successor.generation(), UNOWNED_GENERATION); + let after = stored_index_row(&engine, &row).await; + assert_eq!(after.owner_generation, successor.generation()); + assert_eq!(after.status, BackupStatus::Creating); + assert_eq!( + after.heartbeat_at, before.heartbeat_at, + "the takeover left the heartbeat as the dead owner wrote it" + ); + assert_eq!( + owner.heartbeat().await.expect("heartbeat"), + Heartbeat::Lost, + "the old owner's heartbeat finds another generation" + ); + assert_eq!( + owner + .set_status( + JobStatus::Creating, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched, + "the old generation's write is fenced" + ); + + // The successor heartbeats: a third taker finds a live owner. + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + assert!( + engine + .claim_dead_backup(&key, window) + .await + .expect("claim") + .is_none(), + "the successor's heartbeat is fresh" + ); + + // The successor's AVAILABLE write carries the manifest's source table. + let mut source_table = row.source_table_details_json.clone(); + source_table["item_count"] = serde_json::json!(41); + let fields = StatusFields { + size_bytes: Some(4_096), + item_count: Some(41), + failure_reason: None, + source_table_details_json: Some(source_table.clone()), + completed: true, + }; + assert_eq!( + successor + .set_status(JobStatus::Creating, JobStatus::Available, &fields) + .await + .expect("write"), + WriteOutcome::Matched + ); + let available = stored_index_row(&engine, &row).await; + assert_eq!(available.status, BackupStatus::Available); + assert_eq!(available.size_bytes, 4_096); + assert_eq!(available.item_count, 41); + assert_eq!( + available.source_table_details_json, source_table, + "the write applied the source table" + ); + assert!(available.completed_at.is_some(), "completed is stamped"); + let listed: Vec = engine + .list_backup_rows(Some(BackupStatus::Available), &store) + .try_collect() + .await + .unwrap(); + assert_eq!(listed.len(), 1, "the status index followed: {listed:?}"); + assert!( + engine + .claim_dead_backup(&key, Duration::ZERO) + .await + .expect("claim") + .is_none(), + "an AVAILABLE row is not taken over, whatever its heartbeat's age" + ); + + // A FAILED write without the member leaves the stored source table. + let failed = index_row(&account_id, "Music", "dead-owner-failed", &store); + let claim = engine.claim_backup(&failed).await.expect("claim"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("owner dead".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + let stored = stored_index_row(&engine, &failed).await; + assert_eq!( + stored.source_table_details_json, + failed.source_table_details_json + ); + assert_eq!(stored.failure_reason.as_deref(), Some("owner dead")); +} + +/// The `active_restores` of a `backups_by_arn` row, as the map's entries. +async fn registrations( + engine: &CassandraEngine, + account_id: &str, + backup_arn: &str, +) -> std::collections::HashMap { + use cdrs_tokio::types::AsRust; + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT active_restores FROM {}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!(account_id, backup_arn), + "registrations", + ) + .await + .expect("read the row"); + let map: Option = rows + .first() + .expect("the row exists") + .get_by_name("active_restores") + .ok() + .flatten(); + map.map(|map| map.as_rust().expect("map").unwrap_or_default()) + .unwrap_or_default() +} + +/// `sweep_restore_registrations`: an entry whose target is a CREATING +/// restore target of the backup with the registered table id is kept; one +/// whose target is ACTIVE, one whose target restores from another backup and +/// one whose target has no row are removed, and the swept backup's ARN is +/// returned once. A second sweep finds nothing to do. +#[tokio::test] +async fn test_the_registrations_sweep_removes_stale_entries_and_keeps_a_live_one() { + if crate::helpers::skip_without_cassandra() { + return; + } + let (engine, account_id, _drop_keyspaces) = isolated_engine("sw", false).await; + let store = unique_store(); + let catalog = engine.catalog_keyspace(); + let row = index_row(&account_id, "Music", "registered", &store); + let claim = engine.claim_backup(&row).await.expect("claim"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + let other = index_row(&account_id, "Music", "other", &store); + let other_claim = engine.claim_backup(&other).await.expect("claim"); + assert_eq!( + other_claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + let mut targets = Vec::new(); + for name in ["SwLive", "SwActive", "SwForeign"] { + targets.push( + create_active_table( + &engine, + &account_id, + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + }, + ) + .await, + ); + } + let (live, active, foreign) = (&targets[0], &targets[1], &targets[2]); + // A CREATING target restoring from the backup; a CREATING target + // restoring from the other backup; an ACTIVE table; a name with no row. + for (target, source) in [(live, &row), (foreign, &other)] { + apply( + &engine, + &format!( + "UPDATE {catalog}.tables SET table_status = 'CREATING', \ + status_transition_at = null, restore_source_backup_arn = ? \ + WHERE account_id = ? AND table_name = ? IF table_id = ?" + ), + cdrs_tokio::query_values!( + source.backup_arn.as_str(), + account_id.as_str(), + target.table_name.as_str(), + target.table_id.as_str() + ), + ) + .await; + } + let missing_id = uuid::Uuid::new_v4().to_string(); + for (table_id, name) in [ + (live.table_id.as_str(), "SwLive"), + (active.table_id.as_str(), "SwActive"), + (foreign.table_id.as_str(), "SwForeign"), + (missing_id.as_str(), "SwMissing"), + ] { + apply( + &engine, + &format!( + "UPDATE {catalog}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS" + ), + cdrs_tokio::query_values!(table_id, name, account_id.as_str(), row.backup_arn.as_str()), + ) + .await; + } + // The other backup registers its own live restore: nothing to sweep. + apply( + &engine, + &format!( + "UPDATE {catalog}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS" + ), + cdrs_tokio::query_values!( + foreign.table_id.as_str(), + "SwForeign", + account_id.as_str(), + other.backup_arn.as_str() + ), + ) + .await; + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .len(), + 4 + ); + + let swept = engine.sweep_restore_registrations().await.expect("sweep"); + assert_eq!(swept, std::slice::from_ref(&row.backup_arn)); + let left = registrations(&engine, &account_id, &row.backup_arn).await; + assert_eq!( + left, + std::collections::HashMap::from([(live.table_id.clone(), "SwLive".to_owned())]), + "the live entry is kept and the three stale ones are removed" + ); + assert_eq!( + registrations(&engine, &account_id, &other.backup_arn).await, + std::collections::HashMap::from([(foreign.table_id.clone(), "SwForeign".to_owned())]), + "the other backup's live entry is untouched" + ); + assert!( + engine + .sweep_restore_registrations() + .await + .expect("sweep") + .is_empty(), + "a second sweep finds nothing" + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + Some("SwLive".to_owned()), + "the kept entry is the in-use rule's" + ); +} + +/// The liveness test inside the takeover write: with the owner's beat +/// landing between `claim_dead_backup`'s `LOCAL_SERIAL` read (which found +/// the heartbeat stale) and its LWT, the LWT's `heartbeat_at = ` +/// condition fails and the takeover answers `None`; the row keeps the +/// owner's generation and the owner's next beat confirms. Without that +/// condition the staleness comparison alone would decide, and a live owner's +/// job would be failed by the reconciler. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_beat_between_the_takeovers_read_and_its_write_refuses_the_takeover() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = index_row(&account_id, "Music", "beat-in-the-gap", &store); + let key = row.key(); + let window = Duration::from_secs(1); + let owner = engine.claim_backup(&row).await.expect("claim"); + tokio::time::sleep(Duration::from_millis(1_300)).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_takeover_write(&row.backup_arn); + let taker = { + let engine = engine.clone(); + let key = key.clone(); + tokio::spawn(async move { engine.claim_dead_backup(&key, window).await }) + }; + pause.reached().await; + // The read found the beat stale; the owner beats now. + assert_eq!( + owner.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + pause.release(); + let taken = taker.await.expect("the taker's task").expect("claim"); + assert!( + taken.is_none(), + "the LWT's heartbeat condition refuses a row whose owner beat in the gap" + ); + let after = stored_index_row(&engine, &row).await; + assert_eq!(after.owner_generation, owner.generation()); + assert_eq!(after.status, BackupStatus::Creating); + assert_eq!( + owner.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed, + "the owner's claim is intact" + ); + // Without the beat, the same stale row is taken over. + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_dead_backup(&key, window) + .await + .expect("claim") + .expect("taken over once no beat lands in the gap"); + assert_ne!(successor.generation(), owner.generation()); +} diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs index 752d0e106..ee901e850 100644 --- a/crates/storage-mongodb/src/backup_catalog.rs +++ b/crates/storage-mongodb/src/backup_catalog.rs @@ -11,7 +11,11 @@ //! "$$NOW"`, so no row ever exists unowned and a second claim of the same //! backup trips the unique index on (`account_id`, `backup_id`). Every write //! the job makes carries its generation in the filter; a heartbeat that -//! matches other than exactly one document means the claim is lost. +//! matches other than exactly one document means the claim is lost. The +//! heartbeat is also the liveness test: the reconciler takes over a CREATING +//! row whose heartbeat is older than the stale window on the server's clock +//! with one conditional update that sets a fresh generation and leaves +//! `heartbeat_at` as the dead owner wrote it. //! //! The lock on a row for DeleteBackup is a session transaction that `$inc`s //! `lock_seq` on the row before it reads the status and the in-use rule, so @@ -204,6 +208,21 @@ fn key_filter(key: &BackupKey) -> Document { doc! { "account_id": &key.account_id, "backup_id": &key.backup_id } } +/// `set` as the `$set` stage of a pipeline update: `$$NOW` stays the server's +/// clock and every other value is wrapped in `$literal`, so a string, or a +/// key inside the source table object, is never read as a field path. +fn literal_set(set: Document) -> Document { + set.into_iter() + .map(|(k, v)| { + let v = match v { + Bson::String(s) if s == "$$NOW" => Bson::String(s), + other => doc! { "$literal": other }.into(), + }; + (k, v) + }) + .collect() +} + /// The filter for the rows of `store`. fn store_filter(store: &StoreIdentity) -> Document { doc! { @@ -357,22 +376,19 @@ impl JobClaim for MongoBackupClaim { if let Some(reason) = &fields.failure_reason { set.insert("failure_reason", reason.as_str()); } + if let Some(source) = &fields.source_table_details_json { + let source = bson::to_bson(source).map_err(|e| { + StorageError::Internal(format!("serialize source_table_details_json: {e}")) + })?; + set.insert("source_table_details_json", source); + } if fields.completed { set.insert("completed_at", "$$NOW"); } - // A pipeline update, so `$$NOW` is the server's clock; the - // literal strings are wrapped so the pipeline does not read them - // as field paths. - let set: Document = set - .into_iter() - .map(|(k, v)| { - let v = match v { - Bson::String(s) if s == "$$NOW" => Bson::String(s), - other => doc! { "$literal": other }.into(), - }; - (k, v) - }) - .collect(); + // A pipeline update, so `$$NOW` is the server's clock; every other + // value is wrapped in `$literal` so the pipeline does not read a + // string, or a key of the source table object, as a field path. + let set = literal_set(set); match self .index .update_one(filter, vec![doc! { "$set": set }]) @@ -540,16 +556,7 @@ impl BackupCatalog for MongoEngine { let mut set = row_fields(row)?; set.insert("owner_generation", generation_field(generation)); set.insert("heartbeat_at", "$$NOW"); - let set: Document = set - .into_iter() - .map(|(k, v)| { - let v = match v { - Bson::String(s) if s == "$$NOW" => Bson::String(s), - other => doc! { "$literal": other }.into(), - }; - (k, v) - }) - .collect(); + let set = literal_set(set); match self .backup_index() .update_one(filter, vec![doc! { "$set": set }]) @@ -735,6 +742,45 @@ impl BackupCatalog for MongoEngine { }) } + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + stale_after: Duration, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The heartbeat is the liveness test: a live owner refreshes it + // every period, so a CREATING row whose heartbeat is older than + // the window on the server's clock has no owner, and the one + // write below both tests that and takes the row. The filter + // binds no generation, so the write takes a row whatever + // generation it carries; `heartbeat_at` is left as the dead + // owner wrote it. A second taker before this one resolves the + // row (another process's reconciler; one process runs one + // reconciler) finds the same stale heartbeat and takes the row + // again under its own generation, which fences this claim's + // status write as any takeover fences the dead owner's. + let generation = new_generation(); + let mut filter = key_filter(key); + filter.insert("status", BackupStatus::Creating.as_str()); + filter.insert("$expr", age_is_stale(&["heartbeat_at"], stale_after)); + let update = vec![doc! { "$set": literal_set(doc! { + "owner_generation": generation_field(generation), + }) }]; + let taken = self + .backup_index() + .find_one_and_update(filter, update) + .await + .map_err(internal)?; + Ok(taken.map(|_| { + Box::new(MongoBackupClaim { + index: self.backup_index(), + key: key.clone(), + generation, + }) as Box + })) + }) + } + fn list_backup_rows_older_than<'a>( &'a self, status: BackupStatus, @@ -1238,4 +1284,31 @@ mod tests { assert_eq!(generation_from_field(&doc, "owner_generation"), g); assert_eq!(generation_from_field(&doc! {}, "owner_generation"), 0); } + + /// A pipeline `$set` keeps `$$NOW` as the server's clock and wraps every + /// other value, the source table object included, so none is read as a + /// field path. + #[test] + fn a_pipeline_set_wraps_every_value_but_now_in_a_literal() { + let source = bson::to_bson(&fixture_source_table()).unwrap(); + let set = literal_set(doc! { + "status": "AVAILABLE", + "completed_at": "$$NOW", + "owner_generation": 7_i64, + "source_table_details_json": source.clone(), + }); + assert_eq!(set.get_str("completed_at"), Ok("$$NOW")); + assert_eq!( + set.get_document("status").unwrap(), + &doc! { "$literal": "AVAILABLE" } + ); + assert_eq!( + set.get_document("owner_generation").unwrap(), + &doc! { "$literal": 7_i64 } + ); + assert_eq!( + set.get_document("source_table_details_json").unwrap(), + &doc! { "$literal": source } + ); + } } diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index ab4ad92c4..bdb8b6c2f 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -610,6 +610,7 @@ async fn live_claim_backup_excludes_a_second_owner_and_fences_its_writes() { size_bytes: Some(1234), item_count: Some(5), failure_reason: None, + source_table_details_json: None, completed: true, }; assert_eq!( @@ -832,6 +833,145 @@ async fn index_doc(engine: &MongoEngine, row: &BackupRow) -> Document { .expect("the row exists") } +/// `claim_dead_backup`: a CREATING row is refused while its owner's heartbeat +/// is within the window, and taken over once the window has passed on the +/// server's clock (no hand-dating: the test waits), under a fresh generation +/// and with `heartbeat_at` as the dead owner left it; the old claim then +/// finds its heartbeat lost and its status write unmatched; the successor's +/// heartbeat makes it a live owner a further taker is refused, and its +/// AVAILABLE write carries the source table object and stamps +/// `completed_at`. An AVAILABLE row and a missing row are refused. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_dead_backup_takes_over_a_stale_row_and_refuses_a_live_one() { + let fx = Fixture::new().await; + let window = Duration::from_secs(3); + let dead = row(ACCOUNT, "Music", "a"); + let key = dead.key(); + let owner = fx.engine.claim_backup(&dead).await.expect("claim"); + assert_eq!(owner.heartbeat().await.unwrap(), Heartbeat::Confirmed); + assert!( + fx.engine + .claim_dead_backup(&key, STALE_AFTER) + .await + .unwrap() + .is_none(), + "a live owner" + ); + assert!( + fx.engine + .claim_dead_backup(&key, window) + .await + .unwrap() + .is_none(), + "a heartbeat within the window" + ); + let before = index_doc(&fx.engine, &dead).await; + let heartbeat_before = *before.get_datetime("heartbeat_at").expect("heartbeat_at"); + tokio::time::sleep(window + Duration::from_millis(300)).await; + let successor = fx + .engine + .claim_dead_backup(&key, window) + .await + .unwrap() + .expect("the window has passed on the server's clock"); + assert_ne!(successor.generation(), owner.generation()); + assert_ne!(successor.generation(), UNOWNED_GENERATION); + let after = index_doc(&fx.engine, &dead).await; + assert_eq!( + *after.get_datetime("heartbeat_at").expect("heartbeat_at"), + heartbeat_before, + "the takeover left the heartbeat as the dead owner wrote it" + ); + assert_eq!(after.get_str("status"), Ok("CREATING")); + assert_eq!( + after.get_i64("owner_generation").unwrap(), + i64::try_from(successor.generation()).unwrap() + ); + assert_eq!(owner.heartbeat().await.unwrap(), Heartbeat::Lost); + assert_eq!( + owner + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields::default() + ) + .await + .unwrap(), + WriteOutcome::NotMatched, + "the old generation is fenced" + ); + assert_eq!(successor.heartbeat().await.unwrap(), Heartbeat::Confirmed); + + // A second taker while the row is held: the heartbeat the successor + // just refreshed is within the window, so it is refused; another + // process's reconciler racing the takeover itself, before any + // heartbeat, would take the row again and fence the successor. + assert!( + fx.engine + .claim_dead_backup(&key, window) + .await + .unwrap() + .is_none(), + "a heartbeating successor is a live owner" + ); + + let mut source = dead.source_table_details_json.clone(); + source["item_count"] = serde_json::json!(4242); + let fields = StatusFields { + size_bytes: Some(777), + item_count: Some(4242), + failure_reason: None, + source_table_details_json: Some(source.clone()), + completed: true, + }; + assert_eq!( + successor + .set_status(JobStatus::Creating, JobStatus::Available, &fields) + .await + .unwrap(), + WriteOutcome::Matched + ); + let stored = fx + .engine + .read_backup_row(&arn_of(&dead)) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.status, BackupStatus::Available); + assert_eq!(stored.size_bytes, 777); + assert_eq!(stored.item_count, 4242); + assert_eq!( + stored.source_table_details_json, source, + "the AVAILABLE write carries the source table object" + ); + assert_eq!(stored.source_table().unwrap().item_count, 4242); + assert!(stored.completed_at.is_some()); + assert_eq!(stored.owner_generation, successor.generation()); + + tokio::time::sleep(window + Duration::from_millis(300)).await; + assert!( + fx.engine + .claim_dead_backup(&key, window) + .await + .unwrap() + .is_none(), + "an AVAILABLE row with a stale heartbeat is not CREATING" + ); + let missing = row(ACCOUNT, "Music", "none").key(); + assert!( + fx.engine + .claim_dead_backup(&missing, window) + .await + .unwrap() + .is_none(), + "a missing row" + ); + drop(owner); + drop(successor); + fx.finish().await; +} + /// `list_backup_rows_older_than`: the rows of the store with the status /// whose named clock column is older than the age on the server's clock. A /// row FAILED just now by a dead owner is listed on the heartbeat clock (its diff --git a/crates/storage-mongodb/src/snapshot_engine.rs b/crates/storage-mongodb/src/snapshot_engine.rs index 1b6c9e5ee..e3292a84e 100644 --- a/crates/storage-mongodb/src/snapshot_engine.rs +++ b/crates/storage-mongodb/src/snapshot_engine.rs @@ -69,8 +69,11 @@ //! //! Memory: a batch is at most `batch_size` documents //! (`paging::DEFAULT_FETCH_BATCH`, 1000) and at most the server's 16 MiB -//! reply limit, whichever is reached first, so unlike the SQL backends' -//! row-bounded fetches one batch here is bounded in bytes too. +//! reply limit, whichever is reached first, so one batch here is bounded in +//! bytes by the server. The stream applies no cut of its own at +//! `paging::SNAPSHOT_FETCH_BYTES` (64 MiB): the reply limit is already under +//! that budget, and the byte cut the SQL backends make exists because their +//! row-bounded fetches have no such limit. use extenddb_core::types::Item; use extenddb_storage::error::StorageError; diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs index 58ffc1a39..d7109a12d 100644 --- a/crates/storage-postgres/src/backup_catalog.rs +++ b/crates/storage-postgres/src/backup_catalog.rs @@ -251,7 +251,8 @@ impl JobClaim for PostgresBackupClaim { size_bytes = COALESCE($5, size_bytes), \ item_count = COALESCE($6, item_count), \ failure_reason = COALESCE($7, failure_reason), \ - completed_at = CASE WHEN $8 THEN now() ELSE completed_at END \ + completed_at = CASE WHEN $8 THEN now() ELSE completed_at END, \ + source_table_details_json = COALESCE($10, source_table_details_json) \ WHERE account_id = $1 AND backup_id = $2 AND owner_generation = $3 \ AND status = $9", ) @@ -264,6 +265,7 @@ impl JobClaim for PostgresBackupClaim { .bind(fields.failure_reason.as_deref()) .bind(fields.completed) .bind(from.as_str()) + .bind(fields.source_table_details_json.as_ref()) .execute(session.connection()) .await; match written { @@ -556,6 +558,47 @@ impl BackupCatalog for PostgresEngine { }) } + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + stale_after: Duration, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The same advisory lock the owner took: a session that still + // holds it is alive whatever its heartbeat says (the lock goes + // with the session), and a second taker is refused here. + let Some(mut session) = + try_claim(&self.pool, &backup_lock_key(&key.backup_id), None).await? + else { + return Ok(None); + }; + let generation = session.generation(); + let taken = sqlx::query( + "UPDATE backup_index SET owner_generation = $3 \ + WHERE account_id = $1 AND backup_id = $2 AND status = 'CREATING' \ + AND heartbeat_at < now() - make_interval(secs => $4)", + ) + .bind(&key.account_id) + .bind(&key.backup_id) + .bind(generation_column(generation)) + .bind(stale_after.as_secs_f64()) + .execute(session.connection()) + .await + .map_err(db_err)? + .rows_affected(); + if taken != 1 { + session.close().await; + return Ok(None); + } + Ok(Some(Box::new(PostgresBackupClaim { + session: Mutex::new(session), + generation, + key: key.clone(), + pool: self.pool.clone(), + }) as Box)) + }) + } + fn list_backup_rows_older_than<'a>( &'a self, status: BackupStatus, diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs index f4a5625ed..179302914 100755 --- a/crates/storage-postgres/src/backup_engine.rs +++ b/crates/storage-postgres/src/backup_engine.rs @@ -190,6 +190,17 @@ pub(crate) fn backup_lock_key(backup_id: &str) -> String { format!("backup:{backup_id}") } +/// The TCP keepalive settings a claim session asks the server for, as +/// `(setting, seconds)`: idle 30 s before the first probe, then probes 10 s +/// apart, three unanswered before the connection is dropped. Public for the +/// test that reads them back with `SHOW`. +#[doc(hidden)] +pub const CLAIM_KEEPALIVES: [(&str, u32); 3] = [ + ("tcp_keepalives_idle", 30), + ("tcp_keepalives_interval", 10), + ("tcp_keepalives_count", 3), +]; + /// Take the lock `key` on a new dedicated connection, with a fresh /// generation. `Ok(None)`: another session holds it. /// @@ -234,6 +245,25 @@ pub(crate) async fn try_claim_with( the server setting: {e}" ); } + // The server's TCP keepalive probes toward this process, so the + // advisory lock of a claim whose host vanished (no FIN, no RST) is + // released about a minute after the loss (30 s idle, then three probes + // 10 s apart) instead of after the kernel's default two hours, and the + // row's successor can take the lock once the row's heartbeat is stale. + // The owner's own liveness check is the heartbeat, not these. Each is a + // session setting a managed service may refuse; a refusal is a warning + // and the server's defaults stay. + for (name, value) in CLAIM_KEEPALIVES { + if let Err(e) = sqlx::query(&format!("SET {name} = {value}")) + .execute(&mut conn) + .await + { + tracing::warn!( + "could not set {name} for the claim session; continuing with the server \ + setting: {e}" + ); + } + } let mut taken = false; for attempt in 1..=tries.max(1) { taken = sqlx::query_scalar("SELECT pg_try_advisory_lock(hashtextextended($1, $2))") @@ -254,6 +284,36 @@ pub(crate) async fn try_claim_with( })) } +/// Open a claim session on `key` as a job would and read `settings` back +/// from it with `SHOW`, then end the session: for the test that checks the +/// session settings [`try_claim`] applies. `None` when another session holds +/// the lock. +/// +/// # Errors +/// +/// The claim's own errors, and `Internal` when a `SHOW` fails. +#[cfg(feature = "test-hooks")] +#[doc(hidden)] +pub async fn claim_session_settings( + pool: &sqlx::PgPool, + key: &str, + settings: &[&str], +) -> Result>, StorageError> { + let Some(mut session) = try_claim(pool, key, None).await? else { + return Ok(None); + }; + let mut values = Vec::with_capacity(settings.len()); + for name in settings { + let value: String = sqlx::query_scalar(&format!("SHOW {name}")) + .fetch_one(session.connection()) + .await + .map_err(|e| StorageError::Internal(format!("SHOW {name}: {e}")))?; + values.push(value); + } + session.close().await; + Ok(Some(values)) +} + /// Take the restore lock for `table_name` on a new dedicated connection. async fn try_restore_lock( pool: &sqlx::PgPool, diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs index 651c1072e..8e031f3e9 100755 --- a/crates/storage-postgres/src/lib.rs +++ b/crates/storage-postgres/src/lib.rs @@ -28,7 +28,9 @@ mod password_hash_store; mod pg_util; mod snapshot_engine; #[doc(hidden)] -pub use snapshot_engine::map_fetch_error; +pub use snapshot_engine::{ + MIN_NEXT_PAGE_ROWS, MIN_PAGE_ROWS, SNAPSHOT_SLOT_BOUND, map_fetch_error, page_sql_for, +}; mod stream_engine; #[doc(hidden)] pub mod stream_records_sql; @@ -41,8 +43,11 @@ mod vector_search; mod worker_store; mod workers; +#[cfg(feature = "test-hooks")] +#[doc(hidden)] +pub use backup_engine::claim_session_settings; #[doc(hidden)] -pub use backup_engine::RESTORE_LOCK_SEED; +pub use backup_engine::{CLAIM_KEEPALIVES, RESTORE_LOCK_SEED}; pub use bootstrapper::PostgresBootstrapper; pub use catalog_store::PostgresCatalogStore; pub use config::PostgresStorageConfig; @@ -294,6 +299,9 @@ pub struct PostgresEngine { /// wait; the open needs only a data connection by then, so they delay it /// no further. pub(crate) snapshot_open_slots: Arc, + /// The permits `snapshot_open_slots` was built with, for the refusal + /// that names the bound. + pub(crate) snapshot_slot_limit: usize, /// Whether `data_pool` is the catalog pool (no separate data database). /// The snapshot open uses one transaction on that layout, and /// `DeleteTable` runs its in-use probe and its synchronous drop on the @@ -424,6 +432,7 @@ impl PostgresEngine { snapshot_open_slots: Arc::new(tokio::sync::Semaphore::new(snapshot_slot_count( pool_size, ))), + snapshot_slot_limit: snapshot_slot_count(pool_size), shared_data_pool, #[cfg(feature = "test-hooks")] drop_attempts: Arc::default(), diff --git a/crates/storage-postgres/src/snapshot_engine.rs b/crates/storage-postgres/src/snapshot_engine.rs index 3fbf6f83f..1f6bbeb90 100644 --- a/crates/storage-postgres/src/snapshot_engine.rs +++ b/crates/storage-postgres/src/snapshot_engine.rs @@ -91,46 +91,164 @@ //! layout the first catalog read already did. Its text is the manifest //! marker: the `xmin:xmax:xip` of the snapshot, recorded so a later //! change-log archive can tell which commits the snapshot includes; nothing -//! reopens a snapshot from it. Items are then read through a server-side -//! cursor in batches. +//! reopens a snapshot from it. Items are then read in keyset pages on the +//! same transaction: each page is one query that seeks past the last row's +//! primary key (`WHERE (pk, sk) > ($1, $2) ORDER BY pk, sk LIMIT n`, an +//! index scan on the primary key) and cuts itself at the byte budget +//! ([`PageQuery`]). //! -//! Memory: one `FETCH` is `paging::DEFAULT_FETCH_BATCH` (1000) rows, all of -//! them held as item text at once. At the default 400 KB item limit one -//! row's text can reach 27.7 MB (the measured worst case in #394's line-cap -//! table), so a fetch is bounded by rows, not bytes. A byte budget (keyset -//! paging with a byte cutoff) is required before the stream gets a caller; -//! the backup design assigns it to the backup job PR. +//! Memory: a page holds at most `paging::fetch_batch()` rows (1000) and +//! stops after the row that takes its item text to or past +//! `paging::SNAPSHOT_FETCH_BYTES` (64 MiB), so the stream holds at most the +//! budget plus one row of text in decoded form, plus the one row's text +//! being parsed: the rows are pulled from the connection one at a time and +//! parsed as they arrive (`fetch_page`), so the page's text is never held +//! whole on this side. The cut is made in the page query, so the server +//! returns no row past it; the rows it reads past the cut before `LIMIT n` +//! stops it are at most `n - 1`, and each of them costs the server one +//! serialization of its item text and two spills of it (the two window +//! functions buffer the `n` rows before the outer `WHERE` drops them), so a +//! page costs the server `n` rows of text, not the rows it returns. `n` is +//! [`MIN_PAGE_ROWS`] for the first page and is then sized from the largest +//! row of the last pages, those holding at least a row bound's worth of rows +//! ([`RowBoundMemory`], [`next_page_rows`]: the budget divided by that row, +//! never below [`MIN_NEXT_PAGE_ROWS`]), so a table of large rows pages at +//! the rows the budget holds plus one, and the over-read that remains is the +//! second page, and then the page that follows a row bound's worth of small +//! rows: each asks for the rows the small rows before it justified (the +//! first page's 16, then a row bound's worth) and may meet large ones, so it +//! reads up to its `n` rows of up to the line cap (32 MiB) each; `n` then +//! stays at the floor while large rows keep arriving within a row bound's +//! worth of rows and returns to the row bound a row bound's worth of small +//! rows after the last large one, so such a page recurs at most once per row +//! bound's worth of small rows, on a table whose large rows are rarer than +//! that. The bound is the rows the page reads, `n` of up to the line cap +//! each (measured on a 20,000-row table with 30 such rows every 33rd key +//! among the first 1,000: `docs/design/15-backup-restore-bench-pr8.md`); +//! nothing in this form stops the server at the budget, since a window +//! function runs over the `n` rows before the outer `WHERE` drops them. The +//! floor has a cost of its own: after a page holding a row at the line cap +//! the bound stays at the floor for a row bound's worth of rows, 250 pages of +//! four, whatever those rows hold, so a table whose large rows are rarer +//! than one per row bound pays 250 round trips per large row where one page +//! of memory would page it at the row bound (measured on a 20,000-row table +//! with a 27 MB row at every 1,100th key: 2,585 page queries against 38 with +//! one page of memory, 25.6 against 25.1 s, 1.06 against 1.09 GB of temp; +//! the record). //! -//! Order: the cursor orders by the primary key (`key_order_clause`) only when -//! the text key columns (`pk`, and `sk_s` for an `S` range key) have a -//! collation named "C" or "POSIX", read from `pg_attribute` inside the -//! snapshot transaction. The rule is the name: any other collation, -//! including "default" on a database whose own collation is C, streams -//! unordered, which is complete. A table created before the key columns were -//! declared `COLLATE "C"` keeps the database collation, and ordering it -//! bytewise is a Seq Scan and an external sort of the whole table before the -//! first FETCH (7 MB on disk for 100k rows at `work_mem` 4 MB), with the -//! lock, the connection and the permit held throughout. Such a table is -//! streamed with a plain `SELECT`, reported `key_ordered = false`, and a -//! warning names it. +//! The plan is the planner's choice, not the query's promise. With the +//! planner's estimate of the rows after the seek above `n`, the page is +//! `Limit` over an `Index Scan` on the primary key index with no sort (the +//! 100k-row test table's first page, a page in its middle and a page ten +//! rows from its end, `the_page_query_is_an_index_scan` and +//! `the_last_page_sorts_the_rows_after_the_seek`). On a table of about a +//! thousand rows, a page near its end whose `n` is above the rows left +//! (1,060 rows, 57 left, `n` = 810) gets a `Bitmap Heap Scan` driven by a +//! `Bitmap Index Scan` on the same index and a `Sort` of those rows under +//! the `Limit` (PostgreSQL 16, the same test), so such a page sorts the rows +//! after the seek, at most the rows left, with their text; the seek is +//! served by the primary key index either way. +//! +//! Order: the page orders by the primary key in the key columns' own +//! collation, which is the order the primary key index has, so every table +//! pages by its index. The stream is reported `key_ordered` only when the +//! key shape has a format-defined order (`key_schema_is_orderable`) and the +//! text key columns (`pk`, `sk_s`) have a collation named "C" or "POSIX", +//! read from `pg_attribute` inside the snapshot transaction; the rule is the +//! name, so "default" on a database whose own collation is C streams +//! unordered, which is complete (`open_stream` says why the table is not +//! sorted bytewise instead). use std::sync::Arc; use extenddb_core::types::ScalarAttributeType; use extenddb_core::types::{Item, TableDescription}; +use extenddb_storage::backup_catalog::{JobBound, limit_exceeded_message}; use extenddb_storage::error::StorageError; use extenddb_storage::snapshot::{drop_deleting_indexes, key_schema_is_orderable, paging}; -use extenddb_storage::util::sk_column; +use extenddb_storage::util::sk_column_n; use extenddb_storage::{SnapshotEngine, SnapshotInfo, TableSnapshot}; use futures::future::BoxFuture; use futures::stream::BoxStream; +use sqlx::Row as _; +use sqlx::postgres::PgRow; use sqlx::{PgConnection, Postgres, Transaction}; use tokio::sync::{Notify, OwnedSemaphorePermit}; use crate::PostgresEngine; use crate::data::{all_sort_key_info, data_table_name, index_table_name}; -const CURSOR: &str = "extenddb_snapshot"; +/// Rows the first page asks for, before anything is known about the row +/// size. The server serializes every row the inner `LIMIT` reads, so this +/// bounds the first page's cost at 16 rows of up to the line cap (32 MiB) +/// each; a table of small rows pays one short round trip for it, since the +/// second page is sized from the first ([`next_page_rows`]). +#[doc(hidden)] +pub const MIN_PAGE_ROWS: usize = 16; + +/// The fewest rows a page after the first asks for. The budget divided by +/// the largest row is two for 27 MB rows, while such a page keeps three (the +/// row that reaches the budget is kept), so without a floor every page of +/// such a table would be two rows and no page would reach its cut; at four, +/// a page of such rows reads one row past its cut. +#[doc(hidden)] +pub const MIN_NEXT_PAGE_ROWS: usize = 4; + +/// The largest rows of the last pages, which size the next page's row bound +/// ([`next_page_rows`]): the memory keeps the pages that together hold at +/// least [`RowBoundMemory::rows`] rows (the row bound, 1000 in production), +/// so a large row keeps the bound low for at least a row bound's worth of +/// rows after it rather than for one page. With one page of memory a large +/// row sets the next bound to 4, four small rows set the one after back to +/// 1000, and on a table whose large rows are scattered among small ones the +/// 1000-row page meets large rows again and the cycle repeats every other +/// page (5,555 rows read for 1,000 kept on the measured shape); with the +/// memory the bound stays at 4 while large rows keep arriving within a row +/// bound's worth of rows, so that table pages at 4 after its first 1000-row +/// page (1,900 read for 1,000 kept). +#[derive(Debug, Clone)] +struct RowBoundMemory { + /// How many rows the kept pages must hold at least. + rows: usize, + /// The largest row and the row count of each kept page, oldest first. + pages: std::collections::VecDeque<(usize, usize)>, +} + +impl RowBoundMemory { + fn new(rows: usize) -> Self { + Self { + rows, + pages: std::collections::VecDeque::new(), + } + } + + /// Record a page's largest row (`max_row`, 0 for an empty page) and its + /// row count, and forget the oldest pages the memory no longer needs: + /// the oldest page goes while the pages after it hold at least `rows`. + fn record(&mut self, max_row: usize, rows: usize) { + self.pages.push_back((max_row, rows)); + while self.pages.len() > 1 { + let after_oldest: usize = self.pages.iter().skip(1).map(|&(_, n)| n).sum(); + if after_oldest < self.rows { + break; + } + self.pages.pop_front(); + } + } + + /// The largest row among the kept pages; `None` when none held a row. + fn largest_row(&self) -> Option { + self.pages + .iter() + .map(|&(max_row, _)| max_row) + .max() + .filter(|&b| b > 0) + } +} + +/// The name the snapshot-slot refusal gives its bound: the setting the slot +/// count is derived from (half of `pool_size`). +pub const SNAPSHOT_SLOT_BOUND: &str = "[storage.postgres] pool_size snapshot-slot"; const BEGIN_SNAPSHOT: &str = "BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY"; @@ -239,17 +357,17 @@ impl PostgresEngine { let slot = Arc::clone(&self.snapshot_open_slots) .try_acquire_owned() .map_err(|_| { - StorageError::LimitExceeded( - "too many table snapshots are open on this server; retry when one ends" - .to_owned(), - ) + StorageError::LimitExceeded(limit_exceeded_message(JobBound::Counted { + name: SNAPSHOT_SLOT_BOUND, + limit: self.snapshot_slot_limit, + })) })?; let pinned = if self.shared_data_pool { self.pin_on_one_transaction(account_id, table_name).await? } else { self.pin_behind_barrier(account_id, table_name).await? }; - open_cursor(pinned, slot, table_name).await + open_stream(pinned, slot, table_name).await } /// Shared layout (module doc): one `REPEATABLE READ READ ONLY` @@ -462,53 +580,60 @@ async fn leave_out_gsis_without_data_table( Ok(()) } -/// The common half of the open: decide the order, then `DECLARE` the cursor -/// on the held transaction. -async fn open_cursor( +/// The common half of the open: decide the order and the collation, then +/// hand the held transaction to the keyset stream. +async fn open_stream( mut pinned: Pinned, slot: OwnedSemaphorePermit, table_name: &str, ) -> Result { let ddb_table = data_table_name(&pinned.table_id); - // Ordered only when the text key columns already sort bytewise: on a - // table whose key columns keep the database collation the ORDER BY would - // be a Seq Scan and a full sort before the first FETCH, with the lock, - // the connection and the permit held throughout (module doc). - let mut order_by = key_order_clause(&pinned.description); - if order_by.is_some() { - let not_bytewise: Vec = sqlx::query_scalar( - "SELECT a.attname::text FROM pg_attribute a \ - LEFT JOIN pg_collation c ON c.oid = a.attcollation \ - WHERE a.attrelid = to_regclass($1) AND a.attname = ANY($2) \ - AND NOT a.attisdropped \ - AND COALESCE(c.collname, '') NOT IN ('C', 'POSIX')", - ) - .bind(&ddb_table) - .bind(key_text_columns(&pinned.description)) - .fetch_all(pinned.held.open_conn()?) - .await - .map_err(|e| db_err(&e))?; - if !not_bytewise.is_empty() { - tracing::warn!( - table = table_name, - table_id = %pinned.table_id, - columns = ?not_bytewise, - "table snapshot: key columns do not have collation \"C\"; streaming the \ - table unordered (key_ordered = false) rather than sorting it" - ); - order_by = None; - } + let columns = key_columns(&pinned.description); + // Whether the text key columns already sort bytewise: a collation named + // "C" or "POSIX" in `pg_attribute`, read inside the snapshot transaction. + // The page orders by the primary key in the columns' own collation either + // way, which is the order the primary key index has, so a table created + // before the key columns were declared `TEXT COLLATE "C"` (it keeps the + // database collation, `en_US.utf8` in the default setups, where `a < B`) + // pages by its index and is reported unordered rather than sorted + // bytewise (a Seq Scan and an external sort of the whole table before the + // first row, with the lock, the connection and the permit held + // throughout). `COLLATE "C"` is written into the page query only when the + // columns have it, where it changes nothing and states the order the + // stream relies on (`pk` and `sk_s` bytewise, `sk_n` and `sk_b` by value: + // the order `extenddb_core::backup_format::key_order` defines), and the + // stream is reported ordered only then. + let text_columns: Vec = columns + .iter() + .filter(|c| c.kind == KeyKind::Text) + .map(|c| c.name.clone()) + .collect(); + let not_bytewise: Vec = sqlx::query_scalar( + "SELECT a.attname::text FROM pg_attribute a \ + LEFT JOIN pg_collation c ON c.oid = a.attcollation \ + WHERE a.attrelid = to_regclass($1) AND a.attname = ANY($2) \ + AND NOT a.attisdropped \ + AND COALESCE(c.collname, '') NOT IN ('C', 'POSIX')", + ) + .bind(&ddb_table) + .bind(&text_columns) + .fetch_all(pinned.held.open_conn()?) + .await + .map_err(|e| db_err(&e))?; + let collate_c = not_bytewise.is_empty(); + let orderable = key_schema_is_orderable(&pinned.description); + if orderable && !collate_c { + tracing::warn!( + table = table_name, + table_id = %pinned.table_id, + columns = ?not_bytewise, + "table snapshot: key columns do not have collation \"C\"; streaming the \ + table unordered (key_ordered = false) rather than sorting it" + ); } - let key_ordered = order_by.is_some(); - let select = match &order_by { - Some(cols) => format!("SELECT item_data::text FROM {ddb_table} ORDER BY {cols}"), - None => format!("SELECT item_data::text FROM {ddb_table}"), - }; + let key_ordered = orderable && collate_c; + let query = PageQuery::new(&ddb_table, columns, collate_c); paging::open_pause(paging::OpenStage::BeforeItems).await; - sqlx::query(&format!("DECLARE {CURSOR} NO SCROLL CURSOR FOR {select}")) - .execute(pinned.held.open_conn()?) - .await - .map_err(|e| db_err(&e))?; Ok(TableSnapshot { description: pinned.description, @@ -517,21 +642,10 @@ async fn open_cursor( marker: pinned.marker, }, key_ordered, - items: cursor_stream(pinned.held, slot), + items: keyset_stream(pinned.held, slot, query), }) } -/// The text key columns whose collation decides the order: `pk`, and `sk_s` -/// for an `S` range key (`sk_n` is `NUMERIC` and `sk_b` is `bytea`). -fn key_text_columns(description: &TableDescription) -> Vec { - let sks = all_sort_key_info(&description.key_schema, &description.attribute_definitions); - let mut columns = vec!["pk".to_owned()]; - if matches!(sks.as_slice(), [(_, ScalarAttributeType::S)]) { - columns.push("sk_s".to_owned()); - } - columns -} - /// Drop attribute definitions no key schema (table, GSI, LSI) names, after /// an index has been left out of a description. `CreateTable` refuses a /// definition no key uses. @@ -553,45 +667,334 @@ fn prune_unused_attribute_definitions(description: &mut TableDescription) { .retain(|d| used.contains(&d.attribute_name)); } -/// `ORDER BY` columns for the format's key order, or `None` when the key -/// shape has none (`key_schema_is_orderable`). The open uses the clause only -/// after checking that the text key columns (`pk`, and `sk_s` for an `S` -/// range key) have collation "C" or "POSIX" (`pg_attribute.attcollation`); -/// a table created before the key columns were declared `TEXT COLLATE "C"` -/// keeps the database collation (`en_US.utf8` in the default setups, where -/// `a < B`) and is streamed unordered instead, because `ORDER BY ... COLLATE -/// "C"` on it is a full sort before the first row. On the tables that reach -/// the clause the explicit `COLLATE "C"` therefore does not change the order; -/// it is kept as defence in depth (no test can tell it from its absence), so -/// the query states the order it relies on. `pk` and `sk_s` sort bytewise and -/// `sk_n` (`NUMERIC`) and `sk_b` (`BYTEA`) sort by value: the order -/// `extenddb_core::backup_format::key_order` defines. -fn key_order_clause(description: &TableDescription) -> Option { - if !key_schema_is_orderable(description) { - return None; - } +/// The type a primary-key column of the data table has, which decides how +/// the seek binds its value. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum KeyKind { + /// `pk`, `sk_s`, `sk_s`: `TEXT`. + Text, + /// `sk_n`, `sk_n`: `NUMERIC`, carried as its text. + Numeric, + /// `sk_b`, `sk_b`: `BYTEA`. + Bytes, +} + +/// A primary-key column of the data table. +#[derive(Debug, Clone)] +struct KeyColumn { + name: String, + kind: KeyKind, +} + +/// The primary-key columns of a table's data table, in the index's order: +/// `pk`, then one typed column per range key (`sk_column_n`). +fn key_columns(description: &TableDescription) -> Vec { let sks = all_sort_key_info(&description.key_schema, &description.attribute_definitions); - match sks.as_slice() { - [] => Some("pk COLLATE \"C\"".to_owned()), - [(_, ScalarAttributeType::S)] => Some("pk COLLATE \"C\", sk_s COLLATE \"C\"".to_owned()), - [(_, sk_type)] => Some(format!("pk COLLATE \"C\", {}", sk_column(*sk_type))), - _ => None, + let mut columns = vec![KeyColumn { + name: "pk".to_owned(), + kind: KeyKind::Text, + }]; + for (i, &(_, sk_type)) in sks.iter().enumerate() { + columns.push(KeyColumn { + name: sk_column_n(i, sk_type), + kind: match sk_type { + ScalarAttributeType::S => KeyKind::Text, + ScalarAttributeType::N => KeyKind::Numeric, + ScalarAttributeType::B => KeyKind::Bytes, + }, + }); } + columns } -/// Stream items from the open cursor, one `FETCH` per batch. The held +/// One key column's value from the last row of a page: where the next page +/// starts. +#[derive(Debug, Clone)] +enum KeyValue { + Text(String), + Numeric(String), + Bytes(Vec), +} + +/// The page query of one table: the first page's text and a following +/// page's, built once per open. +/// +/// Three layers. The innermost reads the rows after the seek in primary-key +/// order and stops at `LIMIT n`; the middle computes, over those `n` rows +/// only, the running total of item text in the same order and the count of +/// rows the limit returned (`count(*) OVER ()`, the end-of-stream signal); +/// the outer keeps every row whose predecessors' total is under the byte +/// budget and orders the result, since a window function's output order is +/// not promised to the client. The layering matters: a window function runs +/// before `LIMIT`, so a running total written beside the `LIMIT` would be +/// computed over the whole rest of the table. The innermost `ORDER BY ... +/// LIMIT` behind a row comparison on the primary-key columns is an index +/// scan on the primary key; `item_data::text` is computed once, in the +/// innermost layer, and the other layers read the alias. +#[derive(Debug, Clone)] +struct PageQuery { + first: String, + after: String, + columns: Vec, +} + +impl PageQuery { + fn new(ddb_table: &str, columns: Vec, collate_c: bool) -> Self { + Self { + first: page_sql(ddb_table, &columns, collate_c, false), + after: page_sql(ddb_table, &columns, collate_c, true), + columns, + } + } +} + +/// The text of a page query (see [`PageQuery`]). With `after_key` the query +/// seeks past a key bound as `$1..$k` (text, numeric as text, bytes) and +/// takes the row bound and the byte budget as `$k+1` and `$k+2`; without, as +/// `$1` and `$2`. +fn page_sql(ddb_table: &str, columns: &[KeyColumn], collate_c: bool, after_key: bool) -> String { + let order = |prefix: &str| -> String { + columns + .iter() + .enumerate() + .map(|(i, c)| { + let name = if prefix.is_empty() { + format!("t.{}", c.name) + } else { + format!("{prefix}.k{i}") + }; + if collate_c && c.kind == KeyKind::Text { + format!("{name} COLLATE \"C\"") + } else { + name + } + }) + .collect::>() + .join(", ") + }; + let seek = if after_key { + let left = columns + .iter() + .map(|c| format!("t.{}", c.name)) + .collect::>() + .join(", "); + let right = columns + .iter() + .enumerate() + .map(|(i, c)| { + let p = i + 1; + match c.kind { + KeyKind::Text => format!("${p}::text"), + KeyKind::Numeric => format!("${p}::text::numeric"), + KeyKind::Bytes => format!("${p}::bytea"), + } + }) + .collect::>() + .join(", "); + format!("WHERE ({left}) > ({right}) ") + } else { + String::new() + }; + let k = if after_key { columns.len() } else { 0 }; + let n_param = k + 1; + let budget_param = k + 2; + let inner_select = columns + .iter() + .enumerate() + .map(|(i, c)| format!("t.{} AS k{i}", c.name)) + .collect::>() + .join(", "); + let keys = (0..columns.len()) + .map(|i| format!("k{i}")) + .collect::>() + .join(", "); + let shown = columns + .iter() + .enumerate() + .map(|(i, c)| match c.kind { + KeyKind::Numeric => format!("page.k{i}::text AS c{i}"), + KeyKind::Text | KeyKind::Bytes => format!("page.k{i} AS c{i}"), + }) + .collect::>() + .join(", "); + format!( + "SELECT {shown}, page.item, page.n_total FROM (\ + SELECT {keys}, item, \ + sum(octet_length(item)) OVER (ORDER BY {lim_order} \ + ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running, \ + count(*) OVER () AS n_total \ + FROM (\ + SELECT {inner_select}, t.item_data::text AS item FROM {ddb_table} t \ + {seek}ORDER BY {t_order} LIMIT ${n_param}\ + ) lim\ + ) page \ + WHERE page.running - octet_length(page.item) < ${budget_param} \ + ORDER BY {page_order}", + lim_order = order("lim"), + t_order = order(""), + page_order = order("page"), + ) +} + +/// The page query of `description`'s table, for the crate's tests: the +/// text [`PageQuery`] runs for a page after a key when `after_key`, else the +/// first page's, with `COLLATE "C"` on the text key columns when +/// `collate_c` (what `open_stream` decides from `pg_attribute`). +#[doc(hidden)] +#[must_use] +pub fn page_sql_for( + description: &TableDescription, + table_id: &str, + collate_c: bool, + after_key: bool, +) -> String { + page_sql( + &data_table_name(table_id), + &key_columns(description), + collate_c, + after_key, + ) +} + +/// One page as the database returned it, decoded. +struct Page { + /// The items the byte cut kept, in key order. + items: Vec, + /// The last kept row's key: where the next page starts. `None` for an + /// empty page. + last_key: Option>, + /// Each kept row's text, in bytes, in key order: what the next page's + /// row bound is sized from. + row_bytes: Vec, + /// Rows the innermost `LIMIT` returned, cut or not; fewer than the row + /// bound means the table is exhausted. + n_total: usize, +} + +/// Run the page query and decode its rows as they arrive: the rows are +/// pulled from the connection one at a time (`fetch`), each row's text is +/// parsed into its `Item` and dropped, so the page's text is never held +/// whole; what the page holds is its items. A row whose text does not parse +/// ends the page with `Internal`; a database error is mapped by +/// [`map_fetch_error`]. +async fn fetch_page( + conn: &mut PgConnection, + query: &PageQuery, + after: Option<&[KeyValue]>, + n: usize, + budget: usize, +) -> Result { + use futures::TryStreamExt as _; + let sql = if after.is_some() { + &query.after + } else { + &query.first + }; + let mut q = sqlx::query(sql); + for value in after.into_iter().flatten() { + q = match value { + KeyValue::Text(s) | KeyValue::Numeric(s) => q.bind(s.as_str()), + KeyValue::Bytes(b) => q.bind(b.as_slice()), + }; + } + let mut rows = q + .bind(i64::try_from(n).unwrap_or(i64::MAX)) + .bind(i64::try_from(budget).unwrap_or(i64::MAX)) + .fetch(conn); + let mut page = Page { + items: Vec::new(), + last_key: None, + row_bytes: Vec::new(), + n_total: 0, + }; + while let Some(row) = rows.try_next().await.map_err(|e| map_fetch_error(&e))? { + let row: PgRow = row; + let n_total: i64 = row.try_get("n_total").map_err(|e| db_err(&e))?; + page.n_total = usize::try_from(n_total).unwrap_or(0); + let mut key = Vec::with_capacity(query.columns.len()); + for (i, column) in query.columns.iter().enumerate() { + let name = format!("c{i}"); + key.push(match column.kind { + KeyKind::Text => { + KeyValue::Text(row.try_get(name.as_str()).map_err(|e| db_err(&e))?) + } + KeyKind::Numeric => { + KeyValue::Numeric(row.try_get(name.as_str()).map_err(|e| db_err(&e))?) + } + KeyKind::Bytes => { + KeyValue::Bytes(row.try_get(name.as_str()).map_err(|e| db_err(&e))?) + } + }); + } + let data: String = row.try_get("item").map_err(|e| db_err(&e))?; + drop(row); + let item = serde_json::from_str::(&data) + .map_err(|e| StorageError::Internal(format!("Parse item: {e}")))?; + page.row_bytes.push(data.len()); + drop(data); + page.items.push(item); + page.last_key = Some(key); + } + Ok(page) +} + +/// Rows the next page asks for, from the largest row of the last pages +/// (`memory`, [`RowBoundMemory`]): the rows the budget holds at that row +/// (`budget / largest`), between [`MIN_NEXT_PAGE_ROWS`] and the row bound. +/// The largest row and not the average, because the server serializes every +/// row the inner `LIMIT` reads: after a page of small rows with a few large +/// ones among them the average is small and a bound from it asks the next +/// page for hundreds of rows of a table that may now be large rows only (one +/// such page measured at 5.9 GB of server temp files for the three rows it +/// returned), where the largest row asks for the rows the budget holds of +/// it. A table of small rows pages at the row bound; a table of 27 MB rows +/// at [`MIN_NEXT_PAGE_ROWS`], of which one row per page is read past the cut +/// and dropped by the outer `WHERE`; a table whose large rows are scattered +/// among small ones pages at the row bound until the first page that meets +/// a large row, which reads up to its `n` rows of up to the line cap, and at +/// [`MIN_NEXT_PAGE_ROWS`] from there while large rows keep arriving within a +/// row bound's worth of rows, the bound returning to the row bound a row +/// bound's worth of small rows after the last large one; so every page that +/// follows a row bound's worth of small rows can read `n` rows of up to the +/// line cap, and no page can read more. The floor's cost: after a page +/// holding a row at the line cap the bound stays at the floor for a row +/// bound's worth of rows, 250 pages of four, whatever those rows hold, so a +/// table whose large rows are rarer than one per row bound pays 250 round +/// trips per large row (2,585 page queries against 38 with one page of +/// memory on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 +/// against 25.1 s, the record). A memory that held no row (an empty page) +/// gives the row bound. +fn next_page_rows(memory: &RowBoundMemory, budget: usize, batch: usize) -> usize { + let Some(max_row) = memory.largest_row() else { + return batch; + }; + (budget / max_row).max(MIN_NEXT_PAGE_ROWS).min(batch) +} + +/// Stream items by keyset pages on the held transaction. The held /// transaction lives in the stream's state: the last page commits it (waking /// the poller first), an error rolls it back, and dropping the stream before /// its end rolls it back through `HeldSnapshot`'s drop; each wakes the -/// control-plane poller. -fn cursor_stream( +/// control-plane poller. The row bound and the byte budget are read once +/// here, when the stream is built. +fn keyset_stream( held: HeldSnapshot, slot: OwnedSemaphorePermit, + query: PageQuery, ) -> BoxStream<'static, Result> { struct State { held: HeldSnapshot, /// Released with the transaction: the slot counts the connection. slot: Option, + query: PageQuery, + /// The last row's key; `None` before the first page. + after: Option>, + /// Rows the next page asks for. + n: usize, + batch: usize, + budget: usize, + /// The largest rows of the last pages, which size `n`. + memory: RowBoundMemory, buffer: std::vec::IntoIter, done: bool, } @@ -604,13 +1007,19 @@ fn cursor_stream( self.slot = None; } } + let batch = paging::fetch_batch(); let state = State { held, slot: Some(slot), + query, + after: None, + n: MIN_PAGE_ROWS.min(batch), + batch, + budget: paging::fetch_bytes(), + memory: RowBoundMemory::new(batch), buffer: Vec::new().into_iter(), done: false, }; - let batch = paging::fetch_batch(); Box::pin(futures::stream::unfold( state, move |mut state| async move { @@ -623,38 +1032,43 @@ fn cursor_stream( } let conn = state.held.conn()?; paging::record_fetch(); - let rows: Result, sqlx::Error> = - sqlx::query_as(&format!("FETCH {batch} FROM {CURSOR}")) - .fetch_all(conn) - .await; - let rows = match rows { - Ok(rows) => rows, + let page = fetch_page( + conn, + &state.query, + state.after.as_deref(), + state.n, + state.budget, + ) + .await; + let page = match page { + Ok(page) => page, Err(e) => { state.abandon(); - return Some((Err(map_fetch_error(&e)), state)); + return Some((Err(e), state)); } }; - if rows.len() < batch { + let returned = page.items.len(); + paging::record_page_bytes(page.row_bytes.iter().sum()); + paging::record_rows_past_cut( + u64::try_from(page.n_total.saturating_sub(returned)).unwrap_or(u64::MAX), + ); + // The table is exhausted when the limit returned fewer rows + // than asked and the cut dropped none of them; a short page + // the cut shortened still has rows past its last one. + if page.n_total < state.n && returned == page.n_total { state.done = true; // Close the transaction now rather than at drop so vacuum is // released as soon as the last row is out. let _ = state.held.finish().await; state.slot = None; + } else { + state.after = page.last_key; + state + .memory + .record(page.row_bytes.iter().copied().max().unwrap_or(0), returned); + state.n = next_page_rows(&state.memory, state.budget, state.batch); } - let mut items = Vec::with_capacity(rows.len()); - for (data,) in rows { - match serde_json::from_str::(&data) { - Ok(item) => items.push(item), - Err(e) => { - state.abandon(); - return Some(( - Err(StorageError::Internal(format!("Parse item: {e}"))), - state, - )); - } - } - } - state.buffer = items.into_iter(); + state.buffer = page.items.into_iter(); } }, )) @@ -679,3 +1093,140 @@ pub fn map_fetch_error(e: &sqlx::Error) -> StorageError { } db_err(e) } + +#[cfg(test)] +mod tests { + use super::*; + + fn columns(kinds: &[(&str, KeyKind)]) -> Vec { + kinds + .iter() + .map(|(name, kind)| KeyColumn { + name: (*name).to_owned(), + kind: *kind, + }) + .collect() + } + + fn memory_of(pages: &[(usize, usize)], rows: usize) -> RowBoundMemory { + let mut memory = RowBoundMemory::new(rows); + for &(max_row, n) in pages { + memory.record(max_row, n); + } + memory + } + + /// The row bound follows the largest row of the remembered pages within + /// its clamps: small rows page at the row bound, 27 MB rows at the floor + /// (the budget holds two, the floor is four), a memory that held no row + /// (an empty page) at the row bound, a row bound under the floor (a + /// test's small page) is the bound; and a page of 984 small rows and 3 + /// large ones is sized from the large row, not from the average (which + /// is 82 KB and would ask the next page for about 800 rows of a table + /// that is now large rows only). + #[test] + fn the_row_bound_adapts_within_its_clamps() { + let budget = 64 * 1024 * 1024; + assert_eq!( + next_page_rows(&memory_of(&[(300, 16)], 1000), budget, 1000), + 1000 + ); + // The floor as its value: the budget holds two such rows, a page + // keeps three, and four reads one past the cut. + assert_eq!( + next_page_rows(&memory_of(&[(27_000_000, 3)], 1000), budget, 1000), + 4 + ); + assert_eq!( + next_page_rows(&memory_of(&[(1024 * 1024, 8)], 1000), budget, 1000), + 64 + ); + assert_eq!( + next_page_rows(&memory_of(&[(0, 0)], 1000), budget, 1000), + 1000 + ); + assert_eq!(next_page_rows(&memory_of(&[(300, 3)], 3), budget, 3), 3); + // 984 rows of 300 bytes and 3 of 27 MB: the page's largest row sizes + // the next page. + let next = next_page_rows(&memory_of(&[(27_000_000, 987)], 1000), budget, 1000); + assert!( + next < 8, + "the next page asks for {next} rows after a page of 984 small rows and 3 large ones" + ); + assert!(next >= MIN_NEXT_PAGE_ROWS); + } + + /// The memory keeps the pages that together hold at least the row bound: + /// after a page with a 27 MB row, pages of four small rows keep the bound + /// at the floor until a row bound's worth of them have passed (250 pages + /// of four at a row bound of 1000), and the page after that is sized from + /// the small rows alone; a large row met within that span starts the + /// count again; the memory holds at most the pages it needs (the oldest + /// goes once the pages after it hold the row bound), so it does not grow + /// with the table. + #[test] + fn the_memory_covers_a_row_bound_of_rows() { + let budget = 64 * 1024 * 1024; + let mut memory = RowBoundMemory::new(1000); + memory.record(27_000_000, 83); + for _ in 0..249 { + memory.record(300, 4); + assert_eq!(next_page_rows(&memory, budget, 1000), MIN_NEXT_PAGE_ROWS); + } + // 249 pages of four hold 996 rows after the large page: still kept. + assert_eq!(memory.pages.len(), 250); + memory.record(300, 4); + // 1000 rows after it: the large page is forgotten. + assert_eq!(next_page_rows(&memory, budget, 1000), 1000); + assert_eq!(memory.pages.len(), 250); + // A large row within the span starts the count again. + memory.record(27_000_000, 4); + for _ in 0..249 { + memory.record(300, 4); + } + assert_eq!(next_page_rows(&memory, budget, 1000), MIN_NEXT_PAGE_ROWS); + memory.record(300, 4); + assert_eq!(next_page_rows(&memory, budget, 1000), 1000); + // Pages of the row bound: one page of memory is enough. + let mut memory = RowBoundMemory::new(1000); + memory.record(27_000_000, 3); + memory.record(300, 1000); + assert_eq!(memory.pages.len(), 1); + assert_eq!(next_page_rows(&memory, budget, 1000), 1000); + } + + /// The seek compares the key columns as their own types (numeric through + /// text, bytes as bytea), the first page has no seek, the text columns + /// carry `COLLATE "C"` only when asked, the row bound and the budget + /// take the parameters after the key, and `item_data::text` is computed + /// once in the innermost layer. + #[test] + fn the_page_query_binds_the_key_by_type() { + let cols = columns(&[("pk", KeyKind::Text), ("sk_n", KeyKind::Numeric)]); + let after = page_sql("\"_ddb_t\"", &cols, true, true); + assert!( + after.contains("WHERE (t.pk, t.sk_n) > ($1::text, $2::text::numeric) ORDER BY t.pk COLLATE \"C\", t.sk_n LIMIT $3"), + "{after}" + ); + assert!( + after.contains("< $4 ORDER BY page.k0 COLLATE \"C\", page.k1"), + "{after}" + ); + assert!(after.contains("page.k1::text AS c1"), "{after}"); + assert_eq!(after.matches("item_data::text").count(), 1, "{after}"); + let first = page_sql("\"_ddb_t\"", &cols, false, false); + assert!(!first.contains("WHERE (t.pk"), "{first}"); + assert!(first.contains("ORDER BY t.pk, t.sk_n LIMIT $1"), "{first}"); + assert!(first.contains("< $2 ORDER BY page.k0, page.k1"), "{first}"); + let bytes = columns(&[ + ("pk", KeyKind::Text), + ("sk_b", KeyKind::Bytes), + ("sk2_s", KeyKind::Text), + ]); + let after = page_sql("\"_ddb_t\"", &bytes, true, true); + assert!( + after.contains("> ($1::text, $2::bytea, $3::text) ORDER BY t.pk COLLATE \"C\", t.sk_b, t.sk2_s COLLATE \"C\" LIMIT $4"), + "{after}" + ); + } +} diff --git a/crates/storage-postgres/tests/backup_restore.rs b/crates/storage-postgres/tests/backup_restore.rs index 2966ded7a..85115d440 100644 --- a/crates/storage-postgres/tests/backup_restore.rs +++ b/crates/storage-postgres/tests/backup_restore.rs @@ -2753,7 +2753,7 @@ mod catalog_index { use super::*; - fn store() -> StoreIdentity { + pub(super) fn store() -> StoreIdentity { StoreIdentity { store_kind: StoreKind::Filesystem, store_root: "/var/lib/extenddb/backups".to_owned(), @@ -2898,6 +2898,7 @@ mod catalog_index { size_bytes: Some(1234), item_count: Some(5), failure_reason: None, + source_table_details_json: None, completed: true, }; assert_eq!( @@ -4819,6 +4820,264 @@ mod catalog_index { /// run against this catalog and real filesystem stores: two instances of one /// catalog starting against one empty root, and two instances with /// host-local roots under one `store_root` label. +/// The catalog methods the store-backed backup job and the reconciler call: +/// the takeover of a dead owner's row, the age listings on the database's +/// clock, the AVAILABLE write's `source_table_details_json`, the claim +/// session's settings. +mod backup_job { + use std::time::Duration; + + use extenddb_core::backup_format::BackupArn; + use extenddb_storage::BackupCatalog; + use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, JobStatus, StatusFields, WriteOutcome, + }; + + use super::catalog_index::row as new_row; + use super::*; + + fn arn_of(row: &BackupRow) -> BackupArn { + BackupArn::parse(&row.backup_arn).unwrap() + } + + async fn stored(s: &Scratch, row: &BackupRow) -> BackupRow { + s.engine + .read_backup_row(&arn_of(row)) + .await + .expect("read") + .expect("the row exists") + } + + /// Wait until no session holds the backup's advisory lock (the server + /// releases it a moment after the holding connection closes). + async fn await_lock_free(s: &Scratch, backup_id: &str) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while backup_lock_holder(s, backup_id).await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "the advisory lock outlived its session" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + } + + /// `claim_dead_backup`: refused while the owner's session holds the + /// lock, refused while the row's heartbeat is within the stale window + /// though the lock is free (an owner that died a moment ago), taken over + /// once the heartbeat is older than the window on the database's clock + /// (the window is crossed by waiting, not by dating the row), with a + /// fresh generation and `heartbeat_at` as the dead owner left it; the + /// successor's AVAILABLE write carries the manifest's `source_table` + /// into the row; a row that is no longer CREATING is not taken over. + #[tokio::test] + async fn claim_dead_backup_takes_over_a_stale_row_and_refuses_a_live_one() { + if base_conn().is_none() { + eprintln!( + "SKIP claim_dead_backup_takes_over_a_stale_row_and_refuses_a_live_one: no \ + PostgreSQL" + ); + return; + } + let s = scratch().await; + let window = Duration::from_secs(1); + let row = new_row(ACCOUNT, "Music", "nightly"); + let key = row.key(); + let owner = s.engine.claim_backup(&row).await.expect("claim"); + // Alive: its session holds the lock. + assert!( + s.engine + .claim_dead_backup(&key, window) + .await + .expect("takeover attempt") + .is_none(), + "a row whose owner holds the lock was taken over" + ); + assert_eq!(stored(&s, &row).await.owner_generation, owner.generation()); + // The owner's session ends (a crash): the lock goes, the heartbeat + // stays as written. Within the window the row is not yet dead. + drop(owner); + await_lock_free(&s, &row.backup_id).await; + let left = stored(&s, &row).await; + assert!( + s.engine + .claim_dead_backup(&key, window) + .await + .expect("takeover attempt") + .is_none(), + "a row whose heartbeat is within the stale window was taken over" + ); + assert_eq!( + backup_lock_holder(&s, &row.backup_id).await, + None, + "a refused takeover left its lock session behind" + ); + tokio::time::sleep(window + Duration::from_millis(300)).await; + let successor = s + .engine + .claim_dead_backup(&key, window) + .await + .expect("takeover") + .expect("a stale CREATING row is taken over"); + assert_ne!(successor.generation(), left.owner_generation); + let taken = stored(&s, &row).await; + assert_eq!(taken.status, BackupStatus::Creating); + assert_eq!(taken.owner_generation, successor.generation()); + assert_eq!( + taken.heartbeat_at, left.heartbeat_at, + "the takeover refreshed heartbeat_at" + ); + assert!( + backup_lock_holder(&s, &row.backup_id).await.is_some(), + "the successor holds the lock" + ); + // A second taker is refused by the lock while the successor lives. + assert!( + s.engine + .claim_dead_backup(&key, window) + .await + .expect("second takeover attempt") + .is_none() + ); + // The successor's writes go through: the manifest's source table + // replaces the row's, the FAILED write's `None` leaves it. + let manifest_source = serde_json::json!({"table_name": "Music", "item_count": 42}); + let fields = StatusFields { + size_bytes: Some(10), + item_count: Some(42), + source_table_details_json: Some(manifest_source.clone()), + completed: true, + ..StatusFields::default() + }; + assert_eq!( + successor + .set_status(JobStatus::Creating, JobStatus::Available, &fields) + .await + .expect("write"), + WriteOutcome::Matched + ); + let available = stored(&s, &row).await; + assert_eq!(available.status, BackupStatus::Available); + assert_eq!(available.source_table_details_json, manifest_source); + assert_eq!(available.item_count, 42); + drop(successor); + await_lock_free(&s, &row.backup_id).await; + tokio::time::sleep(window + Duration::from_millis(300)).await; + assert!( + s.engine + .claim_dead_backup(&key, window) + .await + .expect("takeover attempt") + .is_none(), + "an AVAILABLE row was taken over" + ); + // A row that is gone. + assert!( + s.engine + .claim_dead_backup(&new_row(ACCOUNT, "Music", "other").key(), window) + .await + .expect("takeover attempt") + .is_none() + ); + s.cleanup().await; + } + + /// An owner whose claim session the server ended (`pg_terminate_backend`) + /// gets `Unknown` from its next status write, not an error, and its + /// re-read shows the row as it is: the fence the job and the reconciler + /// act on. Once the heartbeat is stale the row is a successor's. + #[tokio::test] + async fn a_terminated_claim_session_reports_unknown_and_the_row_is_reread() { + if base_conn().is_none() { + eprintln!( + "SKIP a_terminated_claim_session_reports_unknown_and_the_row_is_reread: no \ + PostgreSQL" + ); + return; + } + let s = scratch().await; + let row = new_row(ACCOUNT, "Music", "nightly"); + let owner = s.engine.claim_backup(&row).await.expect("claim"); + let pid = backup_lock_holder(&s, &row.backup_id) + .await + .expect("the claim session holds the lock"); + let terminated: bool = sqlx::query_scalar("SELECT pg_terminate_backend($1)") + .bind(pid) + .fetch_one(&s.catalog) + .await + .expect("terminate"); + assert!(terminated); + await_lock_free(&s, &row.backup_id).await; + let outcome = owner + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + size_bytes: Some(1), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("a lost connection is an outcome, not an error"); + assert_eq!(outcome, WriteOutcome::Unknown); + // The re-read runs on the pool, not the dead session. + let read = owner.read().await.expect("re-read"); + assert_eq!(read.status, JobStatus::Creating); + assert_eq!(read.owner_generation, owner.generation()); + // The heartbeat on the dead session is an error the caller treats as + // `Lost`. + assert!(owner.heartbeat().await.is_err()); + s.cleanup().await; + } + + /// The claim session asks the server for TCP keepalives + /// (`CLAIM_KEEPALIVES`), read back from the session with `SHOW`: idle + /// 30 s, then a probe every 10 s, three unanswered, so a vanished host + /// loses its lock after about a minute (design 4(e)). The literals, not + /// the constant, so a constant at the server's defaults (7200 s idle) + /// fails here. + #[tokio::test] + async fn the_claim_session_sets_tcp_keepalives() { + if base_conn().is_none() { + eprintln!("SKIP the_claim_session_sets_tcp_keepalives: no PostgreSQL"); + return; + } + let s = scratch().await; + let names = [ + "tcp_keepalives_idle", + "tcp_keepalives_interval", + "tcp_keepalives_count", + ]; + let values = extenddb_storage_postgres::claim_session_settings( + &s.catalog, + "backup:keepalive-test", + &names, + ) + .await + .expect("open a claim session") + .expect("the lock is free"); + assert_eq!(values, ["30", "10", "3"]); + assert_eq!( + extenddb_storage_postgres::CLAIM_KEEPALIVES, + [ + ("tcp_keepalives_idle", 30), + ("tcp_keepalives_interval", 10), + ("tcp_keepalives_count", 3), + ] + ); + // A pooled connection has the server's defaults, not the claim's. + let pooled: String = sqlx::query_scalar("SHOW tcp_keepalives_idle") + .fetch_one(&s.catalog) + .await + .expect("show"); + assert_ne!( + pooled, "30", + "the test server's default is the claim's value" + ); + s.cleanup().await; + } +} + mod store_identity { use std::sync::Arc; diff --git a/crates/storage-postgres/tests/table_snapshot.rs b/crates/storage-postgres/tests/table_snapshot.rs index 9dbc4a928..e0a99d5ab 100644 --- a/crates/storage-postgres/tests/table_snapshot.rs +++ b/crates/storage-postgres/tests/table_snapshot.rs @@ -555,9 +555,18 @@ async fn half_the_pool_holds_snapshots(shared: bool) { .await .expect("an open over the bound is refused at once, not queued") .expect_err("the sixth open is over the bound"); - assert!( - matches!(err, extenddb_storage::error::StorageError::LimitExceeded(_)), - "{err}" + // The one refusal text every job bound answers, naming this one. + assert_eq!( + err.to_string(), + extenddb_storage::error::StorageError::LimitExceeded( + extenddb_storage::backup_catalog::limit_exceeded_message( + extenddb_storage::backup_catalog::JobBound::Counted { + name: extenddb_storage_postgres::SNAPSHOT_SLOT_BOUND, + limit: 5, + } + ) + ) + .to_string() ); let key: Item = [ ("pk".to_owned(), AttributeValue::S("p000".to_owned())), @@ -1077,6 +1086,662 @@ async fn snapshot_marker_and_key_order(shared: bool) { } on_both_layouts!(snapshot_marker_and_key_order); +/// The keyset pages respect the byte budget on both layouts +/// (`snapshot_conformance::run_page_bounds`), and the rows the server reads +/// past each page's cut are those the row bound lets it: with a budget of +/// eight rows the first page asks for `MIN_PAGE_ROWS` (16) and keeps eight, +/// so it reads eight it drops; every later page is sized from the largest +/// row of the page before (the budget divided by it, eight here), so it +/// reads exactly the rows it keeps and drops none. The figure is asserted +/// exactly: a counter that never moves would pass a bound (lesson 25), and a +/// row bound sized from the average again reads the same eight, so the +/// mixed-page case is pinned by `next_page_rows`'s unit test and by +/// `the_row_bound_remembers_large_rows_among_small_ones`. +async fn pages_are_cut_at_the_byte_budget(shared: bool) { + let s = scratch_with(shared).await; + let bounds = snapshot_conformance::run_page_bounds( + &s.hooks, + &s.engine, + ACCOUNT, + "page_bounds", + || async { + s.engine + .process_control_plane_transitions() + .await + .expect("pump"); + }, + ) + .await; + let past_cut = bounds.rows_past_cut; + let first_page = u64::try_from( + extenddb_storage_postgres::MIN_PAGE_ROWS - snapshot_conformance::PAGE_BOUNDS_ROWS, + ) + .unwrap(); + // The figure the memory record cites beside the RSS numbers. + eprintln!( + "rows read past the cut: {past_cut} over {} pages of {} rows ({first_page} expected, all \ + on the first page)", + bounds.pages, + bounds.budget / bounds.row_bytes + ); + assert_eq!( + past_cut, first_page, + "{past_cut} rows read past the cut over {} pages: the first page reads MIN_PAGE_ROWS and \ + drops {first_page}, and every later page is sized to the rows it keeps", + bounds.pages + ); +} +on_both_layouts!(pages_are_cut_at_the_byte_budget); + +/// Every plan node type under `node`, in depth-first order. +fn plan_node_types(node: &serde_json::Value, into: &mut Vec) { + if let Some(t) = node.get("Node Type").and_then(serde_json::Value::as_str) { + into.push(t.to_owned()); + } + for child in node + .get("Plans") + .and_then(serde_json::Value::as_array) + .into_iter() + .flatten() + { + plan_node_types(child, into); + } +} + +/// The index names every scan node under `node` uses. +fn plan_index_names(node: &serde_json::Value, into: &mut Vec) { + if let Some(name) = node.get("Index Name").and_then(serde_json::Value::as_str) { + into.push(name.to_owned()); + } + for child in node + .get("Plans") + .and_then(serde_json::Value::as_array) + .into_iter() + .flatten() + { + plan_index_names(child, into); + } +} + +/// A 100k-row composite-key table (`pk` = `p00000`..`p02000`, 50 rows each, +/// `sk_n` = 1..100000) filled straight into the data table and analyzed, +/// with its description and a data pool; the engine's `put_item` would take +/// minutes for 100k rows. +async fn hundred_k_table(s: &Scratch) -> (extenddb_core::types::TableDescription, PgPool) { + filled_table( + s, + "hundred_k", + "SELECT 'p' || lpad((i / 50)::text, 5, '0'), i, \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 100000) AS i", + ) + .await +} + +/// One key value bound to a page query's seek: text (numeric keys travel +/// as text) or bytes. +enum Bind { + Text(&'static str), + Bytes(Vec), +} + +/// `EXPLAIN (FORMAT JSON)` of the page query (with `COLLATE "C"` on its text +/// key columns when `collate_c`, as the engine writes it for a table whose +/// columns have that collation) after `after` (`None` for the first page) at +/// row bound `n`: the plan's node types in depth-first order and the index +/// names its scans use, with the plan for the message. +async fn explain_page( + data: &PgPool, + desc: &extenddb_core::types::TableDescription, + collate_c: bool, + after: Option>, + n: i64, +) -> (Vec, Vec, serde_json::Value) { + let budget = i64::try_from(extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES).unwrap(); + let sql = + extenddb_storage_postgres::page_sql_for(desc, &desc.table_id, collate_c, after.is_some()); + let explain_sql = format!("EXPLAIN (FORMAT JSON) {sql}"); + let mut explain = sqlx::query_scalar::<_, serde_json::Value>(&explain_sql); + let seek = after.is_some(); + for value in after.into_iter().flatten() { + explain = match value { + Bind::Text(t) => explain.bind(t), + Bind::Bytes(b) => explain.bind(b), + }; + } + let plan = explain + .bind(n) + .bind(budget) + .fetch_one(data) + .await + .unwrap_or_else(|e| panic!("explain (after a key: {seek}) at n = {n}: {e}")); + let root = &plan[0]["Plan"]; + let mut types = Vec::new(); + plan_node_types(root, &mut types); + let mut indexes = Vec::new(); + plan_index_names(root, &mut indexes); + (types, indexes, plan) +} + +/// Assert that a page query's plan is `Limit` over one `Index Scan` on the +/// primary key index with no sort: the seek and the order both follow the +/// index, so the page costs the server the rows it reads, up to `n`, each +/// serialized once, and not the rows before them. +fn assert_index_scan( + label: &str, + types: &[String], + indexes: &[String], + plan: &serde_json::Value, + pkey: &str, +) { + assert!( + !types.iter().any(|t| t == "Sort"), + "{label}: the plan sorts: {types:?}\n{plan}" + ); + let scans: Vec<&str> = types + .iter() + .map(String::as_str) + .filter(|t| t.ends_with("Scan") && *t != "Subquery Scan") + .collect(); + assert_eq!( + scans, + vec!["Index Scan"], + "{label}: the relation is not read by one index scan: {types:?}\n{plan}" + ); + assert_eq!( + indexes, + vec![pkey.to_owned()], + "{label}: the scan does not use the primary key index\n{plan}" + ); +} + +/// A key shape the page query is run on: the `CreateTable` input, the data +/// table's key columns, the `SELECT` filling 20,000 rows of it, and a key in +/// its middle to seek past. +struct KeyShape { + label: &'static str, + input: extenddb_core::types::CreateTableInput, + columns: &'static str, + rows: &'static str, + middle: Vec, + /// Whether the text key columns have collation "C", so the engine writes + /// `COLLATE "C"` into the query (every table but the legacy one). + collate_c: bool, + /// Whether the stream reports the table ordered (a multi-part range key + /// has no format order; a legacy table's key column has the database + /// collation). + ordered: bool, +} + +fn key_shape_input(name: &str, range: &[(&str, &str)]) -> extenddb_core::types::CreateTableInput { + let mut key_schema = vec![serde_json::json!({"AttributeName": "pk", "KeyType": "HASH"})]; + let mut defs = vec![serde_json::json!({"AttributeName": "pk", "AttributeType": "S"})]; + for (attr, ty) in range { + key_schema.push(serde_json::json!({"AttributeName": attr, "KeyType": "RANGE"})); + defs.push(serde_json::json!({"AttributeName": attr, "AttributeType": ty})); + } + serde_json::from_value(serde_json::json!({ + "TableName": name, + "KeySchema": key_schema, + "AttributeDefinitions": defs, + "BillingMode": "PAY_PER_REQUEST" + })) + .expect("create input") +} + +/// The six key shapes a data table's primary key can have: hash only; an +/// `S`, `N` or `B` range key; a multi-part range key; and a legacy table +/// whose text key column carries the database collation (recollated after +/// the fill, as `legacy_collation_table_streams_unordered_and_complete` +/// does). +fn key_shapes() -> Vec { + vec![ + KeyShape { + label: "hash only", + input: key_shape_input("hash_only", &[]), + columns: "pk, item_data", + rows: "SELECT 'p' || lpad(i::text, 5, '0'), \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad(i::text, 5, '0')), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![Bind::Text("p10000")], + collate_c: true, + ordered: true, + }, + KeyShape { + label: "S range key", + input: key_shape_input("range_s", &[("sk", "S")]), + columns: "pk, sk_s, item_data", + rows: "SELECT 'p' || lpad((i / 50)::text, 5, '0'), 'k' || lpad(i::text, 5, '0'), \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('S', 'k' || lpad(i::text, 5, '0')), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![Bind::Text("p00200"), Bind::Text("k10000")], + collate_c: true, + ordered: true, + }, + KeyShape { + label: "N range key", + input: key_shape_input("range_n", &[("sk", "N")]), + columns: "pk, sk_n, item_data", + rows: "SELECT 'p' || lpad((i / 50)::text, 5, '0'), i, \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![Bind::Text("p00200"), Bind::Text("10000")], + collate_c: true, + ordered: true, + }, + KeyShape { + label: "B range key", + input: key_shape_input("range_b", &[("sk", "B")]), + columns: "pk, sk_b, item_data", + rows: "SELECT 'p' || lpad((i / 50)::text, 5, '0'), decode(lpad(to_hex(i), 8, '0'), 'hex'), \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('B', encode(decode(lpad(to_hex(i), 8, '0'), 'hex'), 'base64')), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![ + Bind::Text("p00200"), + Bind::Bytes(10000_u32.to_be_bytes().to_vec()), + ], + collate_c: true, + ordered: true, + }, + KeyShape { + label: "multi-part range key (S, N)", + input: key_shape_input("multi_part", &[("sk", "S"), ("sk2", "N")]), + columns: "pk, sk_s, sk2_n, item_data", + rows: "SELECT 'p' || lpad((i / 50)::text, 5, '0'), 'k' || lpad((i / 10)::text, 4, '0'), i, \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('S', 'k' || lpad((i / 10)::text, 4, '0')), \ + 'sk2', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![ + Bind::Text("p00200"), + Bind::Text("k1000"), + Bind::Text("10000"), + ], + collate_c: true, + ordered: false, + }, + KeyShape { + label: "legacy collation (pk on the database collation)", + input: key_shape_input("legacy", &[("sk", "N")]), + columns: "pk, sk_n, item_data", + rows: "SELECT 'p' || lpad((i / 50)::text, 5, '0'), i, \ + jsonb_build_object('pk', jsonb_build_object('S', 'p' || lpad((i / 50)::text, 5, '0')), \ + 'sk', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', repeat('x', 100))) \ + FROM generate_series(1, 20000) AS i", + middle: vec![Bind::Text("p00200"), Bind::Text("10000")], + collate_c: false, + ordered: false, + }, + ] +} + +/// On every key shape a table can have, the page query (first page and a +/// page after a key in the table's middle, at the first page's row bound of +/// 16 and at 1000) is `Limit` over one `Index Scan` on the primary key index +/// with no sort, on a 20,000-row table; and the stream over the rows is +/// complete, ordered where the shape has a format order, in 21 round trips +/// (16 rows, then nineteen pages of 1000 and a last of 984). The legacy +/// table carries no `COLLATE "C"` in its query and pages in its index's +/// order. Near the end of a small table the plan differs: +/// `the_last_page_sorts_the_rows_after_the_seek`. +async fn the_page_query_is_an_index_scan_on_every_key_shape(shared: bool) { + let s = scratch_with(shared).await; + for shape in key_shapes() { + let name = shape.input.table_name.clone(); + create_active(&s, shape.input).await; + let desc = s + .engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: name.clone(), + }, + ) + .await + .expect("describe"); + let ddb_table = data_table_name(&desc.table_id); + let data = s.data_pool().await; + sqlx::query(&format!( + "INSERT INTO {ddb_table} ({}) {}", + shape.columns, shape.rows + )) + .execute(&data) + .await + .unwrap_or_else(|e| panic!("{}: fill the table: {e}", shape.label)); + if shape.label.starts_with("legacy") { + sqlx::query(&format!( + "ALTER TABLE {ddb_table} ALTER COLUMN pk TYPE TEXT COLLATE \"default\"" + )) + .execute(&data) + .await + .expect("recollate"); + } + sqlx::query(&format!("ANALYZE {ddb_table}")) + .execute(&data) + .await + .expect("analyze"); + let pkey = format!("{}_pkey", ddb_table.trim_matches('"')); + for n in [ + i64::try_from(extenddb_storage_postgres::MIN_PAGE_ROWS).unwrap(), + 1000, + ] { + let (types, indexes, plan) = explain_page(&data, &desc, shape.collate_c, None, n).await; + assert_index_scan( + &format!("{}, first page at n = {n}", shape.label), + &types, + &indexes, + &plan, + &pkey, + ); + let middle: Vec = shape + .middle + .iter() + .map(|b| match b { + Bind::Text(t) => Bind::Text(t), + Bind::Bytes(v) => Bind::Bytes(v.clone()), + }) + .collect(); + let (types, indexes, plan) = + explain_page(&data, &desc, shape.collate_c, Some(middle), n).await; + assert_index_scan( + &format!("{}, page after a key at n = {n}", shape.label), + &types, + &indexes, + &plan, + &pkey, + ); + } + let sql = + extenddb_storage_postgres::page_sql_for(&desc, &desc.table_id, shape.collate_c, true); + assert_eq!( + sql.contains("COLLATE \"C\""), + shape.collate_c, + "{}: {sql}", + shape.label + ); + data.close().await; + + let fetches_before = extenddb_storage::snapshot::paging::fetches(); + let snapshot = s + .engine + .open_table_snapshot(ACCOUNT, &name) + .await + .expect("open"); + assert_eq!(snapshot.key_ordered, shape.ordered, "{}", shape.label); + let order = extenddb_core::backup_format::KeyOrder::resolve( + &snapshot.description.key_schema, + &snapshot.description.attribute_definitions, + ) + .ok(); + let items: Vec = { + use futures::StreamExt as _; + snapshot + .items + .map(|item| item.expect("item")) + .collect() + .await + }; + assert_eq!(items.len(), 20_000, "{}: every row streams", shape.label); + if shape.ordered { + let order = order.expect("an ordered shape has a format order"); + for pair in items.windows(2) { + assert_eq!( + order.compare(&pair[0], &pair[1]).expect("comparable keys"), + std::cmp::Ordering::Less, + "{}: {:?} is not before {:?}", + shape.label, + pair[0], + pair[1] + ); + } + } + let fetches = extenddb_storage::snapshot::paging::fetches() - fetches_before; + assert_eq!( + fetches, 21, + "{}: a first page of 16, nineteen of 1000 and a last of 984", + shape.label + ); + } +} +on_both_layouts!(the_page_query_is_an_index_scan_on_every_key_shape); + +/// Bytes of item text in a 27 MB row of `scattered_table`: the size of +/// #394's worst-expansion line, so three such rows fill the 64 MiB budget +/// (0, 27, 54 MB before each) and the fourth is the next page's. +const LARGE_ROW_PAD: usize = 27_000_000; + +/// A table of `small` rows of 50-byte text under keys `a0000`.., with a +/// 27 MB row at every `every`th key (`i % every == every - 1`), filled +/// straight into the data table. Key order does not sort items by size, so +/// a table whose large items are a minority is this shape rather than the +/// grouped one (`put-worst`'s `zz` keys after the `a` keys). +async fn scattered_table( + s: &Scratch, + name: &str, + small: usize, + every: usize, +) -> (extenddb_core::types::TableDescription, PgPool) { + filled_table( + s, + name, + &format!( + "SELECT 'a' || lpad(i::text, 4, '0'), i, \ + jsonb_build_object('pk', jsonb_build_object('S', 'a' || lpad(i::text, 4, '0')), \ + 'sk', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', CASE WHEN i % {every} = {} \ + THEN repeat('x', {LARGE_ROW_PAD}) ELSE repeat('x', 50) END)) \ + FROM generate_series(0, {}) AS i", + every - 1, + small - 1 + ), + ) + .await +} + +/// On a table whose 27 MB rows are scattered among small ones (every 33rd +/// key of 300, nine large rows) the row bound's memory holds: the first +/// page reads 16 small rows, the second asks for the row bound (1000), meets +/// the large rows and reads every row left (284) for the 83 it keeps (up to +/// and including the third large one, whose predecessors hold 54 MB), and +/// every page after it asks for the floor (4) until the table ends, since +/// the memory covers a row bound's worth of rows and the table has fewer: +/// 201 rows read past the cut, all on the second page, over 53 round trips, +/// every row once in key order. With one page of memory the bound returns to +/// 1000 after four small rows and the over-read recurs (306 rows past the +/// cut over 7 pages on this table), which is what the rule corrects; the +/// bound that remains is the second page, and then the page that follows a +/// row bound's worth of small rows, each reading up to `n` rows of up to the +/// line cap: the 201 rows here are that second page's. +async fn the_row_bound_remembers_large_rows_among_small_ones(shared: bool) { + let s = scratch_with(shared).await; + let (_desc, data) = scattered_table(&s, "scattered", 300, 33).await; + data.close().await; + let budget = extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES; + extenddb_storage::snapshot::paging::reset_page_counters(); + let fetches_before = extenddb_storage::snapshot::paging::fetches(); + let snapshot = s + .engine + .open_table_snapshot(ACCOUNT, "scattered") + .await + .expect("open"); + assert!(snapshot.key_ordered); + let keys: Vec = { + use futures::StreamExt as _; + snapshot + .items + .map(|item| match item.expect("item").get("pk") { + Some(extenddb_core::types::AttributeValue::S(v)) => v.clone(), + other => panic!("pk is {other:?}"), + }) + .collect() + .await + }; + let expected: Vec = (0..300).map(|i| format!("a{i:04}")).collect(); + assert_eq!(keys, expected, "every row once, in key order"); + let fetches = extenddb_storage::snapshot::paging::fetches() - fetches_before; + let past_cut = extenddb_storage::snapshot::paging::rows_past_cut(); + let first = extenddb_storage_postgres::MIN_PAGE_ROWS; + let second_read = 300 - first; + let second_kept = 99 - first; + let rest: usize = 300 - 99; + eprintln!( + "scattered rows: {past_cut} rows read past the cut over {fetches} pages ({} expected, all on \ + the second page)", + second_read - second_kept + ); + assert_eq!( + past_cut, + u64::try_from(second_read - second_kept).unwrap(), + "rows read past the cut: the second page reads the {second_read} rows left for the \ + {second_kept} it keeps, and no later page asks for more than the floor" + ); + // The floor is written as its value: a floor of 2 (the budget divided by + // a 27 MB row) would page a table of such rows two to a page with no page + // reaching its cut, and show here as 103 round trips. + assert_eq!( + fetches, + u64::try_from(2 + rest.div_ceil(4)).unwrap(), + "round trips: two pages, then {rest} rows at the floor of four" + ); + let largest = extenddb_storage::snapshot::paging::max_page_bytes(); + assert!( + largest >= budget && largest < budget + LARGE_ROW_PAD, + "the largest page holds {largest} bytes" + ); +} +on_both_layouts!(the_row_bound_remembers_large_rows_among_small_ones); + +/// A table filled with `rows` straight into its data table and analyzed, as +/// `hundred_k_table` does, with its description and a data pool; `rows` is +/// the `SELECT` producing `(pk, sk_n, item_data)`. +async fn filled_table( + s: &Scratch, + name: &str, + rows: &str, +) -> (extenddb_core::types::TableDescription, PgPool) { + create_active(s, snapshot_conformance::table_input(name)).await; + let desc = s + .engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: name.to_owned(), + }, + ) + .await + .expect("describe"); + let ddb_table = data_table_name(&desc.table_id); + let data = s.data_pool().await; + sqlx::query(&format!( + "INSERT INTO {ddb_table} (pk, sk_n, item_data) {rows}" + )) + .execute(&data) + .await + .expect("fill the table"); + sqlx::query(&format!("ANALYZE {ddb_table}")) + .execute(&data) + .await + .expect("analyze"); + (desc, data) +} + +/// The plan near the end of a table, where the planner's estimate of the +/// rows after the seek is under the row bound, is the planner's choice and +/// not always the index scan: on the 100k-row table a page after the +/// 99,990th row at `n` = 1000 (ten rows left) is still `Limit` over the +/// `Index Scan`, while on a table of 1,060 rows (1,000 under one key prefix, +/// 60 under another) a page after the third +/// of the last 60 at `n` = 810 is a `Bitmap Heap Scan` driven by a `Bitmap +/// Index Scan` on the primary key index and a `Sort` of the 57 rows after +/// the seek under the `Limit`, so that page sorts the rows it reads, with +/// their text (PostgreSQL 16). The cell pins both so the module doc's +/// statement of them stays measured; the seek uses the primary key index in +/// both, and the sorted page returns the rows after the seek in key order. +async fn the_last_page_sorts_the_rows_after_the_seek(shared: bool) { + let s = scratch_with(shared).await; + let (desc, data) = hundred_k_table(&s).await; + let ddb_table = data_table_name(&desc.table_id); + let pkey = vec![format!("{}_pkey", ddb_table.trim_matches('"'))]; + let (types, indexes, plan) = explain_page( + &data, + &desc, + true, + Some(vec![Bind::Text("p01999"), Bind::Text("99990")]), + 1000, + ) + .await; + eprintln!("100k rows, a page after ('p01999', 99990) at n = 1000: {types:?}"); + assert_eq!( + indexes, pkey, + "100k: the seek does not use the primary key index\n{plan}" + ); + assert!( + types.iter().any(|t| t == "Index Scan") && !types.iter().any(|t| t == "Sort"), + "100k rows: the page near the end is not the index scan the module doc says it is \ + (correct the doc if the planner changed): {types:?}\n{plan}" + ); + data.close().await; + + let (desc, data) = filled_table( + &s, + "small", + "SELECT pk, i, jsonb_build_object('pk', jsonb_build_object('S', pk), \ + 'sk', jsonb_build_object('N', i::text), \ + 'v', jsonb_build_object('S', repeat('x', 50))) \ + FROM (SELECT 'a' || lpad(i::text, 4, '0') AS pk, i FROM generate_series(0, 999) AS i \ + UNION ALL \ + SELECT 'zz' || lpad(i::text, 3, '0'), i FROM generate_series(0, 59) AS i) rows", + ) + .await; + let ddb_table = data_table_name(&desc.table_id); + let pkey = vec![format!("{}_pkey", ddb_table.trim_matches('"'))]; + let (types, indexes, plan) = explain_page( + &data, + &desc, + true, + Some(vec![Bind::Text("zz002"), Bind::Text("2")]), + 810, + ) + .await; + eprintln!("1,060 rows, a page after ('zz002', 2) at n = 810: {types:?}"); + assert_eq!( + indexes, pkey, + "1,060: the seek does not use the primary key index\n{plan}" + ); + assert!( + types.iter().any(|t| t == "Sort") && types.iter().any(|t| t == "Bitmap Heap Scan"), + "1,060 rows: the page near the end is not the bitmap scan and Sort the module doc says \ + PostgreSQL 16 chooses there (correct the doc if the planner changed): {types:?}\n{plan}" + ); + // The sorted page's rows: the 57 after the seek, in key order. + let sql = extenddb_storage_postgres::page_sql_for(&desc, &desc.table_id, true, true); + let rows: Vec<(String, String)> = sqlx::query_as(&format!("SELECT c0, c1 FROM ({sql}) q")) + .bind("zz002") + .bind("2") + .bind(810_i64) + .bind(i64::try_from(extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES).unwrap()) + .fetch_all(&data) + .await + .expect("the last page"); + let expected: Vec<(String, String)> = (3..60) + .map(|i| (format!("zz{i:03}"), i.to_string())) + .collect(); + assert_eq!(rows, expected); + data.close().await; +} +on_both_layouts!(the_last_page_sorts_the_rows_after_the_seek); + /// A hash-only table with no index, for tests that need many of them. fn plain_input(name: &str) -> extenddb_core::types::CreateTableInput { serde_json::from_value(serde_json::json!({ @@ -1253,16 +1918,17 @@ async fn a_restored_table_describes_its_restore(shared: bool) { } on_both_layouts!(a_restored_table_describes_its_restore); -/// The pid of the one backend whose last statement was the stream's `FETCH`. +/// The pid of the one backend whose last statement was the stream's page +/// query (its projection starts with the first key column's alias). async fn stream_backend(s: &Scratch, data: &PgPool) -> i32 { sqlx::query_scalar( "SELECT pid FROM pg_stat_activity WHERE datname = $1 \ - AND state = 'idle in transaction' AND query LIKE 'FETCH%'", + AND state = 'idle in transaction' AND query LIKE 'SELECT page.k0 AS c0%'", ) .bind(s.data_name()) .fetch_one(data) .await - .expect("the stream's backend is idle in its transaction after a FETCH") + .expect("the stream's backend is idle in its transaction after a page query") } /// End the stream's backend from another session, as a server restart or @@ -1436,7 +2102,7 @@ async fn release_wakes_the_active_poller(shared: bool, how: Release) { match snapshot.items.next().await { Some(Ok(_)) => {} Some(Err(_)) => break, - None => panic!("the stream ended without the FETCH error"), + None => panic!("the stream ended without the page query error"), } } kept = Some(snapshot); @@ -1649,7 +2315,7 @@ async fn the_in_catalog_backup_wakes_the_active_poller(shared: bool) { } on_both_layouts!(the_in_catalog_backup_wakes_the_active_poller); -/// A `FETCH` that fails (the backend ended under the stream) ends the stream +/// A page query that fails (the backend ended under the stream) ends the stream /// at the error and frees its permit and connection at once: with the /// errored stream still held a full bound of five opens succeeds. The /// `SnapshotExpired` codes take this branch. @@ -1671,10 +2337,10 @@ async fn a_fetch_error_ends_the_stream_and_frees_its_permit(shared: bool) { match snapshot.items.next().await { Some(Ok(_)) => items += 1, Some(Err(e)) => { - eprintln!("FETCH error after {items} items: {e}"); + eprintln!("page query error after {items} items: {e}"); break; } - None => panic!("the stream ended without reporting the FETCH error"), + None => panic!("the stream ended without reporting the page query error"), } } assert_eq!(items, 5, "the first page, then the error"); diff --git a/crates/storage-sqlite/Cargo.toml b/crates/storage-sqlite/Cargo.toml index 589030ade..96a432a1a 100644 --- a/crates/storage-sqlite/Cargo.toml +++ b/crates/storage-sqlite/Cargo.toml @@ -42,6 +42,11 @@ libsqlite3-sys = { workspace = true } [dev-dependencies] extenddb-storage = { workspace = true, features = ["test-util"] } +# `tests/backup_job_panic.rs` drives the engine's CreateBackup job and +# reconciler against a `SqliteEngine` (the engine depends on `extenddb-storage`, +# not on this crate, so the edge is acyclic). +extenddb-engine = { workspace = true } +extenddb-backup-store = { workspace = true } # The slow-statement test sets sqlx's threshold, a `log` level, and prints # every tracing event of a child process. log = "0.4" diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index 1629f2f53..8fa162aff 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -106,6 +106,49 @@ pub(crate) fn backup_server_slot_count(pool_size: u32) -> usize { .clamp(1, BACKUP_SERVER_SLOTS_MAX) } +/// The pool a server needs so that [`backup_server_slot_count`] admits every +/// job `backup.max_concurrent_jobs` admits: `pool_size` when its slot count +/// already meets the jobs, else the smallest pool whose slot count does, the +/// slot cap included (the rule is stated once, on `SqliteConfig::pool_size`). +/// The in-memory pool is not sized by this: it has one connection by nature. +pub(crate) fn pool_size_for_jobs(pool_size: u32, max_concurrent_jobs: usize) -> u32 { + let wanted = max_concurrent_jobs.min(BACKUP_SERVER_SLOTS_MAX); + if backup_server_slot_count(pool_size) >= wanted { + return pool_size; + } + let slots = u32::try_from(wanted).unwrap_or(u32::MAX); + slots.saturating_add(1).saturating_mul(2) +} + +/// The warning a server logs when [`pool_size_for_jobs`] raised its pool: +/// both numbers, the raised pool's slot count, and what the slots admit. +/// With the jobs within the slot cap the slots admit every job the +/// `max_concurrent_jobs` permit admits; over it the slots are capped at +/// [`BACKUP_SERVER_SLOTS_MAX`], and the line says which jobs the slot bound +/// refuses (the limits document's "above 16 jobs a 17th concurrent job is +/// refused by the slot"). +pub(crate) fn pool_raise_message( + pool_size: u32, + for_jobs: u32, + max_concurrent_jobs: usize, +) -> String { + let slots = backup_server_slot_count(for_jobs); + let admits = if slots < max_concurrent_jobs { + format!( + "the slots are capped at {BACKUP_SERVER_SLOTS_MAX}, so jobs {} to \ + {max_concurrent_jobs} are refused by the slot bound", + slots + 1 + ) + } else { + "the slot bound refuses no job the max_concurrent_jobs permit admitted".to_owned() + }; + format!( + "SQLite pool_size {pool_size} raised to {for_jobs} for backup.max_concurrent_jobs = \ + {max_concurrent_jobs}: a pool of {for_jobs} has {slots} server slots, one per running \ + backup job, so {admits}" + ) +} + /// How the slot refusal names its bound, beside the number: the engine's /// `max_concurrent_jobs` refusal carries the same text with its own name, so /// an operator can tell which bound a request met. The shared text reads @@ -3969,6 +4012,123 @@ mod restore_tests { assert!(raised[0].contains("pool_size >= 3"), "{raised:?}"); } + /// A server whose `backup.max_concurrent_jobs` is over the slot count of + /// its configured pool raises the pool, once, with a warning naming both + /// numbers, so the slot count meets the jobs; a pool whose slots already + /// meet them, and an engine opened with no jobs, are left as configured + /// and log nothing; the jobs over the slot cap raise the pool to the cap. + #[tokio::test] + async fn a_pool_raised_for_the_jobs_is_logged_and_its_slots_meet_them() { + use super::pool_size_for_jobs; + assert_eq!(pool_size_for_jobs(10, 4), 10, "the defaults need no raise"); + assert_eq!(pool_size_for_jobs(4, 4), 10); + assert_eq!(pool_size_for_jobs(4, 1), 4); + assert_eq!(pool_size_for_jobs(4, 2), 6); + assert_eq!(pool_size_for_jobs(2, 0), 2); + // An odd pool whose slots are exactly the jobs is kept: 11 holds 4 + // slots, and a strict comparison would lower it to 10. + assert_eq!(pool_size_for_jobs(11, 4), 11); + assert_eq!(pool_size_for_jobs(9, 4), 10, "9 holds 3 slots"); + assert_eq!(pool_size_for_jobs(10, 16), 34); + assert_eq!( + pool_size_for_jobs(10, 1024), + 34, + "the slot cap bounds the raise" + ); + for (pool, jobs) in [(1, 0), (4, 1), (4, 4), (10, 4), (3, 16), (10, 1024)] { + assert!( + backup_server_slot_count(pool_size_for_jobs(pool, jobs)) >= jobs.min(16), + "pool {pool}, jobs {jobs}" + ); + } + + let target = std::env::var_os("CARGO_TARGET_DIR") + .map(std::path::PathBuf::from) + .unwrap_or_else(|| { + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target") + }); + let path = target.join(format!("backup-test-{}.sqlite", uuid::Uuid::new_v4())); + let path = path.to_str().expect("UTF-8 path").to_owned(); + let (guard, logs) = crate::schema::log_capture::capture(); + let raised = SqliteEngine::new_for_server(&path, 4, 4, "us-east-1", 409_600) + .await + .expect("engine"); + assert_eq!( + raised.backup_server_slot_count, 4, + "the slots meet the jobs" + ); + assert!( + raised.backup_server_slots.available_permits() == 4, + "the semaphore has the raised count" + ); + close_file_engine(raised, std::path::Path::new(&path)).await; + let kept = SqliteEngine::new_for_server(&path, 10, 4, "us-east-1", 409_600) + .await + .expect("engine"); + assert_eq!(kept.backup_server_slot_count, 4); + close_file_engine(kept, std::path::Path::new(&path)).await; + let memory = SqliteEngine::new_for_server(":memory:", 10, 4, "us-east-1", 409_600) + .await + .expect("engine"); + assert_eq!( + memory.backup_server_slot_count, 1, + "an in-memory database has one connection and one slot whatever the jobs" + ); + drop(memory); + drop(guard); + let logs = logs.lock().expect("logs").clone(); + let raised: Vec<&String> = logs + .iter() + .filter(|l| l.contains("pool_size") && l.contains("raised")) + .collect(); + assert_eq!(raised.len(), 1, "{logs:#?}"); + assert!(raised[0].starts_with("WARN"), "{raised:?}"); + assert!( + raised[0].contains("pool_size 4 raised to 10 for backup.max_concurrent_jobs = 4"), + "{raised:?}" + ); + assert!(raised[0].contains("has 4 server slots"), "{raised:?}"); + } + + /// The pool-raise warning says what the raised pool's slots admit: with + /// the jobs within the slot cap, every job the permit admits; over the + /// cap (16), the jobs the slot bound refuses, 17 to `max_concurrent_jobs`, + /// as the limits document states it (a line saying the slot refused no + /// admitted job at 20 jobs was wrong: it refuses the 17th). + #[test] + fn the_pool_raise_warning_names_the_jobs_the_slot_cap_refuses() { + use super::{pool_raise_message, pool_size_for_jobs}; + let at_8 = pool_raise_message(10, pool_size_for_jobs(10, 8), 8); + assert!( + at_8.starts_with( + "SQLite pool_size 10 raised to 18 for backup.max_concurrent_jobs = 8: a pool of \ + 18 has 8 server slots, one per running backup job, so " + ), + "{at_8}" + ); + assert!( + at_8.ends_with( + "so the slot bound refuses no job the max_concurrent_jobs permit admitted" + ), + "{at_8}" + ); + let at_20 = pool_raise_message(10, pool_size_for_jobs(10, 20), 20); + assert!( + at_20.starts_with( + "SQLite pool_size 10 raised to 34 for backup.max_concurrent_jobs = 20: a pool of \ + 34 has 16 server slots, one per running backup job, so " + ), + "{at_20}" + ); + assert!( + at_20.ends_with( + "so the slots are capped at 16, so jobs 17 to 20 are refused by the slot bound" + ), + "{at_20}" + ); + assert!(!at_20.contains("refuses no job"), "{at_20}"); + } + /// A flood of CreateBackups from one account on the default pool of 10 /// does not starve another account's reads: at most /// `backup_server_slot_count` backups run at once (observed as CREATING @@ -4471,6 +4631,7 @@ mod restore_tests { size_bytes: Some(1234), item_count: Some(5), failure_reason: None, + source_table_details_json: None, completed: true, }; assert_eq!( @@ -5143,6 +5304,228 @@ mod restore_tests { ); } + /// `claim_dead_backup` by the slot rule: a CREATING row whose slot no + /// task of this process holds is dead at once, whatever its heartbeat + /// and whatever the stale window (a row the last process left, here + /// inserted with a fresh heartbeat and tested with a window of an + /// hour); a row whose claim a task holds is live and refused; a row + /// whose claim dropped (the job's task ended by any route) is taken + /// over at once. The successor holds the slot: a second taker and a + /// `claim_backup` of the row are refused until the successor's claim + /// drops. The takeover mints a fresh generation and leaves + /// `heartbeat_at` as the dead owner wrote it. The successor's + /// AVAILABLE write carries `source_table_details_json` when `Some`, + /// leaves it when `None`, and stamps `completed_at`. An AVAILABLE row + /// and a missing row are refused, and the slot taken for the test is + /// given back. + #[tokio::test] + async fn claim_dead_backup_takes_over_a_row_whose_slot_no_task_holds() { + let engine = engine().await; + let stale = Duration::from_secs(3600); + let slot_key = |row: &BackupRow| format!("backup:{}", row.backup_id); + + // A row the last process left: inserted directly, no slot held, + // its heartbeat fresh. Dead at once. + let left = row(ACCOUNT, "Music", "left"); + let source_table = serde_json::to_string(&left.source_table_details_json).unwrap(); + sqlx::query(&format!( + "INSERT INTO backup_index (backup_id, account_id, table_name, backup_arn, \ + backup_name, status, store_kind, store_root, store_id, size_bytes, item_count, \ + created_at, source_table_details_json, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, ?, 'CREATING', ?, ?, ?, 0, 0, ?, ?, 7, {NOW})", + NOW = crate::backup_catalog::NOW + )) + .bind(&left.backup_id) + .bind(&left.account_id) + .bind(&left.table_name) + .bind(&left.backup_arn) + .bind(&left.backup_name) + .bind(left.store.store_kind.as_str()) + .bind(&left.store.store_root) + .bind(&left.store.store_id) + .bind(crate::sqlite_util::format_epoch_seconds(left.created_at)) + .bind(&source_table) + .execute(&engine.pool) + .await + .expect("a row of the last process"); + let before = engine + .read_backup_row(&arn_of(&left)) + .await + .unwrap() + .unwrap(); + assert_eq!(before.owner_generation, 7); + assert!(!engine.job_slots.holds(&slot_key(&left))); + let successor = engine + .claim_dead_backup(&left.key(), stale) + .await + .expect("claim") + .expect("a row with no slot holder is dead at once, fresh heartbeat or not"); + assert_ne!(successor.generation(), 7); + assert_ne!(successor.generation(), UNOWNED_GENERATION); + let after = engine + .read_backup_row(&arn_of(&left)) + .await + .unwrap() + .unwrap(); + assert_eq!(after.owner_generation, successor.generation()); + assert_eq!(after.status, BackupStatus::Creating); + assert_eq!( + after.heartbeat_at, before.heartbeat_at, + "the takeover left the heartbeat as the dead owner wrote it" + ); + assert!( + engine.job_slots.holds(&slot_key(&left)), + "the successor holds the slot" + ); + assert!( + engine + .claim_dead_backup(&left.key(), stale) + .await + .expect("claim") + .is_none(), + "a row another task holds is refused" + ); + assert!( + matches!( + engine.claim_backup(&left).await.err(), + Some(StorageError::AlreadyExists(m)) if m.contains("a live job owns") + ), + "a job's claim of a taken-over row meets the successor's slot" + ); + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + // The AVAILABLE write with the manifest's source table: the row + // carries it; a later write with `None` leaves it. + let mut details = left.source_table_details_json.clone(); + details["item_count"] = serde_json::json!(4242); + let fields = StatusFields { + size_bytes: Some(99), + item_count: Some(4242), + failure_reason: None, + source_table_details_json: Some(details.clone()), + completed: true, + }; + assert_eq!( + successor + .set_status(JobStatus::Creating, JobStatus::Available, &fields) + .await + .unwrap(), + WriteOutcome::Matched + ); + let stored = engine + .read_backup_row(&arn_of(&left)) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.status, BackupStatus::Available); + assert_eq!(stored.size_bytes, 99); + assert_eq!(stored.item_count, 4242); + assert_eq!( + stored.source_table_details_json, details, + "the AVAILABLE write carries the source table it was given" + ); + assert_ne!( + stored.source_table_details_json, + left.source_table_details_json + ); + assert!(stored.completed_at.is_some(), "completed_at is stamped"); + assert_eq!( + successor + .set_status( + JobStatus::Available, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("later".to_owned()), + ..StatusFields::default() + } + ) + .await + .unwrap(), + WriteOutcome::Matched + ); + let stored = engine + .read_backup_row(&arn_of(&left)) + .await + .unwrap() + .unwrap(); + assert_eq!( + stored.source_table_details_json, details, + "a write without the member leaves the column" + ); + // Not CREATING any more: refused, and the slot taken for the test + // is given back. + drop(successor); + assert!( + !engine.job_slots.holds(&slot_key(&left)), + "the claim's drop frees the slot" + ); + assert!( + engine + .claim_dead_backup(&left.key(), stale) + .await + .expect("claim") + .is_none(), + "a row that is not CREATING is refused" + ); + assert!( + !engine.job_slots.holds(&slot_key(&left)), + "a refusal leaves no slot behind" + ); + + // A live owner: its claim holds the slot, so it is refused + // whatever the heartbeat; once the claim drops (the task ended), + // the row is dead at once. + let live = row(ACCOUNT, "Music", "live"); + let owner = engine.claim_backup(&live).await.expect("claim"); + sqlx::query( + "UPDATE backup_index SET heartbeat_at = \ + strftime('%Y-%m-%dT%H:%M:%fZ', 'now', '-1 day') WHERE backup_id = ?", + ) + .bind(&live.backup_id) + .execute(&engine.pool) + .await + .expect("age the heartbeat"); + assert!( + engine + .claim_dead_backup(&live.key(), Duration::from_secs(1)) + .await + .expect("claim") + .is_none(), + "a held slot is a live owner, a day-old heartbeat or not" + ); + assert_eq!( + engine + .read_backup_row(&arn_of(&live)) + .await + .unwrap() + .unwrap() + .owner_generation, + owner.generation(), + "a refusal leaves the generation" + ); + drop(owner); + let successor = engine + .claim_dead_backup(&live.key(), stale) + .await + .expect("claim") + .expect("the owner's task ended: dead at once"); + assert_ne!(successor.generation(), UNOWNED_GENERATION); + drop(successor); + + // A missing row: refused, no slot left behind. + let missing = row(ACCOUNT, "Music", "missing"); + assert!( + engine + .claim_dead_backup(&missing.key(), stale) + .await + .expect("claim") + .is_none() + ); + assert!(!engine.job_slots.holds(&slot_key(&missing))); + } + /// `list_backup_rows_older_than`: the rows of a store with a status /// whose clock column is older than the age on the database's clock. /// The two clocks are told apart by a row heartbeated at the start diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs index deba547c8..562f94b0b 100644 --- a/crates/storage-sqlite/src/backup_catalog.rs +++ b/crates/storage-sqlite/src/backup_catalog.rs @@ -4,14 +4,20 @@ //! The catalog index on SQLite: `backup_index`, the restore job fields of //! `table_restores`, `backup_maintenance` and `settings.catalog_id`. //! -//! One process serves a database file (the serve lock enforces it), so a -//! job's claim is an in-process slot ([`JobSlots`]): a key held in a set for +//! One process serves a database file (the serve lock enforces it, on Unix; +//! on a platform where no lock is taken the server binds CreateBackup to the +//! in-catalog path, `extenddb_server::backup_store::in_catalog_create_only`, +//! since two servers on one file would take each other's rows for dead), so +//! a job's claim is an in-process slot ([`JobSlots`]): a key held in a set for //! the job's life and released by a drop guard, so a task that ends by any //! route, a panic included, frees it. A restart frees every slot at once, -//! which is the proof of death this backend needs: a CREATING in-catalog -//! backup row whose slot no task holds is dead, and every RUNNING restore -//! job record of the stopped process is aged at startup so the reconciler's -//! first cycle finds it ([`SqliteEngine::expire_restore_job_heartbeats`]). +//! which is the proof of death this backend needs: a CREATING row (a +//! `backup_index` row or an in-catalog `backups` row) whose slot no task +//! holds is dead, whatever its heartbeat says, so the reconciler's takeover +//! ([`BackupCatalog::claim_dead_backup`]) reads no stale window here, and +//! every RUNNING restore job record of the stopped process is aged at startup +//! so the reconciler's first cycle finds it +//! ([`SqliteEngine::expire_restore_job_heartbeats`]). //! //! Every write a job makes to its row goes through its claim and carries the //! claim's `owner_generation`, so a row a successor took over matches no @@ -189,6 +195,41 @@ impl SqliteJobClaim { } } } + + /// The beat's `UPDATE`, run by a caller that holds the engine write + /// lock. + async fn beat_under_the_lock(&self) -> Result { + let affected = match &self.record { + JobRecord::Backup(key) => { + sqlx::query(&format!( + "UPDATE backup_index SET heartbeat_at = {NOW} \ + WHERE account_id = ? AND backup_id = ? AND owner_generation = ?" + )) + .bind(&key.account_id) + .bind(&key.backup_id) + .bind(generation_column(self.generation)) + .execute(&self.engine.pool) + .await + } + JobRecord::Restore { table_id } => { + sqlx::query(&format!( + "UPDATE table_restores SET heartbeat_at = {NOW} \ + WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" + )) + .bind(table_id) + .bind(generation_column(self.generation)) + .execute(&self.engine.pool) + .await + } + } + .map_err(db_err)? + .rows_affected(); + Ok(if affected == 1 { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + } } impl JobClaim for SqliteJobClaim { @@ -204,39 +245,32 @@ impl JobClaim for SqliteJobClaim { fn heartbeat(&self) -> BoxFuture<'_, Result> { Box::pin(async move { let _writer = self.engine.write_lock.lock().await; - let affected = match &self.record { - JobRecord::Backup(key) => { - sqlx::query(&format!( - "UPDATE backup_index SET heartbeat_at = {NOW} \ - WHERE account_id = ? AND backup_id = ? AND owner_generation = ?" - )) - .bind(&key.account_id) - .bind(&key.backup_id) - .bind(generation_column(self.generation)) - .execute(&self.engine.pool) - .await - } - JobRecord::Restore { table_id } => { - sqlx::query(&format!( - "UPDATE table_restores SET heartbeat_at = {NOW} \ - WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" - )) - .bind(table_id) - .bind(generation_column(self.generation)) - .execute(&self.engine.pool) - .await - } - } - .map_err(db_err)? - .rows_affected(); - Ok(if affected == 1 { - Heartbeat::Confirmed - } else { - Heartbeat::Lost - }) + self.beat_under_the_lock().await + }) + } + + /// The bound starts once the beat holds the engine write lock: the wait + /// for the lock is this process's own queue (a writer holding it past a + /// heartbeat period, a GSI backfill say, delays the beat and does not end + /// the job), and only the `UPDATE` itself is held to `bound`. + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxFuture<'_, Option>> { + Box::pin(async move { + let _writer = self.engine.write_lock.lock().await; + tokio::time::timeout(bound, self.beat_under_the_lock()) + .await + .ok() }) } + /// The wait for the engine write lock is outside the bound (above): a + /// beat parked there confirms the claim for as long as it waits. + fn heartbeat_parks_before_its_bound(&self) -> bool { + true + } + fn set_status<'a>( &'a self, from: JobStatus, @@ -247,11 +281,20 @@ impl JobClaim for SqliteJobClaim { let _writer = self.engine.write_lock.lock().await; let affected = match &self.record { JobRecord::Backup(key) => { + let source_table = fields + .source_table_details_json + .as_ref() + .map(serde_json::to_string) + .transpose() + .map_err(|e| { + StorageError::Internal(format!("source table details: {e}")) + })?; sqlx::query(&format!( "UPDATE backup_index SET status = ?, \ size_bytes = COALESCE(?, size_bytes), \ item_count = COALESCE(?, item_count), \ failure_reason = COALESCE(?, failure_reason), \ + source_table_details_json = COALESCE(?, source_table_details_json), \ completed_at = CASE WHEN ? THEN {NOW} ELSE completed_at END \ WHERE account_id = ? AND backup_id = ? AND owner_generation = ? \ AND status = ?" @@ -260,6 +303,7 @@ impl JobClaim for SqliteJobClaim { .bind(fields.size_bytes) .bind(fields.item_count) .bind(fields.failure_reason.as_deref()) + .bind(source_table) .bind(fields.completed) .bind(&key.account_id) .bind(&key.backup_id) @@ -835,6 +879,55 @@ impl BackupCatalog for SqliteEngine { }) } + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + _stale_after: Duration, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The slot is the liveness test (module doc): a job of this + // process that holds it is alive whatever its heartbeat says, and + // a row without a holder is dead at once, so the stale window is + // not read. Taking the slot is the test and the exclusion of a + // second taker in one step. + let Some(slot) = JobSlots::take(&self.job_slots, backup_slot_key(&key.backup_id)) + else { + return Ok(None); + }; + let generation = new_generation(); + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + // `heartbeat_at` stays as the dead owner wrote it (trait doc). + let taken = sqlx::query( + "UPDATE backup_index SET owner_generation = ? \ + WHERE account_id = ? AND backup_id = ? AND status = 'CREATING'", + ) + .bind(generation_column(generation)) + .bind(&key.account_id) + .bind(&key.backup_id) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + tx.commit().await.map_err(db_err)?; + if taken != 1 { + drop(slot); + return Ok(None); + } + Ok(Some(Box::new(SqliteJobClaim { + engine: self.clone(), + _slot: Some(slot), + _permit: None, + generation, + record: JobRecord::Backup(key.clone()), + }) as Box)) + }) + } + fn list_backup_rows_older_than<'a>( &'a self, status: BackupStatus, @@ -1261,3 +1354,414 @@ mod tests { ); } } + +/// The job under the engine write lock (the maintainer's blocking item on +/// PR 8): a writer that holds the lock past two heartbeat periods delays the +/// beat and the claim re-check and ends no job, and a put that comes while +/// the beat is parked on the lock is not refused. +#[cfg(test)] +mod write_lock_tests { + use std::os::unix::fs::DirBuilderExt; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::time::Duration; + + use extenddb_backup_store::{BackupStore, ByteStream, FilesystemStore, ObjectMeta, StoreError}; + use extenddb_core::backup_format::BackupArn; + use extenddb_core::limits::LimitsConfig; + use extenddb_engine::{ + BackupRuntime, ConfiguredStore, JobSettings, OperationContext, check_configured_store, + dispatch, + }; + use extenddb_storage::backup_catalog::{BackupCatalog, BackupStatus, StoreKind}; + use futures::future::BoxFuture; + use futures::stream::BoxStream; + use serde_json::json; + + use crate::store::SqliteEngine; + + const ACCOUNT: &str = "123456789012"; + + /// A scratch directory under the cargo target directory, removed on + /// drop with the database files beside it. + struct Scratch(std::path::PathBuf); + + impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } + } + + async fn call( + ctx: &OperationContext, + operation: &str, + body: serde_json::Value, + ) -> serde_json::Value { + dispatch(operation, body, ctx, "127.0.0.1:0") + .await + .unwrap_or_else(|e| panic!("{operation}: {e:?}")) + .body + } + + /// A store whose first `put` waits `delay` before it starts, counting + /// the puts started: the job's second data file then comes after the + /// delay, while the test still holds the write lock. + struct SlowFirstPut { + inner: Arc, + delay: Duration, + puts_started: AtomicUsize, + } + + impl BackupStore for SlowFirstPut { + fn put<'a>( + &'a self, + key: &'a str, + body: ByteStream, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + if self.puts_started.fetch_add(1, Ordering::AcqRel) == 0 { + tokio::time::sleep(self.delay).await; + } + self.inner.put(key, body).await + }) + } + + fn put_if_absent<'a>( + &'a self, + key: &'a str, + body: ByteStream, + ) -> BoxFuture<'a, Result, StoreError>> { + self.inner.put_if_absent(key, body) + } + + fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> { + self.inner.get(key) + } + + fn head<'a>( + &'a self, + key: &'a str, + ) -> BoxFuture<'a, Result, StoreError>> { + self.inner.head(key) + } + + fn list<'a>(&'a self, prefix: &'a str) -> BoxStream<'a, Result> { + self.inner.list(prefix) + } + + fn delete_prefix<'a>(&'a self, prefix: &'a str) -> BoxFuture<'a, Result> { + self.inner.delete_prefix(prefix) + } + + fn validate(&self) -> BoxFuture<'_, Result<(), StoreError>> { + self.inner.validate() + } + } + + /// A file-backed engine with a healthy filesystem store under a scratch + /// root, a runtime with `settings`, and a `Music` table of three items. + struct Fixture { + _scratch: Scratch, + engine: SqliteEngine, + root: std::path::PathBuf, + runtime: Arc, + ctx: OperationContext, + } + + async fn fixture( + settings: JobSettings, + wrap: impl FnOnce(Arc) -> Arc, + ) -> Fixture { + let target = std::env::var_os("CARGO_TARGET_DIR") + .map(std::path::PathBuf::from) + .unwrap_or_else(|| { + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../target") + }); + let scratch = Scratch(target.join(format!("backup-write-lock-{}", uuid::Uuid::new_v4()))); + std::fs::create_dir_all(&scratch.0).expect("scratch directory"); + let db_path = scratch.0.join("catalog.sqlite"); + let root = scratch.0.join("backups"); + std::fs::DirBuilder::new() + .mode(0o700) + .create(&root) + .expect("root"); + let engine = SqliteEngine::new( + db_path.to_str().expect("UTF-8 path"), + 4, + "us-east-1", + 409_600, + ) + .await + .expect("file engine"); + crate::migrations::apply_for_tests(&engine.pool).await; + for (key, value) in [ + ("control_plane_delay_seconds", "0"), + ("index_propagation_delay_ms", "0"), + ] { + sqlx::query( + "INSERT INTO settings (key, value) VALUES (?, ?) \ + ON CONFLICT(key) DO UPDATE SET value = excluded.value", + ) + .bind(key) + .bind(value) + .execute(&engine.pool) + .await + .expect("setting"); + } + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES (?, 'default')") + .bind(ACCOUNT) + .execute(&engine.pool) + .await + .expect("account"); + let storage: Arc = Arc::new(engine.clone()); + + let store: Arc = Arc::new( + FilesystemStore::open(&root) + .await + .expect("filesystem store"), + ); + let configured = ConfiguredStore::checking( + StoreKind::Filesystem, + root.to_str().expect("UTF-8 path").to_owned(), + wrap(store), + None, + ); + let outcome = check_configured_store(&configured, &*storage, "sqlite").await; + assert!(outcome.is_ok(), "the store's first check: {outcome:?}"); + configured.record_check(outcome); + let runtime = Arc::new( + BackupRuntime::new(4) + .with_job_settings(settings) + .with_store(configured) + .with_source("sqlite", "test"), + ); + let ctx = OperationContext { + storage: storage.clone(), + backup: runtime.clone(), + limits: Arc::new(LimitsConfig::default()), + region: Arc::from("us-east-1"), + account_id: Arc::from(ACCOUNT), + import_paths: Arc::from(Vec::new()), + export_paths: Arc::from(Vec::new()), + authorized_key_info: Default::default(), + auth_cache: extenddb_auth::AuthCacheRegistry::empty(), + table_key_info_lookup: None, + }; + call( + &ctx, + "CreateTable", + json!({ + "TableName": "Music", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "BillingMode": "PAY_PER_REQUEST" + }), + ) + .await; + for i in 0..3 { + call( + &ctx, + "PutItem", + json!({"TableName": "Music", "Item": {"pk": {"S": format!("k{i}")}}}), + ) + .await; + } + Fixture { + _scratch: scratch, + engine, + root, + runtime, + ctx, + } + } + + impl Fixture { + /// CreateBackup of `Music`, answered CREATING; the backup's ARN. + async fn create_backup(&self) -> BackupArn { + let details = call( + &self.ctx, + "CreateBackup", + json!({"TableName": "Music", "BackupName": "nightly"}), + ) + .await; + assert_eq!( + details["BackupDetails"]["BackupStatus"], "CREATING", + "{details}" + ); + BackupArn::parse( + details["BackupDetails"]["BackupArn"] + .as_str() + .expect("an ARN"), + ) + .expect("a backup ARN") + } + + /// Wait for the job to end, then read its row. + async fn ended_row(&self, arn: &BackupArn) -> extenddb_storage::backup_catalog::BackupRow { + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while self.runtime.running_jobs() > 0 { + assert!( + std::time::Instant::now() < deadline, + "the job did not end after the lock was released" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + self.engine + .read_backup_row(arn) + .await + .expect("read the row") + .expect("the row exists") + } + } + + /// With a heartbeat period of 300 ms, the test takes the engine write + /// lock the moment CreateBackup answers CREATING and holds it for three + /// periods. The timer's first beat and the job's re-check before the + /// manifest queue behind it; their bound starts once they hold the lock + /// (`heartbeat_within`), so both confirm when the lock is released and + /// the backup ends AVAILABLE with its manifest in the store. With the + /// bound over the whole call the first beat ends the job as a lost claim + /// ("the heartbeat did not return within 0 s") and the row stays + /// CREATING. + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn a_writer_holding_the_lock_past_two_periods_delays_the_beat_and_ends_no_job() { + let period = Duration::from_millis(300); + let f = fixture( + JobSettings { + heartbeat_interval: period, + ..JobSettings::default() + }, + |store| store, + ) + .await; + let arn = f.create_backup().await; + // The writer: the lock taken as the row exists, held for three + // periods, past the first beat's tick and its whole-call bound. + let held = f.engine.write_lock.lock().await; + tokio::time::sleep(3 * period).await; + assert_eq!(f.runtime.running_jobs(), 1, "the job waits on the lock"); + drop(held); + + let row = f.ended_row(&arn).await; + assert_eq!( + (row.status, row.failure_reason.clone()), + (BackupStatus::Available, None), + "{row:?}" + ); + assert_eq!(row.item_count, 3); + assert!( + f.root + .join(arn.store_prefix()) + .join("extenddb-manifest.json") + .is_file(), + "the manifest is in the store" + ); + assert!( + row.heartbeat_at > row.created_at, + "a beat landed after the insert ({} > {})", + row.heartbeat_at, + row.created_at + ); + f.engine.pool.close().await; + } + + /// The maintainer's second-round case: a put that comes while the beat + /// is parked on the lock. A data file per item (`data_file_target_bytes` + /// of one byte) and a store whose first put takes four periods: the + /// first data file's check passes on the claim's insert, the timer's + /// first beat parks on the lock the test took as the row appeared, and + /// the second data file's check comes four periods after the insert + /// with no beat confirmed since. The check passes on the beat in flight + /// (`confirms_the_claim`'s third clause), the second put starts under + /// the hold, and once the lock is released the beat lands and the backup + /// ends AVAILABLE with its three data files and its manifest. With the + /// in-flight clause removed the second data file's check refuses the + /// put ("no heartbeat confirmed the claim recently enough for data file + /// 2") and the job stops: this cell then sees one put started and the + /// row left CREATING, since no reconciler runs here (on a server the + /// reconciler fails the row and deletes its prefix, which is what the + /// maintainer measured at the third or fourth data file beside a GSI + /// build). + #[tokio::test(flavor = "multi_thread", worker_threads = 4)] + async fn a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it() { + let period = Duration::from_millis(300); + let slow = Arc::new(std::sync::Mutex::new(None::>)); + let f = fixture( + JobSettings { + heartbeat_interval: period, + data_file_target_bytes: 1, + ..JobSettings::default() + }, + |store| { + let wrapped = Arc::new(SlowFirstPut { + inner: store, + delay: 4 * period, + puts_started: AtomicUsize::new(0), + }); + *slow.lock().unwrap() = Some(Arc::clone(&wrapped)); + wrapped + }, + ) + .await; + let slow = slow.lock().unwrap().take().expect("the wrapped store"); + let arn = f.create_backup().await; + // The writer: the lock taken as the row exists, held until the + // second data file's put has started, which needs the pre-put check + // to pass with the first beat parked behind this hold. + let held = f.engine.write_lock.lock().await; + let at_the_hold = f + .engine + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while slow.puts_started.load(Ordering::Acquire) < 2 { + if std::time::Instant::now() >= deadline || f.runtime.running_jobs() == 0 { + drop(held); + let row = f.ended_row(&arn).await; + panic!( + "the second data file was not put under the hold: {} put(s) started, row {row:?}", + slow.puts_started.load(Ordering::Acquire) + ); + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + let parked = f + .engine + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + assert_eq!( + parked.heartbeat_at, at_the_hold.heartbeat_at, + "no beat landed under the hold: the first beat is parked on the lock" + ); + assert_eq!(parked.status, BackupStatus::Creating); + drop(held); + + let row = f.ended_row(&arn).await; + assert_eq!( + (row.status, row.failure_reason.clone()), + (BackupStatus::Available, None), + "{row:?}" + ); + assert_eq!(row.item_count, 3); + let prefix = f.root.join(arn.store_prefix()); + for name in [ + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + "extenddb-manifest.json", + ] { + assert!(prefix.join(name).is_file(), "{name} is in the store"); + } + assert!( + row.heartbeat_at > at_the_hold.heartbeat_at, + "the parked beat landed once the lock was released ({} > {})", + row.heartbeat_at, + at_the_hold.heartbeat_at + ); + f.engine.pool.close().await; + } +} diff --git a/crates/storage-sqlite/src/config.rs b/crates/storage-sqlite/src/config.rs index b0d947d56..24987988f 100644 --- a/crates/storage-sqlite/src/config.rs +++ b/crates/storage-sqlite/src/config.rs @@ -24,11 +24,17 @@ use serde::Deserialize; /// stream (file-backed or in-memory) holds one. A snapshot or a `CreateBackup` /// that finds no slot free is refused at once with `LimitExceededException`, /// so no backup runs while snapshots hold every slot. With one slot, one held -/// snapshot refuses every `CreateBackup` for as long as it is open. Running -/// backups hold one connection each for their read snapshot plus one shared -/// writer (`n` backups hold `n + 1`), so from `pool_size` 4 up a flood of -/// backups leaves at least half the pool to other requests; with 2 or 3 the -/// one admitted backup holds two of them. +/// snapshot refuses every `CreateBackup` for as long as it is open. A server +/// with `backup.max_concurrent_jobs` over the slot count raises a file-backed +/// `pool_size` to `2 * (max_concurrent_jobs + 1)` (10 at the default of 4, +/// which is the default pool; 34 at most, where the 16-slot cap is reached), +/// logging a warning that names both numbers, so the slot never refuses a job +/// the engine's permit admitted (`backup::pool_size_for_jobs` computes it); +/// an in-memory database has one connection and one slot whatever the jobs. +/// Running backups hold one connection each for their read snapshot plus one +/// shared writer (`n` backups hold `n + 1`), so from `pool_size` 4 up a flood +/// of backups leaves at least half the pool to other requests; with 2 or 3 +/// the one admitted backup holds two of them. #[derive(Debug, Clone, Deserialize)] #[serde(deny_unknown_fields)] pub struct SqliteConfig { diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs index 7c2188096..c7ec54c23 100644 --- a/crates/storage-sqlite/src/lib.rs +++ b/crates/storage-sqlite/src/lib.rs @@ -254,13 +254,18 @@ fn sqlite_server_components_factory( .await .map_err(BackendError::Catalog)?; } - let mut engine = - SqliteEngine::new(&db_path, pool_size, ®ion, options.max_item_size_bytes) - .await - .map_err(|e| BackendError::ConnectionFailed { - backend: "sqlite".to_owned(), - details: e.to_string(), - })?; + let mut engine = SqliteEngine::new_for_server( + &db_path, + pool_size, + options.max_concurrent_jobs, + ®ion, + options.max_item_size_bytes, + ) + .await + .map_err(|e| BackendError::ConnectionFailed { + backend: "sqlite".to_owned(), + details: e.to_string(), + })?; if options.beside_server { return components_of(engine, &db_path).await; } diff --git a/crates/storage-sqlite/src/snapshot_engine.rs b/crates/storage-sqlite/src/snapshot_engine.rs index e79fc6da0..4b719b8fb 100644 --- a/crates/storage-sqlite/src/snapshot_engine.rs +++ b/crates/storage-sqlite/src/snapshot_engine.rs @@ -55,12 +55,23 @@ //! LIMIT n`), which `SQLite` serves from the primary key index. Multi-part //! range keys fall back to rowid order and report `key_ordered = false`. //! -//! Memory: one page is `LIMIT paging::DEFAULT_FETCH_BATCH` (1000) rows, all -//! of them held as item text at once. At the default 400 KB item limit one -//! row's text can reach 27.7 MB (the measured worst case in #394's line-cap -//! table), so a page is bounded by rows, not bytes. A byte budget (keyset -//! paging with a byte cutoff) is required before the stream gets a caller; -//! the backup design assigns it to the backup job PR. +//! Memory: a page ends at `paging::fetch_batch()` rows (1000) or after the +//! row that takes its item text to or past `paging::fetch_bytes()` (64 MiB, +//! the budget `extenddb_storage::snapshot::paging` states), whichever comes +//! first, so a page holds at most the budget plus one row of item text. The +//! cut is applied while the page's rows are pulled from the query one at a +//! time (`fetch_page`): the pull stops at the cut and the statement is +//! reset, so the stream reads no row past it and the next page starts after +//! the last row pulled. The driver's worker thread runs the statement ahead +//! of the pull and buffers the rows it has read: that run-ahead is bounded by +//! the pool's `row_buffer_size` (`store.rs`), not by the budget, at five +//! rows (the four the channel holds and the one the worker holds while its +//! `send` waits), 160 MiB at the 32 MiB line cap, and is the one term of a +//! page's memory the budget does not state. A page cut by the budget is short +//! without being the end of the table; the end is a page under the row bound +//! that the budget did not cut. The in-memory copy is a different bound: it +//! holds the whole table decoded, and the slot count, not the budget, bounds +//! how many copies exist at once. use extenddb_core::types::{Item, ScalarAttributeType, TableDescription}; use extenddb_storage::error::StorageError; @@ -266,9 +277,10 @@ fn keyset_stream( buffer: Vec::new().into_iter(), done: false, }; - // Read the page size once so a change to the test knob during a stream - // cannot end it early or late. + // Read the page size and the byte budget once so a change to either + // test knob during a stream cannot end it early or late. let batch = paging::fetch_batch(); + let budget = paging::fetch_bytes(); let limit = i64::try_from(batch).unwrap_or(i64::MAX); Box::pin(futures::stream::unfold( state, @@ -282,8 +294,16 @@ fn keyset_stream( } let tx = state.tx.as_mut()?; paging::record_fetch(); - let page = fetch_page(tx, &state.table, &state.paging, &state.cursor, limit).await; - let (items, next_cursor) = match page { + let page = fetch_page( + tx, + &state.table, + &state.paging, + &state.cursor, + limit, + budget, + ) + .await; + let Page { items, next, cut } = match page { Ok(page) => page, Err(e) => { state.done = true; @@ -292,10 +312,13 @@ fn keyset_stream( return Some((Err(e), state)); } }; - if let Some(next) = next_cursor { + if let Some(next) = next { state.cursor = next; } - if items.len() < batch { + // A short page shows the end only when the budget did not + // cut it: a cut page is short because the pull stopped, not + // because the table did. + if items.len() < batch && !cut { state.done = true; if let Some(tx) = state.tx.take() { let _ = tx.commit().await; @@ -308,127 +331,118 @@ fn keyset_stream( )) } -/// Query one page of up to `limit` rows for the paging mode and cursor. -async fn query_page( - tx: &mut Transaction<'static, Sqlite>, - table: &str, - paging: &Paging, - cursor: &Cursor, - limit: i64, -) -> Result, StorageError> { - match (paging, cursor) { +/// The SQL of one page for the paging mode and cursor: the key columns and +/// `item_data`, in key order, after the cursor when there is one, `LIMIT ?` +/// last. +fn page_sql(table: &str, paging: &Paging, cursor: &Cursor) -> Result { + Ok(match (paging, cursor) { (Paging::Pk, Cursor::Start) => { - sqlx::query(&format!( - "SELECT pk, item_data FROM {table} ORDER BY pk LIMIT ?" - )) - .bind(limit) - .fetch_all(&mut **tx) - .await + format!("SELECT pk, item_data FROM {table} ORDER BY pk LIMIT ?") } - (Paging::Pk, Cursor::Pk(last)) => { - sqlx::query(&format!( - "SELECT pk, item_data FROM {table} WHERE pk > ? ORDER BY pk LIMIT ?" - )) - .bind(last) - .bind(limit) - .fetch_all(&mut **tx) - .await + (Paging::Pk, Cursor::Pk(_)) => { + format!("SELECT pk, item_data FROM {table} WHERE pk > ? ORDER BY pk LIMIT ?") } (Paging::PkSk { sk_col, .. }, Cursor::Start) => { - sqlx::query(&format!( - "SELECT pk, {sk_col}, item_data FROM {table} ORDER BY pk, {sk_col} LIMIT ?" - )) - .bind(limit) - .fetch_all(&mut **tx) - .await - } - (Paging::PkSk { sk_col, .. }, Cursor::PkSkText(pk, sk)) => { - sqlx::query(&format!( - "SELECT pk, {sk_col}, item_data FROM {table} \ - WHERE (pk, {sk_col}) > (?, ?) ORDER BY pk, {sk_col} LIMIT ?" - )) - .bind(pk) - .bind(sk) - .bind(limit) - .fetch_all(&mut **tx) - .await + format!("SELECT pk, {sk_col}, item_data FROM {table} ORDER BY pk, {sk_col} LIMIT ?") } - (Paging::PkSk { sk_col, .. }, Cursor::PkSkBlob(pk, sk)) => { - sqlx::query(&format!( + (Paging::PkSk { sk_col, .. }, Cursor::PkSkText(..) | Cursor::PkSkBlob(..)) => { + format!( "SELECT pk, {sk_col}, item_data FROM {table} \ WHERE (pk, {sk_col}) > (?, ?) ORDER BY pk, {sk_col} LIMIT ?" - )) - .bind(pk) - .bind(sk) - .bind(limit) - .fetch_all(&mut **tx) - .await + ) } (Paging::Rowid, Cursor::Start) => { - sqlx::query(&format!( - "SELECT rowid, item_data FROM {table} ORDER BY rowid LIMIT ?" - )) - .bind(limit) - .fetch_all(&mut **tx) - .await + format!("SELECT rowid, item_data FROM {table} ORDER BY rowid LIMIT ?") } - (Paging::Rowid, Cursor::Rowid(last)) => { - sqlx::query(&format!( - "SELECT rowid, item_data FROM {table} WHERE rowid > ? ORDER BY rowid LIMIT ?" - )) - .bind(last) - .bind(limit) - .fetch_all(&mut **tx) - .await + (Paging::Rowid, Cursor::Rowid(_)) => { + format!("SELECT rowid, item_data FROM {table} WHERE rowid > ? ORDER BY rowid LIMIT ?") } _ => { return Err(StorageError::Internal( "snapshot cursor does not match its paging mode".to_owned(), )); } - } - .map_err(|e| db_err(&e)) + }) +} + +/// The cursor a page's last row leaves for the next page. +fn cursor_after(row: &sqlx::sqlite::SqliteRow, paging: &Paging) -> Result { + Ok(match paging { + Paging::Pk => Cursor::Pk(row.try_get("pk").map_err(|e| db_err(&e))?), + Paging::PkSk { sk_col, sk_type } => { + let pk: String = row.try_get("pk").map_err(|e| db_err(&e))?; + match sk_type { + ScalarAttributeType::B => Cursor::PkSkBlob( + pk, + row.try_get::, _>(*sk_col).map_err(|e| db_err(&e))?, + ), + ScalarAttributeType::S | ScalarAttributeType::N => Cursor::PkSkText( + pk, + row.try_get::(*sk_col).map_err(|e| db_err(&e))?, + ), + } + } + Paging::Rowid => Cursor::Rowid(row.try_get("rowid").map_err(|e| db_err(&e))?), + }) } -/// One page: items and the cursor for the next page. Fewer items than -/// `limit` means the table is exhausted. +/// One page as [`fetch_page`] returns it. +struct Page { + items: Vec, + /// The cursor for the next page; `None` for an empty page. + next: Option, + /// Whether the byte budget ended the page before `limit` rows. + cut: bool, +} + +/// One page: at most `limit` rows, pulled one at a time from the page query +/// and stopped after the row that takes the page's item text to or past +/// `budget` bytes, so a page holds at most the budget plus one row and at +/// least one row, and the stream pulls no row past the cut (the statement is +/// reset when the row stream is dropped, with the rows the driver's worker +/// read ahead, at most the pool's `row_buffer_size` plus the one the worker +/// holds, five; the next page starts at the cursor the last pulled row +/// leaves). Fewer rows than `limit` +/// without a cut means the table is exhausted. async fn fetch_page( tx: &mut Transaction<'static, Sqlite>, table: &str, paging: &Paging, cursor: &Cursor, limit: i64, -) -> Result<(Vec, Option), StorageError> { - let rows = query_page(tx, table, paging, cursor, limit).await?; - let mut items = Vec::with_capacity(rows.len()); + budget: usize, +) -> Result { + use futures::TryStreamExt as _; + let sql = page_sql(table, paging, cursor)?; + let query = sqlx::query(&sql); + let query = match cursor { + Cursor::Start => query, + Cursor::Pk(pk) => query.bind(pk), + Cursor::PkSkText(pk, sk) => query.bind(pk).bind(sk), + Cursor::PkSkBlob(pk, sk) => query.bind(pk).bind(sk.as_slice()), + Cursor::Rowid(rowid) => query.bind(rowid), + } + .bind(limit); + let mut rows = query.fetch(&mut **tx); + let mut items = Vec::new(); + let mut bytes = 0usize; let mut next = None; - for row in &rows { + let mut cut = false; + while let Some(row) = rows.try_next().await.map_err(|e| db_err(&e))? { let data: String = row.try_get("item_data").map_err(|e| db_err(&e))?; + bytes += data.len(); let item: Item = serde_json::from_str(&data) .map_err(|e| StorageError::Internal(format!("Parse item: {e}")))?; items.push(item); + next = Some(cursor_after(&row, paging)?); + if bytes >= budget { + cut = true; + break; + } } - if let Some(last) = rows.last() { - next = Some(match paging { - Paging::Pk => Cursor::Pk(last.try_get("pk").map_err(|e| db_err(&e))?), - Paging::PkSk { sk_col, sk_type } => { - let pk: String = last.try_get("pk").map_err(|e| db_err(&e))?; - match sk_type { - ScalarAttributeType::B => Cursor::PkSkBlob( - pk, - last.try_get::, _>(*sk_col) - .map_err(|e| db_err(&e))?, - ), - ScalarAttributeType::S | ScalarAttributeType::N => Cursor::PkSkText( - pk, - last.try_get::(*sk_col).map_err(|e| db_err(&e))?, - ), - } - } - Paging::Rowid => Cursor::Rowid(last.try_get("rowid").map_err(|e| db_err(&e))?), - }); - } - Ok((items, next)) + drop(rows); + paging::record_page_bytes(bytes); + Ok(Page { items, next, cut }) } #[cfg(test)] @@ -550,6 +564,33 @@ mod tests { }, ) .await; + // The byte budget: pages are cut after the row that reaches it, every + // row streams once across cut pages, and the rows past a cut are + // never read (the pull stops at the cut), which `rows_past_cut` + // reports as zero here. + let bounds = snapshot_conformance::run_page_bounds( + &hooks, + &*engine, + ACCOUNT, + "snap_bounds", + || async { + engine + .process_control_plane_transitions() + .await + .expect("pump"); + }, + ) + .await; + assert_eq!( + bounds.rows_past_cut, 0, + "SQLite pulls rows one at a time and stops at the cut: {bounds:?}" + ); + assert_eq!( + bounds.pages, + (snapshot_conformance::PAGE_BOUNDS_ITEMS as u64) + .div_ceil(snapshot_conformance::PAGE_BOUNDS_ROWS as u64), + "a page is exactly the budget's rows: {bounds:?}" + ); let pool = engine.pool.clone(); snapshot_conformance::run_refusals( &hooks, diff --git a/crates/storage-sqlite/src/store.rs b/crates/storage-sqlite/src/store.rs index 8202203fc..a8f7c8de0 100644 --- a/crates/storage-sqlite/src/store.rs +++ b/crates/storage-sqlite/src/store.rs @@ -38,17 +38,24 @@ //! so the invariant stays auditable, is init-time bootstrap (in this file and //! in `bootstrapper.rs`), which runs before the server serves traffic. +use std::str::FromStr; use std::sync::Arc; use std::sync::atomic::AtomicU64; use extenddb_storage::error::StorageError; use sqlx::SqlitePool; -use sqlx::sqlite::SqlitePoolOptions; +use sqlx::sqlite::{SqliteConnectOptions, SqlitePoolOptions}; use tokio::sync::Mutex; use crate::INDEX_PROPAGATION_DELAY_QUERY; use crate::sqlite_util::sqlite_url; +/// Rows the driver's worker buffers ahead of the reading task (see +/// `SqliteEngine::new_for_server`). The rows ahead of the reader are one +/// more than this: the channel holds this many and the worker, blocked on +/// its `send`, holds the next row it has already stepped and copied. +const ROW_BUFFER_ROWS: usize = 4; + /// The SQLite storage engine. Implements `StorageEngine` (all data/metadata/ /// stream/worker/backup traits) over a single SQLite database. #[derive(Clone)] @@ -91,15 +98,50 @@ pub struct SqliteEngine { pub(crate) job_slots: Arc, } +/// The connection options of every pool connection. The driver runs each +/// statement on a worker thread that buffers rows ahead of the task reading +/// them (sqlx's `row_buffer_size`, 50 by default) over a bounded channel; +/// the worker steps and copies the next row before its `send` blocks, so +/// the rows ahead of the reader are the buffer plus one. A table snapshot +/// reads rows of up to the 32 MiB line cap and stops a page at the byte +/// budget, so a 50-row buffer is up to 1.6 GB the page never asked for; +/// [`ROW_BUFFER_ROWS`] bounds the run-ahead at five rows (160 MiB at the +/// line cap), which costs ordinary queries a channel hand-off every four +/// rows. The test at the end of this module counts the rows stepped ahead. +pub(crate) fn connect_options(url: &str) -> Result { + Ok(SqliteConnectOptions::from_str(url) + .map_err(|e| StorageError::Connection(e.to_string()))? + .row_buffer_size(ROW_BUFFER_ROWS)) +} + impl SqliteEngine { /// Open (creating if necessary) the SQLite database and build the engine. /// /// `path_or_url` accepts a filesystem path, `:memory:`, or a `sqlite:` URL. + /// The pool is sized for no backup jobs; a server opens the engine with + /// [`Self::new_for_server`]. pub async fn new( path_or_url: &str, pool_size: u32, region: &str, max_item_size_bytes: usize, + ) -> Result { + Self::new_for_server(path_or_url, pool_size, 0, region, max_item_size_bytes).await + } + + /// [`Self::new`] for a server that runs up to `max_concurrent_jobs` + /// backup and restore jobs at once: a file-backed pool is raised, with a + /// warning naming both numbers, until its server slots admit every job + /// the engine's permit admits (`backup::pool_size_for_jobs`; the rule is + /// stated on `SqliteConfig::pool_size`), so no job the permit admitted + /// is refused by the slot. The in-memory pool has one connection by + /// nature and is not raised. + pub async fn new_for_server( + path_or_url: &str, + pool_size: u32, + max_concurrent_jobs: usize, + region: &str, + max_item_size_bytes: usize, ) -> Result { let url = sqlite_url(path_or_url); // Use the same parsed-location classifier as the serve lock. String @@ -107,6 +149,18 @@ impl SqliteEngine { let in_memory = crate::serve_lock::database_file(path_or_url) .map_err(StorageError::Connection)? .is_none(); + let for_jobs = if in_memory { + pool_size + } else { + crate::backup::pool_size_for_jobs(pool_size, max_concurrent_jobs) + }; + if for_jobs != pool_size { + tracing::warn!( + "{}", + crate::backup::pool_raise_message(pool_size, for_jobs, max_concurrent_jobs) + ); + } + let pool_size = for_jobs; // An in-memory database lives only inside its own connection: a second // connection opens a *separate* empty database. So for an in-memory // location we pin the pool to a single connection that is never @@ -129,13 +183,14 @@ impl SqliteEngine { if in_memory { opts = opts.idle_timeout(None).max_lifetime(None); } + let connect = connect_options(&url)?; let pool = opts .after_connect(|conn, _| { Box::pin( async move { crate::sqlite_util::set_connection_pragmas(conn, true).await }, ) }) - .connect(&url) + .connect_with(connect) .await .map_err(|e| StorageError::Connection(e.to_string()))?; @@ -800,3 +855,65 @@ mod d1_write_lock_tests { .await; } } + +#[cfg(test)] +mod row_buffer_tests { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + + use futures::TryStreamExt; + + /// The rows the driver's worker steps ahead of a reader that pulls one + /// row and stops, on a connection with the engine's options: a collation + /// the query's `WHERE` applies to every stepped row counts the steps, a + /// table of large rows (1 MiB each, 40 of them) is read through a plain + /// scan, the reader takes one row and drops the stream, and the steps + /// beyond that row are the buffer plus the one the worker holds, five. + /// With sqlx's default buffer of 50 the count is 51, 1.6 GB of lines at + /// the cap a page never asked for. + #[tokio::test] + async fn the_driver_steps_five_rows_ahead_of_the_pages_reader() { + let steps = Arc::new(AtomicUsize::new(0)); + let counting = Arc::clone(&steps); + let options = super::connect_options("sqlite::memory:") + .expect("options") + .collation("counting", move |a: &str, b: &str| { + counting.fetch_add(1, Ordering::SeqCst); + a.cmp(b) + }); + let mut conn = sqlx::ConnectOptions::connect(&options) + .await + .expect("connect"); + sqlx::query("CREATE TABLE big (pk TEXT COLLATE counting, item_data TEXT)") + .execute(&mut conn) + .await + .expect("create"); + let payload = "x".repeat(1024 * 1024); + for i in 0..40 { + sqlx::query("INSERT INTO big (pk, item_data) VALUES (?, ?)") + .bind(format!("k{i:03}")) + .bind(&payload) + .execute(&mut conn) + .await + .expect("insert"); + } + steps.store(0, Ordering::SeqCst); + // The scan evaluates the comparison on each row it steps, so the + // collation is called once per step; no index, no sort, so no + // comparison runs ahead of the stepping. + let mut rows = sqlx::query("SELECT item_data FROM big WHERE pk <> 'none'").fetch(&mut conn); + let first = rows.try_next().await.expect("fetch").expect("a row"); + drop(first); + // The worker runs on until its channel is full and its next send + // blocks; give it the time to get there. + tokio::time::sleep(std::time::Duration::from_millis(300)).await; + let stepped = steps.load(Ordering::SeqCst); + drop(rows); + assert_eq!( + stepped - 1, + super::ROW_BUFFER_ROWS + 1, + "{stepped} rows stepped for one pulled: the buffer plus the row the worker holds" + ); + assert_eq!(stepped - 1, 5); + } +} diff --git a/crates/storage-sqlite/tests/backup_job_panic.rs b/crates/storage-sqlite/tests/backup_job_panic.rs new file mode 100644 index 000000000..d366de136 --- /dev/null +++ b/crates/storage-sqlite/tests/backup_job_panic.rs @@ -0,0 +1,338 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! A CreateBackup job whose task panics on SQLite leaves a CREATING row that +//! the reconciler's next cycle resolves to FAILED, in the same process, with +//! no restart (design 4(e): at runtime a row whose in-process slot no task +//! holds is dead; PR 8's acceptance). +//! +//! The engine is built the way the server builds it (the backend's server +//! components factory, its workers spawned through the runtime hooks, so a +//! table reaches ACTIVE as it does under `serve`), the store is a filesystem +//! store whose `put` panics, and the request goes through the engine's +//! `dispatch`. The job's task panics at its first put; the task's drop +//! releases the `Arc` and with it the row's `JobSlot`, so the +//! SQLite catalog's `claim_dead_backup` takes the row over at once, whatever +//! its heartbeat says, and the reconciler writes FAILED (no manifest) and +//! then, since the dead owner's heartbeat is within the window, leaves the +//! prefix for a later cycle. + +use std::os::unix::fs::DirBuilderExt; +use std::path::PathBuf; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::time::Duration; + +use extenddb_backup_store::{BackupStore, ByteStream, FilesystemStore, ObjectMeta, StoreError}; +use extenddb_core::backup_format::BackupArn; +use extenddb_core::limits::LimitsConfig; +use extenddb_engine::reconciler::{self, OWNER_DEAD_NO_MANIFEST, ReconcilerState}; +use extenddb_engine::{ + BackupRuntime, ConfiguredStore, OperationContext, check_configured_store, dispatch, +}; +use extenddb_storage::backup_catalog::{BackupStatus, StoreKind}; +use extenddb_storage::server_components::ServerComponentsOptions; +use extenddb_storage::{CancellationToken, WorkerContext}; +use extenddb_storage_sqlite::SqliteConfig; +use futures::future::BoxFuture; +use futures::stream::BoxStream; +use serde_json::json; + +const REGION: &str = "us-east-1"; + +/// A filesystem store whose `put` panics: every other call reaches the real +/// store, so the identity check, the listing and the delete behave as on a +/// server; only the job's first object write dies. +struct PanicOnPut { + inner: FilesystemStore, + puts: AtomicUsize, +} + +impl BackupStore for PanicOnPut { + fn put<'a>( + &'a self, + key: &'a str, + _body: ByteStream, + ) -> BoxFuture<'a, Result> { + self.puts.fetch_add(1, Ordering::SeqCst); + Box::pin(async move { panic!("injected panic in put of {key}") }) + } + + fn put_if_absent<'a>( + &'a self, + key: &'a str, + body: ByteStream, + ) -> BoxFuture<'a, Result, StoreError>> { + self.inner.put_if_absent(key, body) + } + + fn get<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result> { + self.inner.get(key) + } + + fn head<'a>(&'a self, key: &'a str) -> BoxFuture<'a, Result, StoreError>> { + self.inner.head(key) + } + + fn list<'a>(&'a self, prefix: &'a str) -> BoxStream<'a, Result> { + self.inner.list(prefix) + } + + fn delete_prefix<'a>(&'a self, prefix: &'a str) -> BoxFuture<'a, Result> { + self.inner.delete_prefix(prefix) + } + + fn validate(&self) -> BoxFuture<'_, Result<(), StoreError>> { + self.inner.validate() + } +} + +/// A scratch directory under the cargo target directory, removed on drop. +struct Scratch(PathBuf); + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +async fn call( + ctx: &OperationContext, + operation: &str, + body: serde_json::Value, +) -> serde_json::Value { + dispatch(operation, body, ctx, "127.0.0.1:0") + .await + .unwrap_or_else(|e| panic!("{operation}: {e:?}")) + .body +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_job_task_that_panics_leaves_a_row_the_next_cycle_resolves_without_a_restart() { + let scratch = Scratch( + PathBuf::from(env!("CARGO_TARGET_TMPDIR")) + .join(format!("backup-job-panic-{}", uuid::Uuid::new_v4())), + ); + std::fs::create_dir_all(&scratch.0).expect("scratch directory"); + let db_path = scratch.0.join("catalog.sqlite"); + let root = scratch.0.join("backups"); + std::fs::DirBuilder::new() + .mode(0o700) + .create(&root) + .expect("root"); + + // The engine as the server builds it: a dev bootstrap of a fresh file + // (the migrations, the default account), the workers spawned. + let config = SqliteConfig { + path: db_path.to_str().expect("UTF-8 path").to_owned(), + pool_size: 4, + }; + let components = (extenddb_storage_sqlite::backend().server_components)(&config, REGION, { + let mut options = ServerComponentsOptions::default(); + options.bootstrap_if_uninitialized = true; + options.release = "test"; + options.max_concurrent_jobs = 4; + options + }) + .await + .unwrap_or_else(|e| panic!("server components: {e:?}")); + let storage = components.engine; + let shutdown = CancellationToken::new(); + let (_layer, reload_handle) = + tracing_subscriber::reload::Layer::new(tracing_subscriber::EnvFilter::new("info")); + let worker_ctx = WorkerContext { + metrics: Arc::new(extenddb_core::metrics::MetricsCollector::new()), + catalog_store: components.catalog_store.clone(), + reload_handle, + config_log_level: "info".to_owned(), + shutdown: shutdown.clone(), + }; + let hooks = components + .runtime_hooks + .expect("the SQLite backend has runtime hooks"); + let workers = hooks.spawn_workers(&worker_ctx).await; + let account_id: String = { + let pool = sqlx::SqlitePool::connect(&format!("sqlite://{}", db_path.display())) + .await + .expect("a second connection to the catalog"); + sqlx::query("INSERT OR REPLACE INTO settings (key, value) VALUES ('control_plane_delay_seconds', '0.05')") + .execute(&pool) + .await + .expect("setting"); + let id = sqlx::query_scalar("SELECT value FROM settings WHERE key = 'default_account_id'") + .fetch_one(&pool) + .await + .expect("the bootstrapped account"); + pool.close().await; + id + }; + + // The store: a healthy filesystem store (its identity file created by + // the check the server runs at startup) whose `put` panics. + let store = Arc::new(PanicOnPut { + inner: FilesystemStore::open(&root) + .await + .expect("filesystem store"), + puts: AtomicUsize::new(0), + }); + let configured = ConfiguredStore::checking( + StoreKind::Filesystem, + root.to_str().expect("UTF-8 path").to_owned(), + store.clone(), + None, + ); + let outcome = check_configured_store(&configured, &*storage, "sqlite").await; + assert!(outcome.is_ok(), "the store's first check: {outcome:?}"); + configured.record_check(outcome); + let stale_job_after = Duration::from_secs(300); + let runtime = Arc::new( + BackupRuntime::new(4) + .with_stale_job_after(stale_job_after) + .with_store(configured) + .with_source("sqlite", "test") + .with_shutdown(shutdown.child_token()), + ); + let ctx = OperationContext { + storage: storage.clone(), + backup: runtime.clone(), + limits: Arc::new(LimitsConfig::default()), + region: Arc::from(REGION), + account_id: Arc::from(account_id.as_str()), + import_paths: Arc::from(Vec::new()), + export_paths: Arc::from(Vec::new()), + authorized_key_info: Default::default(), + auth_cache: extenddb_auth::AuthCacheRegistry::empty(), + table_key_info_lookup: None, + }; + + call( + &ctx, + "CreateTable", + json!({ + "TableName": "Music", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], + "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "BillingMode": "PAY_PER_REQUEST" + }), + ) + .await; + let deadline = std::time::Instant::now() + Duration::from_secs(30); + loop { + let desc = call(&ctx, "DescribeTable", json!({"TableName": "Music"})).await; + if desc["Table"]["TableStatus"] == "ACTIVE" { + break; + } + assert!( + std::time::Instant::now() < deadline, + "Music did not become ACTIVE: {desc}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + call( + &ctx, + "PutItem", + json!({"TableName": "Music", "Item": {"pk": {"S": "a"}, "n": {"N": "1"}}}), + ) + .await; + + // The request answers CREATING; the job's task then panics at its first + // put and ends. + let details = call( + &ctx, + "CreateBackup", + json!({"TableName": "Music", "BackupName": "nightly"}), + ) + .await; + assert_eq!( + details["BackupDetails"]["BackupStatus"], "CREATING", + "{details}" + ); + let arn_text = details["BackupDetails"]["BackupArn"] + .as_str() + .expect("an ARN") + .to_owned(); + let arn = BackupArn::parse(&arn_text).expect("a backup ARN"); + let deadline = std::time::Instant::now() + Duration::from_secs(30); + while runtime.running_jobs() > 0 { + assert!( + std::time::Instant::now() < deadline, + "the job task did not end" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + assert_eq!( + store.puts.load(Ordering::SeqCst), + 1, + "the task died at its first put" + ); + let row = storage + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + assert_eq!( + row.status, + BackupStatus::Creating, + "the panicking task wrote nothing" + ); + let heartbeat_before = row.heartbeat_at; + assert_eq!( + runtime.free_slots(), + 4, + "the task's drop returned the job permit" + ); + + // One reconciler cycle in the same process: the slot no task holds makes + // the row dead at once, and without a manifest it is FAILED. + let mut state = ReconcilerState::with_shutdown(shutdown.child_token()); + let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a reconciler cycle"); + assert_eq!(report.store_healthy, Some(true), "{report:?}"); + assert_eq!( + report.creating_resolved, + vec![(arn_text.clone(), BackupStatus::Failed)], + "{report:?}" + ); + let row = storage + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + assert_eq!(row.status, BackupStatus::Failed); + assert_eq!(row.failure_reason.as_deref(), Some(OWNER_DEAD_NO_MANIFEST)); + assert!( + row.completed_at.is_some(), + "the FAILED write stamps completed_at" + ); + assert_eq!( + row.heartbeat_at, heartbeat_before, + "the takeover left the heartbeat" + ); + // The dead owner's heartbeat is within the window, so the prefix (empty + // here: the put never landed) is the FAILED-prefix duty's once it is + // stale, not this cycle's. + assert!(report.failed_prefixes_deleted.is_empty(), "{report:?}"); + let described = dispatch( + "DescribeBackup", + json!({"BackupArn": arn_text}), + &ctx, + "127.0.0.1:0", + ) + .await; + match described { + Err(extenddb_core::error::DynamoDbError::BackupNotFoundException(_)) => {} + Err(other) => panic!("DescribeBackup of the FAILED row: {other:?}"), + Ok(answer) => panic!("DescribeBackup of the FAILED row answered {}", answer.body), + } + // A second cycle has nothing left to resolve. + let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a second cycle"); + assert!(report.creating_resolved.is_empty(), "{report:?}"); + + shutdown.cancel(); + for worker in workers { + let _ = tokio::time::timeout(Duration::from_secs(5), worker).await; + } +} diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs index cd81e4e39..96766a6de 100644 --- a/crates/storage/src/backup_catalog.rs +++ b/crates/storage/src/backup_catalog.rs @@ -539,6 +539,10 @@ pub struct StatusFields { pub size_bytes: Option, pub item_count: Option, pub failure_reason: Option, + /// The manifest's `source_table` object, which the AVAILABLE write of a + /// backup row carries so DescribeBackup reports the counts the manifest + /// has; a restore job record has no such column and ignores it. + pub source_table_details_json: Option, pub completed: bool, } @@ -810,10 +814,34 @@ pub trait BackupCatalog: Send + Sync { stale_after: Duration, ) -> BoxFuture<'a, Result>; + /// Take over a CREATING row whose owner is dead, under a fresh + /// generation, and return the claim; the reconciler's way onto a row a + /// job left behind. Dead is `heartbeat_at` older than `stale_after` on + /// the database's clock (on PostgreSQL, once the owner's claim session + /// has ended as well: the advisory lock is tried first, so a stopped + /// owner whose session lives keeps its row), except on SQLite, where a + /// CREATING row whose in-process slot no task holds is dead at once (a + /// restart frees every slot, a job that panics drops its slot) and + /// `stale_after` is not read. + /// `heartbeat_at` is left as the dead owner wrote it, so the stale + /// window of the FAILED-prefix duty counts from the owner's last beat + /// and not from the takeover. `None` when the row is not CREATING, is + /// gone, has a live owner, or is already claimed by another task; the + /// old generation's writes through its [`JobClaim`] find + /// [`WriteOutcome::NotMatched`] from then on. + fn claim_dead_backup<'a>( + &'a self, + key: &'a BackupKey, + stale_after: Duration, + ) -> BoxFuture<'a, Result>, StorageError>>; + /// The rows of `store` with `status` whose `clock` column is older than /// `age` on the database's clock: the maintenance command's test of a /// live owner (a CREATING or FAILED row it does not list under the - /// window on [`StaleClock::Heartbeat`] has one). The comparison is the + /// window on [`StaleClock::Heartbeat`] has one), and the reconciler's + /// listing for the FAILED-prefix delay (`Failed`, + /// [`StaleClock::Heartbeat`]) and the tombstone retention (`Failed` or + /// `Deleted`, [`StaleClock::Completed`]). The comparison is the /// backend's, in its own query, so the engine never compares a row's /// timestamp with a clock of its own. fn list_backup_rows_older_than<'a>( @@ -824,6 +852,17 @@ pub trait BackupCatalog: Send + Sync { age: Duration, ) -> BoxStream<'a, Result>; + /// Remove the restore registrations a backend keeps beside its rows that + /// name a target no longer a CREATING restore target of that backup, and + /// return the ARNs of the backups touched. Only Cassandra keeps such a + /// map (`active_restores`, maintained by lightweight transactions whose + /// removal step can be lost with its owner); every other backend derives + /// the in-use check from the target rows themselves and has nothing to + /// sweep, which is what the default body says. + fn sweep_restore_registrations(&self) -> BoxFuture<'_, Result, StorageError>> { + Box::pin(async { Ok(Vec::new()) }) + } + /// The name of a CREATING restore target registered from the backup, if /// any: the rule under which DeleteBackup and a second restore of the /// backup are refused with `BackupInUseException`. Reads the target's @@ -963,6 +1002,34 @@ pub trait JobClaim: Send + Sync { /// is still held. The caller treats an `Err` as [`Heartbeat::Lost`]. fn heartbeat(&self) -> BoxFuture<'_, Result>; + /// [`Self::heartbeat`] under `bound`: `None` when the beat did not + /// return within it, which the caller treats as a lost claim. The default + /// bounds the whole call. A backend whose beat queues behind a lock of + /// its own process before it reaches the catalog (SQLite's engine write + /// lock, held by a GSI backfill or a large write for as long as it + /// takes) measures the bound from the moment the beat holds that lock, + /// so a writer holding it past a period delays the beat and does not end + /// the job: the queue is this process's, not a sign the claim is lost. + fn heartbeat_within( + &self, + bound: Duration, + ) -> BoxFuture<'_, Option>> { + Box::pin(async move { tokio::time::timeout(bound, self.heartbeat()).await.ok() }) + } + + /// Whether [`Self::heartbeat_within`] can wait outside its bound before + /// the bound starts (SQLite's beat queues behind the engine write lock + /// of its own process, so the wait is this process's and never a sign of + /// a takeover). The engine's pre-put check treats a beat in flight as + /// confirming the claim for as long as it waits where this is `true`, + /// and only within one bound of the beat's start where it is `false` + /// (the default): a process stopped mid-beat and resumed would otherwise + /// pass the check until its timer raised the loss, while another owner + /// may have taken the row over during the stop. + fn heartbeat_parks_before_its_bound(&self) -> bool { + false + } + /// Move the record from `from` to `to`, conditioned on the claim's /// generation and on the record being in `from`. fn set_status<'a>( diff --git a/crates/storage/src/snapshot.rs b/crates/storage/src/snapshot.rs index 10f14d790..0f692e5f9 100644 --- a/crates/storage/src/snapshot.rs +++ b/crates/storage/src/snapshot.rs @@ -65,8 +65,12 @@ //! of each backend's `snapshot_engine.rs` are the record of what the //! snapshots do. -use extenddb_core::backup_format::KeyOrder; -use extenddb_core::types::{IndexStatus, Item, TableDescription}; +use extenddb_core::backup_format::{KeyOrder, SourceTableSchema}; +use extenddb_core::types::{ + BillingMode, GsiInput, IndexStatus, Item, LsiInput, ProvisionedThroughput, + ProvisionedThroughputDescription, TableDescription, TimeToLiveDescription, + VectorIndexSpecification, +}; use futures::future::BoxFuture; use futures::stream::BoxStream; @@ -89,26 +93,37 @@ pub use extenddb_core::backup_format::SnapshotDescriptor as SnapshotInfo; /// so the check can set it and observe its effect without a per-backend /// setter that a test could forget to call. pub mod paging { - /// Rows or documents a backend fetches per round trip. Measured on + /// Rows or documents a backend fetches per round trip, at most. Measured on /// localhost `PostgreSQL` 16 with 318-byte items: 1000 and 10000 read a /// million rows in the same time; 1000 holds less memory. Over a 1 ms /// network link the larger fetch would save about a second per million /// rows, which is why the size is a constant and not a configuration knob. - /// - /// The bound is rows, not bytes. `PostgreSQL`'s `FETCH` and `SQLite`'s - /// `LIMIT` each hold up to this many rows of item text in memory at - /// once, and at the default 400 KB item limit one row's text can reach - /// 27.7 MB (the measured worst case in #394's line-cap table), so one - /// fetch is bounded by rows, not bytes. A byte budget (keyset paging with a byte cutoff) is - /// required before the stream gets a caller; the backup design assigns - /// it to the backup job PR. `MongoDB`'s batches are bounded by the - /// server's 16 MiB reply limit as well as by this count. pub const DEFAULT_FETCH_BATCH: usize = 1000; + /// Bytes of item text a page holds, at most, before the page is cut: a + /// page ends at [`DEFAULT_FETCH_BATCH`] rows or after the row that takes + /// its running total of item text to or past this budget, whichever + /// comes first, so a page holds at most the budget plus one row and at + /// least one row. At the default 400 KB item limit one row's text can + /// reach 27.7 MB (the measured worst case in #394's line-cap table), so a + /// row bound alone would let a page of such rows hold gigabytes; the + /// budget is the term of the backup job's memory formula the stream + /// contributes. `PostgreSQL` applies the cut in its page query (a running + /// sum over the key order, so the server never returns the rows past the + /// cut), `SQLite` while it pulls the page's rows, and `MongoDB` is bounded + /// by the server's 16 MiB reply instead. + pub const SNAPSHOT_FETCH_BYTES: usize = 64 * 1024 * 1024; + #[cfg(feature = "test-hooks")] static OVERRIDE: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); #[cfg(feature = "test-hooks")] + static BYTES_OVERRIDE: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + #[cfg(feature = "test-hooks")] static FETCHES: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + #[cfg(feature = "test-hooks")] + static MAX_PAGE_BYTES: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + #[cfg(feature = "test-hooks")] + static ROWS_PAST_CUT: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); /// The page size a stream is built with. Backends read it once when they /// open a stream, so a change to the knob during a stream cannot end it @@ -128,12 +143,29 @@ pub mod paging { } } + /// The byte budget a stream is built with ([`SNAPSHOT_FETCH_BYTES`], or + /// the `test-hooks` override). Read once per open, like [`fetch_batch`]. + #[must_use] + pub fn fetch_bytes() -> usize { + #[cfg(feature = "test-hooks")] + { + match BYTES_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed) { + 0 => SNAPSHOT_FETCH_BYTES, + n => n, + } + } + #[cfg(not(feature = "test-hooks"))] + { + SNAPSHOT_FETCH_BYTES + } + } + /// Record one round trip to the backend for a page of items. A no-op /// without `test-hooks`. /// /// The counter is trusted: the conformance check reads it to confirm /// the backend paged at the knob's size, so a backend must call this - /// exactly once per round trip (one `FETCH`, one page query, one + /// exactly once per round trip (one page query, one /// `find` or `getMore`), never per item. A backend that ignored the /// knob and called it per item would pass the paging assertions with /// the whole table read in one trip. `MongoDB`'s driver hides `getMore` @@ -145,6 +177,27 @@ pub mod paging { FETCHES.fetch_add(1, std::sync::atomic::Ordering::Relaxed); } + /// Record the bytes of item text one page returned, so the conformance + /// check can read the largest page (`max_page_bytes`). Called once per + /// page by the backends that apply the byte cut; a no-op without + /// `test-hooks`. + pub fn record_page_bytes(bytes: usize) { + #[cfg(feature = "test-hooks")] + MAX_PAGE_BYTES.fetch_max(bytes, std::sync::atomic::Ordering::Relaxed); + #[cfg(not(feature = "test-hooks"))] + let _ = bytes; + } + + /// Record rows the database read for a page and dropped at the byte cut + /// (`PostgreSQL`'s outer `WHERE`; at most the page's row bound less one). + /// A no-op without `test-hooks`. + pub fn record_rows_past_cut(n: u64) { + #[cfg(feature = "test-hooks")] + ROWS_PAST_CUT.fetch_add(n, std::sync::atomic::Ordering::Relaxed); + #[cfg(not(feature = "test-hooks"))] + let _ = n; + } + /// Set the page size for every stream opened afterwards; `0` restores the /// default. #[cfg(feature = "test-hooks")] @@ -152,6 +205,21 @@ pub mod paging { OVERRIDE.store(rows, std::sync::atomic::Ordering::Relaxed); } + /// Set the byte budget for every stream opened afterwards; `0` restores + /// [`SNAPSHOT_FETCH_BYTES`]. + #[cfg(feature = "test-hooks")] + pub fn set_fetch_bytes(bytes: usize) { + BYTES_OVERRIDE.store(bytes, std::sync::atomic::Ordering::Relaxed); + } + + /// Zero [`max_page_bytes`] and [`rows_past_cut`], so a check reads only + /// the pages it drives afterwards. + #[cfg(feature = "test-hooks")] + pub fn reset_page_counters() { + MAX_PAGE_BYTES.store(0, std::sync::atomic::Ordering::Relaxed); + ROWS_PAST_CUT.store(0, std::sync::atomic::Ordering::Relaxed); + } + /// Round trips recorded by every stream in this process so far. #[cfg(feature = "test-hooks")] #[must_use] @@ -159,6 +227,22 @@ pub mod paging { FETCHES.load(std::sync::atomic::Ordering::Relaxed) } + /// The largest page (bytes of item text) recorded since the last + /// [`reset_page_counters`]. + #[cfg(feature = "test-hooks")] + #[must_use] + pub fn max_page_bytes() -> usize { + MAX_PAGE_BYTES.load(std::sync::atomic::Ordering::Relaxed) + } + + /// Rows read past a page's cut, summed since the last + /// [`reset_page_counters`]. + #[cfg(feature = "test-hooks")] + #[must_use] + pub fn rows_past_cut() -> u64 { + ROWS_PAST_CUT.load(std::sync::atomic::Ordering::Relaxed) + } + /// The two points in `open_table_snapshot` at which every backend calls /// [`open_pause`]. Under `test-hooks` the conformance check holds an open /// at one of them while it commits writes and an `UpdateTable`, to check @@ -177,9 +261,8 @@ pub mod paging { /// layout the pause is before the `LOCK TABLE`, in the window where a /// `DeleteTable` can still drop the data table. AfterDefinition, - /// The data snapshot is pinned; the item read (`PostgreSQL`'s `DECLARE - /// ... CURSOR`, `MongoDB`'s `find`, `SQLite`'s first page) has not - /// started. + /// The data snapshot is pinned; the item read (`PostgreSQL`'s and + /// `SQLite`'s first page, `MongoDB`'s `find`) has not started. BeforeItems, } @@ -252,6 +335,116 @@ pub fn key_schema_is_orderable(description: &TableDescription) -> bool { KeyOrder::resolve(&description.key_schema, &description.attribute_definitions).is_ok() } +/// The manifest's `source_table` object for a table, built from the +/// description a snapshot returned and the table's TTL description, which is +/// not part of a [`TableSnapshot`] and is read beside the open +/// (`MetadataEngine::describe_ttl`). One derivation for every backend, so the +/// row a CreateBackup inserts and the manifest its job writes carry the same +/// object; the legacy format's `BackupTableDefinition` is a different shape +/// and stays its own. +/// +/// `billing_mode` is the summary's, PROVISIONED when the description has no +/// summary; `provisioned_throughput` and each GSI's are written only for a +/// PROVISIONED table (an on-demand table's description echoes zeros, which +/// no CreateTable accepts). `vector_indexes` is `None` when the description +/// carries none (a backend without vector support) and the list of +/// specifications otherwise, empty included, so a reader can tell the two +/// apart. `item_count` and `table_size_bytes` are the description's, which +/// are the stats pass's figures: the job replaces `item_count` with the +/// stream's count before it writes the manifest. +/// +/// # Errors +/// +/// [`StorageError::Internal`] when the description's `TableClassSummary` has +/// no `TableClass` string: no in-tree backend writes such a summary, and +/// dropping the class silently would make a backup that restores as a +/// different table than it describes. +pub fn source_table_schema( + description: &TableDescription, + ttl: Option, +) -> Result { + let billing_mode = description + .billing_mode_summary + .as_ref() + .map_or(BillingMode::Provisioned, |s| s.billing_mode); + let provisioned = billing_mode == BillingMode::Provisioned; + let throughput = |d: &ProvisionedThroughputDescription| { + provisioned.then_some(ProvisionedThroughput { + read_capacity_units: d.read_capacity_units, + write_capacity_units: d.write_capacity_units, + }) + }; + let table_class = match &description.table_class_summary { + None => None, + Some(summary) => Some( + summary + .get("TableClass") + .and_then(serde_json::Value::as_str) + .ok_or_else(|| { + StorageError::Internal(format!( + "table {} has a TableClassSummary without a TableClass: {summary}", + description.table_name + )) + })? + .to_owned(), + ), + }; + let global_secondary_indexes = description + .global_secondary_indexes + .iter() + .flatten() + .map(|g| GsiInput { + index_name: g.index_name.clone(), + key_schema: g.key_schema.clone(), + projection: g.projection.clone(), + provisioned_throughput: g.provisioned_throughput.as_ref().and_then(throughput), + }) + .collect(); + let local_secondary_indexes = description + .local_secondary_indexes + .iter() + .flatten() + .map(|l| LsiInput { + index_name: l.index_name.clone(), + key_schema: l.key_schema.clone(), + projection: l.projection.clone(), + }) + .collect(); + let vector_indexes = description.vector_indexes.as_ref().map(|indexes| { + indexes + .iter() + .map(|v| VectorIndexSpecification { + index_name: v.index_name.clone(), + dimensions: v.dimensions, + distance_function: v.distance_function, + vector_attribute: v.vector_attribute.clone(), + search_schema: v.search_schema.clone(), + projection: v.projection.clone(), + }) + .collect() + }); + Ok(SourceTableSchema { + table_name: description.table_name.clone(), + table_id: description.table_id.clone(), + table_arn: description.table_arn.clone(), + table_creation_date_time: description.creation_date_time, + key_schema: description.key_schema.clone(), + attribute_definitions: description.attribute_definitions.clone(), + billing_mode, + provisioned_throughput: throughput(&description.provisioned_throughput), + on_demand_throughput: description.on_demand_throughput.clone(), + table_class, + sse_description: description.sse_description.clone(), + global_secondary_indexes, + local_secondary_indexes, + vector_indexes, + stream_specification: description.stream_specification.clone(), + time_to_live: ttl, + item_count: description.item_count, + table_size_bytes: description.table_size_bytes, + }) +} + /// A consistent view of one table: its definition, how the snapshot was /// taken, and every item as of the same instant. pub struct TableSnapshot { @@ -396,8 +589,199 @@ pub trait SnapshotEngine: Send + Sync { #[cfg(test)] mod tests { use super::*; + use extenddb_core::backup_format::ExtenddbManifest; use extenddb_core::types::GsiDescription; + /// A description as the storage layer reports it: PROVISIONED, one GSI + /// (with throughput), one LSI, one vector index, a stream, a table class + /// and SSE; the table of the manifest fixture, so the schema can be + /// spliced into it. + fn described() -> TableDescription { + serde_json::from_value(serde_json::json!({ + "TableName": "Music", + "TableId": "12345a12-abcd-123a-ab12-1234abc12345", + "TableArn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music", + "TableStatus": "ACTIVE", + "CreationDateTime": 1757400000.0, + "TableSizeBytes": 123456, + "ItemCount": 100000, + "KeySchema": [ + {"AttributeName": "Artist", "KeyType": "HASH"}, + {"AttributeName": "SongTitle", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "Artist", "AttributeType": "S"}, + {"AttributeName": "SongTitle", "AttributeType": "S"}, + {"AttributeName": "Genre", "AttributeType": "S"} + ], + "ProvisionedThroughput": { + "ReadCapacityUnits": 5, "WriteCapacityUnits": 5, "NumberOfDecreasesToday": 0 + }, + "BillingModeSummary": {"BillingMode": "PROVISIONED"}, + "GlobalSecondaryIndexes": [{ + "IndexName": "GenreIndex", + "KeySchema": [{"AttributeName": "Genre", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "INCLUDE", "NonKeyAttributes": ["Year"]}, + "IndexStatus": "ACTIVE", + "ProvisionedThroughput": { + "ReadCapacityUnits": 1, "WriteCapacityUnits": 1, "NumberOfDecreasesToday": 0 + }, + "IndexSizeBytes": 10, "ItemCount": 3, + "IndexArn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music/index/GenreIndex" + }], + "LocalSecondaryIndexes": [{ + "IndexName": "TitleIndex", + "KeySchema": [ + {"AttributeName": "Artist", "KeyType": "HASH"}, + {"AttributeName": "SongTitle", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "KEYS_ONLY"}, + "IndexSizeBytes": 10, "ItemCount": 3, + "IndexArn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music/index/TitleIndex" + }], + "VectorIndexes": [{ + "IndexName": "embeddings", + "VectorAttribute": {"AttributeName": "vec"}, + "Dimensions": 1024, + "DistanceFunction": "COSINE", + "IndexStatus": "ACTIVE", + "IndexSizeBytes": 0, "ItemCount": 0, + "IndexArn": "arn:aws:dynamodb:us-east-1:123456789012:table/Music/index/embeddings" + }], + "StreamSpecification": {"StreamEnabled": true, "StreamViewType": "NEW_AND_OLD_IMAGES"}, + "DeletionProtectionEnabled": false, + "SSEDescription": { + "Status": "ENABLED", "SSEType": "KMS", + "KMSMasterKeyArn": "arn:aws:kms:us-east-1:123456789012:key/abcd" + }, + "TableClassSummary": {"TableClass": "STANDARD"}, + "OnDemandThroughput": {"MaxReadRequestUnits": 100, "MaxWriteRequestUnits": 50} + })) + .expect("a table description") + } + + fn ttl() -> TimeToLiveDescription { + serde_json::from_value(serde_json::json!({ + "TimeToLiveStatus": "ENABLED", + "AttributeName": "expires_at" + })) + .expect("a TTL description") + } + + /// The fixture manifest with `source_table` replaced by `schema`, parsed + /// back through the reader's own checks. + fn manifest_with(schema: &SourceTableSchema) -> Result { + let mut manifest: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + manifest["source_table"] = serde_json::to_value(schema).expect("serialize"); + let bytes = serde_json::to_vec(&manifest).expect("bytes"); + ExtenddbManifest::from_json_bytes(&bytes).map_err(|e| e.to_string()) + } + + /// The schema built from a full description is the fixture's own + /// `source_table`, member for member, and reads back through the + /// manifest's checks. + #[test] + fn a_described_table_becomes_the_fixtures_source_table() { + let schema = source_table_schema(&described(), Some(ttl())).expect("schema"); + let fixture: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + assert_eq!( + serde_json::to_value(&schema).expect("serialize"), + fixture["source_table"] + ); + let manifest = manifest_with(&schema).expect("the manifest reads back"); + assert_eq!(manifest.source_table, schema); + assert_eq!( + manifest.source_table.vector_indexes.as_ref().map(Vec::len), + Some(1), + "the vector index specification is carried" + ); + } + + /// The members the reader requires even when absent (`vector_indexes`, + /// `stream_specification`, `time_to_live`) are written as explicit + /// nulls, so the document reads back; an on-demand table writes no + /// throughput for itself or its GSIs. + #[test] + fn absent_members_are_written_as_nulls_and_on_demand_drops_throughput() { + let mut description = described(); + description.vector_indexes = None; + description.stream_specification = None; + description.table_class_summary = None; + description.billing_mode_summary = Some(extenddb_core::types::BillingModeSummary { + billing_mode: BillingMode::PayPerRequest, + last_update_to_pay_per_request_date_time: None, + }); + let schema = source_table_schema(&description, None).expect("schema"); + let json = serde_json::to_value(&schema).expect("serialize"); + for member in [ + "vector_indexes", + "stream_specification", + "time_to_live", + "table_class", + ] { + assert_eq!(json[member], serde_json::Value::Null, "{member}"); + assert!( + json.as_object().unwrap().contains_key(member), + "{member} must be present as null, not missing" + ); + } + assert_eq!(schema.billing_mode, BillingMode::PayPerRequest); + assert_eq!(schema.provisioned_throughput, None); + assert_eq!( + schema.global_secondary_indexes[0].provisioned_throughput, + None + ); + assert_eq!( + json["global_secondary_indexes"].as_array().map(Vec::len), + Some(1) + ); + assert_eq!( + json["local_secondary_indexes"].as_array().map(Vec::len), + Some(1) + ); + manifest_with(&schema).expect("the manifest reads back"); + } + + /// A description without a billing-mode summary is PROVISIONED with the + /// description's throughput; a table without vector support writes + /// `vector_indexes: null`, one with support and no indexes `[]`. + #[test] + fn billing_defaults_to_provisioned_and_vector_support_is_told_from_no_indexes() { + let mut description = described(); + description.billing_mode_summary = None; + description.vector_indexes = Some(vec![]); + let schema = source_table_schema(&description, None).expect("schema"); + assert_eq!(schema.billing_mode, BillingMode::Provisioned); + assert_eq!( + schema.provisioned_throughput, + Some(ProvisionedThroughput { + read_capacity_units: 5, + write_capacity_units: 5 + }) + ); + assert_eq!(schema.vector_indexes, Some(vec![])); + description.vector_indexes = None; + let schema = source_table_schema(&description, None).expect("schema"); + assert_eq!(schema.vector_indexes, None); + } + + #[test] + fn a_table_class_summary_without_a_class_is_an_error() { + let mut description = described(); + description.table_class_summary = Some(serde_json::json!({"Other": 1})); + let err = source_table_schema(&description, None).expect_err("refused"); + assert!( + matches!(&err, StorageError::Internal(m) if m.contains("TableClassSummary")), + "{err}" + ); + } + #[test] fn deleting_indexes_are_dropped_and_others_kept() { let gsi = |name: &str, status: &str| -> GsiDescription { diff --git a/crates/storage/src/snapshot_conformance.rs b/crates/storage/src/snapshot_conformance.rs index 04b58c9b2..7429836e7 100644 --- a/crates/storage/src/snapshot_conformance.rs +++ b/crates/storage/src/snapshot_conformance.rs @@ -375,11 +375,39 @@ where /// Drain `stream` into `into`, each item within [`DRAIN_TIMEOUT`] (see /// [`next_within`]); `stage` names the drain in the panic. async fn drain_within(stream: &mut S, stage: &str, into: &mut Vec) +where + S: futures::Stream> + Unpin, +{ + drain_at_most(stream, stage, into, usize::MAX).await; +} + +/// [`drain_within`] with a bound on the count: a stream that has yielded +/// more than `at_most` items is repeating rows (a seek that is inclusive +/// instead of exclusive yields the row it was seeked past on every page, and +/// a cursor that never advances yields the first page forever), and the +/// drain panics at once naming `stage` and the first key yielded twice, so +/// the check fails in seconds instead of running until another test's wait +/// on the hooks lock gives out. +async fn drain_at_most(stream: &mut S, stage: &str, into: &mut Vec, at_most: usize) where S: futures::Stream> + Unpin, { while let Some(next) = next_within(stream, stage, into.len()).await { into.push(next.unwrap_or_else(|e| panic!("{stage}: snapshot stream item: {e}"))); + if into.len() > at_most { + let mut seen = std::collections::HashSet::new(); + let repeated = into + .iter() + .map(key_id) + .find(|id| !seen.insert(id.clone())) + .unwrap_or_else(|| "(no key yielded twice)".to_owned()); + panic!( + "{stage}: the stream yielded {} items of a table of {at_most}; the first key \ + yielded twice is {repeated} (the stream repeats rows: an inclusive seek or a \ + cursor that does not advance)", + into.len() + ); + } } } @@ -566,6 +594,26 @@ impl Drop for PageSize { } } +/// Sets the shared snapshot byte budget ([`paging::set_fetch_bytes`]) and +/// restores the default when dropped, as [`PageSize`] does for the row +/// bound; hold [`hooks_lock`] while one is alive. +pub struct ByteBudget(()); + +impl ByteBudget { + /// Set the byte budget to `bytes` until the returned value is dropped. + #[must_use] + pub fn set(bytes: usize) -> Self { + paging::set_fetch_bytes(bytes); + Self(()) + } +} + +impl Drop for ByteBudget { + fn drop(&mut self) { + paging::set_fetch_bytes(0); + } +} + /// What the check expects of the backend under test. #[derive(Debug, Clone, Copy)] pub struct Expectations { @@ -1465,6 +1513,206 @@ async fn key_shapes( } } +/// Items [`run_page_bounds`] fills its table with: more than one row bound's +/// worth at the budget it sets, and ten past a multiple of the rows a page +/// keeps, so the last page is one the cut shortens (a backend that ends the +/// stream on a short page without asking whether the cut shortened it loses +/// those rows). +pub const PAGE_BOUNDS_ITEMS: usize = 610; + +/// Rows of item text one page of [`run_page_bounds`]'s second pass holds at +/// the budget it sets: the budget is this many rows' text. +pub const PAGE_BOUNDS_ROWS: usize = 8; + +/// What [`run_page_bounds`] measured, for a backend's own assertions beside +/// it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct PageBounds { + /// Bytes of item text of one row of the table (every row is the same + /// size). + pub row_bytes: usize, + /// The byte budget the second pass ran under: [`PAGE_BOUNDS_ROWS`] rows. + pub budget: usize, + /// Round trips the second pass made. + pub pages: u64, + /// Rows the backend recorded as read past a cut during the second pass + /// ([`paging::rows_past_cut`]; a backend that cuts in its query records + /// them, one that cuts while pulling rows records none). + pub rows_past_cut: u64, +} + +fn page_bounds_item(i: usize) -> Item { + // Fixed-width keys and a fixed padding, so every row's text is the same + // length and the budget can be set to a whole number of rows. + [ + ("pk".to_owned(), s(format!("p{i:05}"))), + ("sk".to_owned(), n(100_000 + i)), + ("v".to_owned(), s("x".repeat(1000))), + ] + .into_iter() + .collect() +} + +/// Check that the stream pages by bytes: no page holds more than the budget +/// plus one row, and the cut does not lose rows. `hooks` is the caller's +/// [`hooks_lock`] guard; `table` must not exist; `pump` runs while the table +/// becomes `ACTIVE`. +/// +/// The table is [`PAGE_BOUNDS_ITEMS`] rows of one size. A first pass at a +/// budget of one byte reads one row per page, which measures the row's text +/// size as the backend counts it (`row_bytes`) and shows the budget took +/// effect (as many round trips as rows). The second pass sets the budget to +/// [`PAGE_BOUNDS_ROWS`] rows and the row bound to its default, and asserts: +/// every item once (completeness across cut pages, the last one included; +/// both passes stop and fail as soon as the stream has yielded more items +/// than the table holds, naming the key yielded twice, so a stream that +/// repeats rows fails in seconds rather than running until the hooks lock's +/// wait gives out); +/// more round trips than the row bound alone would make (the budget, not the +/// row bound, paged the table: a counter beside the knob, since the stream's +/// contents alone cannot show the knob took effect); the largest page at least the +/// budget (a page was cut) and under the budget plus one row (the rule: a +/// page ends after the row that takes it to or past the budget, so a page +/// cut one row late shows as exactly the budget plus one row); and, when the +/// stream is `key_ordered`, ascending order. +/// +/// # Panics +/// +/// On any violation, with the figures. +pub async fn run_page_bounds( + hooks: &HooksGuard, + engine: &E, + account: &str, + table: &str, + mut pump: F, +) -> PageBounds +where + E: SnapshotEngine + TableEngine + DataEngine + Send + Sync, + F: FnMut() -> Fut, + Fut: std::future::Future, +{ + let _ = hooks; + engine + .create_table(account, table_input(table)) + .await + .expect("create the page-bounds table"); + wait_active(engine, account, table, &mut pump).await; + let ki = key_info(engine, account, table).await; + for i in 0..PAGE_BOUNDS_ITEMS { + put(engine, &ki, page_bounds_item(i)).await; + } + let expected: BTreeSet = (0..PAGE_BOUNDS_ITEMS) + .map(|i| key_id(&page_bounds_item(i))) + .collect(); + + // First pass: one row per page. + let row_bytes = { + let _budget = ByteBudget::set(1); + let _rows = PageSize::set(0); + paging::reset_page_counters(); + let fetches_before = paging::fetches(); + let mut snapshot = engine + .open_table_snapshot(account, table) + .await + .expect("open at a one-byte budget"); + let mut seen = Vec::new(); + drain_at_most( + &mut snapshot.items, + "page bounds, one-byte budget", + &mut seen, + PAGE_BOUNDS_ITEMS, + ) + .await; + drop(snapshot); + assert_eq!(seen.len(), PAGE_BOUNDS_ITEMS, "one-byte budget: items read"); + let fetches = paging::fetches() - fetches_before; + assert!( + fetches >= PAGE_BOUNDS_ITEMS as u64, + "one-byte budget: {fetches} round trips for {PAGE_BOUNDS_ITEMS} rows; the budget \ + did not cut the pages to one row" + ); + let row_bytes = paging::max_page_bytes(); + assert!( + row_bytes > 1000, + "one row of item text measured as {row_bytes} bytes" + ); + row_bytes + }; + + // Second pass: a budget of a whole number of rows, the row bound at its + // default. + let budget = row_bytes * PAGE_BOUNDS_ROWS; + let _budget = ByteBudget::set(budget); + let _rows = PageSize::set(0); + paging::reset_page_counters(); + let fetches_before = paging::fetches(); + let mut snapshot = engine + .open_table_snapshot(account, table) + .await + .expect("open at the row-multiple budget"); + let key_ordered = snapshot.key_ordered; + let order = extenddb_core::backup_format::KeyOrder::resolve( + &snapshot.description.key_schema, + &snapshot.description.attribute_definitions, + ) + .expect("the page-bounds table has an orderable key"); + let mut seen = Vec::new(); + drain_at_most( + &mut snapshot.items, + "page bounds, row-multiple budget", + &mut seen, + PAGE_BOUNDS_ITEMS, + ) + .await; + drop(snapshot); + let pages = paging::fetches() - fetches_before; + + let mut got = BTreeSet::new(); + for item in &seen { + let id = key_id(item); + assert!(got.insert(id.clone()), "page bounds: duplicate item {id}"); + } + assert_eq!( + got, expected, + "page bounds: the stream did not yield every row exactly once (a cut page lost or \ + repeated rows)" + ); + let row_bound_pages = PAGE_BOUNDS_ITEMS.div_ceil(paging::fetch_batch()) as u64; + assert!( + pages > row_bound_pages, + "page bounds: {pages} round trips, no more than the row bound alone would make \ + ({row_bound_pages}); the byte budget did not page the table" + ); + let largest = paging::max_page_bytes(); + assert!( + largest >= budget, + "page bounds: the largest page holds {largest} bytes, under the budget of {budget}; no \ + page was cut by bytes" + ); + assert!( + largest < budget + row_bytes, + "page bounds: the largest page holds {largest} bytes, the budget plus a row or more \ + ({budget} + {row_bytes}); a page is cut one row late" + ); + if key_ordered { + for pair in seen.windows(2) { + assert_eq!( + order.compare(&pair[0], &pair[1]).expect("comparable keys"), + std::cmp::Ordering::Less, + "page bounds: {} is not before {}", + key_id(&pair[0]), + key_id(&pair[1]) + ); + } + } + PageBounds { + row_bytes, + budget, + pages, + rows_past_cut: paging::rows_past_cut(), + } +} + /// Checks that a snapshot cannot be opened on a missing or non-`ACTIVE` /// table. `hooks` is the caller's [`hooks_lock`] guard. `table` must not /// exist; `set_updating` flips that table's catalog diff --git a/devtools/backup-kill-points.py b/devtools/backup-kill-points.py new file mode 100755 index 000000000..fff70ed63 --- /dev/null +++ b/devtools/backup-kill-points.py @@ -0,0 +1,1204 @@ +#!/usr/bin/env python3 +"""The API half of kill-points.sh and bench-rss.sh: load a table, start a +backup, poll it, read the store. + +Usage (every call reads EXTENDDB_TEST_ENDPOINT and the AWS_* credentials +provision-test-credentials exported): + kill-points.py create-table [--items N] [--pad BYTES] [--gsis N] [--threads N] + kill-points.py create-backup -> prints the ARN + kill-points.py status -> CREATING|AVAILABLE|NOTFOUND + kill-points.py wait-available [--timeout S] -> prints size,count,duration + kill-points.py describe -> JSON + kill-points.py put-worst [--count N] [--sets N] [--small N] [--interleave N] + -> the worst-expansion items after N small ones, or at every Nth key among them + kill-points.py put-random [--count N] [--bytes N] +""" +from __future__ import annotations + +import argparse +import base64 +import json +import os +import sys +import threading +import time +from concurrent.futures import ThreadPoolExecutor + +import boto3 +import urllib3 +from botocore.config import Config +from botocore.exceptions import ClientError + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) +ENDPOINT = os.environ.get("EXTENDDB_TEST_ENDPOINT", "") + + +def client(): + return boto3.client( + "dynamodb", + endpoint_url=ENDPOINT, + region_name="us-east-1", + verify=False, + config=Config(retries={"max_attempts": 10, "mode": "standard"}, read_timeout=120), + ) + + +def wait_active(c, name: str) -> None: + while c.describe_table(TableName=name)["Table"]["TableStatus"] != "ACTIVE": + time.sleep(0.2) + # Every GSI ACTIVE too. + while any( + g.get("IndexStatus") != "ACTIVE" + for g in c.describe_table(TableName=name)["Table"].get("GlobalSecondaryIndexes", []) + ): + time.sleep(0.2) + + +def create_table(args) -> None: + c = client() + attrs = [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + ] + gsis = [] + for i in range(args.gsis): + attrs.append({"AttributeName": f"g{i}", "AttributeType": "S"}) + gsis.append( + { + "IndexName": f"by-g{i}", + "KeySchema": [ + {"AttributeName": f"g{i}", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + "Projection": {"ProjectionType": "ALL"}, + } + ) + kwargs = {"GlobalSecondaryIndexes": gsis} if gsis else {} + c.create_table( + TableName=args.name, + KeySchema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + AttributeDefinitions=attrs, + BillingMode="PAY_PER_REQUEST", + **kwargs, + ) + wait_active(c, args.name) + n = args.items + if n == 0: + return + local = threading.local() + started = time.monotonic() + + def write(start: int) -> None: + if not hasattr(local, "c"): + local.c = client() + reqs = [] + for i in range(start, min(start + 25, n)): + item = { + "pk": {"S": f"p{i % 1000:05d}"}, + "sk": {"N": str(i)}, + "pad": {"S": "x" * args.pad}, + } + for g in range(args.gsis): + item[f"g{g}"] = {"S": f"v{(i // (g + 1)) % 97}"} + reqs.append({"PutRequest": {"Item": item}}) + pending = {args.name: reqs} + while pending: + pending = local.c.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + if pending: + time.sleep(0.05) + + with ThreadPoolExecutor(max_workers=args.threads) as pool: + list(pool.map(write, range(0, n, 25))) + wait_active(c, args.name) + print(f"loaded {n} items in {time.monotonic() - started:.1f} s", file=sys.stderr) + + +def put_worst(args) -> None: + """#394's worst-expansion lines (`data_file.rs`'s line-cap table): an + item of 2,262 `NS` sets of the 90 values `1.0E-130`..`9.9E-130` is + 409,429 bytes as the service sizes items and 27.7 MB as a data-file line + (and as stored item text, the server normalizing each number to its + 133-character decimal form), placed after `--small` small rows (their + `a...` keys sort before `zz...`) or, with `--interleave N`, at every Nth + key among them (the shape of a table whose large items are a minority: + key order does not sort items by size).""" + c = client() + every = args.interleave + big_positions = set([i for i in range(args.small) if every and i % every == every - 1][: args.count]) + small = [ + {"PutRequest": {"Item": {"pk": {"S": f"a{i:04d}"}, "sk": {"N": str(i)}, "pad": {"S": "x" * 50}}}} + for i in range(args.small) if i not in big_positions + ] + for i in range(0, len(small), 25): + pending = {args.name: small[i : i + 25]} + while pending: + pending = c.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + values = [f"{a}.{b}E-130" for a in range(1, 10) for b in range(10)] + # MongoDB stores an item as one BSON document, capped at 16 MiB, so + # #394's 2,262-set shape (28.9 MB as BSON) cannot be stored there; --sets + # lowers the count to the largest row the backend takes. + sets = [{"NS": values} for _ in range(args.sets)] + if every: + for i in sorted(big_positions): + c.put_item(TableName=args.name, Item={"pk": {"S": f"a{i:04d}"}, "sk": {"N": str(i)}, "l": {"L": sets}}) + print(f"worst-expansion: {args.small} small items with {len(big_positions)} list-of-{args.sets}-NS items interleaved every {every}", file=sys.stderr) + else: + for i in range(args.count): + c.put_item(TableName=args.name, Item={"pk": {"S": f"zz{i:03d}"}, "sk": {"N": "0"}, "l": {"L": sets}}) + print(f"worst-expansion: {args.small} small items then {args.count} list-of-{args.sets}-NS items", file=sys.stderr) + + +def put_random(args) -> None: + """Incompressible items: `B` attributes of random bytes, `args.bytes` in + all across `args.count` items.""" + c = client() + per = args.bytes // args.count + for i in range(args.count): + c.put_item( + TableName=args.name, + Item={"pk": {"S": f"r{i:05d}"}, "sk": {"N": "0"}, "blob": {"B": os.urandom(per)}}, + ) + print(f"random: {args.count} items of {per} bytes", file=sys.stderr) + + +def create_backup(args) -> None: + c = client() + d = c.create_backup(TableName=args.name, BackupName=args.backup_name)["BackupDetails"] + print(d["BackupArn"]) + print(d["BackupStatus"], file=sys.stderr) + + +def status(args) -> None: + c = client() + try: + print(c.describe_backup(BackupArn=args.arn)["BackupDescription"]["BackupDetails"]["BackupStatus"]) + except ClientError as e: + if e.response["Error"]["Code"] == "BackupNotFoundException": + print("NOTFOUND") + else: + raise + + +def wait_available(args) -> None: + c = client() + started = time.monotonic() + while True: + d = c.describe_backup(BackupArn=args.arn)["BackupDescription"] + s = d["BackupDetails"]["BackupStatus"] + if s == "AVAILABLE": + print( + json.dumps( + { + "size_bytes": d["BackupDetails"]["BackupSizeBytes"], + "item_count": d["SourceTableDetails"]["ItemCount"], + "duration_s": round(time.monotonic() - started, 2), + } + ) + ) + return + if s != "CREATING": + print(f"backup is {s}", file=sys.stderr) + sys.exit(2) + if time.monotonic() - started > args.timeout: + print("timeout", file=sys.stderr) + sys.exit(3) + time.sleep(0.2) + + +def describe(args) -> None: + c = client() + print(json.dumps(c.describe_backup(BackupArn=args.arn)["BackupDescription"], default=str, indent=1)) + + +def main() -> None: + p = argparse.ArgumentParser() + sub = p.add_subparsers(dest="cmd", required=True) + a = sub.add_parser("create-table") + a.add_argument("name") + a.add_argument("--items", type=int, default=100000) + a.add_argument("--pad", type=int, default=200) + a.add_argument("--gsis", type=int, default=2) + a.add_argument("--threads", type=int, default=16) + a.set_defaults(fn=create_table) + a = sub.add_parser("put-worst") + a.add_argument("name") + a.add_argument("--count", type=int, default=20) + a.add_argument("--sets", type=int, default=2262) + a.add_argument("--small", type=int, default=1000) + a.add_argument("--interleave", type=int, default=0, help="place the large items at every Nth key among the small ones instead of after them") + a.set_defaults(fn=put_worst) + a = sub.add_parser("put-random") + a.add_argument("name") + a.add_argument("--count", type=int, default=256) + a.add_argument("--bytes", type=int, default=64 * 1024 * 1024) + a.set_defaults(fn=put_random) + a = sub.add_parser("create-backup") + a.add_argument("name") + a.add_argument("backup_name") + a.set_defaults(fn=create_backup) + a = sub.add_parser("status") + a.add_argument("arn") + a.set_defaults(fn=status) + a = sub.add_parser("wait-available") + a.add_argument("arn") + a.add_argument("--timeout", type=float, default=1800) + a.set_defaults(fn=wait_available) + a = sub.add_parser("describe") + a.add_argument("arn") + a.set_defaults(fn=describe) + args = p.parse_args() + args.fn(args) + + + + + +# =========================================================================== +# PR 9: the restore kill points. +# +# Each run builds one backend's deployment from the hooks binary +# (`--features extenddb-app/backup-test-hooks`, whose feature chain pulls +# `extenddb-storage/test-hooks`, so the engine's restore points and the +# backend's own point are armed), `reconcile_interval_secs = 2`, +# `stale_job_after_secs = 45`; loads a table with one GSI (`by-g`, every +# item carries `g`), takes a backup, then starts a restore paused at the +# named point (`EXTENDDB_BACKUP_PAUSE`), kills the server by PID, restarts +# it without the pause, and asserts the outcome: at every ACTIVE outcome +# the base table's `Scan` count and the GSI's `Scan` count both equal +# `--items` (a resumed Cassandra restore once left the GSI short while the +# base count was exact, so the base count alone is not the assertion). +# Logs under `--work`. +# +# kill-points.py restore \ +# --bin [--items N] [--work DIR] [--second-reach] \ +# [--remove-target] [--vector] [--source-bin ] +# +# `pg_terminate_backend` is not a pause point of its own: the server is +# paused at `restore_after_file_commit`, the session holding the claim's +# advisory lock is terminated while the job is paused (psql must answer a +# row), the script waits past one heartbeat period and releases, and asserts +# the job's lost-claim stop, the record RUNNING with its progress row, the +# reconciler's takeover no earlier than `stale_job_after_secs` after the +# owner's last beat (read from the record at the termination; the beat +# precedes the termination by up to one period), and the exact counts. `--vector` gives the source table a +# vector index; with `restore_between_failed_and_removal` on postgres the +# cell then asserts the removed target's data table and `_ddb_vec_` relation +# are gone from the data database. +# +# Cassandra has no `SnapshotEngine`, so its CreateBackup is the in-catalog +# format and no backup taken on a Cassandra server restores through the +# store-backed job. A Cassandra kill point therefore takes its backup on a +# SQLite source server (`--source-bin`, a SQLite build), copies the prefix +# under the Cassandra target's root and seeds the target's `backup_index` +# row as `devtools/backup-restore-matrix` does (and as `backup sync +# --adopt-store` will), then restores on the paused Cassandra server. The +# matrix script is imported for `copy_prefix`, `prefix_of` and `seed_row`. +# +# `--second-reach` kills at the point's second reach instead of its first: +# the first pause is released (the release file written, then removed once +# the job logs its release), and the server is SIGKILLed when the job +# reaches the point again. For `restore_mongodb_after_insert_many` and +# `restore_cassandra_before_progress` that puts the kill after an earlier +# file's or batch's progress is recorded, so the resume runs the +# re-application path (MongoDB: the one unrecorded file re-applied with +# E11000 suppressed; Cassandra: at most one batch re-applied by the upsert) +# and the outcome asserted is ACTIVE with the exact count, where the +# first-reach kill (no progress recorded, items written) asserts +# `target_not_empty`. +# +# The services are local (PostgreSQL 55432, MongoDB 27117, Cassandra 9042). +# Every server is stopped by the PID this script started, never by a +# pattern over the process list; every scratch database, keyspace and the +# fixed MongoDB databases are dropped. +# =========================================================================== + +import os +import re +import shutil +import signal +import subprocess + +import requests +import urllib.parse # noqa: F401 + +PG = {"host": "127.0.0.1", "port": "55432", "user": "postgres", "password": "devpass"} +# The MongoDB replica set a MongoDB kill point uses (its fixed databases are +# dropped per run, so give the script a replica set of its own). +MONGODB_URI = os.environ.get("EXTENDDB_TEST_MONGODB_URI") or "mongodb://127.0.0.1:27117/?directConnection=true" +ACCOUNT = "123456789012" +ADMIN_PASSWORD = "TestPass1!" +REGION = "us-east-1" + + +def _say(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +def _repo() -> str: + """The ExtendDB checkout whose `devtools/` scripts this script calls + (`provision-test-credentials`, `backup-restore-matrix`): `EXTENDDB_REPO`, + or the working directory when the script is run from the checkout.""" + return os.path.abspath(os.environ.get("EXTENDDB_REPO") or os.getcwd()) + + +def _free_port() -> int: + import socket + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +def _psql(stmt: str, database: str = "postgres") -> str: + env = dict(os.environ, PGPASSWORD=PG["password"]) + r = subprocess.run( + ["psql", "-h", PG["host"], "-p", PG["port"], "-U", PG["user"], "-d", database, + "-v", "ON_ERROR_STOP=1", "-tAq", "-c", stmt], + env=env, capture_output=True, text=True, + ) + if r.returncode != 0: + raise SystemExit(f"psql failed: {r.stderr}\n{stmt}") + return r.stdout.strip() + + +def _sqlite_connect(path: str): + try: + import pysqlite3 as sqlite3 # type: ignore + except ImportError: + import sqlite3 # type: ignore + return sqlite3.connect(path, timeout=10) + + +class Deployment: + """One backend's server for a kill-point run, from the hooks binary.""" + + def __init__(self, backend: str, binary: str, work: str): + self.backend = backend + self.bin = binary + self.work = work + os.makedirs(work, exist_ok=True) + self.config = os.path.join(work, "extenddb.toml") + self.root = os.path.join(work, "root") + self.spool = os.path.join(work, "spool") + self.port = _free_port() + self.endpoint = f"https://127.0.0.1:{self.port}" + self.tag = f"kp{int(time.time()) % 1_000_000}" + self.proc = None + self.creds = {} + + def _free_port_again(self) -> None: + self.port = _free_port() + self.endpoint = f"https://127.0.0.1:{self.port}" + + def init(self) -> None: + env = dict(os.environ, EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD) + if self.backend == "sqlite": + cmd = [self.bin, "init", "--backend", "sqlite", "--sqlite-path", + os.path.join(self.work, "db.sqlite"), "--config", self.config] + elif self.backend == "postgres": + cmd = [self.bin, "init", "--pg-host", PG["host"], "--pg-port", PG["port"], + "--pg-user", PG["user"], "--pg-pass", PG["password"], + "--data-db", f"{self.tag}_data", "--catalog-db", f"{self.tag}_catalog", + "--extenddb-user", f"{self.tag}_app", "--config", self.config] + elif self.backend == "mongodb": + self._mongo_drop() + with open(self.config, "w") as f: + f.write(f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\nconnection_string = "{MONGODB_URI}"\n') + cmd = [self.bin, "init", "--backend", "mongodb", "--config", self.config, "--overwrite"] + elif self.backend == "cassandra": + cmd = [self.bin, "init", "--backend", "cassandra", + "--cassandra-contact-points", "127.0.0.1:9042", + "--cassandra-user", "cassandra", "--cassandra-pass", "cassandra", + "--config", self.config] + else: + raise SystemExit(f"unknown backend {self.backend}") + with open(os.path.join(self.work, "init.log"), "w") as log: + r = subprocess.run(cmd, cwd=self.work, env=env, stdout=log, stderr=subprocess.STDOUT) + if r.returncode != 0: + print(open(os.path.join(self.work, "init.log")).read()) + raise SystemExit(f"init of the {self.backend} server failed") + self._write_config() + + def _write_config(self) -> None: + home = os.path.expanduser("~") + os.makedirs(self.root, mode=0o700, exist_ok=True) + store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{self.root}"\n' + backup = "[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\nmax_concurrent_jobs = 2\ndata_file_target_bytes = 33554432\n" + if self.backend == "sqlite": + text = open(self.config).read() + text = re.sub(r"(?m)^[ \t]*#?[ \t]*port[ \t]*=[ \t]*\d+.*$", f"port = {self.port}", text, count=1) + text = re.sub(r'(?m)^run_dir = .*$', f'run_dir = "{self.work}/run"', text) + text += "\n" + backup + store + open(self.config, "w").write(text) + elif self.backend == "postgres": + text = open(self.config).read() + text = re.sub(r"(?m)^[ \t]*#?[ \t]*port[ \t]*=[ \t]*\d+.*$", f"port = {self.port}", text, count=1) + text = re.sub(r'(?m)^run_dir = .*$', f'run_dir = "{self.work}/run"', text) + text += "\n" + backup + store + open(self.config, "w").write(text) + else: + conn = (MONGODB_URI if self.backend == "mongodb" + else "") + storage = (f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\nconnection_string = "{MONGODB_URI}"\n' + if self.backend == "mongodb" else + # `replication_factor = 1` as `init` writes it for a single + # node: the default is 3, and the data keyspace the first + # CreateTable creates under RF 3 on one node answers every + # lightweight transaction and quorum read with "No nodes + # available in query plan" (the catalog keyspace, created by + # `init` under RF 1, keeps working, so the server starts and + # the table reports ACTIVE before the first write fails). + '[storage]\nbackend = "cassandra"\n\n[storage.cassandra]\n' + 'contact_points = ["127.0.0.1:9042"]\nusername = "cassandra"\npassword = "cassandra"\n' + 'keyspace_prefix = "extenddb"\nreplication_factor = 1\ndatacenter = "datacenter1"\n') + open(self.config, "w").write( + f'[server]\nbind_addr = "127.0.0.1"\nport = {self.port}\nregion = "{REGION}"\n' + f'run_dir = "{self.work}/run"\n\n' + f'[server.tls]\ncert_path = "{home}/.extenddb/tls/cert.pem"\n' + f'key_path = "{home}/.extenddb/tls/key.pem"\n\n' + f'{storage}\n[auth]\nprovider = "builtin"\n\n{backup}\n{store}') + + def start(self, log_name: str, pause: str | None = None) -> None: + env = dict(os.environ) + if pause: + env["EXTENDDB_BACKUP_PAUSE"] = pause + env["EXTENDDB_BACKUP_PAUSE_RELEASE"] = os.path.join(self.work, "release") + self.log_path = os.path.join(self.work, log_name) + self.log = open(self.log_path, "a") + self.proc = subprocess.Popen( + [self.bin, "serve", "--foreground", "--config", self.config], + cwd=self.work, env=env, stdout=self.log, stderr=subprocess.STDOUT) + for _ in range(300): + try: + if requests.get(f"{self.endpoint}/health", verify=False, timeout=2).status_code == 200: + break + except requests.RequestException: + pass + if self.proc.poll() is not None: + break + time.sleep(0.2) + if self.proc.poll() is not None: + raise SystemExit(f"the {self.backend} server exited at start; see {self.log_path}") + _say(f"{self.backend} pid {self.proc.pid} on {self.port} ({log_name})") + + def sigkill(self) -> None: + if self.proc and self.proc.poll() is None: + self.proc.send_signal(signal.SIGKILL) + self.proc.wait(timeout=10) + _say(f"SIGKILLed pid {self.proc.pid}") + + def sigterm(self) -> int: + self.proc.send_signal(signal.SIGTERM) + code = self.proc.wait(timeout=60) + _say(f"SIGTERM pid {self.proc.pid}, exit {code}") + return code + + def stop(self) -> None: + if self.proc and self.proc.poll() is None: + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=60) + except subprocess.TimeoutExpired: + self.proc.send_signal(signal.SIGKILL) + self.proc.wait(timeout=10) + + def provision(self) -> None: + env = dict(os.environ, EXTENDDB_TEST_ENDPOINT=self.endpoint, EXTENDDB_ADMIN_USER="admin", + EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD, EXTENDDB_CA_CERT=os.path.expanduser("~/.extenddb/tls/cert.pem")) + r = subprocess.run([sys.executable, os.path.join(_repo(), "devtools", "provision-test-credentials")], + env=env, capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"provisioning failed: {r.stderr}") + for line in r.stdout.splitlines(): + m = re.match(r"export (\w+)=(.*)", line.strip()) + if m: + self.creds[m.group(1)] = m.group(2) + os.environ[m.group(1)] = m.group(2) + os.environ["EXTENDDB_TEST_ENDPOINT"] = self.endpoint + for key, value in [("control_plane_delay_seconds", "0.05"), ("index_propagation_delay_ms", "0")]: + subprocess.run([self.bin, "settings", "--config", self.config, "set", key, value], + cwd=self.work, capture_output=True) + + def teardown(self) -> None: + self.stop() + if self.backend == "postgres": + _psql(f"DROP DATABASE IF EXISTS {self.tag}_data WITH (FORCE)") + _psql(f"DROP DATABASE IF EXISTS {self.tag}_catalog WITH (FORCE)") + _psql(f"DROP ROLE IF EXISTS {self.tag}_app") + elif self.backend == "mongodb": + self._mongo_drop() + elif self.backend == "cassandra": + try: + from cassandra.auth import PlainTextAuthProvider + from cassandra.cluster import Cluster + cluster = Cluster(["127.0.0.1"], port=9042, + auth_provider=PlainTextAuthProvider("cassandra", "cassandra")) + s = cluster.connect() + for ks in ("extenddb_catalog", f"extenddb_account_{ACCOUNT}"): + s.execute(f"DROP KEYSPACE IF EXISTS {ks}") + cluster.shutdown() + except Exception as e: # noqa: BLE001 + print(f"cassandra teardown: {e}") + + def _mongo_drop(self) -> None: + """Drop the fixed databases and wait until each drop has completed: on + a replica set `dropDatabase` is two-phase and a collection created + while the database is drop-pending fails with code 215 or vanishes + when the drop lands (a server started over a pending drop answered + `ResourceNotFoundException` to `BatchWriteItem` into an ACTIVE + table). The probe creates a collection until the server accepts it, + then drops the collection, which leaves no pending database.""" + from pymongo import MongoClient + from pymongo.errors import OperationFailure + c = MongoClient(MONGODB_URI) + try: + for name in ("extenddb_catalog", "extenddb_data"): + c.drop_database(name) + deadline = time.monotonic() + 120 + for name in ("extenddb_catalog", "extenddb_data"): + while True: + try: + c[name].create_collection("drop_probe") + except OperationFailure as e: + if e.code == 215 and time.monotonic() < deadline: + time.sleep(0.25) + continue + raise + c[name].drop_collection("drop_probe") + break + finally: + c.close() + + +def _ddb(dep: Deployment): + import boto3 + from botocore.config import Config + return boto3.client("dynamodb", endpoint_url=dep.endpoint, region_name=REGION, verify=False, + aws_access_key_id=dep.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=dep.creds["AWS_SECRET_ACCESS_KEY"], + config=Config(retries={"max_attempts": 10, "mode": "standard"}, read_timeout=120)) + + +# The points and which backends they apply to. +RESTORE_POINTS = { + "restore_after_download": ["postgres", "sqlite", "mongodb", "cassandra"], + "restore_after_file_commit": ["postgres", "sqlite", "mongodb", "cassandra"], + "restore_sqlite_mid_batch": ["sqlite"], + "restore_cassandra_before_progress": ["cassandra"], + "restore_mongodb_after_insert_many": ["mongodb"], + "restore_postgres_before_progress_delete": ["postgres"], + "restore_mongodb_before_worker_flip": ["mongodb"], + "restore_between_failed_and_removal": ["postgres", "sqlite", "mongodb", "cassandra"], + "restore_fenced_out": ["postgres"], + "pg_terminate_backend": ["postgres"], + "sigterm": ["postgres", "sqlite", "mongodb", "cassandra"], +} + + +def _wait_table_active(ddb, name: str, timeout: float = 120) -> None: + """Poll DescribeTable until the table and every GSI are ACTIVE, the way + the fidelity suite's `wait_for_active` does, before any load.""" + deadline = time.monotonic() + timeout + while True: + t = ddb.describe_table(TableName=name)["Table"] + gsis = t.get("GlobalSecondaryIndexes") or [] + if t["TableStatus"] == "ACTIVE" and all(g.get("IndexStatus", "ACTIVE") == "ACTIVE" for g in gsis): + return + assert time.monotonic() < deadline, f"{name} did not reach ACTIVE" + time.sleep(0.3) + + +def _batch_write_ready(ddb, name: str, reqs) -> None: + """Write one batch, tolerating the load-readiness window on a freshly + created table: a `ResourceNotFoundException` on MongoDB (a collection not + yet visible) or an `InternalServerError` on Cassandra (schema not settled + on the coordinator) right after the table reports ACTIVE is a harness + race, not a restore defect, so re-poll ACTIVE and retry the batch for up + to 60 s rather than aborting the load.""" + from botocore.exceptions import ClientError + + deadline = time.monotonic() + 240 + pending = {name: reqs} + while pending: + try: + pending = ddb.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + except ClientError as e: + code = e.response["Error"]["Code"] + if code in ("ResourceNotFoundException", "InternalServerError") and time.monotonic() < deadline: + _wait_table_active(ddb, name) + time.sleep(2) + continue + raise + + +def _signed_post(dep: Deployment, operation: str, body: dict): + """A hand-signed request for an operation botocore's model lacks + (CreateTable with VectorIndexes), as `devtools/backup-restore-matrix` + sends it.""" + import json + import boto3 + from botocore.auth import SigV4Auth + from botocore.awsrequest import AWSRequest + body_bytes = json.dumps(body).encode() + headers = {"X-Amz-Target": f"DynamoDB_20120810.{operation}", "Content-Type": "application/x-amz-json-1.0"} + frozen = boto3.Session(aws_access_key_id=dep.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=dep.creds["AWS_SECRET_ACCESS_KEY"]).get_credentials().get_frozen_credentials() + req = AWSRequest(method="POST", url=dep.endpoint, data=body_bytes, headers=headers) + SigV4Auth(frozen, "dynamodb", REGION).add_auth(req) + return requests.post(dep.endpoint, data=body_bytes, headers=dict(req.headers), verify=False) + + +def _create_table_and_backup(dep: Deployment, items: int, big_file: bool, vector: bool = False): + ddb = _ddb(dep) + name = f"kp{dep.tag}" + attrs = [{"AttributeName": "pk", "AttributeType": "S"}, {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}] + body = dict(TableName=name, KeySchema=[{"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}], + AttributeDefinitions=attrs, BillingMode="PAY_PER_REQUEST", + GlobalSecondaryIndexes=[{"IndexName": "by-g", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "ALL"}}]) + if vector: + # ExtendDB's VectorIndexes member is not in botocore's model, so the + # CreateTable goes through a signed POST, as the matrix script's does. + body["VectorIndexes"] = [{"IndexName": "by-emb", "Dimensions": 3, "DistanceFunction": "COSINE", + "VectorAttribute": {"AttributeName": "emb"}, + "Projection": {"ProjectionType": "ALL"}}] + r = _signed_post(dep, "CreateTable", body) + if r.status_code != 200: + raise SystemExit(f"CreateTable with a vector index: {r.status_code} {r.text}") + else: + ddb.create_table(**body) + _wait_table_active(ddb, name) + if dep.backend == "cassandra": + # Cassandra's DDL settles on the coordinator a moment after the table + # reports ACTIVE; a load that starts in that gap meets "No nodes + # available in query plan" or an InternalServerError from the LWT. + # `_batch_write_ready` retries through it; this settle wait shortens it. + time.sleep(3) + # big_file forces more than 10,000 items into one data file for the + # SQLite and Cassandra mid-batch points (each sits between two batches of + # one file); otherwise 4,000-byte items give two or three data files at + # the 32 MiB file target. (A Cassandra deployment never takes the backup + # here: its backup is the in-catalog format, module comment.) + pad = 20 if big_file else 4000 + reqs = [] + for i in range(items): + item = {"pk": {"S": f"p{i % 50:04d}"}, "sk": {"N": str(i)}, "g": {"S": f"g{i % 7}"}, "pad": {"S": "x" * pad}} + if vector and i % 5 == 0: + item["emb"] = {"L": [{"N": "0.1"}, {"N": str(0.01 * (i % 50))}, {"N": "0.3"}]} + reqs.append({"PutRequest": {"Item": item}}) + if len(reqs) == 25 or i == items - 1: + _batch_write_ready(ddb, name, reqs) + reqs = [] + arn = ddb.create_backup(TableName=name, BackupName="kpbackup")["BackupDetails"]["BackupArn"] + deadline = time.monotonic() + 600 + while True: + d = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] + if d == "AVAILABLE": + break + assert d == "CREATING", d + assert time.monotonic() < deadline, "backup did not become AVAILABLE" + time.sleep(0.2) + return ddb, name, arn + + +def _count(ddb, name: str, index: str | None = None) -> int: + """The `Scan` count of the table, or of its GSI `index` (an eventually + consistent read, the only kind a GSI takes).""" + total, start = 0, None + while True: + kw = {"TableName": name, "Select": "COUNT"} + if index: + kw["IndexName"] = index + else: + kw["ConsistentRead"] = True + if start: + kw["ExclusiveStartKey"] = start + r = ddb.scan(**kw) + total += r["Count"] + start = r.get("LastEvaluatedKey") + if not start: + return total + + +def _assert_exact(ddb, name: str, items: int, what: str) -> None: + """The base table and the GSI `by-g` both hold `items` rows. Every item + of `_create_table_and_backup`'s fixture carries `g`, so a GSI short of + the base count is an index row the restore lost (a re-applied batch on + Cassandra deleted its index rows before #742 was corrected).""" + n = _count(ddb, name) + assert n == items, f"{what}: restored {n}, expected {items}" + g = _count(ddb, name, index="by-g") + assert g == items, f"{what}: the GSI by-g holds {g} rows, expected {items}" + + +def _wait_reached(dep: Deployment, point: str, timeout: float = 120, times: int = 1) -> bool: + """True once the log holds `times` lines of the point being reached.""" + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + with open(dep.log_path, errors="replace") as f: + if sum(1 for line in f if f"reached kill point {point}" in line) >= times: + return True + if dep.proc.poll() is not None: + return False + time.sleep(0.2) + return False + + +def _release_first_reach(dep: Deployment, point: str) -> None: + """Release the paused job from the point's first reach and remove the + release file again, so the next reach pauses. The pause polls the file + every 100 ms and the job reaches the point again as soon as the next + file's or batch's writes are done, so the log is polled every 10 ms for + the release line and the file removed at once; a second reach the file + still released (a next write that took less than that) is a setup + failure, not a kill, and the run says so.""" + release = os.path.join(dep.work, "release") + open(release, "w").close() + deadline = time.monotonic() + 30 + while True: + with open(dep.log_path, errors="replace") as f: + released = sum(1 for line in f if f"released from kill point {point}" in line) + if released >= 1: + break + if time.monotonic() > deadline: + raise SystemExit(f"the job was not released from {point}; see {dep.log_path}") + time.sleep(0.01) + os.remove(release) + if released >= 2: + raise SystemExit(f"the second reach of {point} was released before the file was removed; raise --items") + _say(f"{dep.backend}/{point}: first reach released; waiting for the second reach") + + +def _log_has(dep: Deployment, needle: str, timeout: float = 20) -> str | None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + with open(dep.log_path, errors="replace") as f: + for line in f: + if needle in line: + return line + time.sleep(0.25) + return None + + +def _tamper_a_data_file(dep: Deployment, arn: str) -> str: + """Corrupt one data file's bytes under the store root so its sha256 no + longer matches the manifest, which fails the restore in its verification + pass before any item is written (5.3 step 3). Returns the file path.""" + import glob + + files = sorted(glob.glob(os.path.join(dep.root, "**", "data", "*.json.gz"), recursive=True)) + if not files: + raise SystemExit(f"no data files under {dep.root} to tamper") + victim = files[-1] + with open(victim, "r+b") as f: + f.seek(0, os.SEEK_END) + size = f.tell() + f.seek(size // 2) + b = f.read(16) + f.seek(size // 2) + f.write(bytes((x ^ 0xFF) for x in b)) + _say(f"tampered {os.path.relpath(victim, dep.root)}") + return victim + + +def _matrix_module(): + """`devtools/backup-restore-matrix` imported as a module, for the prefix + copy and the `backup_index` seed a Cassandra target needs.""" + import importlib.machinery + import importlib.util + + path = os.path.join(_repo(), "devtools", "backup-restore-matrix") + # The script has no `.py` suffix, so the loader is named explicitly. + loader = importlib.machinery.SourceFileLoader("backup_restore_matrix", path) + spec = importlib.util.spec_from_loader("backup_restore_matrix", loader) + module = importlib.util.module_from_spec(spec) + saved = sys.argv + sys.argv = ["backup-restore-matrix"] + try: + spec.loader.exec_module(module) + finally: + sys.argv = saved + return module + + +def _adopt_backup(source: Deployment, target: Deployment, arn: str) -> None: + """Copy the source's backup prefix under the target's root and seed the + target's `backup_index` row, the matrix's stand-in for `backup sync + --adopt-store`.""" + import json + import types + + mx = _matrix_module() + src_prefix = mx.prefix_of(source.root, arn) + dst_prefix = mx.prefix_of(target.root, arn) + mx.copy_prefix(src_prefix, dst_prefix, target.root) + with open(os.path.join(dst_prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + + def store_id() -> str: + with open(os.path.join(target.root, "extenddb-store.json")) as f: + return json.load(f)["store_id"] + + adapter = types.SimpleNamespace(backend=target.backend, root=target.root, work=target.work, + tag=target.tag, store_id=store_id) + mx.seed_row(adapter, arn, manifest, manifest["backup_size_bytes"], manifest["source_table"]["item_count"]) + _say(f"backup {arn} adopted by the {target.backend} target (prefix copied, backup_index row seeded)") + + +def _remove_target(dep: Deployment, ddb, target: str, arn: str) -> None: + """The operator's remedy for a `target_not_empty` target: `extenddb backup + remove-target --account
    ` run beside the live server (MongoDB + and Cassandra take the maintenance claim; SQLite would need the server + stopped), its output asserted, then the target gone (`DescribeTable` + ResourceNotFound) and `DeleteBackup` of the backup allowed.""" + from botocore.exceptions import ClientError + + # DeleteTable's refusal shows the stuck state the command is for. + try: + ddb.delete_table(TableName=target) + raise AssertionError("DeleteTable of the target_not_empty target was accepted") + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceInUseException", e.response["Error"] + with_ = [dep.bin, "backup", "remove-target", "--account", ACCOUNT, target, "--config", dep.config] + r = subprocess.run(with_, cwd=dep.work, capture_output=True, text=True, timeout=300) + with open(os.path.join(dep.work, "remove-target.out"), "w") as f: + f.write(r.stdout + r.stderr) + assert r.returncode == 0, f"remove-target exited {r.returncode}: {r.stdout}{r.stderr}" + for needle in REMOVE_TARGET_LINES(target): + assert needle in r.stdout, f"remove-target output lacks {needle!r}:\n{r.stdout}{r.stderr}" + try: + ddb.describe_table(TableName=target) + raise AssertionError("the target is still described after remove-target") + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + ddb.delete_backup(BackupArn=arn) + _say(f"{dep.backend}: remove-target removed {target}; DescribeTable ResourceNotFound, DeleteBackup allowed" + f" ({' | '.join(l for l in r.stdout.splitlines() if l.strip())[:300]})") + + +def REMOVE_TARGET_LINES(target: str) -> list[str]: + """The lines `backup remove-target` prints for a removed FAILED target.""" + return [ + "=== extenddb backup remove-target ===", + 'Taking the maintenance claim as "extenddb backup remove-target on ', + f"removed the restore target {target} (account {ACCOUNT}, table id ", + "was FAILED (target_not_empty)", + "The maintenance claim is released.", + ] + + +def _run_restore_point(backend: str, point: str, binary: str, items: int, work: str, + second_reach: bool = False, source_bin: str | None = None, + remove_target: bool = False, vector: bool = False) -> int: + """One restore kill point on one backend. Returns 0 on pass, 1 on a + failed assertion, 2 on setup error. With `second_reach` the SIGKILL is + taken at the point's second reach (module comment). On Cassandra the + backup is taken on a SQLite source (`source_bin`) and adopted by the + target (module comment). With `remove_target` a first-reach run of the + two no-progress points goes on to `extenddb backup remove-target` and + asserts the target gone and the backup deletable. With `vector` the + source table carries a vector index and the tamper cell on PostgreSQL + asserts the removed target's data table and vector relation gone.""" + import boto3 # noqa: F401 + from botocore.exceptions import ClientError + + shutil.rmtree(work, ignore_errors=True) + os.makedirs(work, exist_ok=True) + dep = Deployment(backend, binary, work) + big_file = point in ("restore_sqlite_mid_batch", "restore_cassandra_before_progress") + if second_reach and point in ("sigterm", "pg_terminate_backend", "restore_fenced_out"): + raise SystemExit(f"--second-reach does not apply to {point}") + if vector and (backend != "postgres" or point != "restore_between_failed_and_removal"): + raise SystemExit("--vector applies to restore_between_failed_and_removal on postgres") + source = None + if backend == "cassandra": + if not source_bin: + raise SystemExit("a Cassandra kill point needs --source-bin (a SQLite build) for its backup") + source = Deployment("sqlite", os.path.abspath(source_bin), os.path.join(work, "source")) + rc = 0 + try: + if source is not None: + source.init() + source.start("source.log") + source.provision() + _, name, arn = _create_table_and_backup(source, items, big_file) + source.stop() + _say(f"source (sqlite) backup {arn} of {items} items taken and the source stopped") + dep.init() + if point == "sigterm": + dep.start("serve.log") + elif point == "pg_terminate_backend": + # Not a pause point of its own: the claim session is terminated + # while the job is paused after its first file's commit. + dep.start("serve-paused.log", pause="restore_after_file_commit") + else: + dep.start("serve-paused.log", pause=point) + dep.provision() + if source is not None: + _adopt_backup(source, dep, arn) + ddb = _ddb(dep) + else: + ddb, name, arn = _create_table_and_backup(dep, items, big_file, vector=vector) + target = f"{name}r" + if point == "restore_between_failed_and_removal": + _tamper_a_data_file(dep, arn) + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + vector_relation = None + if vector: + # The registration created the target's catalog rows; read the + # table id and the vector index id so the data relations can be + # probed once the failing restore's target is gone. + vids = _psql("SELECT t.table_id, v.index_id FROM tables t JOIN vector_indexes v ON v.table_id = t.table_id " + f"WHERE t.table_name = '{target}' AND t.account_id = '{ACCOUNT}'", database=f"{dep.tag}_catalog") + assert vids and "|" in vids, f"the target's vector index row was not found: {vids!r}" + vector_relation = tuple(vids.splitlines()[0].split("|")) + _say(f"{backend}/{point}: the target's table id {vector_relation[0]}, vector index id {vector_relation[1]}") + + if point == "sigterm": + time.sleep(0.5) + status = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + if status != "CREATING": + raise SystemExit(f"the restore finished ({status}) before SIGTERM; raise --items") + dep.sigterm() + log = open(dep.log_path, errors="replace").read() + assert "stopped by the server's shutdown" in log, log[-2000:] + assert ": FAILED (" not in log, log[-2000:] + dep.start("serve-restart.log") + dep.provision() + ddb = _ddb(dep) + deadline = time.monotonic() + 240 + while ddb.describe_table(TableName=target)["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the restore did not resume" + time.sleep(0.5) + _assert_exact(ddb, target, items, point) + _say(f"{backend}/{point}: RUNNING across restart, resumed ACTIVE with {items} items (base and GSI exact)") + return 0 + + if point == "pg_terminate_backend": + # The job is paused at restore_after_file_commit (file 0 committed, + # one restore_progress row). Terminate the session that holds the + # claim's advisory lock while the job is paused, wait past one + # heartbeat period so the timer's next beat meets the terminated + # session, then release: the job stops at its next check as a + # lost claim with no further catalog or table access, the record + # stays RUNNING with its progress row, and the same server's + # reconciler takes it over once heartbeat_at is stale (45 s after + # the last beat, so at least stale_job_after_secs after the + # termination) and finishes it. + paused_at = "restore_after_file_commit" + if not _wait_reached(dep, paused_at, timeout=300): + raise SystemExit(f"the kill point {paused_at} was not reached; see {dep.log_path}") + out = _psql("SELECT pg_terminate_backend(l.pid) FROM pg_locks l " + "JOIN pg_stat_activity a ON a.pid = l.pid " + "WHERE l.locktype = 'advisory' AND l.objsubid = 1 " + f"AND a.datname = '{dep.tag}_catalog' AND a.pid <> pg_backend_pid()", database="postgres") + terminated_at = time.monotonic() + assert out.strip() == "t", f"no claim session was terminated (psql answered {out!r})" + # The stale window counts from the owner's last confirmed beat on + # the database's clock, which precedes the termination by up to + # one heartbeat period; read that age now (no beat lands on the + # terminated session, so it stays until the takeover). + beat_age = float(_psql("SELECT extract(epoch FROM now() - heartbeat_at) FROM table_restores", + database=f"{dep.tag}_catalog")) + _say(f"{backend}/{point}: terminated the claim session (advisory objsubid=1) while paused at {paused_at}; " + f"the last beat was {beat_age:.1f} s old") + time.sleep(20) + open(os.path.join(dep.work, "release"), "w").close() + stopped = _log_has(dep, "stopped with no further catalog or table access", timeout=120) + assert stopped, "the job did not stop as a lost claim after its claim session was terminated" + _say(f"{backend}/{point}: the job stopped {time.monotonic() - terminated_at:.1f} s after the termination: " + f"{stopped.strip()[-240:]}") + rec = _psql("SELECT status, file_index, items_applied FROM table_restores", database=f"{dep.tag}_catalog") + prog = _psql("SELECT file_index, items_applied FROM restore_progress ORDER BY 1", database=f"{dep.tag}_data") + assert rec.startswith("RUNNING|"), f"the record is not RUNNING after the lost claim: {rec!r}" + assert prog, "no restore_progress row survived the lost claim" + _say(f"{backend}/{point}: record {rec!r}; progress rows {prog.replace(chr(10), ' ')!r}") + took = _log_has(dep, "taken over by this instance and resumed", timeout=180) + assert took, "the reconciler did not take the record over" + took_after = time.monotonic() - terminated_at + assert took_after + beat_age >= 45, (f"the takeover came {took_after:.1f} s after the termination and " + f"{took_after + beat_age:.1f} s after the last beat, inside stale_job_after_secs (45)") + _say(f"{backend}/{point}: takeover {took_after:.1f} s after the termination, {took_after + beat_age:.1f} s " + f"after the last beat: {took.strip()[-200:]}") + # The pause is the server's, so the successor's resume pauses at + # the same point; release it. + time.sleep(0.5) + open(os.path.join(dep.work, "release"), "w").close() + deadline = time.monotonic() + 240 + while True: + st = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + if st == "ACTIVE": + break + assert time.monotonic() < deadline, f"stuck at {st}" + time.sleep(0.5) + _assert_exact(ddb, target, items, point) + _say(f"{backend}/{point}: lost claim, record RUNNING with its progress row, takeover {took_after:.1f} s " + f"after the termination, resumed ACTIVE with {items} items (base and GSI exact)") + return 0 + + if point == "restore_fenced_out": + # The job pauses after loading every file, with its heartbeat + # timer stopped (restore.rs). The claim session is terminated so + # the advisory lock frees (on PostgreSQL the lock, not the stale + # heartbeat alone, is what the reconciler waits on, 4(e)); the + # same server's reconciler then takes the record over on a free + # permit (max_concurrent_jobs = 2) once the heartbeat is stale and + # finishes the restore. Releasing the paused job last, its own + # DONE write matches no record (the successor's generation fenced + # it out) and it makes no table access. + if not _wait_reached(dep, point): + raise SystemExit(f"the kill point {point} was not reached; see {dep.log_path}") + _say(f"{backend}/{point}: reached; terminating the claim session so its lock frees") + _psql("SELECT pg_terminate_backend(l.pid) FROM pg_locks l " + "JOIN pg_stat_activity a ON a.pid = l.pid " + "WHERE l.locktype = 'advisory' AND l.objsubid = 1 " + f"AND a.datname = '{dep.tag}_catalog' AND a.pid <> pg_backend_pid()", database="postgres") + took = _log_has(dep, "taken over by this instance and resumed", timeout=120) + assert took, "the reconciler did not take the fenced-out record over" + # The server's pause is global, so the successor's resume job also + # pauses at this point with its own heartbeat stopped; release now + # so it finishes. The fenced original then proceeds to its own DONE + # write, which matches no record (the successor bumped the + # generation on takeover) and makes no table access. + open(os.path.join(dep.work, "release"), "w").close() + deadline = time.monotonic() + 240 + while ddb.describe_table(TableName=target)["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the successor did not finish the restore" + time.sleep(0.5) + _assert_exact(ddb, target, items, f"{point} (the successor's table)") + line = _log_has(dep, "no table access was made", timeout=30) + assert line, "the released fenced-out job did not log its NotMatched DONE write" + _say(f"{backend}/{point}: successor finished ACTIVE with {items} items; " + f"the released job's DONE write matched no record ({line.strip()[-80:]})") + return 0 + + # The SIGKILL points: wait for the line, SIGKILL, restart, assert. + if not _wait_reached(dep, point, timeout=300): + raise SystemExit(f"the kill point {point} was not reached; see {dep.log_path}") + if second_reach: + _release_first_reach(dep, point) + if not _wait_reached(dep, point, timeout=300, times=2): + raise SystemExit(f"the kill point {point} was not reached a second time; see {dep.log_path}") + with open(dep.log_path, errors="replace") as f: + released = sum(1 for line in f if f"released from kill point {point}" in line) + if released >= 2: + raise SystemExit(f"the second reach of {point} was released before the file was removed; raise --items") + _say(f"{backend}/{point}: reached a second time; SIGKILL") + else: + _say(f"{backend}/{point}: reached; SIGKILL") + dep.sigkill() + dep.start("serve-restart.log") + dep.provision() + ddb = _ddb(dep) + # The FAILED-target duty waits out the stale window (45 s) before it + # removes the target, so give it more than one window. + deadline = time.monotonic() + 240 + + # These two points pause before the first file's/batch's progress is + # recorded, so the kill leaves inserted items with no recorded + # progress. On resume the engine correctly applies the no-progress + # safety rule (4(e)/5.3 step 7): a non-empty target with no progress + # is FAILED `target_not_empty` and left for the operator, never blindly + # re-applied. (The "re-apply the one unrecorded file/batch with + # suppression" path needs an EARLIER recorded file, which a one-shot + # pause on the first reach cannot set up; `--second-reach` runs it.) + if point in ("restore_mongodb_after_insert_many", "restore_cassandra_before_progress") and not second_reach: + line = _log_has(dep, "target_not_empty", timeout=120) + assert line, f"{point}: the resume did not apply the target_not_empty rule" + st = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + assert st == "CREATING", f"{point}: target is {st}, expected it left CREATING for the operator" + _say(f"{backend}/{point}: resume found no progress and a non-empty target -> " + f"FAILED target_not_empty, target left for the operator ({line.strip()[-90:]})") + if remove_target: + _remove_target(dep, ddb, target, arn) + return 0 + + want_gone = point == "restore_between_failed_and_removal" + while True: + try: + st = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + st = "GONE" + if want_gone and st == "GONE": + break + if not want_gone and st == "ACTIVE": + break + assert time.monotonic() < deadline, f"stuck at {st} for {point}" + time.sleep(0.5) + if not want_gone: + _assert_exact(ddb, target, items, point) + _say(f"{backend}/{point}{' (second reach)' if second_reach else ''}: resumed and ACTIVE with {items} items (base and GSI exact)") + if second_reach: + for needle in ("taken over by this instance and resumed", "resumed from data file", ": ACTIVE, "): + line = _log_has(dep, needle, timeout=5) + if line: + _say(f" {line.strip()[-200:]}") + if point == "restore_postgres_before_progress_delete": + line = _log_has(dep, "restore progress row", timeout=20) + _say(f"{backend}/{point}: {line.strip() if line else 'progress rows swept (no line seen)'}") + else: + line = _log_has(dep, "target", timeout=5) + _say(f"{backend}/{point}: FAILED target removed after the stale window (target gone)") + if vector_relation is not None: + table_id, index_id = vector_relation + left = _psql(f"SELECT to_regclass('\"_ddb_{table_id}\"') IS NOT NULL, " + f"to_regclass('\"_ddb_vec_{index_id}\"') IS NOT NULL", database=f"{dep.tag}_data") + assert left == "f|f", f"the removed target left a data relation behind (data table, vector relation exist: {left!r})" + _say(f"{backend}/{point} (vector): the data table _ddb_{table_id} and the vector relation " + f"_ddb_vec_{index_id} are gone with the target") + return 0 + except AssertionError as e: + print(f"KILL-POINT FAILED {backend}/{point}: {e}") + return 1 + except SystemExit as e: + print(f"KILL-POINT SETUP {backend}/{point}: {e}") + return 2 + finally: + try: + dep.teardown() + except Exception as e: # noqa: BLE001 + print(f"teardown {backend}/{point}: {e}") + if source is not None: + source.stop() + _say(f"logs under {work}") + + +def _restore_main(argv) -> None: + p = argparse.ArgumentParser(prog="kill-points.py restore") + p.add_argument("backend", choices=["sqlite", "postgres", "mongodb", "cassandra"]) + p.add_argument("point") + p.add_argument("--bin", required=True) + p.add_argument("--items", type=int, default=12000) + p.add_argument("--work", required=True) + p.add_argument("--second-reach", action="store_true", + help="SIGKILL at the point's second reach (after an earlier progress record)") + p.add_argument("--source-bin", help="a SQLite build that takes the backup a Cassandra target restores") + p.add_argument("--remove-target", action="store_true", + help="after a first-reach target_not_empty outcome, run `extenddb backup remove-target` and assert the target gone") + p.add_argument("--vector", action="store_true", + help="give the source table a vector index; restore_between_failed_and_removal on postgres then asserts the removed target's vector relation gone") + a = p.parse_args(argv) + if a.point not in RESTORE_POINTS: + raise SystemExit(f"unknown point {a.point}; known: {', '.join(RESTORE_POINTS)}") + if a.backend not in RESTORE_POINTS[a.point]: + raise SystemExit(f"{a.point} does not apply to {a.backend}") + sys.exit(_run_restore_point(a.backend, a.point, os.path.abspath(a.bin), a.items, os.path.abspath(a.work), + second_reach=a.second_reach, source_bin=a.source_bin, + remove_target=a.remove_target, vector=a.vector)) + + +# Dispatch `kill-points.py restore ...` to the restore orchestrator, every +# other subcommand to PR 8's API helper. +if __name__ == "__main__": + if len(sys.argv) > 1 and sys.argv[1] == "restore": + _restore_main(sys.argv[2:]) + else: + main() diff --git a/devtools/backup-kill-points.sh b/devtools/backup-kill-points.sh new file mode 100755 index 000000000..1c1f772fc --- /dev/null +++ b/devtools/backup-kill-points.sh @@ -0,0 +1,284 @@ +#!/usr/bin/env bash +# The backup job's kill points, at the binary level (design +# `docs/design/15-backup-restore.md`: the PR 8 acceptance and the testing +# summary). One deployment per backend, a server built with +# `cargo build --release --features extenddb-app/backup-test-hooks` (SQLite +# and S3 points: `--no-default-features --features sqlite,extenddb-app/backup-test-hooks`), +# `reconcile_interval_secs = 2`, `stale_job_after_secs = 45`, a 100k-item +# composite-key table with two GSIs. Run from a checkout; the API half is +# `devtools/backup-kill-points.py` beside this script. +# +# devtools/backup-kill-points.sh [after_data_file|before_manifest|after_manifest|sigterm ...] +# +# Per point: `after_data_file`, `before_manifest` and `after_manifest` set +# EXTENDDB_BACKUP_PAUSE on the server, run CreateBackup, wait for the "reached +# kill point" line, `kill -9` the server by its PID, and restart it without +# the pause; then the row and the prefix are watched through `extenddb backup +# list` and the store. +# after_data_file: no manifest under the prefix after the kill; the row +# CREATING, then FAILED (SQLite: the first cycle, the slot rule; PostgreSQL: +# after the 45 s window), then the prefix deleted (once the row's heartbeat +# is stale, within the window plus a cycle); DescribeBackup BackupNotFound. +# before_manifest: the three documents (manifest-files, manifest-summary, +# the sha256 sidecar) and the data file under the prefix after the kill +# and no manifest; then as after_data_file (FAILED `owner dead`, the +# prefix deleted once the heartbeat is stale): a prefix without +# `extenddb-manifest.json` is never resolved AVAILABLE, whatever else is +# under it. +# after_manifest: the row AVAILABLE after the cycle with the manifest's +# size and count, the prefix intact, DescribeBackup AVAILABLE. +# sigterm: no pause; a larger table (SIGTERM_ITEMS, default 1,000,000); +# SIGTERM while the row is CREATING; the row FAILED `shutdown`, the prefix +# empty, `drained` in the log; on the S3 store no multipart upload left; +# the exit is timed by polling /proc/ at 50 ms. +# Every run is recorded under $LOGS_DIR/// (LOGS_DIR defaults +# to discussions/kill-points under the checkout, which git ignores). +# +# Environment: EXTENDDB_BIN (the server binary built with the hooks; default +# target/release/extenddb under the checkout), ITEMS (default 100000), +# SIGTERM_ITEMS (default 1000000), LOGS_DIR, PG_PORT (default 55432) and +# S3_ENDPOINT (default http://127.0.0.1:9200, a moto server), the services +# the points use. +# `A && say ... || fail ...`: `say` never fails, so the `fail` branch runs only +# when the check before it fails. +# shellcheck disable=SC2015 +set -u +BACKEND=${1:?backend}; shift +POINTS=("$@"); [ ${#POINTS[@]} -eq 0 ] && POINTS=(after_data_file before_manifest after_manifest sigterm) +HERE=$(cd "$(dirname "$0")" && pwd) +REPO=$(cd "$HERE/.." && pwd) +LOGS=${LOGS_DIR:-$REPO/discussions/kill-points}/$BACKEND +ITEMS=${ITEMS:-100000} +SIGTERM_ITEMS=${SIGTERM_ITEMS:-1000000} +PG_PORT=${PG_PORT:-55432} +S3_ENDPOINT=${S3_ENDPOINT:-http://127.0.0.1:9200} +case $BACKEND in + sqlite|s3|postgres) ;; + *) echo "backend must be sqlite, postgres or s3"; exit 2 ;; +esac +BIN=${EXTENDDB_BIN:-$REPO/target/release/extenddb} +[ -x "$BIN" ] || { echo "build $BIN with --features extenddb-app/backup-test-hooks first (or set EXTENDDB_BIN)"; exit 2; } +"$BIN" version 2>/dev/null | grep -q "backup-test-hooks enabled" || { echo "$BIN was not built with --features extenddb-app/backup-test-hooks"; exit 2; } +KP="python3 $HERE/backup-kill-points.py" +export EXTENDDB_REPO=$REPO +export S3_ENDPOINT +export AWS_ACCESS_KEY_ID=testing AWS_SECRET_ACCESS_KEY=testing AWS_REGION=us-east-1 AWS_EC2_METADATA_DISABLED=true +export EXTENDDB_ADMIN_PASSWORD='TestPass1!' + +say() { echo "[$(date +%H:%M:%S)] $*"; } +# The points run in pipelines (subshells), so a failure is a FAIL line in the +# point's log, which the end of the script greps for. +fail() { say "FAIL: $*"; } + +# --- one deployment per point -------------------------------------------------- +# $1 = point; sets WORK, PORT, ROOT (filesystem) or BUCKET (s3), TAG +deploy() { + local point=$1 + WORK=$LOGS/$point/work; rm -rf "$WORK"; mkdir -p "$WORK"; cd "$WORK" || exit 1 + PORT=$((19000 + RANDOM % 900)) + TAG="kp$(date +%s | tail -c 6)" + case $BACKEND in + sqlite|s3) + "$BIN" init --backend sqlite --sqlite-path "$WORK/db.sqlite" --config extenddb.toml > init.log 2>&1 || { cat init.log; exit 1; } ;; + postgres) + "$BIN" init --pg-host 127.0.0.1 --pg-port "$PG_PORT" --pg-user postgres --pg-pass devpass \ + --data-db "${TAG}_data" --catalog-db "${TAG}_catalog" --extenddb-user "${TAG}_app" --config extenddb.toml > init.log 2>&1 || { cat init.log; exit 1; } ;; + esac + python3 -c "import sys; sys.path.insert(0, '$REPO/tests'); import lifecycle_helpers as lh; lh._patch_config_port('extenddb.toml', $PORT)" + printf '\n[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\n' >> extenddb.toml + case $BACKEND in + sqlite) ROOT=$WORK/backups ;; + postgres) ROOT=$WORK/backups; mkdir -m 0700 "$ROOT" + printf '[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "%s"\n' "$ROOT" >> extenddb.toml ;; + s3) BUCKET="kp-$point-$(date +%s | tail -c 6)" + python3 -c "import boto3; boto3.client('s3', endpoint_url='$S3_ENDPOINT', region_name='us-east-1').create_bucket(Bucket='$BUCKET')" + printf '[backup.store]\nkind = "s3"\n[backup.store.s3]\nbucket = "%s"\nendpoint = "%s"\nforce_path_style = true\nregion = "us-east-1"\n' "$BUCKET" "$S3_ENDPOINT" >> extenddb.toml ;; + esac + export EXTENDDB_TEST_ENDPOINT="https://127.0.0.1:$PORT" +} + +# start [env assignments...] +start() { + local log=$1; shift + env "$@" "$BIN" serve --foreground --config extenddb.toml > "$WORK/$log" 2>&1 & + SPID=$! + for _ in $(seq 1 100); do curl -sk "$EXTENDDB_TEST_ENDPOINT/health" > /dev/null 2>&1 && break; sleep 0.2; done + curl -sk "$EXTENDDB_TEST_ENDPOINT/health" > /dev/null || { fail "server did not start"; tail -5 "$WORK/$log"; kill -9 "$SPID" 2>/dev/null; exit 1; } + say "server pid $SPID on $PORT ($log)" +} + +provision() { + eval "$(cd "$REPO" && EXTENDDB_ADMIN_USER=admin EXTENDDB_CA_CERT=$HOME/.extenddb/tls/cert.pem python3 devtools/provision-test-credentials 2> "$WORK/creds.err")" || { cat "$WORK/creds.err"; exit 1; } + "$BIN" settings --config extenddb.toml set control_plane_delay_seconds 0.05 > /dev/null 2>&1 +} + +# The row as `backup list` prints it (status, size, count, reason). +row() { "$BIN" backup list --config extenddb.toml 2>/dev/null | grep -F "$1" | head -1; } +row_failed() { "$BIN" backup list --failed --config extenddb.toml 2>/dev/null | grep -F "$1" | head -1; } +row_status() { local r; r=$(row "$1"); [ -n "$r" ] && echo "${r%% *}" || { r=$(row_failed "$1"); [ -n "$r" ] && echo "${r%% *}" || echo GONE; }; } + +# The objects under the backup's prefix, one key per line. +prefix_keys() { + local arn=$1 table=$2 id=${1##*/} + case $BACKEND in + sqlite|postgres) [ -d "$ROOT/123456789012/$table/$id" ] && (cd "$ROOT/123456789012/$table/$id" && find . -type f | sed 's|^\./||' | sort) ;; + s3) python3 - "$BUCKET" "123456789012/$table/$id/" <<'EOF' +import boto3, os, sys +s3 = boto3.client("s3", endpoint_url=os.environ["S3_ENDPOINT"], region_name="us-east-1") +for o in s3.list_objects_v2(Bucket=sys.argv[1], Prefix=sys.argv[2]).get("Contents", []): + print(o["Key"][len(sys.argv[2]):]) +EOF + ;; + esac +} +manifest_json() { + local arn=$1 table=$2 id=${1##*/} + case $BACKEND in + sqlite|postgres) cat "$ROOT/123456789012/$table/$id/extenddb-manifest.json" ;; + s3) python3 -c "import boto3,sys; print(boto3.client('s3', endpoint_url='$S3_ENDPOINT', region_name='us-east-1').get_object(Bucket='$BUCKET', Key='123456789012/$table/$id/extenddb-manifest.json')['Body'].read().decode())" ;; + esac +} +multipart_uploads() { + python3 -c "import boto3; print(len(boto3.client('s3', endpoint_url='$S3_ENDPOINT', region_name='us-east-1').list_multipart_uploads(Bucket='$BUCKET').get('Uploads', [])))" +} + +wait_status() { # arn expected timeout + local arn=$1 want=$2 t=$3 s + for _ in $(seq 1 "$t"); do s=$(row_status "$arn"); [ "$s" = "$want" ] && { echo "$s"; return 0; }; sleep 1; done + echo "$s"; return 1 +} + +stop_server() { kill -TERM "$SPID" 2>/dev/null; wait "$SPID" 2>/dev/null; } +teardown() { + case $BACKEND in + postgres) PGPASSWORD=devpass psql -h 127.0.0.1 -p "$PG_PORT" -U postgres -q -c "DROP DATABASE IF EXISTS ${TAG}_data" -c "DROP DATABASE IF EXISTS ${TAG}_catalog" -c "DROP ROLE IF EXISTS ${TAG}_app" > /dev/null 2>&1 ;; + s3) python3 - "$BUCKET" <<'EOF' +import boto3, os, sys +s3 = boto3.client("s3", endpoint_url=os.environ["S3_ENDPOINT"], region_name="us-east-1") +b = sys.argv[1] +for o in s3.list_objects_v2(Bucket=b).get("Contents", []): s3.delete_object(Bucket=b, Key=o["Key"]) +for u in s3.list_multipart_uploads(Bucket=b).get("Uploads", []): s3.abort_multipart_upload(Bucket=b, Key=u["Key"], UploadId=u["UploadId"]) +s3.delete_bucket(Bucket=b) +EOF + ;; + esac + rm -rf "$WORK/db.sqlite" "$WORK/db.sqlite-wal" "$WORK/db.sqlite-shm" "$WORK/backups" "$WORK/backup-spool" "$WORK/run" +} + +# --- the points --------------------------------------------------------------- +run_pause_point() { # after_data_file | before_manifest | after_manifest + local point=$1 table=kp_$1 + deploy "$point" + start serve-paused.log EXTENDDB_BACKUP_PAUSE="$point" + provision + $KP create-table "$table" --items "$ITEMS" 2>> "$WORK/load.log" || { fail "load"; stop_server; teardown; return; } + local arn; arn=$($KP create-backup "$table" "$point") || { fail "CreateBackup"; stop_server; teardown; return; } + say "backup $arn" + for _ in $(seq 1 600); do grep -q "reached kill point $point" "$WORK/serve-paused.log" && break; sleep 0.1; done + grep -q "reached kill point $point" "$WORK/serve-paused.log" || { fail "the kill point was not reached"; stop_server; teardown; return; } + say "reached kill point $point; row before the kill: $(row_status "$arn")" + kill -9 "$SPID"; wait "$SPID" 2>/dev/null + say "SIGKILLed $SPID" + prefix_keys "$arn" "$table" > "$WORK/prefix-after-kill.txt" + say "prefix after the kill: $(tr '\n' ' ' < "$WORK/prefix-after-kill.txt")" + case $point in + after_data_file) + grep -q "extenddb-manifest.json" "$WORK/prefix-after-kill.txt" && fail "a manifest exists after the kill at after_data_file" + grep -q "^data/000001.json.gz$" "$WORK/prefix-after-kill.txt" || fail "the data file is not under the prefix" ;; + before_manifest) + grep -q "^extenddb-manifest.json$" "$WORK/prefix-after-kill.txt" && fail "a manifest exists after the kill at before_manifest" + for k in data/000001.json.gz manifest-files.json manifest-summary.json extenddb-manifest.sha256; do + grep -q "^$k$" "$WORK/prefix-after-kill.txt" || fail "$k is not under the prefix after the kill" + done + [ "$(wc -l < "$WORK/prefix-after-kill.txt")" -eq 4 ] || fail "expected the four objects before the manifest put" ;; + after_manifest) + [ "$(wc -l < "$WORK/prefix-after-kill.txt")" -eq 5 ] || fail "expected the five objects after the manifest put" ;; + esac + [ "$(row_status "$arn")" = CREATING ] || fail "the row is $(row_status "$arn") after the kill, not CREATING" + local killed_at; killed_at=$(date +%s) + start serve-restarted.log + case $point in + after_data_file|before_manifest) + local bound=8; [ "$BACKEND" = postgres ] && bound=70 + local s; s=$(wait_status "$arn" FAILED $bound) && say "FAILED after $(( $(date +%s) - killed_at )) s (bound $bound)" || fail "the row is $s after $bound s, not FAILED" + row_failed "$arn" | grep -o 'failure_reason=.*' | tee "$WORK/reason.txt" + grep -q "owner dead" "$WORK/reason.txt" || fail "the reason does not say the owner was dead" + [ "$($KP status "$arn")" = NOTFOUND ] || fail "DescribeBackup is not BackupNotFound" + # The prefix goes once the dead owner's heartbeat is stale (45 s from + # the last beat before the kill) at the following cycle. + for _ in $(seq 1 70); do [ -z "$(prefix_keys "$arn" "$table")" ] && break; sleep 1; done + [ -z "$(prefix_keys "$arn" "$table")" ] && say "prefix deleted after $(( $(date +%s) - killed_at )) s" || fail "the prefix still holds: $(prefix_keys "$arn" "$table" | tr '\n' ' ')" + grep -E "owner (was|is) dead|deleted .* object\(s\) left under the prefix" "$WORK/serve-restarted.log" | sed 's/\x1b\[[0-9;]*m//g' | cut -c1-260 ;; + after_manifest) + local bound=8; [ "$BACKEND" = postgres ] && bound=70 + local s; s=$(wait_status "$arn" AVAILABLE $bound) && say "AVAILABLE after $(( $(date +%s) - killed_at )) s (bound $bound)" || fail "the row is $s after $bound s, not AVAILABLE" + manifest_json "$arn" "$table" > "$WORK/manifest.json" + local msize mcount rsize rcount + msize=$(python3 -c "import json; m=json.load(open('$WORK/manifest.json')); print(m['backup_size_bytes'])") + mcount=$(python3 -c "import json; m=json.load(open('$WORK/manifest.json')); print(sum(f['item_count'] for f in m['data_files']))") + read -r _ _ _ rsize rcount _ <<< "$(row "$arn")" + say "manifest size=$msize count=$mcount; row size=$rsize count=$rcount" + [ "$msize" = "$rsize" ] && [ "$mcount" = "$rcount" ] && [ "$mcount" = "$ITEMS" ] || fail "the row's size and count are not the manifest's" + [ "$(prefix_keys "$arn" "$table" | wc -l)" -eq 5 ] || fail "the prefix did not survive whole" + [ "$($KP status "$arn")" = AVAILABLE ] || fail "DescribeBackup is not AVAILABLE" + $KP describe "$arn" > "$WORK/describe.json" + grep -E "owner was dead" "$WORK/serve-restarted.log" | sed 's/\x1b\[[0-9;]*m//g' | cut -c1-200 ;; + esac + stop_server + teardown +} + +run_sigterm() { + local table=kp_sigterm + deploy sigterm + start serve.log + provision + # No GSIs on this table: the point is the upload's interruption, and a + # SQLite load with two GSIs runs at a third of the speed. + $KP create-table "$table" --items "$SIGTERM_ITEMS" --gsis 0 2>> "$WORK/load.log" || { fail "load"; stop_server; teardown; return; } + local arn; arn=$($KP create-backup "$table" sigterm) || { fail "CreateBackup"; stop_server; teardown; return; } + say "backup $arn" + for _ in $(seq 1 300); do grep -q "backup $arn: started" "$WORK/serve.log" && break; sleep 0.1; done + sleep 1 + local before; before=$($KP status "$arn") + say "row before SIGTERM: $before" + kill -TERM "$SPID"; local sent; sent=$(date +%s%3N) + # The exit is timed by polling /proc/ at 50 ms (a fixed sleep before + # `wait` would report its own length as the exit time). A job that has not + # ended 5 s after SIGTERM is dumped (every thread's stack) for the record + # before the 30 s drain bound ends it. + local waited=0 + while [ -d "/proc/$SPID" ] && [ -f "/proc/$SPID/status" ] && ! grep -q "^State:.*Z" "/proc/$SPID/status" 2>/dev/null; do + sleep 0.05; waited=$(( waited + 50 )) + if [ "$waited" -eq 5000 ]; then + say "the server is still running 5 s after SIGTERM; dumping its threads to $WORK/stacks.txt" + eu-stack -p "$SPID" > "$WORK/stacks.txt" 2>&1 || true + fi + [ "$waited" -ge 40000 ] && break + done + local gone; gone=$(date +%s%3N) + wait "$SPID" 2>/dev/null + say "server exited $(( gone - sent )) ms after SIGTERM (polled at 50 ms)" + [ "$before" = CREATING ] || fail "the row was $before before SIGTERM: the backup had finished; raise SIGTERM_ITEMS" + sed 's/\x1b\[[0-9;]*m//g' "$WORK/serve.log" | grep -E "backup job\(s\)|drained|FAILED \(shutdown\)" | cut -c1-220 + grep -q "drained" "$WORK/serve.log" || fail "no 'drained' line" + grep -q "FAILED (shutdown); its objects were deleted" "$WORK/serve.log" || fail "no FAILED (shutdown) line with the objects deleted" + local s; s=$(row_status "$arn"); [ "$s" = FAILED ] && say "row FAILED" || fail "the row is $s, not FAILED" + row_failed "$arn" | grep -o 'failure_reason=.*' | tee "$WORK/reason.txt" + grep -q 'failure_reason="shutdown"' "$WORK/reason.txt" || fail "the reason is not shutdown" + local left; left=$(prefix_keys "$arn" "$table"); [ -z "$left" ] && say "prefix empty" || fail "the prefix still holds: $left" + [ "$BACKEND" = s3 ] && { local n; n=$(multipart_uploads); [ "$n" = 0 ] && say "no multipart upload left" || fail "$n multipart upload(s) left"; } + teardown +} + +mkdir -p "$LOGS" +for point in "${POINTS[@]}"; do + say "=== $BACKEND $point ===" + case $point in + after_data_file|before_manifest|after_manifest) run_pause_point "$point" 2>&1 | tee "$LOGS/$point.log" ;; + sigterm) run_sigterm 2>&1 | tee "$LOGS/sigterm.log" ;; + *) echo "unknown point $point"; exit 2 ;; + esac +done +grep -l "FAIL:" "$LOGS"/*.log 2>/dev/null && exit 1 +say "all points passed for $BACKEND" diff --git a/devtools/run-mongodb-tests b/devtools/run-mongodb-tests index f6e67930c..8c0f5db9f 100755 --- a/devtools/run-mongodb-tests +++ b/devtools/run-mongodb-tests @@ -23,6 +23,16 @@ # For tests that talk to MongoDB directly (the storage # crate's snapshot tests). Tear down with # docker rm -f extenddb-runtests-mongo-. +# --backup-store-root DIR +# Configure a filesystem backup store at DIR (created +# 0700 if missing; a fresh directory per run, since the +# store's identity file names the catalog that owns it +# and this runner initializes a new catalog each time): +# CreateBackup is then the store-backed job, which the +# backup suites run with +# EXTENDDB_TEST_BACKUP_STORE=filesystem. Without it the +# server has no [backup.store] and CreateBackup is the +# in-catalog create (EXTENDDB_TEST_BACKUP_STORE=none). # --rust-live Also run the Rust live tests of the storage-mongodb # crate against this runner's replica set. They are # `#[ignore]`d, because they need a replica set, and @@ -60,6 +70,7 @@ OUTPUT_DIR="" KEEP=false MONGO_ONLY=false RUST_LIVE=false +BACKUP_STORE_ROOT="" RUN_TESTS_ARGS=() while [[ $# -gt 0 ]]; do @@ -70,8 +81,9 @@ while [[ $# -gt 0 ]]; do --keep) KEEP=true; shift ;; --mongo-only) MONGO_ONLY=true; KEEP=true; shift ;; --rust-live) RUST_LIVE=true; shift ;; + --backup-store-root) BACKUP_STORE_ROOT="$2"; shift 2 ;; --) shift; RUN_TESTS_ARGS=("$@"); break ;; - -h|--help) sed -n '5,45p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -h|--help) sed -n '5,55p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "error: unknown option: $1" >&2; exit 1 ;; esac done @@ -253,6 +265,21 @@ level = "info" format = "pretty" TOML +if [[ -n "$BACKUP_STORE_ROOT" ]]; then + # The store's root must be owner-only (the store refuses a wider mode). + mkdir -p "$BACKUP_STORE_ROOT" + chmod 0700 "$BACKUP_STORE_ROOT" + cat >> "$CONFIG" < (required) + EXTENDDB_ADMIN_PASSWORD the password from 'extenddb init' (required) + EXTENDDB_TEST_BACKUP_STORE where the server writes its backups: filesystem, + s3 or none (required for --pytest; the Rust + integration suite's backup tests read it too) +Environment (--real-dynamodb): EXTENDDB_TEST_ENDPOINT and EXTENDDB_ADMIN_PASSWORD + unset; AWS credentials from the SDK's chain or AWS_ACCESS_KEY_ID. EOF exit 1 } @@ -187,6 +203,25 @@ validate_local_target() { echo " This is the password from '${prefix_lower} init'." exit 1 fi + + # EXTENDDB_TEST_BACKUP_STORE must say where the server writes its backups. + # The backup tests derive their expected outcome from it (the store-backed + # create backs up vector tables and is asynchronous; the in-catalog create + # refuses them), so a run cannot leave it to be guessed. + if $RUN_PYTEST; then + case "${EXTENDDB_TEST_BACKUP_STORE:-}" in + filesystem|s3|none) ;; + *) + echo "error: --${prefix_lower} --pytest requires EXTENDDB_TEST_BACKUP_STORE to be set." + echo " Set it to the server's [backup.store]: 'filesystem' or 's3' for a configured" + echo " store (a file-backed SQLite server without the section has the defaulted" + echo " filesystem root), 'none' for a server without one (kind = \"disabled\", or" + echo " PostgreSQL, MongoDB and Cassandra without the section)." + echo " Got: '${EXTENDDB_TEST_BACKUP_STORE:-}'" + exit 1 + ;; + esac + fi } validate_real_dynamodb() { @@ -244,22 +279,42 @@ fi echo "" # --- Health check (local targets only) --- +# With EXTENDDB_TEST_BACKUP_STORE at filesystem or s3 the check also waits for +# the backup store component to report healthy: /health answers 200 while the +# component is still "checking", and the backup suites assert the component +# is healthy at their start, so what is waited for here is what they assert. health_check() { local endpoint="$1" local timeout=30 local interval=1 local elapsed=0 + local want_store=false + case "${EXTENDDB_TEST_BACKUP_STORE:-}" in + filesystem|s3) want_store=true ;; + esac echo "=== Health check: ${endpoint}/health ===" while [[ $elapsed -lt $timeout ]]; do - if curl -skf -o /dev/null "${endpoint}/health" 2>/dev/null; then - echo " ✓ Server is healthy (${elapsed}s)" - return 0 + local body + if body=$(curl -skf "${endpoint}/health" 2>/dev/null) && [[ -n "$body" ]]; then + if ! $want_store; then + echo " ✓ Server is healthy (${elapsed}s)" + return 0 + fi + if printf '%s' "$body" | python3 -c 'import json, sys +h = json.load(sys.stdin) +sys.exit(0 if h.get("components", {}).get("backup_store", {}).get("status") == "healthy" else 1)' 2>/dev/null; then + echo " ✓ Server and its backup store are healthy (${elapsed}s)" + return 0 + fi fi sleep "$interval" elapsed=$((elapsed + interval)) done echo " ✗ Server health check failed after ${timeout}s" + if $want_store; then + echo " The backup store component did not report healthy; last: $(curl -sk "${endpoint}/health" 2>/dev/null)" + fi echo " Is the server running? Check: curl -sk ${endpoint}/health" return 1 } @@ -470,7 +525,12 @@ if $RUN_PYTEST; then # CLI lifecycle tests manage their own server; exclude from main suite. # tests/test_backup_store_config.py does too: it runs below, in the CLI # lifecycle list on PostgreSQL and in its own step on a SQLite build. - PYTEST_ARGS=(python3 -m pytest tests/ -v --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py --ignore=tests/test_backup_store_config.py) + # tests/test_backup_job.py starts its own servers as well; CI runs it in + # the integration workflows' backup steps, this script does not run it. + # -rP: the summary repeats the passed tests that printed something (the + # backup cells print which branch ran and which store the server reports), + # which pytest's capture would otherwise keep out of the log. + PYTEST_ARGS=(python3 -m pytest tests/ -v -rP --ignore=tests/python --ignore=tests/test_cli_lifecycle.py --ignore=tests/test_cli_container_readiness.py --ignore=tests/test_cli_migrate_concurrency.py --ignore=tests/test_cli_vector_catalog_migration.py --ignore=tests/test_gsi_async_queue.py --ignore=tests/test_backup_store_config.py --ignore=tests/test_backup_job.py) # Parallel execution (--parallel): distribute by file so module/class-scoped # fixtures stay within a single worker. if [[ -n "$PARALLEL" ]] && python3 -c "import xdist" 2>/dev/null; then diff --git a/docker-compose.cassandra.yml b/docker-compose.cassandra.yml index 1ce90a03c..d71128933 100644 --- a/docker-compose.cassandra.yml +++ b/docker-compose.cassandra.yml @@ -253,7 +253,7 @@ services: - "127.0.0.1:${EXTENDDB_PORT:-18443}:18443" volumes: - extenddb-state:/var/lib/extenddb - stop_grace_period: 15s + stop_grace_period: 80s restart: unless-stopped volumes: diff --git a/docker-compose.mongodb.yml b/docker-compose.mongodb.yml index d9514a6bb..baff12ba0 100644 --- a/docker-compose.mongodb.yml +++ b/docker-compose.mongodb.yml @@ -133,7 +133,7 @@ services: - "127.0.0.1:${EXTENDDB_PORT:-18443}:18443" volumes: - extenddb-state:/var/lib/extenddb - stop_grace_period: 15s + stop_grace_period: 80s restart: unless-stopped volumes: diff --git a/docker-compose.yml b/docker-compose.yml index b4d5a2060..6b00a1f6a 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -99,7 +99,7 @@ services: - "127.0.0.1:${EXTENDDB_PORT:-18443}:18443" volumes: - extenddb-state:/var/lib/extenddb - stop_grace_period: 15s + stop_grace_period: 80s restart: unless-stopped volumes: diff --git a/docs/design/15-backup-restore-bench-pr8.md b/docs/design/15-backup-restore-bench-pr8.md new file mode 100644 index 000000000..51a3ccaa0 --- /dev/null +++ b/docs/design/15-backup-restore-bench-pr8.md @@ -0,0 +1,481 @@ +# PR 8 memory and duration record (`bench-rss.sh`, 2026-10-31 and 2026-11-01) + +This is the record the design cites (section 7, revision item 31 of +`15-backup-restore.md`). `kill-points.py`'s API half is committed as +`devtools/backup-kill-points.py`, with the kill-point driver +`devtools/backup-kill-points.sh`; `bench-rss.sh` and the raw samples +(`/proc//status` every 200 ms per run, the servers' logs) are kept +beside the PR outside the repository, on the measuring host; +the figures below are what they produced. The runs of 2026-11-01, after the +PostgreSQL page's row bound changed, are in the section at the end and in the +sentences they correct; the runs of the same day after the row bound gained +its memory are in the last section. + +What was measured: the release binary (`cargo build --release`, the default +PostgreSQL features; `--no-default-features --features sqlite`; `--features +mongodb-test-hooks`) running `serve --foreground` with a filesystem backup +store (SQLite: the defaulted root beside the database file), sampled from +`/proc//status` every 200 ms (`VmRSS` and `VmHWM`). Each table is loaded +through the wire (`kill-points.py create-table`: BatchWriteItem from 16 +threads, items of a 200-byte `S` pad under a composite key with two GSIs), +then the server is restarted so the process's high-water mark covers the +backup alone; the idle figure is read right before `CreateBackup`, the peak +is `VmHWM` once `DescribeBackup` reports `AVAILABLE`. Samples: +`rss//{host.txt,results.txt,.json,.samples}` with `` the +backend for the record's runs and `sqlite-itemchannel1`, +`sqlite-itemchannel-swapped`, `sqlite-arena1`, `sqlite-worst24`, +`sqlite-rowbuffer4` for the comparison runs below; the series of 2026-11-01 +under `sqlite-worst-{default,arena1}/` and `postgres/`. + +Host (`host.txt`): 16 cores, 30.3 GiB, Linux 5.10.270 (amzn2int); PostgreSQL +16.15 in a container on loopback (55432); the SQLite the binary bundles +(libsqlite3-sys 0.30.1, SQLite 3.46; the host's 3.7.17 is unused); MongoDB +7.0.43 single-node replica set in a container (27117). What was not held +constant, stated before any figure here is read across runs: the host ran +other work during every run (the PostgreSQL runs +beside two 1M-item loads and a wire suite; the SQLite and MongoDB runs beside +the Cassandra integration suite and the kill-point runs), so the durations +are upper bounds and comparable only within one back-to-back pair; the peaks +are the process's own and do not depend on the host's load. Everything is on +loopback: nothing here says what a network database or a remote S3 store does +to the duration; the upload window of an S3 store (`upload_part_size_bytes * +max_concurrent_uploads`) is not in these figures, which use the filesystem +store's one write buffer. The MongoDB binary was built before the pipeline's +last change (the sink's `BrokenPipe` answer after a stop, which touches no +allocation); the PostgreSQL and SQLite figures of the record are from binaries +of the PR's first build (`ITEM_CHANNEL = 4`); the PostgreSQL page changed +after these runs (the row bound from the largest row, the rows parsed as they +arrive), so the PostgreSQL worst-shape row below is the first build's and the +section of 2026-11-01 carries what changed. + +## The bound the acceptance asserts: peak(1M) − peak(100k) < 112 MiB + +The bound the script asserts is 64 + 32 + 16 MiB (the page budget, the line +cap, the channels). MongoDB's "1M" run is 500k: its 100k load took 361 s +and the 500k load 1,604 s through the wire on this host, so a 1M load would +have run for close to an hour; the 500k run is recorded in its place and the +difference is against it. + +| backend | run | items | idle RSS | peak (VmHWM) | peak − idle | AVAILABLE after | backup size | +|---|---|---|---|---|---|---|---| +| PostgreSQL | 100k | 100,000 | 24.6 MiB | 35.9 MiB | 11.3 MiB | 1.85 s | 847,689 B | +| PostgreSQL | 1M | 1,000,000 | 24.8 MiB | 45.4 MiB | 20.6 MiB | 36.1 s | 6,452,803 B | +| PostgreSQL | 1M − 100k | | | **+9.5 MiB** | | | PASS (bound 112) | +| SQLite | 100k | 100,000 | 20.4 MiB | 65.4 MiB | 45.0 MiB | 1.44 s | 847,689 B | +| SQLite | 1M | 1,000,000 | 22.8 MiB | 73.3 MiB | 50.5 MiB | 12.45 s | 6,452,803 B | +| SQLite | 1M − 100k | | | **+7.9 MiB** | | | PASS (bound 112) | +| MongoDB | 100k | 100,000 | 26.8 MiB | 36.4 MiB | 9.6 MiB | 1.85 s | 962,185 B | +| MongoDB | 500k (in the 1M row's place) | 500,000 | 25.7 MiB | 41.2 MiB | 15.5 MiB | 8.81 s | 4,869,108 B | +| MongoDB | 500k − 100k | | | **+4.8 MiB** | | | PASS (bound 112) | + +The page budget is never reached by 200-byte rows (a page is `n` rows, at +most `DEFAULT_FETCH_BATCH` = 1000, about 250 KB of text), and the data files +roll at 64 MiB of uncompressed text without the file being held (the encoder +streams 1 MiB chunks into the put), so the peak over idle is a few tens of +MiB on every backend and moves by under 10 MiB from 100k to 1M rows. SQLite's +45 MiB over idle at 100k is the backend's own: the pool's connections each +hold SQLite's page cache, which a backup's reads fill; it is bounded by the +pool and does not grow with the table (the 1M run adds 7.9 MiB to it); the +constant is `PRAGMA cache_size = -32000` +(`crates/storage-sqlite/src/sqlite_util.rs`), 31 MiB per connection, so the +pool's bound is 10 × 31 MiB at the default pool and 34 × 31 MiB at the raised +one, and the 45 MiB is about one and a half connections' caches filled, not +the pool's. The duration scales +with the items: 36 s for 1M on PostgreSQL under a loaded host (the server's +log line says 36.1 s of job time; 1.9 s for 100k), 12.5 s on SQLite, 8.8 s +for 500k on MongoDB. + +Rows read past the cut on PostgreSQL (the `test-hooks` counter +`paging::rows_past_cut`, `crates/storage-postgres/tests/table_snapshot.rs::pages_are_cut_at_the_byte_budget`, +run with `--nocapture`): on the first build **602 rows over 77 pages of 8 +rows** with `n` = 16 for every page (the row bound from the average row, whose +floor was 16; 8 per page where the budget cut each page in half), on both +layouts; after the paging change **8 rows over 77 pages**, all on the first page (the +row bound is then the budget divided by the largest row, 8, so every later +page reads exactly the rows it keeps), asserted exactly. The SQLite stream +pulls no row past its cut (its `rows_past_cut` is 0 and asserted so by +`snapshot_engine.rs`'s test); the driver's worker reads ahead of it, which +the worst-expansion runs below measure. + +## The worst-expansion run + +`put-worst`: 1,000 small rows (`a0000`.., a 50-byte pad) and then 12 items +of #394's worst-expansion shape (`zz000`..: a list of 2,262 `NS` sets of the +90 values `1.0E-130`..`9.9E-130`; 409,429 bytes as the service sizes an +item; 27.7 MB as a data-file line and as stored item text, every number +normalized to its 133-character decimal form; about 34 MB decoded, 203,580 +`String`s of 133 bytes with their headers). The page the cut gives on such +rows is three rows (the rows whose predecessors' bytes are under 64 MiB: 0, +27.7, 55.4 MB), 83 MB of text, so 12 rows are four pages. MongoDB stores an +item as one BSON document and refuses the shape at 28.9 MB (`BSONObjectTooLarge`, +the 16 MiB cap; `rss-mongodb-worst.out`'s first attempt), so its run holds +12 items of 1,100 sets (13.5 MB lines, `--sets 1100`): there every row is a +page of its own (the server's 16 MiB reply bound) and the shape is half the +size of the other two backends'. + +| backend | build | worst rows | idle RSS | peak (VmHWM) | peak − idle | AVAILABLE after | backup size | +|---|---|---|---|---|---|---|---| +| PostgreSQL | `ITEM_CHANNEL = 4` | 12 | 24.5 MiB | **904.5 MiB** | 880 MiB | 18.2 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 4`, the 15:34 build (before the sink's `BrokenPipe` answer) | 12 | 21.1 MiB | **782.7 MiB** | 762 MiB | 8.16 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 4`, the first build (with the `BrokenPipe` answer) | 12 | 20.7 MiB | 602.6 MiB | 582 MiB | 8.97 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 1` | 12 | 20.9 MiB | 555.4 MiB | 535 MiB | 8.13 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 1`, back to back with the row above it | 12 | 20.4 MiB | 552.2 MiB | 532 MiB | 7.95 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 4`, `MALLOC_ARENA_MAX=1` | 12 | 22.0 MiB | 529.5 MiB | 507 MiB | 8.76 s | 3,793,555 B | +| SQLite | `ITEM_CHANNEL = 4` | **24** | 20.3 MiB | **919.4 MiB** | 899 MiB | 15.7 s | 7,567,821 B | +| MongoDB | `ITEM_CHANNEL = 4`, 1,100 sets | 12 | 25.0 MiB | 498.7 MiB | 474 MiB | 4.7 s | 1,851,153 B | + +The driver's run-ahead on SQLite, measured back to back on one 24-row table +(`rss/sqlite-rowbuffer4/results.txt`; the final tree sets `row_buffer_size` +to four on the pool, `crates/storage-sqlite/src/store.rs`): + +| SQLite build | worst rows | idle RSS | peak (VmHWM) | peak − idle | AVAILABLE after | +|---|---|---|---|---|---| +| `row_buffer_size` 4 (final tree) | 24 | 20.2 MiB | 716.7 MiB | 696 MiB | 15.9 s | +| `row_buffer_size` 50 (sqlx's default) | 24 | 22.5 MiB | 866.2 MiB | 844 MiB | 15.7 s | +| `row_buffer_size` 4, the 1M-row table | 1,000,000 small | 22.3 MiB | 67.3 MiB | 45.0 MiB | 11.43 s | +| `row_buffer_size` 50, the 1M-row table | 1,000,000 small | 21.1 MiB | 67.6 MiB | 46.5 MiB | 11.42 s | + +The samples show the shape of it: with 50 the RSS reaches 850 MiB within a +second of the start (the worker has read every one of the 24 rows while the +stream parses the first page's three), falls to 280 MiB when the first page's +cut drops the statement, and climbs again on the second; with four it climbs +with the pipeline (236, 292, 339, 424 MiB and on) to its peak. A 1M-row +backup takes the same 11.4 s with either, so the pool keeps four. + +The terms, by the pipeline's parts, for a row of `r` = 27.7 MB of text whose +decoded `Item` is `d` ≈ 34 MB: + +- the page's text: on the first build PostgreSQL fetched the page whole + (`fetch_all`, 83 MB as the driver's rows) and copied each row's text out + as it extracted it, about one page plus one row; after the paging change it pulls + the rows from the connection one at a time and parses each as it arrives, + so one row, as SQLite, which pulls the rows one at a time; +- the page's decoded rows (up to 3 × `d` = 102 MB) in the stream's buffer + while the pipeline takes them one by one; +- one row's parse: the text, the deserializer's intermediate value and the + item, `r + 2d` ≈ 96 MB, for the row being parsed; +- the pipeline's items: the feeder's, the item channel's (`ITEM_CHANNEL`) + and the encoder's, (2 + `ITEM_CHANNEL`) × `d`: 204 MB at four, 102 MB at + one; +- the encoder's line buffer (one line, `r`), its gzip window, the chunk + sink's 1 MiB and the chunk channel's four 1 MiB chunks; +- on SQLite, the driver's run-ahead (below). + +At the instant the stream fetches a new page while the pipeline is full the +terms sum to about 400 MB with a four-item channel and about 300 MB with a +one-item channel. The high-water marks are 1.4 to 2.2 times that, and the +difference is the allocator's: the row's 203,580 short strings are allocated +on the tokio worker that parses the row and freed on the blocking thread +that encodes it, and glibc's malloc keeps that memory in its arenas between +pages. Evidence: two builds of the same pipeline code (the table's two 12-row +rows, the 15:34 build and the first build) peaked at 762 and 582 MiB over +idle on one shape, two runs of one first-build binary at 12 rows peaked at +574 and 805 MiB (the terms do not vary; the allocator's state does), and +`MALLOC_ARENA_MAX=1` took the same run to 507 MiB. The 24-row run on SQLite +(899 MiB over idle against 582 to 762 for 12 rows) adds one term the 12-row +runs hid: sqlx's SQLite driver runs each statement on a worker thread that +buffers rows ahead of the reading task (`row_buffer_size`, 50 by default), +so while the stream parses its page's three rows the worker has already read +the rows after them, up to 50 or the table's end, whichever comes first (12 +rows in the first runs, 24 in this one, 50 × the line cap, 1.6 GB, on a +large table of such rows); the page's cut drops the statement and frees them. +`rows_past_cut` is 0 because it counts the rows the stream pulls, not the +rows the driver read. The final tree sets the +pool's `row_buffer_size` to four (the table above): the run-ahead is then at +most five rows (the four the channel holds and the one the worker has stepped +and copied while its `send` waits; 160 MiB at the line cap) and the SQLite +peak is the pipeline's terms and the allocator's share. + +The item channel. The proposal was a one-item channel to save the +three items the four-item channel holds. Measured back to back on SQLite on +one loaded 1M-row table, each build once in each order +(`rss/sqlite-itemchannel1/results.txt`, `rss/sqlite-itemchannel-swapped/results.txt`): + +| order | `ITEM_CHANNEL = 4` | `ITEM_CHANNEL = 1` | +|---|---|---| +| one-item build first (18:22) | 12.45 s | 21.85 s | +| four-item build first (18:33) | 12.24 s | 21.41 s | + +A one-item channel costs a 1M-row backup 75% of its duration (every item is +a hand-off between the feeder task and the encoder's blocking thread) and +saves about 50 MiB of the worst-shape peak (582 against 532 MiB over idle, +back to back), so `ITEM_CHANNEL` stays at four and `writer.rs` says why. + +The terms do not grow with the number of small rows (the 1M run) and grow +with the row (every term but the budget and the chunks is a multiple of `r` +or `d`), which the line cap (32 MiB) and `max_item_size_bytes` bound; the +process's peak, with glibc's default arenas, grows with the number of large +rows a job reads, and does not under `MALLOC_ARENA_MAX=1` (the series of +2026-11-01 below). Section 7 of the design states these terms with the figures; +revision item 31 carries the correction (the earlier text, "the budget plus +one row of text, plus its decoded form", named the first term alone). + +## The incompressible run + +`put-random`: 256 items each holding a 256 KiB `B` attribute of +`os.urandom` bytes (64 MiB in all; 85 MB as base64 text). + +| backend | idle RSS | peak (VmHWM) | peak − idle | AVAILABLE after | backup size | +|---|---|---|---|---|---| +| PostgreSQL | 24.6 MiB | 145.1 MiB | 120.5 MiB | 5.9 s | 68,016,482 B | +| SQLite | 22.7 MiB | 97.5 MiB | 74.8 MiB | 4.3 s | 68,016,702 B | +| MongoDB | 25.6 MiB | 115.8 MiB | 90.2 MiB | 4.9 s | 68,016,522 B | + +The backup's size (68 MB) is the data's: gzip shrank nothing. The peak over +idle is one page of text (the 85 MB fits one page: 256 rows of 342 KB, which +PostgreSQL holds whole and SQLite a row at a time) plus the decoded items +(the 64 MiB of bytes) plus the channels; the encoder did not hold the file +(a 68 MB file through a 4 MiB chunk window), which is what this run proves. + +## Durations for the record (the design's RTO baseline, PR 11) + +CreateBackup to AVAILABLE, the `DescribeBackup` poll's figure, a composite +key with two GSIs, 200-byte items, a filesystem store on the same disk, the +host loaded as said above: + +| backend | 100k items | 1M items | 500k items | +|---|---|---|---| +| PostgreSQL 16.15 (loopback, separate data and catalog databases) | 1.85 s | 36.1 s | | +| SQLite 3.46 (file) | 1.44 s | 12.45 s (12.24 and 12.45 s in the comparison runs, no GSIs) | | +| MongoDB 7.0 (loopback replica set) | 1.85 s | not run | 8.81 s | + +The worst shape (12 rows of 27.7 MB, 332 MB of text): 18.2 s on PostgreSQL +on the first build and 16.5 s after the paging change (the section of 2026-11-01), +8.0 to 9.0 s on SQLite; the incompressible 64 MiB: 5.9, 4.3 and 4.9 s. The +loads themselves (the wire, 16 threads of BatchWriteItem): 100k in 217 s on +SQLite with two GSIs and 1M in 6,531 s; 1M in 463 s without GSIs; MongoDB +100k in 361 s and 500k in 1,604 s. + +## 2026-11-01: the allocator series and the PostgreSQL page's cost + +The worst shape (1,000 small rows then N rows of 27.7 MB) in two series of +six runs, each series on one SQLite release build, each run a fresh +deployment, `bench-rss.sh sqlite worst` with `WORST_COUNT` 12, 24, 48, first +with glibc's default arenas and then with `MALLOC_ARENA_MAX=1` in the +server's environment. Series 1: the build before the feeder's line +measurement was added (it measures no line at the default item limit, so the +pipeline's allocation is the same), a quiet host, samples under +`rss-series1/rss/`. Series 2: the final build, with the kill-point runs of the +same day running beside it on the host, samples under +`rss/sqlite-worst-{default,arena1}/`; each binary's md5 is recorded beside +its samples: + +| allocator | worst rows | series 1, quiet host: idle / peak / over idle, AVAILABLE after | series 2, final build, loaded host: idle / peak / over idle, AVAILABLE after | +|---|---|---|---| +| default arenas | 12 | 24.0 / 489.6 / **466 MiB**, 8.74 s | 22.7 / 729.3 / **707 MiB**, 9.15 s | +| default arenas | 24 | 23.7 / 611.4 / **588 MiB**, 18.07 s | 20.6 / 720.4 / **700 MiB**, 17.89 s | +| default arenas | 48 | 22.4 / 782.1 / **760 MiB**, 36.56 s | 23.2 / 1304.5 / **1,281 MiB**, 35.96 s | +| `MALLOC_ARENA_MAX=1` | 12 | 22.1 / 503.3 / **481 MiB**, 9.57 s | 24.0 / 476.2 / **452 MiB**, 9.99 s | +| `MALLOC_ARENA_MAX=1` | 24 | 23.5 / 475.6 / **452 MiB**, 19.55 s | 20.6 / 500.8 / **480 MiB**, 19.32 s | +| `MALLOC_ARENA_MAX=1` | 48 | 22.0 / 501.5 / **480 MiB**, 39.45 s | 22.2 / 477.6 / **455 MiB**, 39.67 s | + +The terms of the section above at this row (`r` = 26.4 MiB, `d` ≈ 32.4 MiB: +the page's three decoded rows 97 MiB, one row's parse 91 MiB, the pipeline's +six items 195 MiB, the line buffer 26 MiB, the chunks 5 MiB, the five-row +run-ahead 132 MiB) sum to 545 MiB on SQLite. With the default arenas the peak +over idle is 0.86 to 1.39 of that sum in the first series and 1.28 to 2.35 in the +second, grows with the rows in both, and differs by 240 to 520 MiB between +the two series at the same count (runs of the first build on a loaded host +reached 1,096 MiB at 48 rows, and its 12-row runs of one binary spread over +574 to 805 MiB); under one arena it is 0.83 to 0.88 of the sum in both series and +flat from 12 to 48 rows. So the pipeline's terms do not grow with the table, +the growth and the spread under the default arenas are the allocator's +retention of the strings freed on the encoder's thread (how much it keeps +depends on the host's state at the time, which is why the host's load moves +it), and the bound the design states (section 7) is the terms at the line +cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, which the admin guide and +the deployment guide's service unit now carry; with the default arenas the +design records the growth and names 2.35 of the terms at 48 rows as the +largest measured. The duration grows with the rows as it should (the 48-row +table is 1.3 GB of text) and is the same under either setting. + +The PostgreSQL page (the probe `pg_adapt_probe.py`, kept with the scripts, +run twice with the first build's and the changed row-bound rule on a +1,000-small-then-60-worst table, `EXPLAIN (ANALYZE, BUFFERS)` of page 3): + +| row bound rule | page 2 (`n` = 1000, reads 984 small + 16 worst, keeps 987) | page 3: `n` asked | rows read / past the cut | temp written (8 KiB blocks) | page 3 time | pages 4, 5 (`n`, time) | +|---|---|---|---|---|---|---| +| the first build: budget / average row, floor 16 | 4.6 s | 810 | 57 / 54 | 757,844 (5.9 GB) | 18.2 s (43.7 s on a loaded host) | 16, 4.4 s each (13 rows past the cut) | +| the change: budget / largest row, floor 4 | 14.0 s | 4 | 4 / 1 | 26,564 (208 MiB) | 1.13 s | 4, 1.2 s each (1 row past the cut) | + +Page 2 is the over-read of this shape (it asks for the rows the small rows +justified and meets the large ones: up to `n` rows of up to the line cap); +its two timings differ by the host, not the rule, since both runs issue the +same query. On a table whose large rows are scattered among small ones that +page is not the only one: with the bound sized from the previous page alone +it recurs every other page, which the last section measures and the row +bound's memory corrects. The PostgreSQL release build with the changed page under +`bench-rss.sh postgres 100k 1m worst` with `GSIS=0` (one deployment, the +runs in that order, samples under `postgres/`): 100k, idle +23.7 MiB, peak 30.4 MiB, 1.87 s; 1M, idle 22.6 MiB, peak 34.4 MiB, 13.27 s +(peak(1M) − peak(100k) = +4.0 MiB, bound 112, PASS; the record's 36.1 s was +under two 1M loads and a wire suite); the worst shape at 12 rows, peak +790.2 MiB (`VmHWM`) after 16.5 s, against the first build's 904.5 MiB and 18.2 s, +with the page-text term gone and each large-row page reading one row past +its cut rather than thirteen (the idle before this run, 122.7 MiB, is the +restarted process's startup work on a table of 27.7 MB rows, not the 1M +load: on a fresh deployment holding only the worst table the restarted +process idled at 138 and 164 MiB in two later runs, climbing from 79 to +168 MiB within two seconds of the listener answering and before +`CreateBackup`, so the over-idle figure, 667 MiB, is not comparable with +the record's 880; the `VmHWM` is). The backup sizes of the `GSIS=0` tables (366,631 and 3,017,864 +bytes) are smaller than the record's because the items carry no GSI +attributes; the backup reads the base table either way. The plan near the end of a table is the planner's: on +the 100k-row test table the index scan holds ten rows from the end, and on a +1,060-row table a page after the third of the last 60 at `n` = 810 is a +bitmap scan of the primary key index and a `Sort` of the 57 rows after the +seek (PostgreSQL 16; `the_last_page_sorts_the_rows_after_the_seek` pins +both). + +## 2026-11-01, later: the row bound's memory, the server-side walk that was measured and not taken, the allocator on PostgreSQL + +The shape the sections above do not cover is a table whose large rows are +scattered among small ones, which is what a table whose large items are a +minority looks like (key order does not sort items by size; `put-worst` +groups its `zz` keys after the `a` keys). With the row bound sized from the +previous page alone, a page that holds a 27 MB row sets the next bound to 4, +four small rows set the one after back to 1000, and that page meets large +rows again: the cycle repeats every other page for the whole table. + +The SQL probe (`pg_memory.py`, kept with the scripts; the window-form page +query as `page_sql` writes it, a complete walk of a 1,000-row table with 30 +rows of 27 MB text at every 33rd key, PostgreSQL 16.15, `work_mem` 4 MB, +one plain run per rule, the server's `temp_bytes` for that run): + +| row bound rule | pages | rows the inner `LIMIT` read for 1,000 kept | `temp_bytes` | wall | +|---|---|---|---|---| +| the largest row of the previous page | 22 (`n` 16, 1000, 4, 1000, 4, …) | 5,555 | 13.38 GB | 71.5 s | +| the largest row of the last four pages | 50 (16, 1000, 4, 4, 4, 4, 1000, …) | 5,555 | 13.38 GB | 64.6 s | +| the largest row of the last pages holding at least 1,000 rows | 228 (16, 1000, then 4 to the end) | 1,901 | 3.89 GB | 22.4 s | + +Four pages of memory read the same rows as one: the four pages after a +large row hold 16 small rows, so the bound returns to 1000 as before and +the 1000-row page recurs every fifth page instead of every other. A memory +in rows holds the bound at 4 for a row bound's worth of rows after the last +large one, so on this table it never returns to 1000, and the one over-read +left is the first page that meets the large rows (984 rows read for 83 +kept). That is the rule the tree carries (`RowBoundMemory`, +`snapshot_engine.rs`): the bound that remains is the second page, and then +the page that follows a row bound's worth of small rows, each reading up to +`n` rows of up to the line cap, and such a page recurs at most once per row +bound's worth of small rows (on a table whose large rows are rarer than one +per thousand keys, at every large row). `the_row_bound_remembers_large_rows_among_small_ones` +pins the figure on a 300-row table with nine such rows: 201 rows read past +the cut, all on the second page, over 53 round trips (306 over 7 with one +page of memory; 103 round trips with a floor of 2). + +The release binary (`pg-interleave.sh`, kept with the scripts; one +deployment with a filesystem store, two tables of 20,000 small rows loaded +through the wire with 30 worst-expansion items, at keys `a0032`, `a0065`, +… `a0989` in one and `zz000`.. in the other, the server restarted before +each backup, the data database's `temp_bytes` during it, both binaries on +the same tables back to back; host load 3.6 to 8.6): + +| table | row bound rule (binary) | AVAILABLE after | data database `temp_bytes` | server `VmHWM` | +|---|---|---|---|---| +| interleaved | the previous page (`80f2d287`) | 72.0 s | 9.21 GB | 905 MiB | +| interleaved | the memory (`e020f82e`, the tree) | 54.6 s | 3.18 GB | 900 MiB | +| grouped | the previous page | 39.8 s | 2.85 GB | 1,143 MiB | +| grouped | the memory | 38.9 s | 2.85 GB | 1,066 MiB | + +The interleaved table's remaining 16 s and 3.2 GB over the grouped one are +the second page, whose `LIMIT 1000` reads keys 17 to 1,016, which hold all +30 large rows (3 kept, 27 past the cut), for the 83 it keeps; the grouped +table is unchanged (its one 1000-row page meets small rows). A second pair on the same host +earlier that hour: 77.4 s and 9.21 GB for the previous-page rule on the +interleaved table, 38.9 s and 2.85 GB grouped. The `VmHWM` figures are +under glibc's default arenas and vary by about 200 MiB between runs; they +are reported, not rested on. + +Measured and not taken: a page query that stops at the budget on the server, +a recursive walk of the primary key one index probe per row +(`WITH RECURSIVE … LATERAL … ORDER BY LIMIT 1`, the running total +carried from step to step, the walk ending after the row that reaches the +budget or at `n` steps), so the server serializes exactly the rows the page +returns and reads none past the cut. On the same two tables and binaries it +took the interleaved table in 38.8 s with 2.51 GB of temp and the grouped +one in 38.0 s with 2.51 GB (the three rows of each large-row page spilled +three times: the work table, the CTE's materialization, the sort), and the +first-page and middle-page plans on the six key shapes were the two index +probes with no other scan of the relation. Its cost is one index probe per +row, which a table of small rows pays on every row: the 1M-row backup +(`bench-rss.sh postgres 100k 1m worst`, `GSIS=0`, the two binaries back to +back on one loaded table) took 23.07 s against 11.85 s for the window form +(+95%; the 100k run 1.64 s; the worst shape at 12 rows 13.4 s and 658 MiB +`VmHWM` against 17.2 s and 553 MiB). The rule for the choice was a 1M-row +duration under 21 s (within 60% of 13 s), so the window form with the +memory stays; the walk's query, tests and figures are kept with the scripts +(`option1/`). A walk in chunks of a few rows per step would bound the +over-read at the chunk less one and cost a probe per chunk; it was not +measured. + +The 1M-row backup with the memory rule, the same back-to-back method: 13.69 s +against 12.06 s for the previous-page rule (`VmHWM` 36.7 against 36.3 MiB; +peak(1M) − peak(100k) = +4.1 MiB, bound 112, PASS); the rule changes no +page of a table with no large row (every page is the row bound either way), +and the tree's binary carries that day's other changes to the job's +validator and feeder as well, so the difference is the host's and theirs, +not the page's. + +What the memory costs where the previous-page rule had no over-read: after +a page holding a row at the line cap the bound stays at the floor for a row +bound's worth of rows, 250 pages of four, whatever those rows hold, so a +table whose large rows are rarer than one per row bound pays 250 round +trips per large row where one page of memory pages it at the row bound (a +1000-row page holding one 27 MB row and 999 small ones is under the budget +and kept whole). Measured on a 20,000-row table with a 27 MB row at every +1,100th key (18 large rows, loaded through the wire with `kill-points.py +put-worst --small 20000 --count 18 --interleave 1100`; the PostgreSQL +release build against a one-line build with one page of memory, the data +table's `idx_scan` counting the page queries; `pg-rare.sh`, kept with the +scripts): 2,585 page queries against 38, 25.6 and 24.9 s in two runs +against 25.1 s, 1.06 against 1.09 GB of temp, the same 5,821,368-byte +backup; the SQL mirror (`pg_probe.py`) walks the same shape in 2,508 pages +against 38 with 0 rows past the cut under either rule. On loopback the +extra round trips cost about 0.3 ms each and disappear under the large rows' +own serialization (about 1.3 s per 27 MB row); on a 1M-row table with 900 +such rows they are 125,000 round trips, about 40 s on loopback and about two +minutes on a 1 ms link, against the 20 minutes the large rows themselves +take. A bound that doubles after each page without a large row (4, 8, 16, +…) would reach the row bound after eight small pages instead of 250 with +the same rows read past the cut on the three shapes the mirror walked; it +was not taken, the cost above being bounded and the rule stated. +The worst shape at 12 rows under the default arenas: 787.8 MiB `VmHWM` in +17.7 s against 750.1 MiB in 17.5 s (the idle before the first of the two, +241.6 MiB, is the restarted process's startup work on the table of 27.7 MB +rows, the caveat of the previous section). + +The allocator on PostgreSQL (`MALLOC_ARENA_MAX=1 bench-rss.sh postgres +worst`, `GSIS=0`, the tree's binary, a fresh deployment): the worst shape +at 12 rows peaked at **339.9 MiB `VmHWM`** in 18.7 s (idle 144.2 MiB, the +startup work above; another measurement of the same shape on a build of the +same day: 346.3 MiB, 208 MiB over an idle of 138.0 MiB, 22.4 s), against +750 to 788 MiB under the default arenas. The terms of the section above on +PostgreSQL at this row are 413 MiB (545 MiB less the five-row run-ahead, +which PostgreSQL has not), so the one-arena peak is 0.82 of the terms here, +0.84 in the other measurement, under the 1.2 the design states, which was +measured on SQLite alone until now. + +What one arena costs the request path. On the SQLite release build with a +50k-item composite-key table, one boto3 client per thread, three +alternating repeats, 20 s per operation (`sqlite-ab.sh`, +`workload.py`, kept with the scripts; host load 2.4 to 5.2): GetItem at +8 threads 624, 621, 616 req/s against 605, 588, 598 under one arena +(−3.8%); Query of a 50-item partition at 8 threads 414, 407, 406 against +398, 391, 395 (−3.5%); Scan without Limit at 4 threads 47.7k, 46.7k, 46.0k +items/s against 44.2k, 42.7k, 43.9k (−6.9%); Scan with Limit 1000 over the +table 46.1k, 42.9k, 43.6k against 40.7k, 40.4k, 39.3k (−9.2%). On the +PostgreSQL release build (`pg-ab.sh`, the same workload and method, the +tree's binary, host load 2.6 to 3.8): GetItem 600, 631, 606 against 617, +623, 617 (+1.1%); Query 398, 413, 407 against 402, 401, 408 (−0.6%); Scan +46.5k, 46.2k, 47.3k against 46.1k, 45.8k, 45.0k (−2.1%); paged Scan 45.0k, +44.3k, 45.9k against 44.3k, 44.1k, 44.1k (−2.1%). glibc with one arena +serializes every allocation of the worker threads on one lock; SQLite's +request path runs in the process (the JSON parse and render per item and +the database's own work), where PostgreSQL's spends its time on the +database's round trip, which is why the setting shows on SQLite and is +within the spread between repeats on PostgreSQL but for the scans. A Python +client bounds these rates; the server's own share of the loss may be larger +under a faster client. The deployment guide's service unit carries the line +commented out, with the condition (a host that backs up tables of large +items) and this cost beside it. diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 020f068d0..51f66ecd8 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -77,7 +77,7 @@ Three store facts that rules of sections 4 and 5 rest on, and the one method a l ### 3. Storage trait changes -`SnapshotEngine` and `TableSnapshot` are `crates/storage/src/snapshot.rs` and the four `snapshot_engine.rs` files (PR 5, #395); the module docs there carry the per-backend mechanism, `snapshot.rs`'s trait docs state what `description` is and which errors end a stream, and `docs/troubleshooting.md` carries the operator side of the PostgreSQL DeleteTable change. The fetch size of every backend's stream is one shared constant, `extenddb_storage::snapshot::paging::DEFAULT_FETCH_BATCH` (1000), with a `test-hooks` override and a fetch counter the conformance check reads. +`SnapshotEngine` and `TableSnapshot` are `crates/storage/src/snapshot.rs` and the four `snapshot_engine.rs` files (PR 5, #395); the module docs there carry the per-backend mechanism, `snapshot.rs`'s trait docs state what `description` is and which errors end a stream, and `docs/troubleshooting.md` carries the operator side of the PostgreSQL DeleteTable change. The fetch size of every backend's stream is one shared constant, `extenddb_storage::snapshot::paging::DEFAULT_FETCH_BATCH` (1000), with a `test-hooks` override and a fetch counter the conformance check reads; beside it PR 8 adds the byte budget `SNAPSHOT_FETCH_BYTES` (64 MiB): on PostgreSQL and SQLite a page ends at the row bound or after the row that takes its item text to or past the budget, whichever comes first, so a page holds at least one row and at most the budget plus one row, with the cut applied in the page query on PostgreSQL (a keyset page in the snapshot's transaction, three layers with the cut in the outer one; the server reads up to `n - 1` rows past the cut, PR 8's section states the bound) and while the rows are pulled on SQLite (PR 8's section states the page and what it costs the server); a MongoDB batch is under the budget by the server's 16 MiB reply bound and applies no cut. - Cassandra: `open_table_snapshot` returns `Unsupported`. Cassandra has no `SnapshotEngine`, so the engine routes CreateBackup there to the legacy create (`LegacyBackupReader::create`, the #384 in-catalog path) whether or not `[backup.store]` is configured, the path every backend takes while no store is configured (section 7); the off-host path is never taken there. That stays until PR 14, by which time either a token-range `SnapshotEngine` with a documented non-point-in-time `SnapshotInfo.kind` (`cassandra_token_scan`) has been accepted as a design change to this section, or Cassandra backups are documented as unavailable. RFC-0003 (backend acceptance criteria) has no backup criterion today; the criterion this trait implies ("a backend that cannot fix an instant across a table returns `Unsupported` rather than a best-effort stream") is to be proposed there in a separate docs change, not in PR 5. @@ -94,13 +94,13 @@ The invariants and state machines every later PR honours, by lettered part; the (b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, per-batch progress, `failure_reason` and `completed_at` (PR 6's schema). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `restore_jobs_total{outcome="failed"}`. -(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10), and the previous owner then finds the store unhealthy and stops acting on it. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. +(c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10), and the previous owner then finds the store unhealthy and stops acting on it. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL, PR 8's review; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. (d) Identities. The ARN encodes partition, region, account, table, and backup id, and the key prefix is derived from account, table and id deterministically by `BackupArn::store_prefix`, under the configured store's prefix, and no row stores it, so restore, sync, purge and the reconciler derive it the same way and `describe_backup(arn)` works from the store alone after a sync. The engine, the writer and sync all parse and format ARNs through one implementation, `BackupArn` in `extenddb_core::backup_format` (#394); the engine keeps #384's region handling: it resolves a backup under the ARN as written and under the ARN re-spelled in its own region, and refuses a partition other than its own. `backup_creation_date_time` and `created_at` are the id's instant, so the ARN, the manifest, the API's reported time and the ListBackups cursor agree by construction. One row is addressed by `BackupKey` (`account_id`, `backup_id`, `backup_arn`): PostgreSQL, SQLite and MongoDB key `backup_index` by (`account_id`, `backup_id`), Cassandra keys `backups_by_arn` by (`account_id`, `backup_arn`); no method takes a bare backup id, and the engine treats a row whose `account_id`, `table_name` or `backup_arn` (after that re-spelling) differs from the request's as absent. One target is addressed by `RestoreTargetKey` (`account_id`, `table_name`, `table_id`), the table id guarding against a table re-created under the name. -(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the one liveness rule the three share. On PostgreSQL the advisory lock is the claim's proof against a second owner, not the liveness test: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler reads the heartbeat alone and never probes the lock. Staleness is always measured on the database's clock, never on an application host's. The reconciler takes the same claim as a worker, with a fresh `owner_generation` that fences the dead job's generation out, leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write errors with an unknown outcome, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7). A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. +(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. -(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4; this release's in-catalog create and restore check before their row, since the backends commit it and detach, revision item 29) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other. +(f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. (g) The store, the rows and sync. The store is the source of truth; the row is a cache. Sync considers only rows of the configured store (`store_kind`, `store_root`, `store_id`); rows for another store are reported, not touched, and `backup sync --forget-store ` is the explicit opt-in to drop them: it deletes that root's catalog rows outright, leaving no DELETED tombstone, so no deletion duty ever acts on that store's bytes. Sync reports a row whose manifest is gone and marks it DELETING only with `--prune-missing`; without a resolving option (PR 10) it never modifies CREATING, DELETING, DELETED or FAILED rows nor adopts their prefixes, and it never adopts a prefix without a manifest, one whose sidecar does not match, or one whose listing lacks a data file the manifest lists or holds it at another size. Neither Describe nor Restore reads the store for an ARN that has no row: a backup that exists only in the store is BackupNotFound until `backup sync` adopts it, so these rules are the only path from a prefix to an AVAILABLE row. ListBackups lists rows of another store like any other (the response shape has no marker), and Describe answers such a row by its lifecycle state as it answers any row (AVAILABLE described, DELETING and DELETED as DELETED, CREATING as CREATING, FAILED not found), with no warning: the store a row belongs to is the operator's concern (`backup list` prints every row's store), not the client's. DeleteBackup and Restore check the row's own state first, as for a row of the configured store (a CREATING row is BackupInUseException, a DELETED or FAILED row BackupNotFoundException, and for Restore a DELETING row too), and then refuse what is left, an AVAILABLE row and for DeleteBackup a DELETING one (whose deletion only its own store's server can finish), with a ValidationException saying the backup is in a store other than the configured one, without naming either root. No API error names a store root. @@ -114,12 +114,12 @@ Per operation, the contract: the request checks in their order, the error each r #### 5.1 CreateBackup -1. Validate the request as #384 does and in its order, every check before the permit: `TableName` resolved (a table ARN is accepted), the table exists and is ACTIVE for the account (`TableNotFoundException`, "Table not found: ", otherwise, whether or not a permit is free), the server's region fits a backup ARN (`check_backup_arn_fits`: a `server.region` outside `[a-z0-9-]` is a ValidationException naming the setting, since no backup ARN can be formed for any table under it). The store-prefix rules on the name (over 255 bytes, or made only of dots; `check_store_prefix_table_name`) are the store-backed writer's (PR 8), applied before it writes and naming the rule whatever the name's length: the in-catalog create writes no prefix, and the service backs up any table it can create (measured: a table named `...` backs up), so the engine applies no name rule in front of `LegacyBackupReader::create`. #384's `refuse_backup_of_vector_table` is not among the off-host path's checks (the manifest records vector indexes, section 1); the engine keeps the refusal in front of `LegacyBackupReader::create`, the path of every backend without `[backup.store]` and of Cassandra always (section 7), and the tests that change are PR 8's and PR 9a's sections. A second CreateBackup on a table that already has a CREATING backup is allowed and takes its own snapshot; the `BackupInUseException` the service models on CreateBackup is not raised. Then reserve a job slot: `max_concurrent_jobs` is a per-instance semaphore the request acquires a permit from, waiting up to 30 s for one (`JobPermit::acquire_within`, `PERMIT_WAIT`; the bound the PostgreSQL and MongoDB restore slots of earlier releases waited, and the service accepts bursts over the count: five restores and six CreateBackup calls sent together were all accepted), the maintenance claim of section 4 being the cross-instance exclusion, and the permit lives with the job; if none is freed within the bound return `LimitExceededException` before opening anything. The request holds nothing while it waits, since the permit is the last of its checks and the snapshot is opened after it; RestoreTableFromBackup and the legacy create take a permit from the same semaphore and return the same error (5.3 step 1, section 3). The default is 4 (section 7), so one running restore does not turn every other backup operation on the instance into `LimitExceededException`. -2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, so the backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` taken from the snapshot's description; then check the maintenance claim (section 4; this release's in-catalog create checks it before its row, revision item 29): if it is held, set the row FAILED with reason `maintenance`, release the snapshot, the lock and the permit, and return `LimitExceededException`; otherwise return `BackupDetails` with BackupStatus CREATING. Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written. +1. Validate the request as #384 does and in its order, every check before the permit: `TableName` resolved (a table ARN is accepted), the table exists and is ACTIVE for the account (`TableNotFoundException`, "Table not found: ", otherwise, whether or not a permit is free), the server's region fits a backup ARN (`check_backup_arn_fits`: a `server.region` outside `[a-z0-9-]` is a ValidationException naming the setting, since no backup ARN can be formed for any table under it), and on the store-backed path the table name fits a store prefix (`check_store_prefix_fits`: a name over 255 bytes or made only of dots, the store-prefix rules, is a ValidationException naming the rule, applied before the ARN is formed so the limit is named whatever the name's length; the in-catalog create writes no prefix and applies no name rule, as the service backs up any table it can create: a table named `...` backs up, measured). #384's `refuse_backup_of_vector_table` is not among the off-host path's checks (the manifest records vector indexes, section 1); the engine keeps the refusal in front of `LegacyBackupReader::create`, the path of every backend without `[backup.store]` and of Cassandra always (section 7), and the tests that change are PR 8's and PR 9a's sections. A second CreateBackup on a table that already has a CREATING backup is allowed and takes its own snapshot; the `BackupInUseException` the service models on CreateBackup is not raised. Then, on the store-backed path, the store must be healthy (section 7: a store that is unhealthy or whose first check is pending refuses with the ValidationException naming its kind and the category, before the permit); then reserve a job slot: `max_concurrent_jobs` is a per-instance semaphore the request acquires a permit from, waiting up to 30 s for one (`JobPermit::acquire_within`, `PERMIT_WAIT`; the bound the PostgreSQL and MongoDB restore slots of earlier releases waited, and the service accepts bursts over the count: five restores and six CreateBackup calls sent together were all accepted; the maintenance claim of section 4 being the cross-instance exclusion), and the permit lives with the job; if none is freed within the bound return `LimitExceededException` before opening anything. The request holds nothing while it waits, since the permit is the last of its checks and the snapshot is opened after it; RestoreTableFromBackup and the legacy create take a permit from the same semaphore and return the same error (5.3 step 1, section 3). The default is 4 (section 7), so one running restore does not turn every other backup operation on the instance into `LimitExceededException`. +2. Open the table snapshot (`SnapshotEngine::open_table_snapshot`) in the request. The pin happens during the request, after the request's checks and after any wait for a permit in step 1 (up to 30 s), so the backup is the table at the open: it holds every write acknowledged before the open and none acknowledged after it, a write acknowledged while the request waited for a permit included (measured in PR 8's second review round: a `PutItem` acknowledged 1.01 s after a `CreateBackup` that waited for a permit was in the backup, 2 of 2 runs; the service, measured in PR 8's review, held every write acknowledged before `CreateBackup` was sent and the 18 acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed: an instant inside the request there too). Mint the backup id from the millisecond the open returned, just after the pin (`format_backup_id`), and the owner generation, then, through `BackupCatalog::claim_backup` (section 3), insert the catalog row with status CREATING, `created_at` = the id's instant, `owner_generation`, `heartbeat_at` = the database's `now()`, and `source_table_details_json` the manifest's `source_table` object built from the snapshot's description and the table's TTL, which is read beside the open and not under the snapshot (`extenddb_storage::snapshot::source_table_schema`; its `item_count` and `table_size_bytes` are the description's figures, the stats pass's, until the AVAILABLE write replaces the object with the manifest's, whose count is the stream's and whose size is the DynamoDB sizes of the items written, summed, so DescribeBackup's `ItemCount` and `TableSizeBytes` are the table's figures as last counted while CREATING and the backup's exact figures once AVAILABLE); then return `BackupDetails` with BackupStatus CREATING. The maintenance claim (section 4) is checked by the request before the permit, as the in-catalog create checks it (revision item 29), so a held claim refuses the request with `LimitExceededException` and inserts no row: a row written FAILED with reason `maintenance` for the refusal carried the insert's heartbeat, and `backup purge`'s wait counted it as a live job for the stale window (revision item 32). The open and the insert are selected against the shutdown token, so a server that is stopping neither waits on an open behind a table's lock nor inserts a row; the request's `ServiceUnavailable` for that case reaches a client only when the token is cancelled while the listener still serves, which is a registration loss (the restartable worker set stopping, `BackupRuntime::cancel_jobs`), since on SIGTERM the listener's 5 s connection drain closes the connection before `drain_then_release` cancels the jobs' token (measured: a `CreateBackup` waiting in the open at SIGTERM saw its connection closed 8.2 s after sending). Everything from the open to the hand-off in step 3 runs in a detached task whose result the request awaits, so a client that disconnects mid-request cannot leave a row whose owner was never spawned; if the row insert fails the snapshot is dropped, the lock released, the permit returned, and nothing was written. 3. The claim: section 4(e) states the rules; what the claim is on each backend, the claim connection's settings and the hand-off to the worker are PR 6's and PR 8's sections. -4. The job streams items from the snapshot into gzip data files through `DataFileEncoder`; a file closes once its `uncompressed_bytes` reaches `data_file_target_bytes` (default 64 MiB), so a file exceeds the knob by less than one line, at most `MAX_LINE_BYTES` (32 MiB, section 1), which is also the knob's minimum. Every item is checked, before it is encoded, with the item validator PR 8 adds to `extenddb-core` (`extenddb_core::validation::validate_restorable_item`, with this instance's `max_item_size_bytes`; the restore binds the same function, 5.3 step 5), so no backup that no restore can load is reported AVAILABLE; that is a regression from main for a table that already holds a value #618 let in (Drawbacks), and the reason reaches the operator only (`backup list --failed` and the log); the client sees CREATING and then BackupNotFound. The heartbeat is written on a timer (`heartbeat_interval_secs`, default 15) for as long as the job task is alive, independent of upload progress; before each put the job checks that its latest heartbeat confirmed the claim within `heartbeat_interval_secs` and starts no put otherwise; a heartbeat that does not confirm the claim, or that has not returned within `heartbeat_interval_secs`, aborts the job without further store writes, deletes included. `stale_job_after_secs` must exceed several heartbeat periods; startup refuses a value under three times the period the jobs run at, which until the knob reaches the jobs is the larger of `heartbeat_interval_secs` and the 15 s constant `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL` every backend's heartbeat loop reads (so 45 s at the least), which is what makes that detection precede a reconciler's deletion. After the last file: write `manifest-files.json`, `manifest-summary.json`, `extenddb-manifest.sha256` (the order #394's `BackupDocuments::build` doc gives); re-check the claim; then write `extenddb-manifest.json` and update the row to AVAILABLE with size and count (`JobClaim::set_status(CREATING, AVAILABLE)`, conditional on the generation and on `status = CREATING`). An update whose outcome is unknown is followed by a re-read of the row, not treated as a failure; one that matches zero rows ends the job with no store access. Store writes are not fenced by the generation; the re-check narrows the window in which a fenced-out job can still write a manifest, and the FAILED tombstone (section 4) closes what remains: sync never adopts a prefix whose row is FAILED, and the reconciler never deletes a prefix while the row's `heartbeat_at` is younger than `stale_job_after_secs` on the catalog database's clock, so a late manifest from a fenced-out job lands under a tombstone that sync will not adopt. -5. On any failure, including `SnapshotExpired`: set the row FAILED with the reason, conditional on the generation and on `status = CREATING`, so a job whose AVAILABLE update committed with its response lost cannot turn a backup a client has seen AVAILABLE into a deleted one. Only if that write reports exactly one row matched does the job stop its uploads, await its in-flight puts and `delete_prefix` the partial backup (which on S3 also aborts any upload still open under it); a write that matches zero rows, or any error on the claim connection, ends the job with no further store access of any kind, because the holder of the claim owns the prefix. If the delete fails the reconciler retries it (section 4). A backup never resumes from a cursor position, so a lost snapshot means a new CreateBackup. DescribeBackup returns BackupNotFound; ListBackups omits it. -6. Shutdown: the snapshot lives in the process and dies with it, so a cancelled job can never complete. On cancellation the worker runs step 5 with reason `shutdown`, and the server's shutdown awaits each job's failure path within the drain window before the runtime stops: runtime shutdown sends no abort for a put still in flight (section 2), so step 5's `delete_prefix` is what sweeps its upload. If the catalog is unreachable it leaves the row CREATING for the reconciler; if only the store is unreachable the row is FAILED and the reconciler deletes the prefix later. Crash: on startup and every reconciler cycle, a CREATING row is resolved when its owner is dead, by the test and the claim of section 4(e); the prefix of a FAILED row is deleted only once `heartbeat_at` is older than `stale_job_after_secs`, the delay step 4 relies on. A restarted process cannot finish a CREATING backup; AVAILABLE from an existing manifest or FAILED are the only two outcomes. +4. The job streams items from the snapshot into gzip data files through `DataFileEncoder`; a file closes once its `uncompressed_bytes` reaches `data_file_target_bytes` (default 64 MiB), so a file exceeds the knob by less than one line, at most `MAX_LINE_BYTES` (32 MiB, section 1), which is also the knob's minimum. Every item is checked, before it is encoded, with the item validator PR 8 adds to `extenddb-core` (`extenddb_core::validation::validate_restorable_item`, with this instance's `max_item_size_bytes`, hash and range key sizes and attribute name size (`RestorableLimits`); the restore binds the same function, 5.3 step 5), so no backup that no restore can load is reported AVAILABLE. An item over the default item limit has its data file line measured against the line cap through a counting writer that holds no line, and one over the cap (only possible above the default item limit) is refused through the same count-and-log path as a #618 item (`RestorableItemReason::LineTooLong`, whose reason names the cap, the line's own length being unknown once the count stops); that is a regression from main for a table that already holds a value #618 let in (Drawbacks), and the reason reaches the operator only (`backup list --failed` and the log); the client sees CREATING and then BackupNotFound. The heartbeat is written on a timer (`heartbeat_interval_secs`, default 15) for as long as the job task is alive, independent of upload progress; before each put the job checks that no loss has been raised and that its latest heartbeat confirmed the claim within two periods (one for the timer, one for the beat to return: a beat sent at the tick returns up to one period later) or that a beat is in flight and has not failed, and starts no put otherwise (the in-flight clause is for a beat parked outside its bound on SQLite's engine write lock, behind a writer holding it past two periods: there the beat waits on this process's own queue, one process serves the file and the reconciler's liveness test is the in-process slot, so a parked beat cannot coincide with a takeover; on the other backends the bound covers the whole call and the clause holds only within one period of the beat's start, the backend saying which through `JobClaim::heartbeat_parks_before_its_bound`, since a process stopped mid-beat and resumed keeps the beat in flight until its timer runs again, a window of tens of microseconds in which another owner may hold the row (the maintainer measured 64 and 85 µs of passing checks with the clause unbounded); a beat that returns Lost, fails or outlives its bound raises the loss, which the check reads first, revision item 32); the claim's insert stands for the first confirmed beat (the job takes the instant before `claim_backup` and the heartbeat's watch starts from it), so the maintenance check between the insert and the job's start counts against the two periods and a first put that comes later than two periods after the insert, with no beat in flight, is refused as a lost claim, the row left to the reconciler; a heartbeat that does not confirm the claim, or that has not returned within one period, aborts the job without further store writes, deletes included, and without a catalog write either, the claim connection being the one that failed (the row is left to the reconciler). `stale_job_after_secs` must exceed several heartbeat periods; startup refuses a value under three times the period the jobs run at, the larger of `heartbeat_interval_secs` and the 15 s constant `extenddb_storage::backup_catalog::JOB_HEARTBEAT_INTERVAL` (so 45 s at the least): the store-backed job heartbeats at that larger value, the in-catalog paths at the constant until PR 14 removes them; two periods stay inside three, so a job whose claim connection has died stops its puts before a reconciler's deletion; on MongoDB and Cassandra a process stopped across the window (SIGSTOP) resumes the put it had in flight after the takeover, which the FAILED tombstone and the FAILED-prefix retry of step 5 and section 4 cover; on PostgreSQL its claim session holds the advisory lock while it is stopped, so no takeover happens while the session lives; on resume the job either completes under its own generation or, when the stop outlasted two heartbeat periods (one, with a beat in flight at the stop), stops as a lost claim with no catalog write (the beat in flight is reported late, or the next put finds the last confirmed beat over two periods old), its session closes and the reconciler resolves the row (4(e)). After the last file: write `manifest-files.json`, `manifest-summary.json`, `extenddb-manifest.sha256` (the order #394's `BackupDocuments::build` doc gives); re-check the claim; then write `extenddb-manifest.json` and update the row to AVAILABLE with size and count (`JobClaim::set_status(CREATING, AVAILABLE)`, conditional on the generation and on `status = CREATING`). An update whose outcome is unknown is followed by a re-read of the row, not treated as a failure (4(e): one more write when the row is still CREATING under the job's generation); one that matches zero rows ends the job with no store access. Store writes are not fenced by the generation; the re-check narrows the window in which a fenced-out job can still write a manifest, and the FAILED tombstone (section 4) closes what remains: sync never adopts a prefix whose row is FAILED, and the reconciler never deletes a prefix while the row's `heartbeat_at` is younger than `stale_job_after_secs` on the catalog database's clock, so a late manifest from a fenced-out job lands under a tombstone that sync will not adopt. +5. On any failure, including `SnapshotExpired`: set the row FAILED with the reason, conditional on the generation and on `status = CREATING`, so a job whose AVAILABLE update committed with its response lost cannot turn a backup a client has seen AVAILABLE into a deleted one. The pipeline ends its puts before the FAILED write (the failing put has ended, and a stopped encoder ends the body of the one running, which the store then fails or completes; nothing is started after the failure) and the feeder stops reading at once when the uploader fails, so the write does not wait for the snapshot stream's next page, and there is no put in flight at the write; only if that write reports exactly one row matched does the job `delete_prefix` the partial backup (which on S3 also aborts any upload still open under it); a write that matches zero rows, or any error on the claim connection, ends the job with no further store access of any kind, because the holder of the claim owns the prefix. If the delete fails the reconciler retries it (section 4). A backup never resumes from a cursor position, so a lost snapshot means a new CreateBackup. DescribeBackup returns BackupNotFound; ListBackups omits it. +6. Shutdown: the snapshot lives in the process and dies with it; a job cancelled before its manifest put never completes; one cancelled after it completes AVAILABLE, since the manifest is the commit record. On cancellation the worker runs step 5 with reason `shutdown`, and the server's shutdown awaits each job's failure path within the drain window before the runtime stops: runtime shutdown sends no abort for a put still in flight (section 2), so step 5's `delete_prefix` is what sweeps its upload. If the catalog is unreachable it leaves the row CREATING for the reconciler; if only the store is unreachable the row is FAILED and the reconciler deletes the prefix later. Crash: on startup and every reconciler cycle, a CREATING row is resolved when its owner is dead, by the test and the claim of section 4(e); the prefix of a FAILED row is deleted only once `heartbeat_at` is older than `stale_job_after_secs`, the delay step 4 relies on; on a row the reconciler itself just failed that window is counted from the dead owner's last beat, so the FAILED-prefix duty of the same cycle, which runs after the dead-owner duty, deletes it as soon as that beat is stale. A restarted process cannot finish a CREATING backup; AVAILABLE from an existing manifest or FAILED are the only two outcomes. #### 5.2 DescribeBackup, ListBackups, DeleteBackup @@ -157,8 +157,8 @@ Export gains the store as a target and uses `open_table_snapshot`, which fixes t ```toml [backup] data_file_target_bytes = 67108864 # uncompressed bytes per data file; a file closes once it reaches this, so it exceeds it by less than one line (5.1 step 4); minimum 32 MiB, the line cap -heartbeat_interval_secs = 15 # job heartbeat timer (5.1 step 4); #384's restore heartbeat interval -stale_job_after_secs = 300 # heartbeat age after which a PostgreSQL, MongoDB or Cassandra job is dead (4(e)), and the delay before a reconciler deletes a dead job's prefix on every backend; at least three times the period the jobs run at, max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL), so 45 at the least until the knob reaches the jobs (5.1 step 4); a new value (#384's restore grace is 60 s) +heartbeat_interval_secs = 15 # the store-backed job's heartbeat timer, floored at JOB_HEARTBEAT_INTERVAL (15 s), which the in-catalog paths keep (5.1 step 4); #384's restore heartbeat interval +stale_job_after_secs = 300 # heartbeat age after which a PostgreSQL, MongoDB or Cassandra job is dead (4(e)), and the delay before a reconciler deletes a dead job's prefix on every backend; at least three times the period the jobs run at, max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL), so 45 at the least (5.1 step 4); a new value (#384's restore grace is 60 s) failed_row_retention_secs = 604800 # FAILED tombstones kept for the operator, 7 days reconcile_interval_secs = 60 # reconciler cycle (section 4) max_concurrent_jobs = 4 # backup and restore jobs running at once on this instance, legacy creates and restores included; a further CreateBackup or RestoreTableFromBackup waits up to 30 s for a slot and is then LimitExceeded, the text naming the bound it met; at most 1024; on PostgreSQL, pool_size and catalog_pool_size are raised to at least 2 * max_concurrent_jobs + 10 (below) @@ -169,9 +169,9 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from: a backup holds the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store), the read path, the gzip window and the manifest's data-file entries until the manifest is written; with the byte budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) the read path holds at most one page, the budget plus one row of text, plus its decoded form, and neither term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. -A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (RestoreTableFromBackup and DeleteBackup of an off-host backup, and CreateBackup once it writes to the store in PR 8; Describe and List, legacy backups and this release's in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. +A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job keys; PR 7 adds `store` and its validation) and `BackupStoreConfig` in `crates/backup-store`; defaults above are the ones they encode. @@ -185,7 +185,7 @@ Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job k ### 9. Observability -Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`; the existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the backup worker and the reconciler like the other workers. +Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`, and `/metrics` emits them under those names as JSON (`render_metrics`, `docs/design/06-component-server.md`: the endpoint is not Prometheus exposition format; the deployment guide tells operators to convert); the snake-case spellings above are the names a converter would give them, not a rendering the server does. PR 8 adds `BackupJobs` (one count per job end, `operation` = the outcome, `succeeded`, `failed` or `cancelled`), `BackupJobDuration` (the job's duration in microseconds, like every duration the collector records, with the same `operation`), `BackupBytesUploaded` (bytes put to the store), `BackupSnapshotAge` (seconds since the snapshot instant, set at each confirmed heartbeat) and `BackupStoreErrors` (`operation` = the store call that failed: the job's `put` and `delete_prefix`, and the reconciler's `get` and `list` of the manifest probe and the deleting duties and its `delete_prefix`, an absent object not counted), with `QuerySource::BackupJob`; `BackupBytesUploaded` is recorded for a job that ended AVAILABLE, and `WorkerErrorCount` under `BackupJob` for a FAILED job and not for one the shutdown cancelled (which records `BackupJobs` with `cancelled` and neither worker metric); the restore variants are PR 9a's. The existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the reconciler like the other workers and by the backup job at its end under `BackupJob`. ### 10. Documentation @@ -203,15 +203,19 @@ Which document and which row each PR changes; what each row says is in the ownin | differences doc: restore-interruption row | PR 8 | | differences doc: Import and Export rows | PR 13 | | differences doc: new row for CreateBackup of a table holding an `N` value #618 let in | PR 8 | -| limits doc: backup rows | PR 7 (the knobs) and PR 8 (the line cap and file size) | +| differences doc: where-backups-live and backup-creation rows (the store-backed, asynchronous create) | PR 8 | +| limits doc: backup rows | PR 7 (the knobs) and PR 8 (the line cap and file size, the SQLite pool raise, the backup-contents row, the "Backup store" row rewritten: CreateBackup writes to the store and is refused while the store is unhealthy) | | differences and limits docs: the override rows stay | PR 9a confirms; no edit | | admin guide: database names | PR 11 | +| admin guide: the `[backup]` rows (`data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs` with the memory figures) and the `[backup.store]` section (the store-backed create, the job's two log lines, the unhealthy-store refusal) | PR 8 | +| deployment guide: the service unit's `MALLOC_ARENA_MAX=1` line | PR 8 | +| `docs/design/README.md` row 15b (the measurement record) | PR 8 | | usage guide: "backups not supported" line | PR 8 | -| upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 | +| upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 (and PR 8 for the store-backed backups a restored catalog no longer lists) | | upgrade manual: CreateBackup without `[backup.store]`, `kind = "disabled"`, the SQLite default root, shared store roots and `--adopt-store` | PR 7 (and PR 10 for `--adopt-store`) | | upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 | | upgrade manual: PR 14's migration refusal, `--force-drop-legacy-backups`, CreateBackup without a store after PR 14 | PR 14 | -| upgrade manual: the #618 regression on the off-host path and the way out | PR 8 | +| upgrade manual: the #618 regression on the off-host path and the way out; the `[backup.store]` paragraph's asynchronous create and the restore of a store-backed backup deferred to PR 9a | PR 8 | | `docs/troubleshooting.md` DeleteTable entry and admin guide `control_plane_delay_seconds` row | #395 (in its diff) | | ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded; the admin and upgrade manuals' backup sections | PR 11 | | `docs/design/README.md` line 22 | PR 2 | @@ -243,7 +247,7 @@ Documentation: the differences doc's PITR row, rewritten as section 6 says. ### PR 6: catalog index, `BackupCatalog`, `JobClaim`, the legacy reader Goal: the catalog index of section 4 on every backend and the engine-side Describe, List and Delete over it, with the legacy reader beside it. Its line names the interfaces and carries the acceptance: -- PR 6: catalog index per section 4, the `BackupCatalog` trait and `JobClaim` of section 3 implemented by every backend, the `backup_index` row (a new table or collection on PostgreSQL, SQLite and MongoDB; the reused legacy tables with `store_kind` set, plus `backup_index_by_status`, on Cassandra) and the restore job fields on #384's registration record, with the migration's backfill of `status`, `heartbeat_at` (the epoch, not NULL) and a sentinel `owner_generation` on the registrations that exist (migration id 6 on every backend: a file body on PostgreSQL, code bodies on SQLite, MongoDB and Cassandra, Cassandra's executing `0006_backup_catalog_index.cql` before its backfill; the MongoDB `backup_index` collection and `tables` fields, section 4; `RESTORE_LOCK_SEED` made `pub` with `#[doc(hidden)]` so its copy in `tests/backup_restore.rs` binds to it; the `store_id` column, the `backup_maintenance` record and Cassandra's write order across its three backup tables), the `[backup]` struct in `crates/config` with the job keys of section 7 (everything but `store`, which PR 7 adds) and the check that `stale_job_after_secs` is at least three times the effective heartbeat period, `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)` (section 7), since this PR's reconciler and permit read them, `max_concurrent_jobs` (default 4, at most 1024) replacing the per-backend restore slot semaphores (`restore_server_slot_count` on PostgreSQL and MongoDB removed; SQLite's `backup_server_slot_count` kept for snapshot opens until PR 8, which folds it into `max_concurrent_jobs`; the legacy create tries it without waiting and answers the one `LimitExceededException` text, `limit_exceeded_message`) with the request order of 5.1 step 1 and 5.3 step 1, the permit last, and the reconciler period and maintenance-claim check of section 4, engine-side Describe, List (the handler's existing paging, filtering and cursor exceptions kept, over the catalog index merged with the legacy reader through `list_account_backup_rows`, and a table ARN accepted in `TableName` as main's CreateBackup accepts it), Delete (BackupInUse on a CREATING row and while a restore target registered from the backup is CREATING; DELETING ordering; a ValidationException naming no root for a row of another store) over the new rows, `LegacyBackupReader::create` kept on every backend as the CreateBackup path while no store is configured, with its deprecation warning, and on Cassandra whatever the store configuration, with the `LegacyBackupReader` trait and fallback, the legacy restore registering through the new job fields and, with a dead owner, marked FAILED and its target removed through `remove_restore_target` (section 3); the engine adopts `BackupArn`, addresses every backup row by account and backup id (`BackupKey`; `backup_index` keyed by (`account_id`, `backup_id`), no `store_key_prefix` column) and every restore target by account, name and table id, and derives the API's source-table shapes from `source_table_details_json`; the `catalog_id`, written at init and by the migration for an existing catalog; DeleteBackup's in-use check and DELETING write in one transaction under the row's lock (`start_backup_delete`), and on Cassandra `active_restores` serving new rows; the legacy create taking a `max_concurrent_jobs` permit, and Cassandra's legacy create heartbeated so the reconciler removes a dead one's CREATING row; the job row's `kind` and store columns; #384's MongoDB claim markers, Cassandra `restore_owner`, PostgreSQL `RestoreOwner` and their startup sweeps are replaced by the generation column and the reconciler; `RESTORE_LOCK_SEED` becomes the shared seed. Acceptance: all existing backup tests pass unchanged against backups created by the PR 0 binary (run the binary, not the unit), the existing paging tests and `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` included (the vector tests change in PR 8 and 9a, not here); a catalog at #384's version holding a CREATING restore target left by SIGKILL migrates with that registration's `heartbeat_at` at the epoch, and the first reconciler pass removes the target, run on PostgreSQL and on MongoDB; ListBackups with a table ARN in `TableName` returns what the table name returns; DescribeBackup of an ARN of account B that carries the id of account A's backup returns BackupNotFound; a DeleteBackup and a legacy restore of one backup serialize, main's `delete_backup_winning_before_restore_leaves_no_target` and `restore_registration_winning_before_delete_returns_backup_in_use` pass unchanged for the legacy path, the `backup_index` row lock observed on PostgreSQL by a two-transaction test of `start_backup_delete` (one holding the row `FOR UPDATE` while it registers a CREATING target, the delete waiting and then answering BackupInUse); the same two orderings run against the engine's DeleteBackup and RestoreTableFromBackup over an off-host backup (a `backup_index` row), gated on that lock, a DeleteBackup that wins leaving the registration BackupNotFound and no target and a registration that wins leaving DeleteBackup BackupInUse, deferred to PR 9a, which adds the first restore of a `backup_index` row (on this PR no restore of one exists); the reconciler's FAILED-target duty of 4(e) on a FAILED record whose target is still CREATING, listed once stale and removed; a server built before this PR, started against the migrated catalog, is the mixed-version case section 10 forbids and is not tested. Depends on PR 4 (`BackupArn`) and PR 5 (`SnapshotInfo`). +- PR 6: catalog index per section 4, the `BackupCatalog` trait and `JobClaim` of section 3 implemented by every backend, the `backup_index` row (a new table or collection on PostgreSQL, SQLite and MongoDB; the reused legacy tables with `store_kind` set, plus `backup_index_by_status`, on Cassandra) and the restore job fields on #384's registration record, with the migration's backfill of `status`, `heartbeat_at` (the epoch, not NULL) and a sentinel `owner_generation` on the registrations that exist (migration id 6 on every backend: a file body on PostgreSQL, code bodies on SQLite, MongoDB and Cassandra, Cassandra's executing `0006_backup_catalog_index.cql` before its backfill; the MongoDB `backup_index` collection and `tables` fields, section 4; `RESTORE_LOCK_SEED` made `pub` with `#[doc(hidden)]` so its copy in `tests/backup_restore.rs` binds to it; the `store_id` column, the `backup_maintenance` record and Cassandra's write order across its three backup tables), the `[backup]` struct in `crates/config` with the job keys of section 7 (everything but `store`, which PR 7 adds) and the check that `stale_job_after_secs` is at least three times the effective heartbeat period, `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)` (section 7), since this PR's reconciler and permit read them, `max_concurrent_jobs` (default 4, at most 1024) replacing the per-backend restore slot semaphores (`restore_server_slot_count` on PostgreSQL and MongoDB removed; SQLite's `backup_server_slot_count` kept for snapshot opens, which PR 8 keeps as the backend's admission bound and raises the pool for (section 7); the legacy create tries it without waiting and answers the one `LimitExceededException` text, `limit_exceeded_message`) with the request order of 5.1 step 1 and 5.3 step 1, the permit last, and the reconciler period and maintenance-claim check of section 4, engine-side Describe, List (the handler's existing paging, filtering and cursor exceptions kept, over the catalog index merged with the legacy reader through `list_account_backup_rows`, and a table ARN accepted in `TableName` as main's CreateBackup accepts it), Delete (BackupInUse on a CREATING row and while a restore target registered from the backup is CREATING; DELETING ordering; a ValidationException naming no root for a row of another store) over the new rows, `LegacyBackupReader::create` kept on every backend as the CreateBackup path while no store is configured, with its deprecation warning, and on Cassandra whatever the store configuration, with the `LegacyBackupReader` trait and fallback, the legacy restore registering through the new job fields and, with a dead owner, marked FAILED and its target removed through `remove_restore_target` (section 3); the engine adopts `BackupArn`, addresses every backup row by account and backup id (`BackupKey`; `backup_index` keyed by (`account_id`, `backup_id`), no `store_key_prefix` column) and every restore target by account, name and table id, and derives the API's source-table shapes from `source_table_details_json`; the `catalog_id`, written at init and by the migration for an existing catalog; DeleteBackup's in-use check and DELETING write in one transaction under the row's lock (`start_backup_delete`), and on Cassandra `active_restores` serving new rows; the legacy create taking a `max_concurrent_jobs` permit, and Cassandra's legacy create heartbeated so the reconciler removes a dead one's CREATING row; the job row's `kind` and store columns; #384's MongoDB claim markers, Cassandra `restore_owner`, PostgreSQL `RestoreOwner` and their startup sweeps are replaced by the generation column and the reconciler; `RESTORE_LOCK_SEED` becomes the shared seed. Acceptance: all existing backup tests pass unchanged against backups created by the PR 0 binary (run the binary, not the unit), the existing paging tests and `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` included (the vector tests change in PR 8 and 9a, not here); a catalog at #384's version holding a CREATING restore target left by SIGKILL migrates with that registration's `heartbeat_at` at the epoch, and the first reconciler pass removes the target, run on PostgreSQL and on MongoDB; ListBackups with a table ARN in `TableName` returns what the table name returns; DescribeBackup of an ARN of account B that carries the id of account A's backup returns BackupNotFound; a DeleteBackup and a legacy restore of one backup serialize, main's `delete_backup_winning_before_restore_leaves_no_target` and `restore_registration_winning_before_delete_returns_backup_in_use` pass unchanged for the legacy path, the `backup_index` row lock observed on PostgreSQL by a two-transaction test of `start_backup_delete` (one holding the row `FOR UPDATE` while it registers a CREATING target, the delete waiting and then answering BackupInUse); the same two orderings run against the engine's DeleteBackup and RestoreTableFromBackup over an off-host backup (a `backup_index` row), gated on that lock, a DeleteBackup that wins leaving the registration BackupNotFound and no target and a registration that wins leaving DeleteBackup BackupInUse, deferred to PR 9a, which adds the first restore of a `backup_index` row (on this PR no restore of one exists); the reconciler's FAILED-target duty of 4(e) on a FAILED record whose target is still CREATING, listed once stale and removed; a server built before this PR, started against the migrated catalog, is the mixed-version case section 10 forbids and is not tested. Depends on PR 4 (`BackupArn`) and PR 5 (`SnapshotInfo`). Invariants honoured: 4(a) and (b) (the two records and their transitions), 4(d) (every row by `BackupKey`, every target by `RestoreTargetKey`), 4(e) (the claim), 4(f) (the maintenance claim its reconciler period checks), 5.2 (DeleteBackup's lock and ordering against a registration), and the Goals' unchanged request and response shapes. @@ -364,20 +368,20 @@ Documentation rows it corrects: limits doc backup rows (the knobs); upgrade manu ### PR 8: CreateBackup job, worker and reconciler Goal: the CreateBackup job of 5.1 on the filesystem store, with the reconciler and its full duty list, the byte budget on the snapshot streams and the item validator. Its line names the interfaces and carries the acceptance: -- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, enforced in each page's query, PostgreSQL's cursor replaced by the keyset page under the same snapshot, with the key columns returned under qualified aliases, an outer `ORDER BY` and `n` starting at 16 and sized from the previous page's average row (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, an off-host round trip of a vector table added and `test_create_backup_of_a_vector_table_is_refused` expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which the harness exports and CI sets (5.1 step 1), FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend; SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs returns `LimitExceededException`; a 1M-item run records duration and peak RSS, and RSS must not grow from the 100k run to the 1M run (on every backend but `sqlite-memory`, whose open copies the table); the memory test with #394's worst-expansion items stays within section 7's formula, with the worst lines placed after a page of small rows, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows read past each PostgreSQL page's cut stay under its `n` and are recorded beside the RSS figure; `EXPLAIN` of the test table's page query shows an index scan; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on each backend; with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS must match the formula in section 7 in each, proving the encoder streams into the upload and the byte budget and channel bound hold at the largest line. Depends on PR 4, 5, 6, 7. +- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, in the page query on PostgreSQL and while the rows are pulled on SQLite, PostgreSQL's cursor replaced by the three-layer keyset page under the same snapshot (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, the create half of an off-host round trip of a vector table added (CreateBackup succeeds, DescribeBackup reaches AVAILABLE with the table's features, ListBackups lists it, and the restore answers the ValidationException naming the store until PR 9a adds the restore half) and `test_create_backup_of_a_vector_table_is_refused` expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which every CI pytest job that runs the suite sets (`filesystem` on the PostgreSQL job, which appends a filesystem store after `init`, and on the SQLite job, the defaulted root; `s3` on the SQLite job against moto; `none` on the MongoDB and Cassandra jobs) and `devtools/run-tests --pytest` refuses to run without (5.1 step 1); the restore cells of `tests/test_backup_restore_fidelity.py` skip on a server with a store, naming PR 9a, until its restore job lands; FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend with a `SnapshotEngine` (PostgreSQL, SQLite, MongoDB; Cassandra keeps the legacy create and its existing cells); SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs waits for the slot (up to 30 s, item 28) and is answered once the first job ended, `LimitExceededException` when the wait outlives the bound; a 1M-item run records duration and peak RSS, and the 1M run's peak must be within 112 MiB of the 100k run's (the budget, the line cap and the channels; on every backend but `sqlite-memory`, whose open copies the table; on MongoDB a 500k run stands in for the 1M run, its load alone taking 27 minutes through the wire); the memory test with #394's worst-expansion items, with the worst lines placed after a page of small rows, has its peak over idle under 1.2 times the sum of section 7's terms at the measured row (the decoded rows in the page and the pipeline, one row's parse, the line buffer, the chunks and SQLite's run-ahead) with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one SQLite binary and at 12 rows on PostgreSQL, and its peak with glibc's default arenas recorded beside it at the same counts, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows a PostgreSQL page reads past its cut are at most `n - 1`, measured by `EXPLAIN (ANALYZE, BUFFERS)` on a 1,000-small-then-60-worst table (4 read, 1 past the cut at the floor) and pinned by `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips on a 300-row table with nine 27 MB rows); `EXPLAIN` of the page query on the six key shapes a table can have (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation) shows `Limit` over one `Index Scan` on the primary key and no other scan of the relation where the planner's estimate is above `n`, and the bitmap scan with a `Sort` near the end of a small table (PostgreSQL 16), both pinned by a test and stated in the module doc; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on PostgreSQL and SQLite (the two backends `tests/test_backup_job.py`'s deployment fixture starts; the fixture's MongoDB deployment destroys and initializes the catalog of a replica set of the suite's own, since the database names are fixed, and the `mongodb-backup-store` job runs the suite and the backup selection against a filesystem store) (an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`; UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone); with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS is recorded against the terms of section 7 in each, proving the encoder streams into the upload (the incompressible run's 68 MB file passes through a 4 MiB chunk window) and that the byte budget and the channel bound hold at the largest line (under `MALLOC_ARENA_MAX=1` the worst-expansion run's peak does not grow with the number of worst rows, 452, 480 and 455 MiB over idle at 12, 24 and 48 rows on the final SQLite binary; with glibc's default arenas it does, 707, 700 and 1,281 MiB in the same series, the allocator's retention and not the pipeline's terms, which the one-arena series shows do not grow). Depends on PR 4, 5, 6, 7. Invariants honoured: 4(a) (every transition of the backup row a job or the reconciler performs), 4(c) (the reconciler acts only on rows of the configured store and on none while it is unhealthy), 4(e) (the claim, the fencing of every status write, the FAILED-write gate before any delete), 4(f) (the duties that run under a maintenance claim), 5.1 as written, section 2's three store facts, section 7's memory bound and pool rule, and the Drawbacks' #618 regression. -Interfaces it adds: `extenddb_core::validation::validate_restorable_item` (5.1 step 4), `extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES`, the worker and the reconciler in `crates/server/src/workers.rs`, the metrics of section 9; the reconciler claims through PR 6's `BackupCatalog::claim_backup`. The snapshot and the claim connection are handed to the backup worker over an in-process channel; the worker is a task in `crates/server/src/workers.rs`, spawned with the server's cancellation token and retained handle like the existing workers. It is needed because four write paths store an `N` value such as `1E126` or `abc` unvalidated on main (#618): BatchWriteItem, TransactWriteItems, UpdateItem on an absent item (which creates the item from its `Key`) and ImportTable, the list #394's `key_order.rs` doc gives. So that one run finds every offending item, the first refusal stops encoding and uploading but not validation: the job reads and checks the rest of the snapshot, logs each refused item (its key, attribute path and reason, with the backup ARN) up to 1000 items and counts the rest, and then fails the backup (step 5) with a FAILED reason that gives the number of refused items and the first one, naming #618 only when that item's refusal is an `N` value; so no backup that no restore can load is reported AVAILABLE. The byte budget (section 7 states the bound): PR 8 replaces #395's cursor on PostgreSQL with pages bounded by bytes in the query, each page a keyset `SELECT` in the same `REPEATABLE READ` transaction, ordered by the data table's primary key and starting after the last key of the page before, with a running byte sum that cuts the page at `SNAPSHOT_FETCH_BYTES` (64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a configuration key in v1) and always returns at least one row; the snapshot, the lock and which tables stream `key_ordered` are unchanged. On SQLite a page stops at `DEFAULT_FETCH_BATCH` rows or after the row that reaches `SNAPSHOT_FETCH_BYTES` of item text, whichever comes first, and holds at least one row. The cut rule exists in SQL on PostgreSQL and in Rust on SQLite, so the check that no page holds more than the budget plus one row lives in `extenddb_storage::snapshot_conformance`, which both run; a MongoDB batch is already bounded by the server's 16 MiB reply. -Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds an off-host round trip of a vector table beside it, and `test_create_backup_of_a_vector_table_is_refused` (`tests/test_backup_restore_fidelity.py`) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs the suite sets: it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. +Interfaces it adds: `extenddb_core::validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits)` (5.1 step 4; the key argument is a schema every table has, multi-part keys included, each key attribute checked with the format's own `check_key_value`; `RestorableLimits` is built from `&LimitsConfig`: the item size, the hash and range key sizes and the attribute name size), `RestorableItemReason::{AttributeNameTooLong, LineTooLong}` (the latter carrying `{ cap }`), `SHOWN_VALUE_BYTES` (4096, the bound on each text an error carries other than a key value) and `SHOWN_KEY_VALUE_BYTES` (12,288, the bound on a key value), `extenddb_core::limits::DEFAULT_MAX_ITEM_SIZE_BYTES` (the default item limit, the feeder's threshold), `extenddb_core::backup_format::encode_item_line_len_bounded(item, cap)` (the line's length up to `cap`, measured without a buffer), `extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES` with `fetch_bytes()`, `record_page_bytes` and `record_rows_past_cut` (counters under `test-hooks`), `extenddb_storage::snapshot::source_table_schema` (5.1 step 2), the job (`crates/engine/src/backup/job.rs` with `job/{writer,heartbeat,failure}.rs`) and the reconciler's new duties (`crates/engine/src/backup/reconciler.rs`) in the engine's backup module, the server handing the engine the shutdown token, the metrics collector, the job settings (`JobSettings`) and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, the metrics of section 9, and on `BackupCatalog`: `claim_dead_backup(key, stale_after)` (the takeover of 4(e); `claim_backup` inserts and takes over nothing), `list_backup_rows_older_than(status, store, clock, age)` (`StaleClock::{Heartbeat, Completed}`, the backend's clock; a row with `completed_at` unset is never older on the completed clock), `sweep_restore_registrations` (Cassandra's `active_restores` sweep, a default empty body elsewhere) and `StatusFields.source_table_details_json` (the AVAILABLE write carries the manifest's `source_table`). The request's detached task opens the snapshot, claims the row, checks the maintenance claim and returns the response through a one-shot channel, then continues as the job with the same owned values, so nothing but the response crosses a channel between the request and the job (the pipeline's item and chunk channels and its blocking encoder are the job's own); the runtime's task tracker holds every job, and the server's shutdown cancels and awaits the jobs' failure paths (`BackupRuntime::drain_jobs`, a 30 s bound) before it drains its workers. It is needed because four write paths store an `N` value such as `1E126` or `abc` unvalidated on main (#618): BatchWriteItem, TransactWriteItems, UpdateItem on an absent item (which creates the item from its `Key`) and ImportTable, the list #394's `key_order.rs` doc gives. So that one run finds every offending item, the first refusal stops encoding and uploading but not validation: the job reads and checks the rest of the snapshot, logs each refused item (its key, attribute path and reason, with the backup ARN) up to 1000 items and counts the rest, and then fails the backup (step 5) with a FAILED reason that gives the number of refused items and the first one, naming #618 only when that item's refusal is an `N` value; so no backup that no restore can load is reported AVAILABLE. The byte budget (section 7 states the bound): PR 8 replaces #395's cursor on PostgreSQL with pages bounded by bytes in the query, each page a keyset `SELECT` in the same `REPEATABLE READ` transaction, ordered by the data table's primary key and starting after the last key of the page before, in three layers (the innermost takes `n` rows after the key under `LIMIT`, since a window function runs before `LIMIT`; the middle adds the running byte sum and the rows' count; the outer keeps the rows whose predecessors' bytes are under `SNAPSHOT_FETCH_BYTES` (64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a configuration key in v1) and orders them), so a page holds at least one row and at most the budget plus one, and the stream ends when the inner query returned fewer rows than asked and the cut dropped none of them; `n` starts at 16 and is then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows (`RowBoundMemory`), never under 4 and never over `DEFAULT_FETCH_BATCH`; the rows read past the cut (at most `n - 1`) are read by the server and dropped by the outer layer, and each costs the server one serialization of its item text and two spills of it (the window functions buffer the `n` rows before the outer layer drops them), so a page costs the server `n` rows of text, not the rows it returns: sized from the average, a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned; sized from the largest row it asks for 4 (208 MiB, 1.1 s); the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, at most once per row bound's worth of small rows; the key columns come back under qualified aliases so the next cursor is the row's own typed key, and the outer `ORDER BY` states a guarantee the executor gives anyway (a mutant without it passes every behavioural test, and the query text pins it); the snapshot, the lock and which tables stream `key_ordered` are unchanged. On SQLite a page stops at `DEFAULT_FETCH_BATCH` rows or after the row that reaches `SNAPSHOT_FETCH_BYTES` of item text, whichever comes first, and holds at least one row; the rows are pulled one at a time, so the stream pulls none past the cut, while the driver's worker thread reads at most five rows ahead of the pull (sqlx's `row_buffer_size`, set on the pool to four from its default of 50, which read every row of a 24-row worst-shape table ahead of the first page, plus the row the worker holds while its `send` waits; 160 MiB at the line cap), and the stream ends only when a page is short of the row bound and no cut happened. The cut rule exists in SQL on PostgreSQL and in Rust on SQLite, so the check that no page holds more than the budget plus one row lives in `extenddb_storage::snapshot_conformance`, which both run; a MongoDB batch is already bounded by the server's 16 MiB reply. +Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds the create half of an off-host round trip of a vector table beside it (PR 9a adds the restore half), and `test_create_backup_of_a_vector_table_is_refused` (`tests/test_backup_restore_fidelity.py`) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs a suite creating backups sets it (the Rust integration suite reads it the same way, `tests/rust/src/backup_restore.rs`): it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. -State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target
    ` (PR 10), the one command that removes a FAILED restore's CREATING target and its record, through `remove_restore_target`, under the maintenance claim like the other writing commands, since DeleteTable refuses a restore target (5.3 step 6); a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove a CREATING restore target that has no job row (`remove_restore_target`); remove `restore_progress` rows whose table is not a CREATING restore target; remove spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. +State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (the resume is PR 9a's, with the restore job; this release's reconciler logs such a record once; once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target
    ` (PR 10), the one command that removes a FAILED restore's CREATING target and its record, through `remove_restore_target`, under the maintenance claim like the other writing commands, since DeleteTable refuses a restore target (5.3 step 6); a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove `restore_progress` rows whose table is not a CREATING restore target and spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3), both from PR 9a, which adds the records they read (the duty that removed a CREATING restore target with no job row is withdrawn: on every backend the job record is the registration itself, so no such state exists after migration 6); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. -Kill points (testing summary): between a data file upload and the manifest; between the manifest put and the AVAILABLE update; SIGTERM during a backup. Main tests that must pass unchanged: #384's backup tests other than the vector tests named above. Documentation rows it corrects: the differences doc's backup consistency row (its MongoDB half, "not point-in-time" today, which the snapshot read concern of section 3 makes one; its Cassandra half stays true), its vector-backup row ("Backing up and restoring a table that has vector indexes": supported on the off-host path from PR 8 and 9a, refused only into a target without vector support, CreateBackup on the legacy create, without `[backup.store]` or on Cassandra, still refused as on main, the legacy reader's behaviour kept for legacy backups; 5.1 step 1), its restore-interruption row (one reconciler with the duties above in place of the per-backend sweeps), a new row for CreateBackup of a table holding an `N` value #618 let in (the backup fails on the off-host path, where main backs it up and restores it; 5.1 step 4), the limits doc's line cap and file size rows, the usage guide's "backups not supported" line; upgrade manual: on the off-host path a backup of a table holding an `N` value the wire refuses (#618) fails where main backs it up, which meets file-backed SQLite without a `[backup]` section and any deployment once a store is configured (5.1 step 4), the reason reaches only the operator (`backup list --failed`, and the log, which names each offending item's key and attribute path with the backup ARN a tenant's support case can cite), and the way out is to rewrite each named item. +Kill points (testing summary): between a data file upload and the manifest; between the three documents and the manifest (`before_manifest`); between the manifest put and the AVAILABLE update; SIGTERM during a backup. Main tests that must pass unchanged: #384's backup tests other than the vector tests named above. Documentation rows it corrects: the differences doc's backup consistency row (its MongoDB half, "not point-in-time" today, which the snapshot read concern of section 3 makes one; its Cassandra half stays true), its vector-backup row ("Backing up and restoring a table that has vector indexes": supported on the off-host path from PR 8 and 9a, refused only into a target without vector support, CreateBackup on the legacy create, without `[backup.store]` or on Cassandra, still refused as on main, the legacy reader's behaviour kept for legacy backups; 5.1 step 1), its restore-interruption row (one reconciler with the duties above in place of the per-backend sweeps), a new row for CreateBackup of a table holding an `N` value #618 let in (the backup fails on the off-host path, where main backs it up and restores it; 5.1 step 4), the limits doc's line cap and file size rows, the usage guide's "backups not supported" line; upgrade manual: on the off-host path a backup of a table holding an `N` value the wire refuses (#618) fails where main backs it up, which meets file-backed SQLite without a `[backup]` section and any deployment once a store is configured (5.1 step 4), the reason reaches only the operator (`backup list --failed`, and the log, which names each offending item's key and attribute path with the backup ARN a tenant's support case can cite), and the way out is to rewrite each named item. ### PR 9a: restore job Goal: the restore job of 5.3 on the same backend over a filesystem store. Its line names the interfaces and carries the acceptance: -- PR 9a: restore job per 5.3, same backend, filesystem store: request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target_table_id, claim, file_index, start_at, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL migration 7 for `restore_progress`, reserved here) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `max_item_size_bytes` (`N` normalization included); items read through `ExtenddbManifest::reader_prehashed`; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write, DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB via duplicate-key suppression); SIGKILL at each kill point of the testing summary leaves a table that either finishes with the right count or is gone (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other; a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers. Depends on PR 8. +- PR 9a: restore job per 5.3, same backend, filesystem store (PR 8 leaves it the engine's `check_catalog_row_for_restore`, which answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" for a row of the configured store, the 24 restore cells of `tests/test_backup_restore_fidelity.py` and `tests/test_auth_permissions.py`'s restore-authorization cell skipped on a server with a store, and the vector wire test's restore assertion, all four to replace with the restore): request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target_table_id, claim, file_index, start_at, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL migration 7 for `restore_progress`, reserved here) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `RestorableLimits` (item, hash key, range key and attribute name sizes; `N` normalization included); items read through `ExtenddbManifest::reader_prehashed`; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write, DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB via duplicate-key suppression); SIGKILL at each kill point of the testing summary leaves a table that either finishes with the right count or is gone (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other; a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers. Depends on PR 8. Invariants honoured: 4(b) (the restore record and its transitions), 4(d) (every target by `RestoreTargetKey`), 4(e) (the claim, the FAILED-write gate before the target is removed, the PostgreSQL stale window), 4(f) (the job checks the maintenance claim after its row exists), 5.2 (the registration under the backup row's lock), 5.3 as written, section 1's duplicate rule and vector refusal, and the Goals' fidelity list. Interfaces it adds: `bulk_load` (section 3) and the restore methods of `BackupCatalog`: @@ -462,7 +466,7 @@ Phase 3, later (v1.x): - Unit: manifest serde and golden files, key mapping and ARN parsing, store conformance. - Integration, every backend, every PR: round trip with full configuration preservation, composite keys, GSIs and LSIs, vector indexes where supported, legacy fallback, PITR surface. - Consistency: concurrent-writer snapshot test on every backend. -- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. These extend the SIGKILL harness the migration tests already use. Backup kill points: between a data file upload and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after a file's transaction commits and before the next file starts; MongoDB between `insert_many` and the progress write (the path that exercises E11000 suppression and the count check); Cassandra between a batch's last write and its progress write; SQLite mid-batch; PostgreSQL between the ACTIVE flip and the progress-row deletion; MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip, where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7; a fenced-out restore job that resumes after a successor claimed the row (its FAILED write matches zero rows and the successor's table survives); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). +- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with `devtools/backup-kill-points.sh` and its API half `devtools/backup-kill-points.py` (a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store; the clippy job lints the feature so the pause code compiles on every push), not by a pytest cell; the restore kill points extend the SIGKILL harness the migration tests already use. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after a file's transaction commits and before the next file starts; MongoDB between `insert_many` and the progress write (the path that exercises E11000 suppression and the count check); Cassandra between a batch's last write and its progress write; SQLite mid-batch; PostgreSQL between the ACTIVE flip and the progress-row deletion; MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip, where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7; a fenced-out restore job that resumes after a successor claimed the row (its FAILED write matches zero rows and the successor's table survives); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). - Distribution matrix for the filesystem default path: unprivileged local launch, the npm launcher, the container image with and without a volume, `sqlite-memory`. - Cross-backend matrix: nightly. - Live DynamoDB: Phase 2, measured. @@ -475,7 +479,7 @@ Phase 3, later (v1.x): - Every failure path deletes the prefix or leaves it unreachable from List, and the row never says AVAILABLE for an incomplete prefix. - Restore writes no item before every data file's checksum has verified; a verification failure deletes the CREATING target table, which never accepted a read or a write; the request refuses before creating the table when the target cannot represent the manifest or the key ranges overlap. - A file's data and its progress record commit in one transaction on PostgreSQL and SQLite (per 10k-item batch on SQLite, with the items-applied count), so no item is applied twice; on MongoDB the progress record follows the insert, re-application of the one unrecorded file is idempotent by duplicate-key suppression (code 11000 only; any other write error fails the restore), and the post-resume document count is checked against the manifest; on Cassandra the progress record follows each batch under a lightweight transaction and at most one batch is re-applied. A key-ordered restore runs the per-file key-range check on every file, and an unordered one ends with a count of the target equal to the manifest's item count. On PostgreSQL the ACTIVE flip precedes the deletion of progress rows; on MongoDB the job's DONE write and `restore_backfill_pending` are one transaction, and the worker's ACTIVE flip is conditioned on that flag and CREATING, not on the job's generation. -- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write errors with an unknown outcome, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched. +- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched. - The ARN is parsed by `BackupArn` once, in the engine, before any catalog or store access; no other code splits ARN strings. - Account scoping is checked on the ARN before any store access, every backup row is read by account and backup id and accepted only when its account, table and ARN equal the request's, and every restore-target method takes the target's account, name and table id. - The backup job validates every item with `validate_restorable_item` before encoding it, keeps validating after the first refusal, and the restore binds the same function. @@ -494,7 +498,7 @@ Phase 3, later (v1.x): - A restore from an S3 store spools the whole compressed backup to local disk before loading any of it. - One core at full utilization for the duration of a backup on compression; `max_concurrent_jobs` and `spawn_blocking` keep it off the request threads but do not make it free on a 2-vCPU host. - Two code paths for one release while legacy backups remain readable. -- A backup of a table holding an `N` value #618 let in fails on the off-host path, where main backs it up (5.1 step 4). +- A backup of a table holding an `N` value #618 let in, or an attribute name over the limit that `UpdateItem` and a `TransactWriteItems` `Update` stored before #433, fails on the off-host path, where main backs it up (5.1 step 4). ## Alternatives considered @@ -550,3 +554,6 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 29. (2026-10-30, PR 7 as implemented.) A store problem never gates the in-catalog create: with a configured store that is unhealthy, CreateBackup writes the in-catalog format as it does with no store, and only the store-backed create of PR 8 is refused by `require_healthy_store`; a defaulted SQLite root whose identity check fails (a catalog re-created by `destroy` and `init` under the old root's `extenddb-store.json`) is the same warning as a failed validation, leaving the store unconfigured, and `destroy` and the admin guide say the file must go with the catalog (section 7, PR 7). An S3 store's `validate` and identity check run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports; a filesystem store's run before the listener; the configuration errors that need no request still stop startup; a `validate` refusal of an S3 bucket after the listener makes the store unhealthy (`failed validation`) rather than stopping startup (section 7, 9, PR 7). `/health`'s `reason` is the problem's category alone; the ids and ARNs go to the operator's log line (section 9, PR 7). The identity file is read up to 64 KiB. The S3 root label is built from the endpoint the client uses, in one spelling. A check that cannot read the catalog leaves the store's health unchanged. An existing spool must be owner-only. The maintenance claim is heartbeated on a timer from the take to the release, and PostgreSQL's `check_maintenance_claim` tests the advisory lock before the heartbeat's age (4(f), PR 7). The jobs' side of the claim is implemented in the engine for the in-catalog create and restore, before the permit and, since those paths commit their row inside the backend, before the row exists; the ordering 5.1 step 2 states applies to the store-backed job PR 8 adds (5.1, 5.3, PR 7). `backup list` opens the SQLite catalog without the serve lock, reads the identity file without creating it and runs on an unhealthy store; on SQLite `backup purge` runs the reconciler's duties with a zero window before its wait; a CREATING row with a dead owner is purged as any row, through `remove_dead_creating_backup_row`, a `BackupCatalog` method of PR 7; a FAILED row with a young heartbeat is waited on; the dry run exits 0 (PR 7, PR 10). Cassandra's claim reads run at `LOCAL_SERIAL` and an insert whose answer was lost is settled by a re-read of the generation (PR 7). The PR 10 comment on `backup_index_by_status` is corrected: migration 6's entry carries `store_kind` and `store_id`. The body sentences that paraphrased the earlier rules are edited with this item (section 7's configuration-error paragraph, 4(f) and 5.1 step 2 and 5.3 step 4's claim check, PR 6's `store_root` row, PR 7's interfaces and acceptance), and the filesystem store's startup checks are bounded by a timeout that leaves the store `checking` when a mount stops answering after the root is opened (a root that does not answer at startup holds the server before the listener, as the database directory would). Also: the PostgreSQL check takes its shared lock at transaction scope so a cancelled request leaves no lock on the pool; a dropped `HeldClaim` stops its timer; the purge's dry run is a read like `list`; a CREATING row is removed before its prefix; an empty `filesystem.path` is left for `open` to refuse; the spool meets the root's whole rule, parents included; `backup list` reads the catalog's status without the serve lock. The defaulted SQLite root's warning holds only while the catalog has no backup rows for the root (`read_store_ids(Filesystem, root)`: the upgrade, fresh and re-init cases); with rows for it the root has been used and is handled as an explicitly configured one, so its backups never vanish behind a warning: a failure of its identity check or manifest probe is an unhealthy store (reported, its rows' restores and deletes refused, rechecked each cycle), and a refusal of its `open` or `validate` a startup error naming the root (section 7, PR 7's acceptance). The server's `configure` carries the reason for an absent store out (`NotConfigured`), and `backup list` and `backup purge` print it, so a defaulted root that failed its check is no longer reported as "[backup.store] is absent or disabled". The store's check runs under a per-kind bound in every reconciler cycle (`check_configured_store`: 5 s for a filesystem root, the startup bound; 30 s for a bucket, under the SDK's 60 s per-attempt timeout so a timed-out check is dropped before the SDK retries its request), recording a transport failure naming the bound, and the cycle selects the check against the shutdown token; an unbounded check let a stalled S3 endpoint hold each cycle for the SDK's whole retry budget and a stalled mount hold it for good, with shutdown hitting the drain timeout (section 7). A filesystem root's `validate` reports no transport class; an I/O failure of its probe write or of resolving the root at startup degrades the store (`failed validation`) as the same condition one cycle later does, while the owner-only and existence refusals stay configuration errors (section 7). A transport failure's operator line carries the store error's whole cause chain (the SDK's top link is a bare class). Every store error of a check that is neither a transport failure nor a denial is `failed validation`, the category `validate` gives it, so a bucket deleted under a running server reads as a bucket missing at startup. A filesystem root that is or contains the database directory is refused in `resolve_store`, beside the spool rule (section 7). A filesystem store's synchronous root checks run on the blocking pool so the startup bound fires whichever call stalls. The checking store's refusal says its first check is pending; `/health`'s reason list carries `catalog unreadable`; `record_check` reports whether the category changed. Purge's wait set is CREATING and FAILED rows with a live heartbeat and RUNNING restores, DELETING rows being finished rather than waited on, and its refusal rule is that set at the wait's end; 4(f), the PR 7 "Destroy and purge" paragraph and the PR 10 claim paragraph (`MAINTENANCE_LOCK_KEY`, `pg_try_advisory_xact_lock_shared` then the heartbeat age, `LOCAL_SERIAL`) and the PR 6 and PR 7 signatures (`check_maintenance_claim(stale_after)`, `maintenance_claim(holder, stale_after) -> Box`) are edited to the code with this item. A row a DeleteBackup moves to DELETING or DELETED, or removes, between purge's read and its turn is finished and counted as deleted, not left (the deletion is one idempotent sequence on both sides). On SQLite purge waits and removes CREATING rows with a zero stale window (under the serve lock every owner is dead, so a young heartbeat proves nothing), and the server's startup under the serve lock removes a maintenance claim a stopped command left, so a killed purge costs no stale window; on every backend purge releases the claim on `SIGINT` or `SIGTERM`, at once during the wait and before the next row during the deletion. The wait's three failures (jobs still running at the limit, the claim lost, the catalog unreadable) and the signal are reported each as itself. PostgreSQL's factory starts the GSI queue workers only for a server, not for a command reading beside one. Purge's restore part of the wait set is the RUNNING `restore_jobs` records alone: the dead-job listing it reads with a zero window also returns the FAILED records whose target is still CREATING (the FAILED-target duty's, 4(e)), which are the reconciler's to finish and no job runs under, so they are not waited on; the wait finishes at once on a stopped deployment that holds one. On SQLite the zero-window pass resolves the kinds this release writes; a RUNNING `restore_jobs` record of kind `manifest`, which PR 9a introduces, is left for the reconciler that resumes it and purge waits on it and refuses naming the table, the one state the PR 10 paragraph's "never makes it wait or refuse" does not cover. The reconciler finishes the configured store's DELETING rows while the maintenance claim is held, as 4(f) lists. The binary ends its tokio runtime with a bounded shutdown (5 s) after `serve` and after every command: a filesystem store's check abandoned at its bound leaves its call parked on a blocking thread until the mount answers, and a plain drop of the runtime joined that thread, so a server whose root had stopped answering drained its workers at SIGTERM and then stayed alive until SIGKILL. Each call of a store check runs as a task of its own that an abandoned check does not take with it, and the store records that a call is in flight; no check starts while one is (the store is recorded `unreachable` again, with the reason), so a mount that stays stalled parks one thread rather than one per cycle. A command's identity read (`backup list`, the dry run) runs under the store's check bound as the server's check does, where the unbounded read waited out the SDK's three attempts of 60 s on an endpoint that accepts and never answers. An empty `filesystem.path` is compared with nothing in `resolve_store` (compared as written it was the working directory, refused by name as the database directory when the server starts there) and left for `open` to refuse. A root that refuses `link(2)` with `EPERM` is reported `failed validation` (the README and the deployment guide said `unreachable`). `tests/test_backup_store_config.py` is excluded from `devtools/run-tests`'s pytest run, as its docstring says; a catalog whose rows cannot be read keeps a defaulted root configured, and that arm is tested. On SQLite the purge's own reconciler cycle is told that its caller holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) and takes the held branch without reading the claim; read through the zero dead-owner window, the command's own claim was seen held only when its heartbeat and the check fell in one millisecond, so the branch was decided by scheduler timing (PR 7, PR 10). Purge's test of a live owner is the backend's `list_backup_rows_older_than` on the heartbeat clock, a `BackupCatalog` method of PR 7 that PR 8's reconciler also lists by, where it compared `heartbeat_at` with this host's clock, which a host running ahead of the database read as a shorter wait, not a longer one (4(e), 4(f), PR 7). On SQLite a command under the serve lock takes over a maintenance claim row whatever its heartbeat says, and PostgreSQL's take tries the exclusive lock again over about 10 ms, so a job's one-statement shared lock does not refuse it (PR 10's claim paragraph). On MongoDB a command that opens the catalog beside a server (`backup list`, the dry run) starts no key repair of documents stored without their key (#473); the server does. On PostgreSQL every `backup` command registers as a server does and its registration's session carries the command's name as `application_name`, so a migration that waits for it names it, and a catalog the factory refuses ends the command with the message and exit code `serve` gives (3 behind, 6 while a migration runs). The jobs' refusal while the claim is held names a backup maintenance command, not `purge`, since every maintenance command holds the one claim. The console banner shows a filesystem root in the canonical spelling the rows and the log line use, with the configured spelling only where it differs (PR 7). After #335 and #681 merged, PR 7 is rebased onto PR 6's rebased head: MongoDB's and Cassandra's `serve` register through the numbered runner (#681), so the backup commands register there too, as on PostgreSQL, named after the command (MongoDB's `extenddb_servers` document carries a `command` field; Cassandra's `extenddb_servers` row, which has no column for it, carries the name in `release` after the release and ` as `, a column being a catalog migration), refuse with the same exit codes, and release the registration when they exit, where a registration left to the runtime hooks the command drops would be deleted only by a task the command's exit may never run, and `migrate --yes` would then wait out the row's or document's 45 s; MongoDB's factory starts the key repair in its `build_components`, after `register_and_check`, only for a server (`start_key_repair`); the reconciler, which takes the backup runtime, runs in the worker set the registration supervisor stops and starts (#666) (PR 7). 30. (2026-11-04, PR 7, second review round.) DeleteBackup of a store-backed row answers the DELETED description once the DELETING write has committed and leaves a failed object deletion to the reconciler, where it answered InternalServerError (5.2). The store's bound covers the store's calls alone, each through one adapter (`ConfiguredStore::calls`): the catalog's reads run beside them, so a slow catalog costs the check time and never marks the store unhealthy, and the deletions DeleteBackup and the reconciler run go through the same adapter, under the same bound and the in-flight rule (section 7; the in-flight mark now means a call abandoned at its bound has not returned). After the second round's review: a filesystem root's calls run one at a time (`ConfiguredStore::call_slot`), a call that finds another running waiting for it under the bound and then having the bound for its own call (third round: one bound over the wait and the call abandoned a 3 s call behind a 3 s call on a healthy root), so two DeleteBackup requests on a stalled mount park one thread between them where each parked its own (the in-flight mark alone was set only once the first call reached its bound); the mark is set under the call task's own lock, only while the task still runs, so a task that ended between the bound's last poll and the mark leaves no mark with nothing to clear it. A store whose startup `validate` failed is built unvalidated, so the next check runs `validate` again rather than reporting the store healthy on a re-read of the identity file. A defaulted SQLite root whose startup checks did not finish keeps its rule (`ConfiguredStore::defaulted`): the reconciler applies it to the first check's outcome and sets a failing root with no rows aside (`set_aside`), as the startup check would have (section 7). The manifest probe reads the newest AVAILABLE row through `BackupCatalog::newest_available_backup_row` rather than listing every one. Purge's wait reads RUNNING restores by status (`BackupCatalog::list_running_restore_jobs`), with no age, so a record stamped in the listing's instant or ahead of the clock holds the wait (4(f)); its deletion loop lives in the engine (`maintenance::purge_rows`) and stops before its next row at a lost claim or a signal; `HeldClaim::release` is bounded (10 s). On SQLite a zero window selects the dead-owner listings and removals by status alone, so a heartbeat stamped ahead of the clock is not a live owner under the serve lock (PR 10's claim paragraph). The backup commands run none of the startup recovery a server runs (`ServerComponentsOptions::startup_recovery`), `purge --confirm` included, which still takes SQLite's serve lock; the signal listener is up before the open in every subcommand, so a signal during the open ends the command with its registration released. PostgreSQL's `release` unlocks the advisory lock before closing the session, so a check right after the release reads the claim free; Cassandra's takeover re-reads on a lost answer as the insert does; MongoDB's claim upsert that matched a document with no `owner_generation` is taken. `tests/test_backup_store_config.py` runs in CI: on the SQLite job with moto, and in the PostgreSQL CLI lifecycle list. Two rows join `docs/differences-from-dynamodb.md`: the maintenance claim's LimitExceededException and the unhealthy store's ValidationException. + +31. (2026-10-31, PR 8 as implemented.) The job and the reconciler live in the engine's backup module (`crates/engine/src/backup/{job,reconciler}.rs`), not in `crates/server/src/workers.rs`, and nothing crosses an in-process channel between the request and the job but the response (the pipeline's item and chunk channels are the job's own): the request's detached task opens the snapshot, claims the row, checks the maintenance claim and answers through a one-shot channel, then continues as the job; the server hands the engine the shutdown token, the metrics collector, the job settings and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, and `drain_jobs` (30 s) runs before the workers drain (PR 8); a stopped pipeline's sink answers the encoder's write with `BrokenPipe`, which nothing retries, where an `Interrupted` answer (std's "try again") kept a job cancelled in the middle of a 300k-row table spinning in the compressor's trailer write through the whole drain with its row left CREATING; the job ends within milliseconds of the signal on every store (5.1 step 5, PR 8). The reconciler's takeover is `BackupCatalog::claim_dead_backup(key, stale_after)`, not `claim_backup`, which inserts; `list_backup_rows_older_than(status, store, clock, age)` with `StaleClock::{Heartbeat, Completed}` carries the database's clock as a backend predicate for the FAILED-prefix delay and the tombstone retention, so the engine never holds a clock; `sweep_restore_registrations` is the Cassandra `active_restores` sweep with a default empty body; `StatusFields` gains `source_table_details_json`, which the job's and the reconciler's AVAILABLE writes fill with the manifest's `source_table` (4(e), PR 8). The PostgreSQL page is three layers, since a window function runs before `LIMIT`, and the stream ends when the inner query returned fewer rows than asked and none was cut (a page cut short of the row bound is not the end); the outer `ORDER BY` is a stated guarantee the executor gives anyway; SQLite's stream pulls no row past its cut (the driver's worker reads at most five rows ahead of it: the four `row_buffer_size` on the pool buffers and the one it holds while its `send` waits), so the rows-past-cut figure is PostgreSQL's alone (section 3, PR 8). `heartbeat_interval_secs` reaches the store-backed job as `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`; the in-catalog paths keep the constant until PR 14, which corrects item 27's "until the knob reaches the jobs" (5.1 step 4, section 7). The job's pre-put check is two periods, not one (a beat sent at the tick returns up to a period later), which stays inside the three-period floor; the check measures from the instant taken before the claim's insert, not from the heartbeat handle's start, so an unbounded maintenance check (a catalog query held behind the pool) cannot admit a first put on a row already past `stale_job_after`; when the pipeline returns no put is in flight, so the failure path's `delete_prefix` follows the FAILED write with nothing to await; a lost or stalled heartbeat ends the job with no catalog write, the claim connection being the one that failed; each unknown outcome of a status write is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row; within one cycle the duties run dead-owner first and FAILED-prefix after it, so a row the cycle just failed loses its prefix in that cycle once the dead owner's beat is stale (5.1 steps 4 to 6, 4(e)). SQLite's snapshot slot is kept as the backend's admission bound and the pool raised to `2 * (max_concurrent_jobs + 1)`, at most 34, which is what item 27's "folds it into `max_concurrent_jobs`" became (section 7, PR 6). On SQLite `claim_dead_backup` reads no window: a row whose slot no task holds is dead at once, and the engine calls it on every CREATING row each cycle (one set probe per live row); the dead owner's handle cannot observe `NotMatched` after a takeover on SQLite or PostgreSQL, whose slot and lock are the owner's life, and the engine reaches a taken-over row through `Unknown` and the re-read (4(e)). On MongoDB and Cassandra a successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row in that gap; the generation fence makes the loser's write `NotMatched` and one resolution lands; Cassandra's takeover condition carries the heartbeat it read (`IF owner_generation = AND backup_status = 'CREATING' AND heartbeat_at = `), which puts the liveness test inside the write; Cassandra's `list_backup_rows_older_than` reads the coordinator clock before its scan where `list_dead_restore_jobs` reads it after, both conservative; MongoDB's `Completed` listing requires the column present, since `age_is_stale` alone reads an absent column as the epoch (4(e), PR 8). MongoDB's stream applies no cut of its own (the server's 16 MiB reply bound is under the budget) and feeds no page-bytes counter, so `run_page_bounds` runs on PostgreSQL and SQLite only (section 3, PR 8); Cassandra's registrations sweep is one range read of `backups_by_arn` per cycle, enough while a catalog holds thousands of backups and not more, and the `backup_size_bytes` of `backups_by_account` and `backups_by_table` is written at the claim and not refreshed by `set_status`, which the index listings do not read (they re-read the row), as before PR 8 (PR 6). The spool sweep and the `restore_progress` sweep land with PR 9a, which adds the records they read, and the duty that removed a CREATING restore target with no job row is withdrawn: the job record is the registration on every backend (section 4, PR 8). The vector "round trip" PR 8 adds is its create half, the restore half being PR 9a's; the vector wire test asserts the store's ValidationException on the restore until then, and the fidelity suite's 24 restore cells skip on a server with a store (PR 8, PR 9a). `source_table.time_to_live` is read beside the open, outside the snapshot, and may trail it by the open's duration; DescribeBackup's `ItemCount` is the stats pass's figure while CREATING and the stream's count once AVAILABLE (5.1 step 2). Section 9 lists the metrics as `/metrics` emits them, the `MetricName` variants PR 8 adds (`BackupJobs`, `BackupJobDuration` in microseconds, `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors`, the outcome and the store call in the operation dimension) as JSON under those names, and says the snake-case spellings are a converter's names and not a rendering the server does (section 9). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the pytest jobs and the Rust integration jobs (`filesystem` on PostgreSQL, which appends a store after `init`, and on SQLite, the defaulted root; `s3` on the SQLite job against moto; `none` on MongoDB and Cassandra and on the PostgreSQL Rust integration jobs; `run-integration-dev-mode` runs `tests/test_dev_mode_authorization.py` alone, which creates no backup, without it) and `devtools/run-tests --pytest` refuses without it (section 10, PR 8). PostgreSQL's snapshot-slot refusal now goes through `limit_exceeded_message` naming `[storage.postgres] pool_size snapshot-slot` (item 26's one-implementation rule). The acceptance's "on each backend" for the 100k backup reads "on each backend with a `SnapshotEngine`", Cassandra keeping the legacy create, and for the #618 cells "on PostgreSQL and SQLite", the two deployments `tests/test_backup_job.py`'s fixture starts (the MongoDB deployment and its CI job came with the review, revision item 32); an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`, and UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone (PR 8). The 1M-item RSS runs and the kill points were measured on a 16-core, 30 GiB host (PostgreSQL 16.15 and MongoDB 7.0 in containers on loopback, the bundled SQLite 3.46) and the figures recorded with the PR in `docs/design/15-backup-restore-bench-pr8.md`; the memory bound holds (the 1M run's peak within 10 MiB of the 100k run's on PostgreSQL and SQLite, and the 500k run's within 5 MiB on MongoDB, where the 500k load alone took 27 minutes through the wire on that host; the bound is 112 MiB). Section 7's memory paragraph is rewritten to the terms the worst-expansion run measured: the earlier "the budget plus one row of text, plus its decoded form" left out the page's decoded rows waiting for the pipeline, the items in the feeder, the item channel and the encoder, one row's parse through the deserializer's intermediate value, and the line buffer, and it said nothing of the allocator; with 27.7 MB rows (three to a page) the terms sum to about 400 MB and the high-water mark over idle was 880 MiB on PostgreSQL and 580 to 760 MiB on SQLite across runs (507 MiB under `MALLOC_ARENA_MAX=1`), with MongoDB measured at 474 MiB on 13.5 MB rows, its 16 MiB document cap refusing the full shape; SQLite's driver run-ahead was bounded (sqlx's `row_buffer_size`, 50 rows by default, set to four on the pool: the 24-row run peaked at 844 MiB over idle with 50 and 696 MiB with four, back to back, and a 1M-row backup took 11.4 s with either), and whether the peak then grows with the number of worst rows is settled by the 12, 24 and 48-row series of the 2026-11-01 sentences below (it does with glibc's default arenas and does not under `MALLOC_ARENA_MAX=1`), and a one-item channel saved about 50 MiB of it for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders), so `ITEM_CHANNEL` stays at four. The PR 8 acceptance's "stays within section 7's formula" and "RSS must match the formula" read "under 1.2 times the sum of section 7's terms at the measured row with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one binary, the default-arena peak recorded beside it" (section 7, PR 8). The body sentences each of these corrects are edited with this item (4(e), 5.1 steps 2, 4, 5 and 6, section 7's knob comments, memory paragraph and pool paragraph, section 9, section 10's table, PR 6's slot sentence, PR 8's acceptance, interfaces, page paragraph and duty list, PR 9a's lead-in); earlier items keep their wording, as item 22 requires. (2026-11-01, PR 8 as measured and corrected.) The PostgreSQL page's row bound `n` is sized from the previous page's largest row, not its average, and the floor after the first page is 4 (`MIN_NEXT_PAGE_ROWS`; `MIN_PAGE_ROWS`, 16, is the first page's alone), because the server serializes and spills every one of the `n` rows the inner `LIMIT` reads, so a page costs the server `n` rows of text and not the rows it returns: after a page of 984 small rows and 3 large ones the average sized the next page at 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned, where the largest row sizes it at 4 (208 MiB of temp, 1.1 s), and a page of 27 MB rows now reads one row past its cut rather than thirteen (its steady state 1.2 s a page, 4.4 s before); the over-read that remains is the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, which the sentence of the same day below bounds (section 3, 7, PR 8). The `EXPLAIN` acceptance holds for a page with the planner's estimate above `n` and not everywhere: on a table of 1,060 rows a page near the end whose `n` is above the rows left gets a bitmap scan of the primary key index and a `Sort` of the rows after the seek under the `Limit` (PostgreSQL 16), while on the 100k-row table the index scan holds ten rows from the end; both are pinned by a test and stated in the module doc (PR 8). PostgreSQL's `fetch_page` pulls the page's rows from the connection one at a time and parses each as it arrives, so the page's text is no longer a term of section 7 (the record's 83 MB at the worst shape); the item channel and the decoded page are unchanged; with that change the PostgreSQL worst-shape run peaked at 790 MiB (`VmHWM`) in 16.5 s against the head's 905 MiB in 18.2 s, and its 1M run's peak was 4.0 MiB over the 100k run's (section 7, PR 8). `run_page_bounds` fails at once, naming the key yielded twice, when a stream has yielded more items than the table holds, so an inclusive seek fails in 3 s where it ran until another test's lock wait gave out (PR 8). SQLite's run-ahead is five rows, not four: sqlx's worker steps and copies the next row before its `send` on the four-row channel blocks, 160 MiB at the line cap (section 7, PR 8). The allocator: the peak over idle of the worst-expansion run grows with the number of large rows under glibc's default arenas and is flat under `MALLOC_ARENA_MAX=1`, which is documented for operators (the admin guide's `[backup]` rows, the deployment guide's service unit, section 7) rather than set from the binary; section 7's memory paragraph is the sizing rule (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite, times the allocator's factor, times `max_concurrent_jobs`), measured at 27.7 MB rows, 1,000 small rows first, 12, 24 and 48 such rows back to back, in two series of six runs each on one SQLite binary: 466, 588 and 760 MiB over idle with the default arenas and 481, 452 and 480 MiB under `MALLOC_ARENA_MAX=1` on a quiet host, then 707, 700 and 1,281 MiB and 452, 480 and 455 MiB on the final build with the kill-point runs beside it (the terms' sum at that row 545 MiB; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows), which replaces this item's earlier "the peak did not move between 12 and 24 worst rows" (section 7, PR 8). The #618 cells run on PostgreSQL and SQLite; the RSS acceptance states its 112 MiB bound and MongoDB's 500k run; the testing summary says the backup kill points are run by hand with the posted script, a `before_manifest` point added (the three documents and the data file under the prefix, no manifest, FAILED `owner dead` and the prefix deleted as at `after_data_file`) and the SIGTERM exit timed by polling (55 ms on SQLite, not the script's 5 s sleep); 5.1 step 1 names the store-health check in its order, before the permit; the record is committed as `docs/design/15-backup-restore-bench-pr8.md` and cited by that path, with its SQLite cache explanation corrected to `cache_size = -32000` (31 MiB per connection, `sqlite_util.rs`) and its two 12-row rows labelled by build; the differences document's backup-creation figures are the record's and its restore-interruption row says the prefix of a SQLite row goes once the dead owner's heartbeat is stale, not in the first cycle; the usage guide's backups row says RestoreTableFromBackup restores in-catalog backups only this release and the upgrade manual says a store-backed backup is listed, described and deleted but its restore answers the ValidationException until the restore job lands (section 5.1, 7, 10, testing summary, PR 8). The harness: `tests/test_abac.py`'s backup cell waits for AVAILABLE before DeleteBackup; the job-bound cell's two diagnostics say what each case is; the fidelity suite reads `/health` once and checks that the server's `backup_store` component is the kind `EXTENDDB_TEST_BACKUP_STORE` names and healthy (absent for `none`), the `none` branch of the vector cell captures the ARN of a create accepted against expectation, `_failed_rows` decodes Rust's `Debug` escapes, a cell pins the whole key of a refused item with a UUID hash key and a ten-digit `N` range key in the log line and in `backup list --failed`; the CI start steps of the PostgreSQL, SQLite and S3 jobs wait for the `backup_store` component to be healthy, the pytest lines carry `-rP` so a passed cell's printed branch reaches the log, and the S3 job dumps moto's log on failure and names what its by-name step covers (PR 8). Also: `validate_restorable_item` takes `&RestorableLimits` (the item size, hash key size, range key size and attribute name size of `LimitsConfig`), not `max_item_size_bytes` alone, so the restore's "every item against the target's limits" (5.3 step 5) is the same call; every untrusted text its error carries (a key value, an attribute name, a path segment, the number the wire's reason echoes) is bounded on its own at 4096 bytes of the text the message writes (a key value at 12,288, six times the hash key limit, so a key of control characters is whole too; 4096 is a round figure that keeps a path segment to four times the longest key attribute name), with a 1024-byte prefix and the length for a longer one, so a key of any legal size appears whole in the log line and the FAILED reason; every such text, the attribute path and the echoed number included, is written as a JSON string's content, with U+0085, U+2028 and U+2029 escaped as `\u0085`, `\u2028`, `\u2029` beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log, for a byte-oriented reader and for one that breaks lines the Unicode way (Python's `str.splitlines`) alike; an item whose data file line would exceed `MAX_LINE_BYTES` under a raised `max_item_size_bytes` is refused by the job's feeder through the same count-and-log path (`RestorableItemReason::LineTooLong`, whose reason names the cap, the line's own length being unknown once the count stops), the line measured on the blocking pool for items over the default item limit only, through a counting writer that holds no line, and such an item's validation (the number rule over every member, CPU work proportional to the item's size) run on the blocking pool with it, an item within the default validated on the feeder's own task, so the encoder's own refusal is never the first to see it; a put that fails while the snapshot stream is between pages ends the pipeline at once (the uploader cancels a token the feeder selects on), so the FAILED write does not wait for the stream; the completion log line carries the stream's item count; a job the shutdown cancelled records `BackupJobs{cancelled}` and neither `WorkerLastSuccess` nor `WorkerErrorCount`; `BackupJobDuration` is microseconds from the snapshot's open to the end of the job's last catalog or store call (the AVAILABLE write, or the FAILED write and the prefix's deletion); the reconciler's `CycleReport` counts a dead-CREATING resolution whose write matched no row as left to its owner (`creating_left_to_owner`), and its cycle line counts the dead legacy restores and the FAILED-record targets apart; `reconciler.rs`'s module doc states the FAILED-prefix duty's timing as the code has it, in this cycle on PostgreSQL, MongoDB and Cassandra and once the dead owner's last beat is stale on SQLite (5.1 steps 4 and 5, section 9, PR 8). (2026-11-01, PR 8 as measured and corrected, the row bound's memory.) The sentence above that the over-read happens "once" was false on a table whose large rows are scattered among small ones: with the bound sized from the previous page alone, a large row set the next bound to 4, four small rows set the one after back to 1000, and that page met large rows again, so the 1000-row page recurred every other page for the whole table (5,555 rows read for 1,000 kept, 13.4 GB of server temp files, on a 1,000-row table with 30 such rows; on the release binary the same bytes took 1.9 times as long and 3.2 times the temp files by key layout alone). The bound is now sized from the largest row of the last pages that together hold at least a row bound's worth of rows (`RowBoundMemory`), so it stays at the floor while large rows keep arriving within that span and returns to the row bound a row bound's worth of small rows after the last large one: 1,901 rows read for 1,000 kept and 3.9 GB on the probe (a memory of four pages read the same 5,555 as one page, since four pages of four rows are sixteen rows), and the interleaved table's backup 54.6 s and 3.2 GB of temp against 72.0 s and 9.2 GB, the grouped table unchanged. The bound that remains, stated as it is: the page that follows a row bound's worth of small rows reads up to `n` rows of up to the line cap, at most once per row bound's worth of small rows, and nothing in the window form stops the server at the budget. A page query that does stop there, a recursive walk of the primary key one index probe per row, was measured and not taken: it read exactly the rows it returned on every shape (the interleaved table in 38.8 s with 2.5 GB of temp) and cost one probe per row, 23.1 s against 11.9 s for the 1M-row backup; the record carries both. `the_row_bound_remembers_large_rows_among_small_ones` pins the memory's figures on a 300-row table with nine such rows (201 rows read past the cut over 53 round trips), `the_page_query_is_an_index_scan_on_every_key_shape` pins the index scan on the six key shapes (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation, each at `n` 16 and 1000, first page and a page after a key, and the stream over 20,000 rows), and `page_sql_for` takes the collation flag `open_stream` decides, so the legacy table's query is tested as the engine writes it (section 3, 7, PR 8). Section 7's rule is scoped: the terms are at the default `max_item_size_bytes`, where no stored item's line exceeds the cap, and on an instance that raises it they are at the largest line a stored item expands to; the feeder's measurement of an item over the default limit holds no line (a counting writer that stops at the cap), where the earlier text said its buffer was bounded by the cap and the code held the whole line (section 7, PR 8). The one-arena factor is measured on both backends: the PostgreSQL worst shape at 12 rows peaked at 340 MiB `VmHWM` under `MALLOC_ARENA_MAX=1` against 750 to 788 MiB under the default arenas, 0.82 of the terms at that row on PostgreSQL (413 MiB, no run-ahead), and the setting's cost on the request path is measured: 3.5 to 9.2% on SQLite (GetItem, Query, Scan, paged Scan) and about 2% on PostgreSQL (the scans, 2.1%), so the deployment guide's service unit carries the line commented out with the condition and the cost, and section 7 states both (section 7, PR 8). 4(e)'s unknown-outcome rule reads as `write_status` has it: each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read that shows the row in the target status counts as matched, anything else as matching no row; the review checklist's bullet says the same (4(e), PR 8). The PR 8 duty list marks the resume of a dead RUNNING restore as PR 9a's, this release's reconciler logging such a record once; section 10's table gains PR 8's rows (the admin guide's `[backup]` rows and `[backup.store]` section, the deployment guide's service unit, the upgrade manual's asynchronous-create and rollback paragraphs, the design README's row 15b); the PR 9a lead-in names `RestorableLimits`; the ratios are given to two places (0.86 to 2.35; 2.35 at 48 rows); the SIGTERM exit is 55 ms in every statement; the record attributes the PostgreSQL worst run's idle to the restarted process's startup work on a table of 27.7 MB rows, not to the 1M load (sections 4, 7, 10, PR 8, PR 9a). (2026-11-01, PR 8 as corrected.) Section 3's parenthesis on the PostgreSQL cut, the PR 8 bullet, its two paging acceptance cells and the Interfaces paragraph's `n` sentence described the recursive walk that was measured and not taken (one row per step, no row read past the cut, two index probes) and the rule before the memory (`n` from the previous page's largest row, the over-read "once"); each is corrected to the shipped form: the three-layer window page with up to `n - 1` rows read past the cut, `n` sized from the memory, the first paging cell naming the record's `EXPLAIN (ANALYZE, BUFFERS)` figure (4 read, 1 past the cut at the floor) and `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips), the `EXPLAIN` cell the plans the tests pin (`Limit` over one `Index Scan`; the bitmap scan with a `Sort` near the end of a small table) (section 3, PR 8). The over-read's statement gains the second page, which follows the first page's 16 rows and asks for the row bound: the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, each up to `n` rows of up to the line cap, said the same way in the PR 8 page paragraph, the module doc, the test's doc and the record (the earlier sentence of this item said "the page that follows a row bound's worth of small rows" alone); the floor's cost is stated beside it: after a page holding a row at the line cap the bound stays at the floor for a row bound's worth of rows, 250 pages of four, whatever those rows hold, so a table whose large rows are rarer than one per row bound pays 250 round trips per large row where one page of memory pages it at the row bound (2,585 page queries against 38 on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, 1.06 against 1.09 GB of temp; the record), and the record's "changes no page of a small-row table" reads "of a table with no large row"; the one-page-memory figure the test's doc, the record and the PR text cite is 306 rows past the cut over 7 pages (section 7, PR 8). 4(e) said the PostgreSQL reconciler "reads the heartbeat alone and never probes the lock" and named "the one liveness rule the three share": PostgreSQL's `claim_dead_backup` tries the advisory lock before it tests the heartbeat, so it takes over only a row whose heartbeat is stale and whose lock is free; a stopped owner (SIGSTOP, a paused VM) whose session lives keeps its row until the session ends or it finishes, and a vanished host's lock is released when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two; 4(e) and 5.1 step 4 now say so (the SIGSTOP sentence scoped to MongoDB and Cassandra, with the PostgreSQL outcome: no takeover while the session lives; on resume the job completes under its own generation or, when the stop outlasted two heartbeat periods, stops as a lost claim with no catalog write and the reconciler resolves the row), the trait doc of `claim_dead_backup`, the admin guide's `stale_job_after_secs` row and the differences document's dead-owner clause gain "on PostgreSQL, once the owner's claim session has ended as well", and item 27's "the heartbeat alone" stands for PR 6's `remove_dead_creating_backup_row`, which `backup purge` still uses (4(e), 5.1 step 4, PR 8). The one-arena cost on PostgreSQL reads "about 2% (the scans, 2.1%)" in section 7, this item, the admin guide, the deployment guide and the PR text, the record's scans being −2.1%; section 7 labels the base of each ratio (the peak over idle on SQLite, the `VmHWM` on PostgreSQL, about 0.47 over idle); the admin guide's `max_concurrent_jobs` row restates section 7's sizing rule (1.2 times the terms per job under `MALLOC_ARENA_MAX=1`, no bound measured under the default arenas, the PostgreSQL figure as `VmHWM`, 2.35) and its `stale_job_after_secs` row the dead-row outcomes (AVAILABLE with a complete manifest, FAILED without, a restore job's target removed); the upgrade manual's rollback paragraph names `backup sync --adopt-store`, as 4(c) requires (section 7, 10, PR 8). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the Rust integration jobs included: the SQLite Rust integration job's file-backed server has the defaulted root, so its CreateBackup is the asynchronous one, and `tests/rust`'s backup tests assumed the synchronous create and left their backups; the suite now waits for AVAILABLE, deletes what it makes and reads the variable as the Python suite does, and `run-tests`' usage, `CONTRIBUTING.md` and the developer test guide name the variable and the refusal (section 10, PR 8). The PR 8 kill-point list names `before_manifest`, and section 10's limits-doc row the rewritten "Backup store" row (PR 8). 5.1 step 6 says a job cancelled after its manifest put completes AVAILABLE, the manifest being the commit record, where it said a cancelled job can never complete (5.1 step 6, PR 8). On PostgreSQL a job stopped and resumed after more than two heartbeat periods stops as a lost claim with no catalog write, its session's end frees the advisory lock and the reconciler resolves the row, where 5.1 step 4 said it resumes and completes; the unknown-outcome rule's second write is stated as the rule of a backend whose claim connection survives the outcome (MongoDB, Cassandra), PostgreSQL's failing with the session; the reconciler's failed `get`, `list` and `delete_prefix` calls are `BackupStoreErrors` as the job's are; and `extenddb stop` waits for the sum of the server's shutdown bounds, the 30 s job drain included (4(e), 5.1 step 4, section 9, PR 8). After #335 and #681 merged, PR 8 is rebased onto PR 7's rebased head: the server's shutdown drains the jobs before the registration supervisor's return and the worker set's stop (#666), a job still running at the drain's bound keeps the serving registration as a worker that could not be stopped does, so a migration cannot start beside it, `extenddb stop` waits for the sum of the bounds that path has (76 s: 5 s of connections, 30 s of jobs, 10 s for the supervisor, 16 s for the workers' stop, 10 s for the release, 5 s for the runtime), and the reconciler's withdrawn duty is stated after migration 6 (PR 8). After #433 (attribute-name size on every write path) merged, the Drawbacks line names an attribute name over the limit that `UpdateItem` and a `TransactWriteItems` `Update` stored before it beside the #618 values (Drawbacks, PR 8). +32. (2026-11-04, after the maintainer's review of PR 8.) The heartbeat's one-period bound is placed by the backend (`JobClaim::heartbeat_within`, a provided method that bounds the whole call; SQLite's override takes the engine write lock first and bounds only the `UPDATE`, so a writer holding that lock past a period, a GSI backfill say, delays the beat and the claim re-check and ends no job; the re-check's confirmation counts as a beat for the pre-put check; after the maintainer's second round, the pre-put check also passes while a beat is in flight and has not failed, since a backup large enough to reach a put while the beat was parked on the lock past two periods stopped at that put, the maintainer's 180,000-item backup beside a GSI build twice at the third or fourth data file, and the reconciler then failed the row and deleted its objects; the safety argument is in 5.1 step 4 and `heartbeat.rs`'s module doc, and the tests are a SQLite cell whose second data file is put under a hold of the lock, `a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it`, and an engine cell with a claim whose beat parks, `a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost`), and `heartbeat.rs`'s SIGSTOP text says that a job with no beat in flight resumes under its own generation once its next beat succeeds. The store-backed create checks the maintenance claim before the permit, as the in-catalog paths do, and inserts no row for a refusal (5.1 step 2 and 4(f) are edited: a FAILED `maintenance` row carried the insert's heartbeat and `backup purge`'s wait counted it as a live job for the stale window). Before every put the job checks the store as the runtime holds it (healthy, and its identity the row's) and on a failure fails the row with the store's reason and makes no store call, the prefix's deletion included (4(c)'s swapped-root case). The open and the insert are selected against the shutdown token. The uploader cancels the feeder's token before it awaits an encoder, so a put that fails mid-file ends the job while the stream is held. A data file whose stored size differs from the encoder's compressed count fails the job (the dead-owner probe refuses that difference). The AVAILABLE write sets `TableSizeBytes` from the stream as it sets `ItemCount` (the differences document names both). `failed_row_retention_secs` is at least `stale_job_after_secs` (a FAILED row removed sooner left a stopped owner's late object with no row to delete it under). A registration lost while serving cancels the jobs' token, which the runtime replaces for the jobs that start under the new registration; the shutdown sequence after the listener is one function the server's tests drive with a job that outlives a short drain. The plain-HTTP listener's drain is bounded as the TLS listener's is. CreateBackup is bound to the in-catalog path on SQLite where the serve lock is not taken (not Unix), since the slot claim is sound only with one process per file. 4(c) states the gap of the different-roots guard before the first AVAILABLE row. The compose files' `stop_grace_period` is 80 s. The kill-point scripts are committed under `devtools/` and the clippy job lints `backup-test-hooks`; a MongoDB job runs the backup suites against a filesystem store. The tests added are the review's: the lock held past two periods on SQLite, the mid-file put failure with the stream held, the unhealthy and renamed store mid-job, the key-ordered stream out of order, `BackupStoreErrors` for the job's `delete_prefix` and the probe's `list`, the re-read under another generation, the two-period sample, the nesting rule through lists and mixed levels, the keepalive literals, the driver's five-row run-ahead, the beat between the takeover's read and its LWT on Cassandra, `pool_size_for_jobs(11, 4)`, and the retention floor. After the third round the clause is bounded where the beat does not park before its bound (`JobClaim::heartbeat_parks_before_its_bound`, `true` on SQLite only): a beat in flight for more than a period on PostgreSQL, MongoDB or Cassandra confirms nothing, since a process stopped mid-beat and resumed would otherwise pass the check until its timer raised the loss (`a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks`). diff --git a/docs/design/README.md b/docs/design/README.md index f9806b336..4fb516ca3 100755 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -20,6 +20,7 @@ This directory contains the authoritative design documents for ExtendDB (extendd | 12 | [Auth / Authz Cache](12-auth-authz-cache.md) | In-process cache for IAM data (credentials, policies, tags, table key info); companion to 05 and 06 | | 13 | [MongoDB Storage](13-storage-mongodb.md) | MongoDB storage backend: engine and catalog traits over the official driver, replica-set transactions | | 15 | [Backup and Restore](15-backup-restore.md) | Overview of the off-host backup series: the invariants, the per-operation contracts and a short section per later PR; the format, store and snapshot documentation lives with their crates (`crates/core/src/backup_format`, `crates/backup-store`, `crates/storage/src/snapshot.rs`) | +| 15b | [Backup memory and duration record, PR 8](15-backup-restore-bench-pr8.md) | The measurements section 7 and revision item 31 of the design cite: peak RSS and duration of the 100k, 1M, worst-expansion and incompressible backups per backend, the allocator series, the PostgreSQL page's cost | ## How to Use These Docs diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index a6088e3fa..4b9960450 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -91,7 +91,7 @@ from a request sent to that service or backend. | Vector index update propagation | Eventually consistent, the same model as a GSI | Matches DynamoDB. Maintenance is queued on the same propagation queue as async GSIs, so a search immediately after a write may not see it. Governed by the same `index_propagation_delay_ms` setting; unlike a GSI there is no per-index override. A value of 0 applies maintenance inline in the write's own transaction, which is stricter than the service and exists so a test can assert steady state without waiting. Inline additionally requires the table's propagation queue to be empty: while rows queued during an index build are still draining, a new write queues behind them instead of overtaking them, so a brief search-after-write window exists even at 0 until the queue drains. That window is ordering, not data loss; the write is applied, in order, by the queue worker. That zero behaves differently while an index is still building, and the two backends differ: **PostgreSQL** applies inline only to an index that is already `ACTIVE`, and defers a write to a building index whatever the delay says, because the write must not reach the index ahead of the backfill's older snapshot of the same item; **SQLite** does not check the status on this path, so with a zero delay a write during a backfill is applied inline and bypasses the hold that keeps the two ordered. The SQLite behaviour is tracked as a defect (F-20), not intended, and it is reachable only with the zero delay, which is a test setting. | | `SearchVectors` score at extreme magnitudes (PostgreSQL backend only) | Returns the true distance as a number for any vector of finite components | The score is bounded to a finite value instead, and the bound is not a measured service answer. pgvector accumulates distances in single precision, so magnitudes far below `f32::MAX` overflow inside the extension: Euclidean above about 9.2e18, dot product above about 1.8e19, and cosine at both ends, above about 1.8e19 and below about 3.7e-23, which is where a component's single-precision square rounds to zero. A non-finite score cannot be serialised as JSON at all, so the result is bounded in SQL: `1e308` for an overflowed distance, `-1e308` for an overflowed negated inner product, which the score contract negates so a client sees `1e308` in `Score`, and 1.0 for a cosine that comes back NaN. Ranking is unaffected at the overflow end, because each bound sits at the end its metric overflows towards, so the farthest row stays farthest and the most similar stays most similar; two rows that both overflow tie, and the tie breaks on the base key. At the underflow end cosine loses resolution rather than being bounded, and which side is tiny decides how much. For a tiny **query** vector, its norm underflows while the inner product usually does not, so the quotient is an infinity that pgvector clamps and the reported distance collapses to one of 0, 1 or 2 following the sign of the inner product, with the 1.0 substitute firing only when the vectors are exactly orthogonal and the quotient is therefore 0/0 (measured). For a tiny **stored** vector it is worse: the stored norm is computed in single precision and reaches zero, so the zero-vector guard fires and that row reports 1.0 at every angle, parallel included. A corpus of tiny embeddings therefore loses ranking altogether rather than losing resolution. For a tiny query vector, by contrast, ranking still separates nearer-than-orthogonal from farther and loses resolution only within each half. The SQLite backend owns its own arithmetic, computes in double precision, and reports the true value, which is why this row is scoped to PostgreSQL. | | Vector index deletion window | `UpdateTable` Delete leaves the index in `DELETING` long enough to observe, then removes it | No observable `DELETING` window on either backend: the catalog row is removed inside the `UpdateTable` transaction, so a `DescribeTable` immediately afterwards already omits the index. The index's data table is dropped after that commit, in a separate transaction, and the two backends handle a failure there differently. **PostgreSQL** treats it as best effort, because the data table lives in a different database entirely: a failure is logged and skipped rather than failing the request. **SQLite** propagates it, so a failed drop returns an error from an `UpdateTable` whose catalog change has already committed, which is the more surprising outcome of the two: the index is gone from the catalog and the caller saw a failure. So an operator debugging a leftover `_ddb_vec_*` table should look for that warning rather than assume the delete was incomplete. | -| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Not supported. On PostgreSQL and SQLite, the two backends with vector indexes, `CreateBackup` of such a table is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend", and no backup is created, because a restore cannot carry the index data across. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | +| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Partial. With a configured `[backup.store]`, `CreateBackup` of such a table succeeds on PostgreSQL and SQLite, the two backends with vector indexes: the backup is written to the store and its manifest records the index specifications, and `DescribeBackup` and `ListBackups` report it as any other backup (wire test on both backends). Without a store (the section absent on PostgreSQL, or `kind = "disabled"`), where `CreateBackup` writes the in-catalog format, it is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend" and no backup is created, because that format cannot carry the index data across. `RestoreTableFromBackup` of any backup in the store answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" until a later release's restore job, which restores a vector table into a backend with vector support and refuses it into one without. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | | `SearchVectors` endpoint | Served only on `search-dynamodb..amazonaws.com`. The standard `dynamodb..amazonaws.com` endpoint answers the same request with HTTP 400 `UnknownOperationException` ("This operation is not supported by this endpoint"); every control-plane and item operation stays on the standard endpoint. Signing is unchanged either way (service name `dynamodb`, target prefix `DynamoDB_20120810`) | Served on the same endpoint as every other operation, so a client is pointed at one ExtendDB endpoint for all of them. Two consequences worth knowing: an SDK that resolves a separate search hostname from its endpoint ruleset needs its endpoint overridden to reach ExtendDB, and ExtendDB does **not** reproduce the service's refusal, so a test asserting `UnknownOperationException` for vector search on the base endpoint passes against Amazon DynamoDB and fails here. | | `SearchVectors` result order for equal distances | Measured unstable: three identical searches returned tied rows in three different orders, and a top-k that truncates a tie group keeps an arbitrary subset of it | Deterministic on both backends, by different means. **PostgreSQL** sorts explicitly on the base table's full primary key after the score, so the order is a property of the query. **SQLite** issues no `ORDER BY` and resolves ties by scan order through a stable top-k, so its order is a property of the plan rather than something the query guarantees. Either way a client re-issuing an identical search sees the same order, which is stricter than the service rather than divergent in outcome. Do not rely on the two backends agreeing on which subset of a truncated tie group they keep. | | Vector indexes on the MongoDB backend | Vector indexes and `SearchVectors` are available | Not supported at all. That backend provides no vector search implementation, so the engine's capability gate refuses `CreateTable` and `UpdateTable` carrying `VectorIndexes` and every `SearchVectors` request, with the same capability message a PostgreSQL deployment without pgvector returns. Every other vector row in this document describes the PostgreSQL and SQLite backends. | @@ -122,15 +122,16 @@ Behaviour stated as measured, and every number below, was observed with release | Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only *(from the code and storage-level tests; not checked on a running server)* | | `DeleteBackup` and `DeleteTable` during a restore | `DeleteTable` on the `CREATING` target is `ResourceInUseException`. `DeleteBackup` during a restore was not measured | While the target is `CREATING`, `DeleteTable` on it is `ResourceInUseException` on all four backends, as on the service. `DeleteBackup` on its source backup is `BackupInUseException` on all four: that refusal is ExtendDB's contract, not measured against the service. Both succeed once the target is `ACTIVE`. How the restore and `DeleteBackup` are ordered differs *(from the code)*. PostgreSQL and SQLite register the restore in the transaction that creates the target, holding the backup row, so a concurrent `DeleteBackup` is ordered strictly before (the restore then finds no backup) or after (it finds the restore). MongoDB runs the restore's registration and `DeleteBackup`'s claim each as one session transaction that first increments a lock field on the backup's catalog document while it is `AVAILABLE`, so the two conflict and one orders wholly before the other: a `DeleteBackup` that wins leaves the registration no `AVAILABLE` backup (`BackupNotFoundException`), a registration that wins makes the `DeleteBackup` `BackupInUseException`. Cassandra uses two lightweight transactions on different partitions: the target's row is inserted first (`IF NOT EXISTS`, which reserves the name), then the restore is registered on the backup's row (`IF backup_status = 'AVAILABLE'`), which `DeleteBackup` claims on the same partition; a concurrent `DescribeTable` can therefore see a `CREATING` target for a restore that then fails with `BackupNotFoundException`, after which the target is removed. On MongoDB `DeleteBackup` writes the backup `DELETING` before it drops the data and `DELETED` after; a `DeleteBackup` whose request fails or is cancelled after the drop started leaves the backup `DELETING`, which `DescribeBackup`, `ListBackups` and a restore treat as gone (`BackupNotFoundException`), and a retry of `DeleteBackup` finishes the drop and answers `DELETED` *(from the code and storage-level tests)*. MongoDB reads the catalog from the primary with read concern `local` regardless of the connection string, which the lock and the job records rely on; the connection string must use `readPreference=primary` (any other read preference is refused at startup). The write concern follows the connection string and must be `majority` (the server default): a connection string with `w=1` is refused at startup, because a `DELETING` write or a job record acknowledged under `w=1` just before a primary failover can be rolled back | | Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | -| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part | +| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | | Point-in-time recovery | 35-day window: `UpdateContinuousBackups` enables continuous backups, and `RestoreTableToPointInTime` restores to any second in the window | Not supported; use on-demand backups (`CreateBackup` and `RestoreTableFromBackup`) instead. `DescribeContinuousBackups` reports `ContinuousBackupsStatus: ENABLED` (the service reports this unconditionally) with `PointInTimeRecoveryStatus: DISABLED` and no restorable-time window. `UpdateContinuousBackups` with `PointInTimeRecoveryEnabled: true` returns `ContinuousBackupsUnavailableException`; with `false` it returns the same description. `RestoreTableToPointInTime` checks the source table, then returns `PointInTimeRecoveryUnavailableException` ("Point in time recovery is not enabled for table ''"), the exception the service models on that operation. A table that does not exist is `TableNotFoundException` ("Table not found: ") on all three operations, as on the service. Same on all four backends | -| Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace. Take database-level backups for off-host copies until the backup store program ships | -| Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE` on all four backends. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw it `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB reports `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s); a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed writer of a later release refuses it before it writes, and the in-catalog format of this release carries it as earlier releases did. Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the check runs after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | +| Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
    //`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | +| Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` and `TableSizeBytes` in `SourceTableDetails` are the table's figures as last counted by the stats pass while `CREATING`, which is 0 and 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact figures once `AVAILABLE`: the items written and the sum of their DynamoDB sizes; the service reported 0 and 0 in every fresh-table backup measured). `CREATING` lasts the whole upload (the durations below), and while it lasts `DeleteBackup` and `RestoreTableFromBackup` of the backup answer `BackupInUseException` "Backup is being created: " on all three backends; the 16 service backups measured were `AVAILABLE` 12 to 90 ms after `CreateBackup` answered, so a client that deletes or restores a backup right after creating it polls `DescribeBackup` first here. The backup is the table as it stood when its snapshot opened, which is inside the request but after the request's checks (the table's state, the store's health, the maintenance claim) and after any wait for a job permit, up to 30 s (below): every write acknowledged before the open is in the backup and none acknowledged after it, so a `PutItem` acknowledged while the request waited for a permit is in the backup (measured: a write acknowledged 1.01 s after a `CreateBackup` that waited for a permit was in it, 2 of 2 runs), and `BackupCreationDateTime` is the open's millisecond. Measured on the service: a restored backup contained every write acknowledged before `CreateBackup` was sent and the 18 writes acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed, so the service takes the table at an instant inside the request as well. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). A `CreateBackup` still waiting in its snapshot open or row insert when the server stops writes no row; it is answered `ServiceUnavailable` "The server is shutting down. Retry the request." only after a registration loss, where the jobs' token is cancelled while the listener still serves, since on SIGTERM the listener's 5 s connection drain closes the connection before that token is cancelled (measured: the connection closed 8.2 s after the request was sent). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). Without a store, and on Cassandra, a table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s): the in-catalog format writes no store prefix. With a store, a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it with `ValidationException` before it opens a snapshot or writes a row ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"). Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | | `DeleteBackup` response | The full `DescribeBackup` description with `BackupStatus` `DELETED`, including `SourceTableDetails.ProvisionedThroughput` (a required member) and `SourceTableFeatureDetails` (measured) | The same on all four backends: the engine reads the backup's recorded definition before the delete and fills the description as `DescribeBackup` does, with `BackupStatus` `DELETED`. A backup whose definition this build cannot read or that does not parse is still deleted and described without feature details (logged at warn), which means a damaged `PROVISIONED` backup is described with on-demand `ProvisionedThroughput` `{0, 0}`: ExtendDB's choice, deletability over the throughput member of a row that is being removed, where `DescribeBackup` on the same row is an `InternalServerError`; a failure to reach the database fails the request and deletes nothing. | | A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | -| A store-backed backup while the configured store is unhealthy | Not applicable: managed storage | `DeleteBackup` and `RestoreTableFromBackup` of a backup in the configured `[backup.store]`, while that store is unhealthy or its first check is pending, answer HTTP 400 `ValidationException` "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", the kind (`filesystem`, `s3`) and the category in parentheses being the store's (`/health` reports the same category); before the first check of a store checked after the listener: "The s3 backup store configured on this server has not passed a check yet (its first check is pending); the operation is refused until it does". `DescribeBackup` and `ListBackups` still describe such a backup. Once `DeleteBackup`'s DELETING write has committed the request answers the description with `BackupStatus` `DELETED` whatever the object deletion does; a deletion the store does not answer within its bound is left DELETING for the reconciler, logged, and the request still answers DELETED. No backup lives in a store on this release's write path (CreateBackup writes the in-catalog format), so these answers need a row a later release wrote. | +| A store-backed backup while the configured store is unhealthy | Not applicable: managed storage | `CreateBackup` on PostgreSQL, SQLite and MongoDB with a configured `[backup.store]`, and `DeleteBackup` and `RestoreTableFromBackup` of a backup in that store, while the store is unhealthy or its first check is pending, answer HTTP 400 `ValidationException` "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", the kind (`filesystem`, `s3`) and the category in parentheses being the store's (`/health` reports the same category); before the first check of a store checked after the listener: "The s3 backup store configured on this server has not passed a check yet (its first check is pending); the operation is refused until it does". `DescribeBackup` and `ListBackups` still describe such a backup. Once `DeleteBackup`'s DELETING write has committed the request answers the description with `BackupStatus` `DELETED` whatever the object deletion does; a deletion the store does not answer within its bound is left DELETING for the reconciler, logged, and the request still answers DELETED. `CreateBackup`'s refusal comes after the table's ACTIVE probe and the name checks and before the maintenance check and the job permit, so it takes no slot and writes no row; the in-catalog create (no store, Cassandra) touches no store and is not refused. | | `DescribeBackup` when the definition cannot be read | Not applicable | A definition row that does not parse, a renamed column, or a revoked privilege (`Internal` from the backend) is an `InternalServerError`, as it is for `RestoreTableFromBackup` on the same read; only a definition version this build does not know is described without `SourceTableFeatureDetails`. | -| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot. How *(from the code)*: PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a MongoDB or Cassandra table while it is not being written if the backup must be consistent | +| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | +| `CreateBackup` of a table holding an `N` value the wire did not validate | Every stored value is a valid number, and a backup holds the table as it is | With a configured `[backup.store]` the backup fails. Four write paths of earlier builds stored an `N` value such as `1E126` or `abc` without validating it (#618: `BatchWriteItem`, `TransactWriteItems`, `UpdateItem` on an absent item in the key it creates, `ImportTable`), and `UpdateItem` and a `TransactWriteItems` `Update` of builds before #433 stored an attribute name over 65,535 bytes without checking it, and the store-backed job validates every item before encoding it with the rule the restore applies (`validate_restorable_item`: every `N` and `NS` member a valid number, the nesting depth, the key shape (each key attribute present, of its declared type, not empty, an `N` key normalized), the item size against the instance's `max_item_size_bytes`, the hash and range key sizes and the attribute name sizes against the instance's limits, and for an item over the default item limit the data file line against the format's 32 MiB cap), so no backup that no restore can load is reported `AVAILABLE`. After the first refusal the job reads and validates the rest of the snapshot, logs each refused item with its key, attribute path and reason beside the backup ARN (up to 1,000, then a count), and fails the backup with a reason giving the count and the first item, naming #618 only when that item's refusal is an `N` value ("3 item(s) cannot be restored; first: key {...}, attribute amount: Number overflow... (#618)"); an item over `max_item_size_bytes` (stored under a higher limit) fails it the same way with a reason naming the size and the limit and not #618. The client sees `CREATING` and then `BackupNotFound`; the reason reaches the operator (`extenddb backup list --failed`, and the log, which a tenant's support case can cite; every tenant-supplied text in the line is written as a JSON string's content, control characters and the Unicode line separators U+0085, U+2028 and U+2029 escaped, so a newline or a line separator in an attribute name or a number value cannot forge a second log record); the way out is to rewrite or delete each named item through the wire (a bad range key cannot be rewritten, since a PutItem with that key is refused; `DeleteItem` removes it). The in-catalog create (no store, Cassandra) backs such a table up and restores it as before (wire tests on PostgreSQL and SQLite for each path) | | `ListBackups` paging | At most 100 summaries per page, with `LastEvaluatedBackupArn`; `Limit` 1 to 100. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list. `TableName` is checked as the model says (3 to 255, `[a-zA-Z0-9_.-]+`, the "N validation error(s) detected" texts). An `ExclusiveStartBackupArn` of another account is `ValidationException` "Invalid Request: The supplied ExclusiveStartBackupArn does not match the account"; of another region, or of a table other than `TableName`'s, `ValidationException` ("does not match the region", "does not match the table name predicate"); a time bound before the epoch is `ValidationException`; a lower bound above the upper bound is `ValidationException` "Time range lower bound must be less than or equal to upper bound" (all measured) | The same paging on all four backends: `Limit` outside 1 to 100 is `ValidationException`, `ExclusiveStartBackupArn` resumes after the given backup, and summaries are ordered by creation time then ARN. Summaries carry `TableId`. `TimeRangeLowerBound` (inclusive) and `TimeRangeUpperBound` (exclusive) keep backups created in that range, and `BackupType` keeps backups of that type (`USER` when absent, every type for `ALL`); every ExtendDB backup is `USER`, so `SYSTEM` and `AWS_BACKUP` list nothing. The filters apply before paging, so `Limit` and the cursor count only backups that pass. Measured on SQLite and PostgreSQL with whole-second bounds between two backups taken 2 s apart; the millisecond edges are from the engine's unit tests. The request checks above are the service's, with its texts where captured: the account text in full; the region and table-name texts complete the captured fragments on the account text's pattern ("Invalid Request: The supplied ExclusiveStartBackupArn does not match the region" / "... the table name predicate"), checked in that order after the ARN's shape; a bound before the epoch is refused with ExtendDB's own text naming the member; equal bounds list nothing. An `ExclusiveStartBackupArn` that is not in the listing (excluded by the filters, or deleted since the previous page) is placed by the creation time in its id; one whose id carries no time (the 13-digit ids of earlier builds, with or without the hex suffix: only the 14-digit form carries one) is then a `ValidationException`. A cursor in another region is accepted only when it is the stored ARN of a backup of the account (or of `TableName`), which is what `LastEvaluatedBackupArn` hands out for a backup created under an earlier `server.region`, so paging passes such a backup; any other cursor in another region is refused ("does not match the region"). If that backup is deleted between two pages, its ARN is no longer a stored one and the next page is refused the same way (a same-region cursor would be placed by its time instead); start the listing again. Under a `server.region` outside `[a-z0-9-]` the cursor is still the stored ARN: a listing of backups created under a valid region pages, and a cursor that is the ARN of a backup an earlier release created under the invalid region is refused with "Invalid Request: BackupArn is not valid". `TableName` is a table name or a table ARN, as `CreateBackup` accepts it (the service accepts both, measured on DynamoDB): the ARN form lists what the name lists, an ARN whose region is not the server's (compared as written) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice), another account's ARN is `AccessDeniedException`, and the cursor's table check runs against the resolved name. A numeric `TableName` is refused with the text the service uses for a map ("Start of structure or map found where not expected"), not measured for a number. Every `ListBackups` request reads all of the account's backups (or all of the table's, with `TableName`) from storage and pages them in memory, so walking N backups 100 at a time reads about N²/100 rows in total *(from the code)* | | Backup ARN and creation time | Backup id is 14-digit zero-padded epoch milliseconds, `-`, 8 hex digits; `BackupCreationDateTime` has sub-second precision | The same shape on all four backends; `BackupCreationDateTime` is the millisecond in the ARN | | Running two servers against one MongoDB catalog | Not applicable | MongoDB uses the fixed database names `extenddb_catalog` and `extenddb_data` whatever `--catalog-db` and `--data-db` say, and takes no serve lock: two servers on one replica set share one catalog and both start. Each server's reconciler acts only on a restore whose job record's heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, so one server does not remove another live server's restore; every heartbeat a restore writes, the registration's first one included, is stamped with the database's `$$NOW` inside the registration's transaction, so a host whose clock is behind the database's plays no part in that judgement *(from the code and storage-level tests)*. Running more than one server per catalog is otherwise unsupported (see the deployment guide) | diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index 9caaaa099..376072943 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -128,12 +128,13 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | Limit | DynamoDB Value | Status | Notes | |-------|---------------|--------|-------| -| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | -| `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed and backed up whatever its name, as on the service: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed writer of a later release refuses it before it writes, and the in-catalog format of this release carries it (see the differences document's backup creation row). Both members may also be the table's ARN; an ARN whose region is not the server's is `ValidationException` "1 validation error detected: Invalid AWS region in ''", the service's text. The value is echoed in the message as sent | +| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a server raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34 (the slots are capped at 16), with a warning naming both numbers, so the slot bound refuses no job the `max_concurrent_jobs` permit admitted (no change at the defaults: a pool of 10 has 4 slots for 4 jobs); above 16 jobs a 17th concurrent job is refused by the slot. A `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); PostgreSQL bounds open table snapshots at half its (raised) pool the same way, naming `[storage.postgres] pool_size snapshot-slot`; a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | +| `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed and backed up whatever its name, as on the service: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it before it opens a snapshot or writes a row, and the in-catalog create (no store, Cassandra) carries it (see the differences document's backup creation row). Both members may also be the table's ARN; an ARN whose region is not the server's is `ValidationException` "1 validation error detected: Invalid AWS region in ''", the service's text. The value is echoed in the message as sent | | `BackupArn` | 37 to 1024 characters | Maximum enforced; minimum implied | Surrounding ASCII whitespace is trimmed first, then over 1024 bytes, not of the backup ARN shape, a control or non-ASCII character, or a region outside `[a-z0-9-]`, an account that is not digits, or a table name outside `[a-zA-Z0-9_.-]` is `ValidationException` "Invalid Request: BackupArn is not valid"; the right shape with an id that is not a backup id is "Invalid Backup ARN"; a digit-only account that is not the caller's is `AccessDeniedException`. No minimum is checked: the shortest string of the right shape is longer than 37 characters. Another region of the caller's account resolves (the server's region is tried first, then the ARN as written). `ListBackups`'s `ExclusiveStartBackupArn` must also match the account, the region, and `TableName` when given | | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | -| Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where ExtendDB's backups will live from a later release's CreateBackup on (this release's writes the in-catalog format and the store serves restores and deletes of the backups it holds): a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses the restore and delete of store-backed backups (CreateBackup, which writes the in-catalog format in this release, is not refused) with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | -| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service. Backups are stored inside the database the server uses; take database-level backups for off-host copies | +| Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB (Cassandra, any backend without a store, and SQLite on a platform where the serve lock is not taken (not Unix), write the in-catalog format) and where restores and deletes of store-backed backups read: a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses CreateBackup where it writes to the store and the restore and delete of store-backed backups with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | +| Backup data file | Not specified | N/A | A data file holds one item per line as DynamoDB JSON, gzip-compressed, in key order on a table whose key schema is orderable (`item_order` in the manifest); a line is at most 32 MiB (`MAX_LINE_BYTES`, which holds the largest line a 400 KB item can expand to, 27.7 MB); a file closes at the first item that takes its uncompressed bytes to or past `[backup] data_file_target_bytes` (default 64 MiB, minimum 32 MiB, the line cap), so it exceeds the knob by less than one line. The snapshot streams on PostgreSQL and SQLite page the table by key and cut each page at 64 MiB of item text (`SNAPSHOT_FETCH_BYTES`, a constant), so a page holds at most that plus one row; MongoDB's batches are bounded by the server's 16 MiB reply | +| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root (the differences document's "Where backups live" row); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | | Point-in-time recovery | 35-day window | N/A | Not supported: `UpdateContinuousBackups` refuses to enable it (`ContinuousBackupsUnavailableException`) and `RestoreTableToPointInTime` refuses every restore (`PointInTimeRecoveryUnavailableException`); use on-demand backups (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index 36ccd41c7..e33c62e3d 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -196,7 +196,7 @@ still reports it as running, with the PID shown as unknown. Pass `serve --foreground --write-pid-file` to write one anyway, which restores both commands at the cost of needing a writable `run_dir`. 8. Spawn background tasks (log level polling, throttling polling, GSI delay polling, stream cleanup, TTL expiry, metrics persistence) -9. On SIGTERM/SIGINT: drain connections (5s timeout), exit +9. On SIGTERM/SIGINT: drain connections (5s timeout), end the running backup jobs through their failure path (30s), stop the background tasks (5s, then abort) and give up the serving registration, exit ## Catalog Model diff --git a/docs/manuals/03-usage-guide.md b/docs/manuals/03-usage-guide.md index c165b0dc5..63c45d01b 100755 --- a/docs/manuals/03-usage-guide.md +++ b/docs/manuals/03-usage-guide.md @@ -571,7 +571,7 @@ extenddb reproduces DynamoDB error responses exactly. Common errors: | PartiQL | ✗ (not planned) | | DAX | ✗ (not applicable) | | Global Tables | ✗ (future) | -| On-demand backups (CreateBackup / RestoreTableFromBackup) | ✓ | +| On-demand backups (CreateBackup / RestoreTableFromBackup) | Partial: CreateBackup ✓ (to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB; in-catalog without one and on Cassandra); RestoreTableFromBackup ✓ for in-catalog backups only, store-backed backups from a later release (their restore answers a `ValidationException` naming the store this release; the upgrade manual) | | Point-in-time recovery | ✗ (`UpdateContinuousBackups` refuses to enable it; see differences-from-dynamodb.md) | | Import/Export (local filesystem) | ✓ (FileSource instead of S3) | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 047494eed..28ceef124 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -40,7 +40,7 @@ Or manually: kill ``` -extenddb handles SIGTERM and SIGINT gracefully — it drains active connections for up to 5 seconds before exiting. +extenddb handles SIGTERM and SIGINT gracefully: it drains active connections for up to 5 seconds (the TLS listener and the plain-HTTP listener of a dev-mode build alike), then waits up to 30 seconds for its running backup jobs to end (a job the stop cancels runs its failure path, which marks its row FAILED and deletes what it wrote to the store; on a large S3 prefix that takes longer than the other bounds), then up to 10 seconds for its registration supervisor to return, up to 16 seconds to stop its background workers (5 seconds for them to return, 1 second for the ones then aborted, and 10 seconds more behind a stop the supervisor was still running), up to 10 seconds to give up its serving registration (kept, to expire by itself, while a background worker or a backup job may still run), and 5 seconds for its runtime's blocking threads, and exits. With no backup job running the job wait ends at once, and in an ordinary stop the supervisor, the workers and the release each take moments. A `CreateBackup` that is still waiting in its snapshot open or row insert when the stop begins writes no row, and its client sees the connection closed by the drain (measured: 8.2 s after the request was sent): the jobs' token, which turns such a request into `ServiceUnavailable` "The server is shutting down. Retry the request.", is cancelled after the drain, so that answer reaches a client only after a registration loss, when the token is cancelled while the listener still serves. `extenddb stop` waits up to 76 seconds, the sum of these bounds, before it reports that the server did not exit. Note that `extenddb stop` reads the PID file, which a server started with `serve --foreground` does not write by default. Stop a foreground server through @@ -53,13 +53,15 @@ SIGTERM directly. Alternatively start it with ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health -# {"status":"healthy","components":{}} +# {"components":{},"status":"healthy"} +# (the order of the keys depends on the build: the default PostgreSQL build +# writes "status" first, the SQLite build "components" first; read the keys) # with a backup store configured: -# {"status":"healthy","components":{"backup_store":{"status":"healthy","kind":"filesystem"}}} +# {"components":{"backup_store":{"kind":"filesystem","status":"healthy"}},"status":"healthy"} # while it cannot be reached: -# {"status":"degraded","components":{"backup_store":{"status":"unhealthy","kind":"s3","reason":"unreachable"}}} +# {"components":{"backup_store":{"kind":"s3","reason":"unreachable","status":"unhealthy"}},"status":"degraded"} # right after startup, before an S3 store's first check has returned: -# {"status":"healthy","components":{"backup_store":{"status":"checking","kind":"s3"}}} +# {"components":{"backup_store":{"kind":"s3","status":"checking"}},"status":"healthy"} ``` `status` is `healthy`, or `degraded` when a component is unhealthy; the HTTP @@ -147,17 +149,17 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau | Key | Default | Description | |-----|---------|-------------| -| `data_file_target_bytes` | `67108864` | Uncompressed bytes per backup data file (minimum 32 MiB); read by the store-backed backup writer | -| `heartbeat_interval_secs` | `15` | The job heartbeat timer. The in-catalog backup and restore paths of this release heartbeat every 15 s whatever this says | -| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead and the reconciler fails the job and removes its restore target. Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the jobs of this release heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | -| `failed_row_retention_secs` | `604800` | How long FAILED backup rows are kept for the operator (7 days) | -| `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup | -| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller | +| `data_file_target_bytes` | `67108864` | Uncompressed bytes per backup data file (minimum 32 MiB, the cap on one line): a data file closes at the first item that takes it to or past this, so it exceeds it by less than one line | +| `heartbeat_interval_secs` | `15` | The job heartbeat timer. The store-backed backup job heartbeats every `max(heartbeat_interval_secs, 15)` s and starts no store write unless a beat confirmed its claim within two periods or a beat is still in flight (on SQLite a beat waits behind any writer holding the engine write lock, a GSI backfill say, for as long as it holds it, and a backup goes on writing meanwhile; on the other backends a beat in flight counts for one period from its start), the claim's insert counting as the first beat; the in-catalog backup and restore paths heartbeat every 15 s whatever this says | +| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead restore job is FAILED and its target removed. Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | +| `failed_row_retention_secs` | `604800` | How long FAILED and DELETED backup rows of the configured store are kept once their objects are gone (7 days); `extenddb backup list --failed` prints the FAILED ones with their reason. At least `stale_job_after_secs`: the reconciler deletes a stopped job's late object only under its row, so the row outlives the window in which that job can still write; a smaller value is refused at startup | +| `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup. Each cycle checks the store, resolves backup and restore jobs whose owner is dead, finishes deletions, deletes the objects of FAILED backups once their heartbeat is older than `stale_job_after_secs`, and removes tombstones past `failed_row_retention_secs` (the differences document's restore-interruption row lists the duties) | +| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen) | | `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | #### [backup.store] -The store CreateBackup's backups will live in, and the one restores and deletes of store-backed backups read. The section's keys, defaults and the refusals of a bad value are the backup store crate's (`crates/backup-store/src/config.rs`); the table below is what the server adds. One definition of the shape exists, so an unknown key anywhere under `[backup]` is refused at load. +The store CreateBackup writes to on PostgreSQL, SQLite and MongoDB, and the one restores and deletes of store-backed backups read. The section's keys, defaults and the refusals of a bad value are the backup store crate's (`crates/backup-store/src/config.rs`); the table below is what the server adds. One definition of the shape exists, so an unknown key anywhere under `[backup]` is refused at load. | Key | Default | Description | |-----|---------|-------------| @@ -165,11 +167,11 @@ The store CreateBackup's backups will live in, and the one restores and deletes | `filesystem.path` | `backups/` beside a SQLite database file | The root directory of a filesystem store. It must exist, be owned by the extenddb user with mode `0700`, and sit under directories owned by root or that user which others cannot write to; `validate` refuses anything else at startup. A relative path, like a relative `spool_path`, is resolved against the working directory the server was started from (the daemon's own working directory is `/`) | | `s3.bucket`, `s3.prefix`, `s3.region`, `s3.endpoint`, `s3.force_path_style`, `s3.allow_insecure_endpoint`, `s3.upload_part_size_bytes`, `s3.max_concurrent_uploads` | | The S3 store; see the crate's `config` module doc. Credentials come from the AWS SDK's chain and are never in the file | -When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call; with one configured, this release's CreateBackup still writes the in-catalog format (the store-backed writer is a later release's) and the store serves restores and deletes of the backups it holds. Backups of either format stay listable, describable, restorable and deletable. +When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call. With one configured, CreateBackup on PostgreSQL, SQLite and MongoDB writes the backup to the store: the request takes the table's snapshot (after its checks and after any wait for a job permit, up to 30 s, so the backup is the table at that open and a write acknowledged during the wait is in it), writes the catalog row and answers `CREATING`, and a job of the server uploads gzip data files and the manifest under `/
    //` below the root and marks the row `AVAILABLE` (the job's log lines carry the ARN: `backup : started, items in the table as last counted by the stats pass, key-ordered` (or `unordered`, for a table whose key shape or column collation has no format order) at the start, the stats pass's count, which is 0 on a table the pass has not reached, and `backup : AVAILABLE, items, bytes written in s` at the end, with the backup's own count); Cassandra, which has no table snapshot, keeps the in-catalog create with a store configured. Backups of either format stay listable, describable and deletable, and in-catalog backups restorable; the restore of a store-backed backup is a later release's and answers `ValidationException` until then. -Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check's store calls are bounded (5 s for a filesystem root, 30 s for a bucket); the catalog's own reads (the catalog id, the rows' store ids, the newest AVAILABLE row) run beside the calls and not under the bound, so a slow catalog costs the check time and never marks the store unhealthy. A store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), a root's calls run one at a time so that calls started before the first reached its bound (two `DeleteBackup` requests together, a `DeleteBackup` beside a check) wait for it under the bound, then have the bound for their own call (a slow call behind another finishes, where one bound over both would abandon it), and park no thread of their own, and `stop` or SIGTERM ends the server within 5 s with that thread still parked. The deletions `DeleteBackup` and the reconciler run against the store go through the same bound and rule: `DeleteBackup` answers the description with `BackupStatus` `DELETED` once its DELETING write has committed, and a deletion the store does not answer within the bound, or that leaves an object under the prefix, leaves the row DELETING for the reconciler and is logged, never an error to the client. A default SQLite root whose startup checks did not finish starts `checking` with its rule kept: a first check that fails while the catalog has no rows for it is the same warning and leaves the store unconfigured. CreateBackup is not refused by an unhealthy store in this release: it writes the in-catalog format, which touches no store; the store-backed create of a later release is. +Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), CreateBackup (where it writes to the store) and the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check's store calls are bounded (5 s for a filesystem root, 30 s for a bucket); the catalog's own reads (the catalog id, the rows' store ids, the newest AVAILABLE row) run beside the calls and not under the bound, so a slow catalog costs the check time and never marks the store unhealthy. A store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned check's call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), a root's calls run one at a time so that calls started before the first reached its bound (two `DeleteBackup` requests together, a `DeleteBackup` beside a check) wait for it under the bound, then have the bound for their own call (a slow call behind another finishes, where one bound over both would abandon it), and park no thread of their own, and `stop` or SIGTERM still ends the server, the runtime giving that thread 5 s before the process exits without it (after the job wait of up to 30 s when a backup job is running; see Stopping). The deletions `DeleteBackup` and the reconciler run against the store go through the same bound and rule: `DeleteBackup` answers the description with `BackupStatus` `DELETED` once its DELETING write has committed, and a deletion the store does not answer within the bound, or that leaves an object under the prefix, leaves the row DELETING for the reconciler and is logged, never an error to the client. A default SQLite root whose startup checks did not finish starts `checking` with its rule kept: a first check that fails while the catalog has no rows for it is the same warning and leaves the store unconfigured. CreateBackup on Cassandra is not refused by an unhealthy store: its in-catalog create touches no store. -The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no rows for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id); point every instance at one shared root before the first backup. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. +The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. `extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. Neither command runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database under the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse; the one exception is a RUNNING restore of kind `manifest`, which no release before PR 9a writes: the command leaves it for the reconciler that resumes it, waits on it as on any RUNNING restore and refuses naming the table; and a server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra both commands open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. @@ -932,6 +934,8 @@ JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions ```bash curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health # {"status":"healthy","components":{}} +# (the key order depends on the build; the earlier Health Check section +# shows the components a backup store adds) ``` The `components` object carries the `backup_store` component when a backup diff --git a/docs/manuals/06-developer-test-guide.md b/docs/manuals/06-developer-test-guide.md index 4e97f24b0..3eefa6ea2 100755 --- a/docs/manuals/06-developer-test-guide.md +++ b/docs/manuals/06-developer-test-guide.md @@ -150,6 +150,20 @@ devtools/run-tests --extenddb --pytest --filter test_put_item 1. A running extenddb server: `./target/debug/extenddb serve --config extenddb.toml` 2. `EXTENDDB_TEST_ENDPOINT=https://localhost:18443` 3. `EXTENDDB_ADMIN_PASSWORD=` +4. `EXTENDDB_TEST_BACKUP_STORE=`, required for `--pytest` + (and so for `--all`) and read by the Rust integration suite's backup + tests: where the server under test writes its backups. `filesystem` for a + server with a `[backup.store]` filesystem root, which a file-backed SQLite + server without the section has by default (`backups/` beside its database + file); `s3` for an S3 store; `none` for a server without a store + (`kind = "disabled"`, or PostgreSQL, MongoDB and Cassandra without the + section). With a configured store CreateBackup is asynchronous and this + release refuses to restore its backups, so the backup tests derive their + expected outcome from the variable and fail rather than guess when it is + unset; `run-tests --pytest` refuses before any test runs with + `error: --extenddb --pytest requires EXTENDDB_TEST_BACKUP_STORE to be set.` + The suites check the value against the server's `/health` `backup_store` + component, so a wrong value fails too. The `run-tests` script automatically: - Performs health check on the target endpoint @@ -185,8 +199,8 @@ The `run-tests` script automatically: The primary test suite is in Python, testing extenddb through the same AWS SDK interface that customers use. ```bash -# Via run-tests script (recommended) -devtools/run-tests --extenddb --pytest +# Via run-tests script (recommended; EXTENDDB_TEST_BACKUP_STORE as above) +EXTENDDB_TEST_BACKUP_STORE=filesystem devtools/run-tests --extenddb --pytest # Run in parallel (uses 1/3 of CPU cores by default) devtools/run-tests --extenddb --pytest --parallel diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 8593fded4..6c81ffbdc 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -6,7 +6,7 @@ This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 6, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), and migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)). -Configure `[backup.store]` before the release that stops creating in-catalog backups. In this release CreateBackup writes the in-catalog format whether or not a store is configured, and logs a deprecation warning naming `[backup.store]` on each call when none is; a later release's CreateBackup writes to the store and refuses without one. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. +Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described and deleted like any other, but its restore is a later release's: until the restore job lands, RestoreTableFromBackup of a backup in the configured store answers `ValidationException` ("is in a backup store; this release restores in-catalog backups only"), while in-catalog backups, those of the previous release included, restore as before; a deployment that needs to restore the backups it takes this release stays on the in-catalog format (`kind = "disabled"`, below) until then. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. @@ -45,9 +45,11 @@ A migration is additive when the previous release serves the catalog correctly w ### Migration 6: the backup catalog index +A backup of a table that holds an `N` value the wire did not validate, or an attribute name over the limit, fails on the store-backed path where the previous release backed it up. Four write paths stored such a value (`1E126`, `abc`) unvalidated (#618: BatchWriteItem, TransactWriteItems, UpdateItem on an absent item in the key it creates, ImportTable), UpdateItem and a TransactWriteItems Update of releases before #433 stored an attribute name over 65,535 bytes unchecked, and the store-backed job validates every item with the rule the restore applies, so no backup that no restore can load is reported `AVAILABLE`. This meets file-backed SQLite without a `[backup.store]` section (the defaulted root) and any deployment once a store is configured. The client sees `CREATING` and then `BackupNotFound`; the reason reaches the operator only: `extenddb backup list --failed` prints the FAILED row with the count of refused items and the first one, and the server log names each refused item's key and attribute path beside the backup ARN (up to 1,000 per backup), which a tenant's support case can cite (every tenant-supplied text in the line is written as a JSON string's content, control characters and the Unicode line separators U+0085, U+2028 and U+2029 escaped, so a newline or a line separator in an attribute name or a number value cannot forge a second log record). The way out is to rewrite or delete each named item through the wire, which validates the value (a bad range key cannot be rewritten, since a PutItem with that key is refused; `DeleteItem` removes it); the in-catalog create of a server without a store still backs such a table up. + Migration 6 adds the catalog index of backups kept in a backup store, turns each restore registration into a restore job record, adds the maintenance claim record (`backup_maintenance`) and writes the catalog's random id (`settings.catalog_id`). On SQLite and PostgreSQL the index is the `backup_index` table and the job record is new columns of `table_restores`; on MongoDB the index is the `backup_index` collection and the job record a `restore_job` field of each restore target's `tables` document; on Cassandra the index is new columns of `backups_by_arn` and the `backup_index_by_status` table, and the job record new `restore_*` columns of `tables`. Stop every server built before this migration before running `extenddb migrate --yes`: such a server neither reads the index nor marks the restore registrations it writes, so a restore it ran during the upgrade would be owned by nobody. A restore in flight at that stop is left as a RUNNING job record whose owner is dead; the first reconciler pass of the new server (within `backup.reconcile_interval_secs`, default 60 s) marks it FAILED and removes its CREATING target, so the name is free again and the table has to be restored anew. On Cassandra the backfill reads `tables` with a filtered scan and writes each registration with a lightweight transaction, which takes seconds on a catalog of ordinary size. On Cassandra and MongoDB, which have no transaction around a body, every write of the migration is safe to send again: a `migrate --yes` that fails part way leaves migration 6 unrecorded, `serve` keeps refusing the catalog, and the next `migrate --yes` runs the body again and finishes. -A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). No backup store exists yet in this release, so the restored catalog loses nothing a server wrote outside the catalog. +A rollback to the previous binary needs two things: the catalog restored from the pre-upgrade backup (the previous binary refuses a catalog carrying migration 6, exit status 4), and the `[backup]` section removed from the configuration file, which the previous binary refuses as an unknown field before it reaches the catalog (`unknown field 'backup'`). The pre-upgrade backup predates every backup this release wrote to the store, so a restored catalog has no `backup_index` row for them while the store keeps their bytes under its root; the previous binary serves neither (it does not read the store), and a later release's `backup sync --adopt-store` re-adopts them (the store's identity file names the pre-rollback catalog, so the re-upgraded catalog finds the store unhealthy until it is adopted; PR 10). In-catalog backups taken after it are lost with the rows, as before. ## Writing a New Migration diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index d8e664c5c..198c6e781 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -120,7 +120,13 @@ exec extenddb serve --foreground --config "$CONFIG" `extenddb stop` cannot stop a foreground server, because there is no PID file to read; stop the container instead, which delivers `SIGTERM` and triggers the same -graceful shutdown. Outside a container, where you may want `stop` to work, add +graceful shutdown. With a backup job running the shutdown takes up to 35 +seconds before the workers stop, and up to 76 seconds in all when the database +is slow to answer (the admin guide's Stopping section); a runtime whose grace +period is shorter (Docker's default is 10 s) kills the process during the job +drain, and the reconciler resolves the job's row as it does after a crash. The +repository's compose files set `stop_grace_period: 80s`, above that sum; a +compose file of your own needs the same. Outside a container, where you may want `stop` to work, add `--write-pid-file` — it writes the PID file to the usual `run_dir` path, which then has to be writable. @@ -261,6 +267,13 @@ User=extenddb Group=extenddb Restart=on-failure RestartSec=5 +# On a host that backs up tables of large items, one malloc arena keeps a +# backup job's high-water mark at the pipeline's terms whatever the number of +# large items it reads (the admin guide's [backup] max_concurrent_jobs row +# gives the figures). It costs the request path a few percent (3.5 to 9.2% +# on GetItem, Query and Scan on SQLite, about 2% on PostgreSQL (the scans, +# 2.1%)), so the line is left to the operator: +#Environment=MALLOC_ARENA_MAX=1 [Install] WantedBy=multi-user.target @@ -297,7 +310,7 @@ pool_size = 50 # Increase for higher concurrency Ensure PostgreSQL `max_connections` accommodates the total pool size plus overhead. -Backup and restore work is bounded per server by `[backup] max_concurrent_jobs` (default 4): `CreateBackup` and `RestoreTableFromBackup` jobs together, on every backend, and a request that finds every slot held waits up to 30 s for one, holding nothing meanwhile, and is then refused with `LimitExceededException` and creates nothing. On PostgreSQL each running job holds one pooled connection for its copy plus one dedicated connection outside the pool for its claim, and startup raises `pool_size` and `catalog_pool_size` to at least `2 * max_concurrent_jobs + 10` with a warning naming both numbers (the admin guide's `[backup]` table). On SQLite a `CreateBackup` also holds its snapshot connection for its duration, and a server admits at most `pool_size / 2 - 1` of them at once (between 1 and 16; 4 for the default `pool_size` of 10), so that half the pool stays free for ordinary requests; a `CreateBackup` over that bound is refused at once with the same `LimitExceededException`. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore goes on in the background; the MongoDB driver does not enforce `waitQueueTimeoutMS`, so the admission bound is what keeps a flood of restores from holding every pooled connection. +Backup and restore work is bounded per server by `[backup] max_concurrent_jobs` (default 4): `CreateBackup` and `RestoreTableFromBackup` jobs together, on every backend, and a request that finds every slot held waits up to 30 s for one, holding nothing meanwhile, and is then refused with `LimitExceededException` and creates nothing. On PostgreSQL each running job holds one pooled connection for its copy plus one dedicated connection outside the pool for its claim, and startup raises `pool_size` and `catalog_pool_size` to at least `2 * max_concurrent_jobs + 10` with a warning naming both numbers (the admin guide's `[backup]` table). On SQLite a `CreateBackup` also holds its snapshot connection for its duration, and a server admits at most `pool_size / 2 - 1` of them at once (between 1 and 16; 4 for the default `pool_size` of 10), so that half the pool stays free for ordinary requests, and raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34, so that bound refuses no job `max_concurrent_jobs` admits (the admin guide's `max_concurrent_jobs` row; above 16 jobs the slot refuses the seventeenth); a `CreateBackup` over that bound is refused at once with the same `LimitExceededException`. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore goes on in the background; the MongoDB driver does not enforce `waitQueueTimeoutMS`, so the admission bound is what keeps a flood of restores from holding every pooled connection. ### Cassandra Connection Pool diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 81fbbc7b3..ca1996766 100755 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -240,9 +240,9 @@ journalctl -t extenddb -f **Fix:** If permission denied, run `extenddb stop` as the same user that started extenddb. If the process already exited, the stale PID file will be cleaned up on the next `extenddb stop` invocation. -### `extenddb (pid ) did not exit within 10s after SIGTERM` +### `extenddb (pid ) did not exit within 76s after SIGTERM` -**Cause:** The server received SIGTERM but did not shut down within the 10-second timeout. It may be stuck draining long-running requests or waiting on a database operation. +**Cause:** The server received SIGTERM but did not shut down within the 76-second timeout, the sum of its shutdown bounds (5 s to drain connections, 30 s for running backup jobs to end, 36 s to stop its background workers and give up its serving registration, 5 s for its runtime). It may be stuck draining long-running requests or waiting on a database operation. **Fix:** Send SIGKILL to force termination: `kill -9 `. Check PostgreSQL for long-running queries that may be blocking shutdown. diff --git a/extenddb.sample.toml b/extenddb.sample.toml index 34d07a562..d4ff16b12 100755 --- a/extenddb.sample.toml +++ b/extenddb.sample.toml @@ -179,7 +179,7 @@ # Backup and restore jobs and the reconciler. Every key has a default; the # section may be left out. stale_job_after_secs must be at least three # heartbeat periods, the larger of heartbeat_interval_secs and the 15 s the -# jobs of this release heartbeat at (45 at the least); max_concurrent_jobs is +# in-catalog jobs heartbeat at (45 at the least); max_concurrent_jobs is # at most 1024; on PostgreSQL pool_size and catalog_pool_size are raised to # at least 2 * max_concurrent_jobs + 10. # data_file_target_bytes = 67108864 # Uncompressed bytes per data file (minimum 32 MiB) @@ -193,9 +193,10 @@ # # it defaults to backup-spool/ beside the database file # [backup.store] -# Where CreateBackup's backups will live, from a later release (this release's -# CreateBackup writes in-catalog backups with any store, and the store serves -# restores and deletes of the backups it holds). Absent on file-backed SQLite: a filesystem +# Where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB +# (CreateBackup answers CREATING and a job of the server uploads; Cassandra +# writes in-catalog backups with or without a store), and where restores and +# deletes of store-backed backups read. Absent on file-backed SQLite: a filesystem # store at backups/ beside the database file. Absent on PostgreSQL, MongoDB, # Cassandra and an in-memory SQLite database: no store, and CreateBackup # writes in-catalog backups with a deprecation warning. kind = "disabled" diff --git a/tests/rust/src/backup_arn_scoping.rs b/tests/rust/src/backup_arn_scoping.rs index 42e5a2905..6dc8714f6 100644 --- a/tests/rust/src/backup_arn_scoping.rs +++ b/tests/rust/src/backup_arn_scoping.rs @@ -12,8 +12,11 @@ //! //! These tests drive the ARN directly rather than provisioning a second account, //! so they exercise the resolution rule without needing a second set of -//! credentials. +//! credentials. Every backup they create is waited to AVAILABLE before it is +//! described or deleted (`backup_restore::create_available_backup`: a +//! store-backed ExtendDB creates asynchronously) and deleted at the end. +use crate::backup_restore::create_available_backup; use crate::test_base::*; use aws_sdk_dynamodb::types::{ AttributeDefinition, BillingMode, KeySchemaElement, KeyType, ScalarAttributeType, @@ -126,14 +129,7 @@ async fn describe_backup_with_own_arn_still_resolves() { let table = format!("BackupScopeOwn_{}", ts()); make_table(&table).await; - let create = c - .create_backup() - .table_name(&table) - .backup_name("scope-own") - .send() - .await - .unwrap(); - let arn = create.backup_details().unwrap().backup_arn().to_string(); + let arn = create_available_backup(c, &table, "scope-own").await; let resp = c.describe_backup().backup_arn(&arn).send().await.unwrap(); assert_eq!( @@ -159,16 +155,12 @@ async fn describe_backup_with_own_nonexistent_arn_is_backup_not_found() { let table = format!("BackupScopeMiss_{}", ts()); make_table(&table).await; - let create = c - .create_backup() - .table_name(&table) - .backup_name("scope-miss") - .send() - .await - .unwrap(); - let real = create.backup_details().unwrap().backup_arn().to_string(); + let real = create_available_backup(c, &table, "scope-miss").await; // Same account/region/table prefix, a backup id that was never issued. - let missing = format!("{}/00000000000000-deadbeef", real.rsplit_once('/').unwrap().0); + let missing = format!( + "{}/00000000000000-deadbeef", + real.rsplit_once('/').unwrap().0 + ); let err = c .describe_backup() @@ -194,18 +186,16 @@ async fn describe_backup_after_delete_is_backup_not_found() { let table = format!("BackupScopeDel_{}", ts()); make_table(&table).await; - let create = c - .create_backup() - .table_name(&table) - .backup_name("scope-del") - .send() - .await - .unwrap(); - let arn = create.backup_details().unwrap().backup_arn().to_string(); + let arn = create_available_backup(c, &table, "scope-del").await; c.delete_backup().backup_arn(&arn).send().await.unwrap(); - let err = c.describe_backup().backup_arn(&arn).send().await.unwrap_err(); + let err = c + .describe_backup() + .backup_arn(&arn) + .send() + .await + .unwrap_err(); assert_eq!( err_code(&err), Some("BackupNotFoundException"), @@ -214,6 +204,7 @@ async fn describe_backup_after_delete_is_backup_not_found() { c.delete_table().table_name(&table).send().await.ok(); } + #[tokio::test] async fn backup_ids_are_not_derived_from_the_timestamp_alone() { let c = client(); @@ -222,14 +213,7 @@ async fn backup_ids_are_not_derived_from_the_timestamp_alone() { let mut arns = Vec::new(); for i in 0..3 { - let create = c - .create_backup() - .table_name(&table) - .backup_name(format!("scope-id-{i}")) - .send() - .await - .unwrap(); - arns.push(create.backup_details().unwrap().backup_arn().to_string()); + arns.push(create_available_backup(c, &table, &format!("scope-id-{i}")).await); } for arn in &arns { diff --git a/tests/rust/src/backup_restore.rs b/tests/rust/src/backup_restore.rs index e7b93884a..4c09373db 100755 --- a/tests/rust/src/backup_restore.rs +++ b/tests/rust/src/backup_restore.rs @@ -2,12 +2,104 @@ // SPDX-License-Identifier: Apache-2.0 //! Backup and restore integration tests — mirrors Java `BackupRestoreTests`. +//! +//! The service creates a backup asynchronously; so does ExtendDB with a +//! configured `[backup.store]` (CreateBackup answers CREATING and a job +//! uploads it). A backup still CREATING refuses DeleteBackup +//! (`BackupInUseException`) and restore, so every test that goes on to +//! describe, delete or restore a backup it just created waits for AVAILABLE +//! first ([`wait_for_backup_available`]), and deletes the backups it made so +//! the server's store does not keep them. Where a backup lives decides what +//! RestoreTableFromBackup answers this release; the tests read it from +//! `EXTENDDB_TEST_BACKUP_STORE` ([`backup_store`]), as the Python suite does. use crate::test_base::*; use aws_sdk_dynamodb::types::{ AttributeDefinition, BillingMode, KeySchemaElement, KeyType, PointInTimeRecoverySpecification, ScalarAttributeType, }; +use aws_sdk_dynamodb::Client; +use std::time::{Duration, Instant}; + +/// Where the server under test writes its backups, from +/// `EXTENDDB_TEST_BACKUP_STORE`: `filesystem` or `s3` (a configured +/// `[backup.store]`, under which a backup is in the store and this release +/// refuses to restore it, PR 9a adding the restore) or `none` (the +/// in-catalog create, which restores). Against the service the variable is +/// not read (the restore runs). Against ExtendDB a test whose outcome depends +/// on it fails when it is unset, with the text the Python fixture prints, +/// rather than accept either outcome. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum BackupStore { + Filesystem, + S3, + None, +} + +pub fn backup_store() -> BackupStore { + const VALUES: [&str; 3] = ["filesystem", "s3", "none"]; + if is_real_dynamodb() { + return BackupStore::None; + } + let raw = std::env::var("EXTENDDB_TEST_BACKUP_STORE").unwrap_or_default(); + match raw.trim() { + "filesystem" => BackupStore::Filesystem, + "s3" => BackupStore::S3, + "none" => BackupStore::None, + "" => panic!( + "EXTENDDB_TEST_BACKUP_STORE is unset; set it to one of {VALUES:?} to say where the \ + server under test writes its backups" + ), + other => panic!( + "EXTENDDB_TEST_BACKUP_STORE is {other:?}; set it to one of {VALUES:?} to say where \ + the server under test writes its backups" + ), + } +} + +/// Poll DescribeBackup until the backup is AVAILABLE. Any other status than +/// CREATING is a failure: a backup that fails is BackupNotFound, never +/// FAILED, on the wire, so the describe itself errors then. +pub async fn wait_for_backup_available(c: &Client, arn: &str) { + let deadline = Instant::now() + Duration::from_secs(120); + loop { + let resp = c + .describe_backup() + .backup_arn(arn) + .send() + .await + .unwrap_or_else(|e| { + panic!("describe_backup of {arn} while waiting for AVAILABLE: {e:?}") + }); + let status = resp + .backup_description() + .and_then(|b| b.backup_details()) + .map(|d| d.backup_status().as_str().to_owned()) + .unwrap_or_default(); + if status == "AVAILABLE" { + return; + } + assert_eq!(status, "CREATING", "backup {arn} is {status}"); + assert!( + Instant::now() < deadline, + "backup {arn} not AVAILABLE in time" + ); + tokio::time::sleep(Duration::from_millis(250)).await; + } +} + +/// CreateBackup, then wait for AVAILABLE; the backup's ARN. +pub async fn create_available_backup(c: &Client, table: &str, backup_name: &str) -> String { + let create = create_backup_with_retry(c, table, backup_name).await; + let arn = create.backup_details().unwrap().backup_arn().to_string(); + wait_for_backup_available(c, &arn).await; + arn +} + +/// DeleteBackup after the test; a backup already gone is not a failure. +pub async fn delete_backup_quietly(c: &Client, arn: &str) { + c.delete_backup().backup_arn(arn).send().await.ok(); +} async fn make_table(name: &str) { let c = client(); @@ -71,8 +163,27 @@ async fn create_backup_happy_case() { let resp = create_backup_with_retry(c, &table, "test-backup-1").await; let details = resp.backup_details().unwrap(); assert!(!details.backup_arn().is_empty()); - assert_eq!(details.backup_status().as_str(), "AVAILABLE"); + // The in-catalog create answers AVAILABLE; the service and a store-backed + // ExtendDB answer CREATING and the backup reaches AVAILABLE later. + let status = details.backup_status().as_str(); + assert!( + status == "AVAILABLE" || status == "CREATING", + "CreateBackup answered {status}" + ); + let arn = details.backup_arn().to_string(); + wait_for_backup_available(c, &arn).await; + let desc = c.describe_backup().backup_arn(&arn).send().await.unwrap(); + assert_eq!( + desc.backup_description() + .unwrap() + .backup_details() + .unwrap() + .backup_status() + .as_str(), + "AVAILABLE" + ); + delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&table).send().await.ok(); } @@ -101,13 +212,13 @@ async fn describe_backup() { let table = format!("DescBackup_{}", ts()); make_table(&table).await; - let create = create_backup_with_retry(c, &table, "desc-backup").await; - let arn = create.backup_details().unwrap().backup_arn().to_string(); + let arn = create_available_backup(c, &table, "desc-backup").await; let resp = c.describe_backup().backup_arn(&arn).send().await.unwrap(); let desc = resp.backup_description().unwrap(); assert_eq!(desc.backup_details().unwrap().backup_arn(), arn.as_str()); + delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&table).send().await.ok(); } @@ -129,12 +240,14 @@ async fn list_backups() { let table = format!("ListBackup_{}", ts()); make_table(&table).await; - create_backup_with_retry(c, &table, "list-1").await; - create_backup_with_retry(c, &table, "list-2").await; + let first = create_available_backup(c, &table, "list-1").await; + let second = create_available_backup(c, &table, "list-2").await; let resp = c.list_backups().table_name(&table).send().await.unwrap(); assert!(resp.backup_summaries().len() >= 2); + delete_backup_quietly(c, &first).await; + delete_backup_quietly(c, &second).await; c.delete_table().table_name(&table).send().await.ok(); } @@ -156,8 +269,7 @@ async fn delete_backup() { let table = format!("DelBackup_{}", ts()); make_table(&table).await; - let create = create_backup_with_retry(c, &table, "del-backup").await; - let arn = create.backup_details().unwrap().backup_arn().to_string(); + let arn = create_available_backup(c, &table, "del-backup").await; let resp = c.delete_backup().backup_arn(&arn).send().await.unwrap(); let desc = resp.backup_description().unwrap(); @@ -185,31 +297,60 @@ async fn restore_table_from_backup() { .unwrap(); } - let create = create_backup_with_retry(c, &table, "restore-backup").await; - let arn = create.backup_details().unwrap().backup_arn().to_string(); + let arn = create_available_backup(c, &table, "restore-backup").await; let restored = format!("Restored_{}", ts()); - let resp = c + let restore = c .restore_table_from_backup() .target_table_name(&restored) .backup_arn(&arn) .send() - .await - .unwrap(); - assert_eq!( - resp.table_description() - .unwrap() - .table_status() - .unwrap() - .as_str(), - "CREATING" - ); - - wait_for_active(c, &restored).await; - let scan = c.scan().table_name(&restored).send().await.unwrap(); - assert_eq!(scan.count(), 5); + .await; + match backup_store() { + // A backup in the configured store is listed, described and deleted + // but not restored this release (the restore job is PR 9a's). + BackupStore::Filesystem | BackupStore::S3 => { + let err = restore.expect_err("a store-backed backup restored"); + assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); + let msg = err_msg(&err); + assert!( + msg.contains("is in a backup store"), + "the refusal does not name the store: {msg}" + ); + println!("restore of a store-backed backup refused: {msg}"); + assert!( + c.describe_table() + .table_name(&restored) + .send() + .await + .is_err(), + "the refused restore created {restored}" + ); + } + BackupStore::None => { + let resp = restore.unwrap(); + assert_eq!( + resp.table_description() + .unwrap() + .table_status() + .unwrap() + .as_str(), + "CREATING" + ); + + wait_for_active(c, &restored).await; + let scan = c.scan().table_name(&restored).send().await.unwrap(); + assert_eq!(scan.count(), 5); + println!( + "restore of an in-catalog backup completed with {} items", + scan.count() + ); + + c.delete_table().table_name(&restored).send().await.ok(); + } + } - c.delete_table().table_name(&restored).send().await.ok(); + delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&table).send().await.ok(); } diff --git a/tests/rust/src/restore_active_completeness.rs b/tests/rust/src/restore_active_completeness.rs index 83644ddc0..93ca7bcd5 100644 --- a/tests/rust/src/restore_active_completeness.rs +++ b/tests/rust/src/restore_active_completeness.rs @@ -23,12 +23,19 @@ //! The dataset is sized so the copy takes longer than the transition window on //! a loaded server. On very fast or idle hardware, raising `ITEMS` widens the //! window. +//! +//! On a server with a configured `[backup.store]` (`EXTENDDB_TEST_BACKUP_STORE` +//! at `filesystem` or `s3`) the restore of a backup this server created is +//! refused this release (the restore job is PR 9a's), so the race test pins +//! that refusal in place of the race. +use crate::backup_restore::{ + backup_store, delete_backup_quietly, wait_for_backup_available, BackupStore, +}; use crate::test_base::*; use aws_sdk_dynamodb::types::{ - AttributeDefinition, AttributeValue, BillingMode, GlobalSecondaryIndex, - KeySchemaElement, KeyType, Projection, ProjectionType, PutRequest, ScalarAttributeType, - Select, WriteRequest, + AttributeDefinition, AttributeValue, BillingMode, GlobalSecondaryIndex, KeySchemaElement, + KeyType, Projection, ProjectionType, PutRequest, ScalarAttributeType, Select, WriteRequest, }; use std::time::Duration; @@ -159,20 +166,36 @@ async fn restored_table_has_all_items_when_first_active() { .await .unwrap(); let arn = backup.backup_details().unwrap().backup_arn().to_string(); - // Wait until the backup is AVAILABLE. - for _ in 0..240 { - let d = c.describe_backup().backup_arn(&arn).send().await.unwrap(); - if d.backup_description() - .and_then(|b| b.backup_details()) - .map(|b| b.backup_status().as_str() == "AVAILABLE") - .unwrap_or(false) - { - break; - } - tokio::time::sleep(std::time::Duration::from_millis(250)).await; - } + wait_for_backup_available(c, &arn).await; let dst = format!("RestoreRaceDst_{}", ts()); + // A backup in the configured store is not restored this release (the + // restore job is PR 9a's): the race has nothing to observe, and the test + // pins the refusal instead. + if matches!(backup_store(), BackupStore::Filesystem | BackupStore::S3) { + let err = c + .restore_table_from_backup() + .target_table_name(&dst) + .backup_arn(&arn) + .send() + .await + .expect_err("a store-backed backup restored"); + assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); + let msg = err_msg(&err); + assert!( + msg.contains("is in a backup store"), + "the refusal does not name the store: {msg}" + ); + println!("restore of a store-backed backup refused: {msg}"); + assert!( + c.describe_table().table_name(&dst).send().await.is_err(), + "the refused restore created {dst}" + ); + delete_backup_quietly(c, &arn).await; + c.delete_table().table_name(&src).send().await.ok(); + return; + } + println!("restore of an in-catalog backup: racing the observer"); // Race an observer against the in-flight restore: the moment it sees // ACTIVE, it counts. Returns None if ACTIVE never arrives within the // bound, which means the restore itself failed. @@ -254,6 +277,7 @@ async fn restored_table_has_all_items_when_first_active() { // above was still in flight (or just after, if the copy was fast). let observed = observer.await.unwrap(); + delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&src).send().await.ok(); c.delete_table().table_name(&dst).send().await.ok(); diff --git a/tests/test_abac.py b/tests/test_abac.py index b56b85036..1e10a80b2 100755 --- a/tests/test_abac.py +++ b/tests/test_abac.py @@ -35,6 +35,7 @@ from conftest import wait_for_active, wait_for_deleted from management_helpers import ManagementClient +from test_backup_restore_fidelity import _delete_backup_when_available def _require_auth_env() -> tuple[str, str, str]: endpoint = os.environ.get("EXTENDDB_TEST_ENDPOINT", "").strip() admin_user = os.environ.get("EXTENDDB_ADMIN_USER", "").strip() @@ -1446,10 +1447,10 @@ def describe_backup(): "BackupDescription"]["BackupDetails"]["BackupArn"] == backup_arn finally: if backup_arn is not None: - try: - admin_client.delete_backup(BackupArn=backup_arn) - except Exception: - pass + # On a server with a backup store the backup is CREATING for + # the job's duration and DeleteBackup answers BackupInUse; + # deleting at once leaked a backup on every store-backed run. + _delete_backup_when_available(admin_client, backup_arn) try: admin_client.delete_table(TableName=table) except Exception: diff --git a/tests/test_auth_permissions.py b/tests/test_auth_permissions.py index 157f4cec3..3c2e33465 100755 --- a/tests/test_auth_permissions.py +++ b/tests/test_auth_permissions.py @@ -18,6 +18,7 @@ from __future__ import annotations import os +import time import uuid import boto3 @@ -1364,6 +1365,13 @@ def source_backup(self): pass if backup_arn is not None: try: + # A store-backed backup is CREATING until its job ends + # and refuses DeleteBackup until then. + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and admin_client.describe_backup( + BackupArn=backup_arn + )["BackupDescription"]["BackupDetails"]["BackupStatus"] == "CREATING": + time.sleep(0.2) admin_client.delete_backup(BackupArn=backup_arn) except Exception: pass @@ -1438,6 +1446,14 @@ def test_restore_needs_the_item_actions_on_the_target(self, source_backup): """RestoreTableFromBackup on the backup plus the seven item actions on the target, each required; RestoreTableFromBackup on the target is neither required nor evaluated.""" + if os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() in ("filesystem", "s3"): + # The denials hold on any server; the restores that must run are + # of a backup in the configured store, which RestoreTableFromBackup + # refuses until the restore job lands (PR 9a). + pytest.skip( + "RestoreTableFromBackup of a backup in the configured store is refused " + "until the restore job lands (PR 9a)" + ) admin_client, table, backup_arn = source_backup user = f"u-{uuid.uuid4().hex[:8]}" ak, sk = self._user_with_key(user) diff --git a/tests/test_authz_resource_arn.py b/tests/test_authz_resource_arn.py index 4d81c5955..6c3b07808 100644 --- a/tests/test_authz_resource_arn.py +++ b/tests/test_authz_resource_arn.py @@ -38,6 +38,7 @@ from conftest import wait_for_active from management_helpers import ManagementClient +from test_backup_restore_fidelity import _delete_backup_when_available def _require_auth_env() -> tuple[str, str, str]: @@ -122,7 +123,13 @@ def _user_with_policy(mgmt: ManagementClient, account_id: str, policy: dict) -> @pytest.fixture(scope="module") def table(auth_env, mgmt, account_id, region): """A table with a GSI, a stream, one item, and a backup. Returns the - canonical table, index, stream, and backup ARNs.""" + canonical table, index, stream, and backup ARNs. The cells deny + DeleteBackup and DeleteTable to other users, so the backup and the table + are deleted here once the module is done: on a server with a configured + `[backup.store]` the backup is a complete set of objects under the store, + and one that is left behind stays until someone deletes it (CI runners + die with theirs; a developer's server keeps them). The backup is deleted + once AVAILABLE, since a backup still CREATING refuses DeleteBackup.""" endpoint = auth_env[0] ak, sk = _user_with_policy(mgmt, account_id, ALLOW_ALL) admin = _client(endpoint, ak, sk, region) @@ -147,13 +154,18 @@ def table(auth_env, mgmt, account_id, region): desc = admin.describe_table(TableName=name)["Table"] table_arn = desc["TableArn"] backup_arn = admin.create_backup(TableName=name, BackupName="arnbk")["BackupDetails"]["BackupArn"] - return { + yield { "name": name, "table_arn": table_arn, "index_arn": table_arn + "/index/gsi1", "stream_arn": desc.get("LatestStreamArn"), "backup_arn": backup_arn, } + _delete_backup_when_available(admin, backup_arn) + try: + admin.delete_table(TableName=name) + except ClientError: + pass class TestExplicitDenyOnResourceArn: diff --git a/tests/test_backup_job.py b/tests/test_backup_job.py new file mode 100644 index 000000000..90be4e39b --- /dev/null +++ b/tests/test_backup_job.py @@ -0,0 +1,662 @@ +# SPDX-License-Identifier: Apache-2.0 + +"""The store-backed CreateBackup job at the server's boundary. + +Each test starts its own `extenddb serve --foreground` on an isolated +deployment with a filesystem backup store (SQLite: the default root beside +the database file; PostgreSQL: a root under the test's directory) and reads +the API's answers, the server's log, `extenddb backup list --failed` and the +store's root: + +- a second CreateBackup while one job runs with `max_concurrent_jobs = 1` + waits for the slot and is answered only once the first job has ended (the + first backup AVAILABLE), and a third is accepted at once afterwards; a wait + that outlives the 30 s bound is LimitExceededException naming the bound; +- a table holding an `N` value the wire let in (#618: `1E126` or `abc` in a + non-key attribute, `1E126` in the range key of a key-ordered table) written + through each of the four paths that store it unvalidated (BatchWriteItem, + TransactWriteItems, UpdateItem on an absent item in the key it creates, + ImportTable) gives a FAILED row, no manifest under the root, three log + lines naming each item's key and attribute, and a FAILED reason giving the + count and the first item and naming #618; +- an item over `max_item_size_bytes` (stored under the default limit, the + server restarted with a lower one) gives a reason that does not name #618; +- a refused item with a UUID hash key and a ten-digit epoch `N` range key is + named by its whole key, and the wire's reason for its value, in the log + line and in `backup list --failed`. + +The binary at `EXTENDDB_BINARY` decides which deployment runs: a SQLite build, +a PostgreSQL build (with `EXTENDDB_TEST_PG_CONNECTION_STRING`) or a MongoDB +build (with `EXTENDDB_TEST_MONGODB_URI`, a replica set of this suite's own: the +catalog's database names are fixed, so each test destroys the previous test's +catalog and initializes its own). Excluded from the backend-agnostic suite; run +by name. +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import threading +import time +import uuid +from concurrent.futures import ThreadPoolExecutor + +import boto3 +import pytest +import requests +from botocore.auth import SigV4Auth +from botocore.awsrequest import AWSRequest +from botocore.exceptions import ClientError + +import lifecycle_helpers as lh +from test_backup_store_config import ( + ADMIN_PASSWORD, + EXTENDDB_BINARY, + _append, + _client, + _pg_build, + _provision, + _Server, + _sqlite_build, + _SqliteDeployment, + _wait_backup_available, + _wait_health, +) + +ACCOUNT = "123456789012" +# The job's bound in the slot-wait cell: items of this count and padding keep +# the first job running for longer than the second request's round trip (0.4 s +# for 30,000 400-byte items on a SQLite build of this host), so the second +# request is sent while the first job holds the one slot and has to wait for +# it. The passing branch is the second request admitted once the first job +# ended (the first backup AVAILABLE at its answer, and seen CREATING while it +# waited); a second answered while the first is still CREATING is a broken +# bound; a refusal at the wait's bound is the slow-host branch, which the cell +# accepts; a first backup already AVAILABLE before the second is sent is a +# probe that came too late, where the cell asks for more items. +BUSY_ITEMS = int(os.environ.get("EXTENDDB_TEST_BACKUP_BUSY_ITEMS", "30000")) +LIMIT_TEXT = ( + "Too many backup and restore jobs are running on this server " + "(at most 1 at once, the [backup] max_concurrent_jobs bound); retry later" +) + + +class _Deployment: + """A deployment with a filesystem store, its config and its root.""" + + def __init__(self, backend: str, config: str, port: int, root: str, cert: str, cwd: str): + self.backend = backend + self.config = config + self.port = port + self.root = root + self.cert = cert + self.cwd = cwd + + def run(self, *args): + import subprocess + + return subprocess.run( + [EXTENDDB_BINARY, args[0], "--config", self.config, *args[1:]], + cwd=self.cwd, + capture_output=True, + text=True, + stdin=subprocess.DEVNULL, + timeout=180, + ) + + def set_limit(self, key: str, value: str) -> None: + """Set a key of the init template's `[limits]` table.""" + with open(self.config) as f: + text = f.read() + assert text.count("[limits]\n") == 1, text + text = text.replace("[limits]\n", f"[limits]\n{key} = {value}\n", 1) + with open(self.config, "w") as f: + f.write(text) + + +def _mongodb_build() -> bool: + if not os.path.isfile(EXTENDDB_BINARY): + return False + res = subprocess.run([EXTENDDB_BINARY, "version"], capture_output=True, text=True, timeout=30) + return "(mongodb)" in res.stdout + + +def _mongodb_deployment(base: str, uri: str) -> tuple[str, int]: + """A catalog on the replica set at `uri`, initialized afresh: the config + stub names the connection string `init` reads, the previous test's + catalog (the database names are fixed per replica set) is destroyed first, + and the generated config is given a free port.""" + config = os.path.join(base, "extenddb.toml") + with open(config, "w") as f: + f.write(f'[storage]\nbackend = "mongodb"\n[storage.mongodb]\nconnection_string = "{uri}"\n') + env = os.environ.copy() + env["EXTENDDB_ADMIN_PASSWORD"] = ADMIN_PASSWORD + common = dict(cwd=base, capture_output=True, text=True, stdin=subprocess.DEVNULL, timeout=120, env=env) + # Nothing to destroy on the first test; `destroy` then exits non-zero. + subprocess.run([EXTENDDB_BINARY, "destroy", "--config", config, "--yes"], **common) + res = subprocess.run( + [EXTENDDB_BINARY, "init", "--backend", "mongodb", "--config", config, "--overwrite"], **common + ) + assert res.returncode == 0, res.stdout + res.stderr + port = lh._find_free_port() + lh._patch_config_port(config, port) + return config, port + + +@pytest.fixture() +def deployment(request, tmp_path) -> _Deployment: + """The build's deployment: SQLite with the defaulted root, or PostgreSQL + or MongoDB with a filesystem root under the test's directory. All read + `reconcile_interval_secs = 1`, `max_concurrent_jobs = 1` and import from + the test's directory (ImportTable reads `//`).""" + base = os.path.realpath(str(tmp_path)) + if _sqlite_build(): + d = _SqliteDeployment(str(tmp_path)) + config, port, cert = d.config, d.port, d.cert + root = os.path.join(base, "backups") + store = "" + backend = "sqlite" + elif _pg_build(): + cli_env = request.getfixturevalue("cli_env") + result = lh._run_extenddb( + "init", *lh._init_args(cli_env), config=cli_env["config_path"], + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(cli_env["config_path"], cli_env["port"]) + config, port = cli_env["config_path"], cli_env["port"] + cert = os.path.join(cli_env["tls_dir"], "cert.pem") + root = os.path.join(base, "backups") + os.mkdir(root, 0o700) + store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n' + backend = "postgres" + elif _mongodb_build(): + uri = os.environ.get("EXTENDDB_TEST_MONGODB_URI") + if not uri: + pytest.skip("EXTENDDB_TEST_MONGODB_URI is unset (a replica set of this suite's own)") + config, port = _mongodb_deployment(base, uri) + cert = os.path.expanduser("~/.extenddb/tls/cert.pem") + root = os.path.join(base, "backups") + os.mkdir(root, 0o700) + store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n' + backend = "mongodb" + else: + pytest.skip("EXTENDDB_BINARY is neither a SQLite, a PostgreSQL nor a MongoDB build") + os.mkdir(os.path.join(base, ACCOUNT)) + _append( + config, + "[backup]\nreconcile_interval_secs = 1\nmax_concurrent_jobs = 1\n" + + store + + f'[import]\npaths = ["{base}"]\n', + ) + return _Deployment(backend, config, port, root, cert, base) + + +def _started(d: _Deployment, log_path: str) -> tuple[_Server, object, dict]: + """The server started on the deployment, a client and its credentials.""" + server = _Server(d.config, log_path) + try: + _wait_health( + d.port, + lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy", + server=server, + ) + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + except BaseException: + # A server whose store never came up (or whose provisioning failed) + # would otherwise keep running after the test's failure. + server.stop() + raise + return server, _client(d.port, creds), creds + + +def _signed_post(d: _Deployment, creds: dict, operation: str, body: dict) -> requests.Response: + """A hand-signed request for an operation botocore's model would refuse + (ImportTable's `FileSource`).""" + body_bytes = json.dumps(body).encode("utf-8") + headers = { + "X-Amz-Target": f"DynamoDB_20120810.{operation}", + "Content-Type": "application/x-amz-json-1.0", + } + endpoint = f"https://127.0.0.1:{d.port}" + frozen = boto3.Session( + aws_access_key_id=creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=creds["AWS_SECRET_ACCESS_KEY"], + ).get_credentials().get_frozen_credentials() + aws_req = AWSRequest(method="POST", url=endpoint, data=body_bytes, headers=headers) + SigV4Auth(frozen, "dynamodb", "us-east-1").add_auth(aws_req) + return requests.post(endpoint, data=body_bytes, headers=dict(aws_req.headers), verify=False) + + +def _wait_active(ddb, name: str, timeout: float = 60.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if ddb.describe_table(TableName=name)["Table"]["TableStatus"] == "ACTIVE": + return + time.sleep(0.05) + raise AssertionError(f"{name} did not become ACTIVE") + + +def _create_composite_table(ddb, name: str, gsi: bool = False) -> None: + """A key-ordered table: `pk` S hash key, `sk` N range key.""" + attrs = [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + ] + kwargs: dict = {} + if gsi: + attrs.append({"AttributeName": "g", "AttributeType": "S"}) + kwargs["GlobalSecondaryIndexes"] = [ + { + "IndexName": "by-g", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "ALL"}, + } + ] + ddb.create_table( + TableName=name, + KeySchema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + AttributeDefinitions=attrs, + BillingMode="PAY_PER_REQUEST", + **kwargs, + ) + _wait_active(ddb, name) + + +def _wait_backup_gone(ddb, arn: str, timeout: float = 60.0) -> None: + """Poll DescribeBackup until the backup is BackupNotFound (its row FAILED + or removed); an AVAILABLE backup fails the test.""" + deadline = time.monotonic() + timeout + while True: + try: + status = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] + except ClientError as e: + assert e.response["Error"]["Code"] == "BackupNotFoundException", e.response["Error"] + return + assert status == "CREATING", f"backup {arn} is {status}" + assert time.monotonic() < deadline, f"backup {arn} still CREATING after {timeout} s" + time.sleep(0.1) + + +_RUST_ESCAPES = {"n": "\n", "r": "\r", "t": "\t", "0": "\0", "\\": "\\", '"': '"', "'": "'"} + + +def _rust_debug_string(quoted: str) -> str: + """The string a Rust `{:?}` of a `String` printed: the quotes removed and + its escapes undone (`\\n`, `\\"`, `\\\\` and `\\u{hex}` for a character + `Debug` does not print as itself). `json.loads` refuses the `\\u{..}` + form, so a reason holding such a character (a key the client wrote with + one) is decoded here.""" + assert quoted.startswith('"') and quoted.endswith('"'), quoted + out: list[str] = [] + i, body = 0, quoted[1:-1] + while i < len(body): + c = body[i] + if c != "\\": + out.append(c) + i += 1 + continue + e = body[i + 1] + if e == "u": + close = body.index("}", i + 3) + out.append(chr(int(body[i + 3 : close], 16))) + i = close + 1 + else: + out.append(_RUST_ESCAPES[e]) + i += 2 + return "".join(out) + + +def _failed_rows(d: _Deployment) -> dict[str, str]: + """`extenddb backup list --failed`: each FAILED row's ARN to its reason.""" + res = d.run("backup", "list", "--failed") + assert res.returncode == 0, res.stdout + res.stderr + rows: dict[str, str] = {} + for line in res.stdout.splitlines(): + m = re.match(r"^FAILED\s.*?\s(arn:\S+)\s.*failure_reason=(\".*\")$", line) + if m: + rows[m.group(1)] = _rust_debug_string(m.group(2)) + return rows + + +def _prefix(d: _Deployment, table: str, arn: str) -> str: + return os.path.join(d.root, ACCOUNT, table, arn.rsplit("/", 1)[1]) + + +def _assert_no_manifest(d: _Deployment, table: str, arn: str) -> None: + prefix = _prefix(d, table, arn) + if os.path.isdir(prefix): + left = os.listdir(prefix) + assert "extenddb-manifest.json" not in left, left + + +def _restored_item_lines(log: str, arn: str) -> list[str]: + return [ + line for line in log.splitlines() if f"backup {arn}: item cannot be restored: " in line + ] + + +def test_second_create_backup_over_the_job_bound_waits_for_the_slot(deployment, tmp_path): + """With `max_concurrent_jobs = 1`, a CreateBackup sent while one job runs + waits for the slot (up to 30 s) and is answered only once the first job + has ended: when its answer arrives the first backup is AVAILABLE, never + still CREATING (a second accepted while the first is CREATING fails the + cell: the bound did not hold). A first backup already AVAILABLE when the + second is sent means the probe came too late and asks for more items. A + wait that outlives the bound (a slow host) is LimitExceededException with + the bound's text, which the cell accepts and reports. The first job + finishes AVAILABLE with every item and a GSI in its + SourceTableFeatureDetails, and a third CreateBackup is accepted at once + (the slot is free).""" + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name = f"busy-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load(d, creds, name, BUSY_ITEMS) + first = ddb.create_backup(TableName=name, BackupName="first")["BackupDetails"] + assert first["BackupStatus"] == "CREATING", first + before = ddb.describe_backup(BackupArn=first["BackupArn"])["BackupDescription"]["BackupDetails"]["BackupStatus"] + if before == "AVAILABLE": + pytest.fail( + "the first backup was already AVAILABLE before the second CreateBackup was sent, " + "so the probe checks nothing; raise EXTENDDB_TEST_BACKUP_BUSY_ITEMS" + ) + + # The second request from its own client and thread: it blocks on the + # slot while the main thread watches the first backup. + def second_request(): + client = _client(d.port, creds) + sent = time.monotonic() + try: + return client.create_backup(TableName=name, BackupName="second")["BackupDetails"], None, time.monotonic() - sent + except ClientError as e: + return None, e.response["Error"], time.monotonic() - sent + + with ThreadPoolExecutor(max_workers=1) as pool: + pending = pool.submit(second_request) + # The first backup is CREATING for as long as the second waits + # (a poll that sees it AVAILABLE is the first job's end, after + # which the second is admitted at once). + seen_creating = 0 + while not pending.done(): + status = ddb.describe_backup(BackupArn=first["BackupArn"])["BackupDescription"]["BackupDetails"]["BackupStatus"] + seen_creating += status == "CREATING" + time.sleep(0.05) + second, refused, waited = pending.result() + status_at_answer = ddb.describe_backup(BackupArn=first["BackupArn"])["BackupDescription"]["BackupDetails"]["BackupStatus"] + if refused is not None: + # The slow-host branch: the first job outlived the 30 s wait. + assert refused["Code"] == "LimitExceededException", refused + assert refused["Message"] == LIMIT_TEXT, refused + assert waited >= 25, f"a refusal before the wait's bound: {waited:.1f} s" + print(f"branch: the second request waited {waited:.1f} s and was refused at the bound ({d.backend})") + else: + assert status_at_answer == "AVAILABLE", ( + f"the second CreateBackup was answered while the first job was still running " + f"(the first backup is {status_at_answer}): the max_concurrent_jobs bound did not hold" + ) + # The second request waited: the first was CREATING while it was + # pending (a first job that ended before the thread connected + # would pass the branch above without any wait). + assert seen_creating > 0, ( + "the first backup was AVAILABLE before the second request was sent, so the probe " + "checked nothing; raise EXTENDDB_TEST_BACKUP_BUSY_ITEMS" + ) + assert second["BackupStatus"] in ("CREATING", "AVAILABLE"), second + print( + f"branch: the second request waited {waited:.1f} s for the slot and was admitted; " + f"the first was seen CREATING {seen_creating} time(s) meanwhile ({d.backend})" + ) + _wait_backup_available(ddb, second["BackupArn"], timeout=120) + _wait_backup_available(ddb, first["BackupArn"], timeout=120) + desc = ddb.describe_backup(BackupArn=first["BackupArn"])["BackupDescription"] + assert desc["SourceTableDetails"]["ItemCount"] == BUSY_ITEMS, desc["SourceTableDetails"] + gsis = desc["SourceTableFeatureDetails"]["GlobalSecondaryIndexes"] + assert [g["IndexName"] for g in gsis] == ["by-g"], gsis + assert os.path.isfile(os.path.join(_prefix(d, name, first["BackupArn"]), "extenddb-manifest.json")) + third = ddb.create_backup(TableName=name, BackupName="third")["BackupDetails"] + assert third["BackupStatus"] == "CREATING", third + _wait_backup_available(ddb, third["BackupArn"], timeout=120) + listed = sorted(b["BackupName"] for b in ddb.list_backups(TableName=name)["BackupSummaries"]) + expected = ["first", "third"] if refused is not None else ["first", "second", "third"] + assert listed == expected, listed + finally: + server.stop() + + +def _load(d: _Deployment, creds: dict, name: str, n: int) -> None: + """Write n composite-key items with BatchWriteItem from 8 threads, one + client each: a boto3 client shared across threads fails at random with + TLS record-layer errors that look like server faults.""" + local = threading.local() + + def write(start: int) -> None: + if not hasattr(local, "client"): + local.client = _client(d.port, creds) + requests_ = [ + { + "PutRequest": { + "Item": { + "pk": {"S": f"p{i % 500:04d}"}, + "sk": {"N": str(i)}, + "g": {"S": f"g{i % 50}"}, + "pad": {"S": "x" * 400}, + } + } + } + for i in range(start, min(start + 25, n)) + ] + pending = {name: requests_} + while pending: + pending = local.client.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + if pending: + time.sleep(0.05) + + with ThreadPoolExecutor(max_workers=8) as pool: + list(pool.map(write, range(0, n, 25))) + + +# The three items each path stores: `1E126` and `abc` in a non-key attribute +# and `1E126` in the range key (an `abc` range key is refused by the number +# column on PostgreSQL and SQLite, so no path stores one). UpdateItem +# validates the values its expression sets, so its three items carry the bad +# value in the key it creates. +BAD_ITEMS = [ + {"pk": {"S": "a"}, "sk": {"N": "1"}, "amount": {"N": "1E126"}}, + {"pk": {"S": "a"}, "sk": {"N": "2"}, "note": {"N": "abc"}}, + {"pk": {"S": "b"}, "sk": {"N": "1E126"}}, +] +BAD_KEYS = [ + {"pk": {"S": "a"}, "sk": {"N": "1E126"}}, + {"pk": {"S": "b"}, "sk": {"N": "1E127"}}, + {"pk": {"S": "c"}, "sk": {"N": "1E128"}}, +] +GOOD_ITEMS = [{"pk": {"S": "z"}, "sk": {"N": str(i)}, "amount": {"N": str(i)}} for i in range(5)] + + +def _write_batch(d, ddb, creds, name): + _create_composite_table(ddb, name) + ddb.batch_write_item( + RequestItems={name: [{"PutRequest": {"Item": i}} for i in BAD_ITEMS + GOOD_ITEMS]} + ) + return BAD_ITEMS + + +def _write_transact(d, ddb, creds, name): + _create_composite_table(ddb, name) + ddb.transact_write_items( + TransactItems=[{"Put": {"TableName": name, "Item": i}} for i in BAD_ITEMS + GOOD_ITEMS] + ) + return BAD_ITEMS + + +def _write_update(d, ddb, creds, name): + _create_composite_table(ddb, name) + for key in BAD_KEYS: + ddb.update_item( + TableName=name, + Key=key, + UpdateExpression="SET note = :v", + ExpressionAttributeValues={":v": {"S": "created from the key"}}, + ) + for item in GOOD_ITEMS: + ddb.put_item(TableName=name, Item=item) + return BAD_KEYS + + +def _write_import(d, ddb, creds, name): + path = os.path.join(d.cwd, ACCOUNT, f"{name}.json") + with open(path, "w") as f: + for item in BAD_ITEMS + GOOD_ITEMS: + f.write(json.dumps({"Item": item}) + "\n") + resp = _signed_post( + d, + creds, + "ImportTable", + { + "FileSource": {"Path": path}, + "InputFormat": "DYNAMODB_JSON", + "TableCreationParameters": { + "TableName": name, + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + ], + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + "BillingMode": "PAY_PER_REQUEST", + }, + }, + ) + assert resp.status_code == 200, resp.text + assert resp.json()["ImportTableDescription"]["ImportStatus"] == "COMPLETED", resp.text + _wait_active(ddb, name) + return BAD_ITEMS + + +WRITE_PATHS = { + "batch_write_item": _write_batch, + "transact_write_items": _write_transact, + "update_item": _write_update, + "import_table": _write_import, +} + + +def _key_json(item: dict) -> str: + return json.dumps({"pk": item["pk"], "sk": item["sk"]}, separators=(",", ":")) + + +@pytest.mark.parametrize("path", sorted(WRITE_PATHS)) +def test_an_item_the_wire_let_in_fails_the_backup(deployment, tmp_path, path): + """Three items with an `N` value #618 let in, stored through one write + path, and five that restore: CreateBackup answers CREATING, the job + validates every item (three log lines, one per item, with its key and + attribute path), the row is FAILED with a reason giving the count and + the first item and naming #618, DescribeBackup is BackupNotFound, + ListBackups omits it, and no manifest is under the root.""" + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name = f"n618-{path.replace('_', '')}-{uuid.uuid4().hex[:6]}" + bad = WRITE_PATHS[path](d, ddb, creds, name) + stored = ddb.scan(TableName=name, ConsistentRead=True)["Items"] + assert len(stored) == len(bad) + len(GOOD_ITEMS), stored + details = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] == "CREATING", details + arn = details["BackupArn"] + _wait_backup_gone(ddb, arn) + assert ddb.list_backups(TableName=name)["BackupSummaries"] == [] + _assert_no_manifest(d, name, arn) + reason = _failed_rows(d)[arn] + first = sorted(bad, key=lambda i: (i["pk"]["S"], float(i["sk"]["N"])))[0] + assert reason.startswith(f"{len(bad)} item(s) cannot be restored; first: key {_key_json(first)}, attribute "), reason + assert reason.endswith(" (#618)"), reason + if "sk" in first and first["sk"]["N"] == "1E126": + assert ", attribute sk: " in reason, reason + else: + assert ", attribute amount: " in reason, reason + log = server.log_text() + lines = _restored_item_lines(log, arn) + assert len(lines) == len(bad), lines + logged_keys = {re.search(r"key (\{.*?\}\}), attribute", line).group(1) for line in lines} + assert logged_keys == {_key_json(i) for i in bad}, logged_keys + assert all(re.search(r"attribute (sk|amount|note): ", line) for line in lines), lines + assert f"backup {arn}: FAILED ({reason}); its objects were deleted" in log, log + assert "more item(s) cannot be restored" not in log + finally: + server.stop() + + +def test_a_refused_item_with_a_uuid_and_epoch_key_is_named_whole(deployment, tmp_path): + """One refused item with a realistic key, a UUID hash key and a ten-digit + epoch range key, and `amount: abc`: the log line and the FAILED reason + carry the whole key as DynamoDB JSON (`{"pk":{"S":""},"sk":{"N": + "1761955200"}}`), not a shortened form, and the wire's reason for the + value, so an operator can find the item from either.""" + d = deployment + server, ddb, _ = _started(d, str(tmp_path / "serve.log")) + try: + name = f"n618-key-{uuid.uuid4().hex[:6]}" + _create_composite_table(ddb, name) + item = {"pk": {"S": str(uuid.uuid4())}, "sk": {"N": "1761955200"}, "amount": {"N": "abc"}} + ddb.batch_write_item(RequestItems={name: [{"PutRequest": {"Item": i}} for i in [item] + GOOD_ITEMS]}) + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_gone(ddb, arn) + key = _key_json(item) + assert f'"pk":{{"S":"{item["pk"]["S"]}"}}' in key and '"sk":{"N":"1761955200"}' in key, key + value_reason = "attribute amount: The parameter cannot be converted to a numeric value: abc" + reason = _failed_rows(d)[arn] + assert reason == f"1 item(s) cannot be restored; first: key {key}, {value_reason} (#618)", reason + lines = _restored_item_lines(server.log_text(), arn) + assert len(lines) == 1, lines + assert f"backup {arn}: item cannot be restored: key {key}, {value_reason}" in lines[0], lines[0] + finally: + server.stop() + + +def test_an_item_over_max_item_size_bytes_fails_the_backup_without_naming_618(deployment, tmp_path): + """A 300 KB item stored under the default `max_item_size_bytes`, the + server restarted with `[limits] max_item_size_bytes = 65536`: the backup + is FAILED with a reason naming the item's size and the limit and not + #618, and the server's own PutItem refuses such an item.""" + d = deployment + name = f"big-{uuid.uuid4().hex[:8]}" + server, ddb, _ = _started(d, str(tmp_path / "serve.log")) + try: + _create_composite_table(ddb, name) + ddb.put_item(TableName=name, Item={"pk": {"S": "big"}, "sk": {"N": "1"}, "pad": {"S": "x" * 300000}}) + ddb.put_item(TableName=name, Item={"pk": {"S": "small"}, "sk": {"N": "1"}, "pad": {"S": "x"}}) + # The same table backs up whole under the default limit. + arn = ddb.create_backup(TableName=name, BackupName="under-default")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn) + finally: + server.stop() + d.set_limit("max_item_size_bytes", "65536") + server, ddb, _ = _started(d, str(tmp_path / "serve2.log")) + try: + with pytest.raises(ClientError) as refused: + ddb.put_item(TableName=name, Item={"pk": {"S": "big2"}, "sk": {"N": "1"}, "pad": {"S": "x" * 100000}}) + assert refused.value.response["Error"]["Code"] == "ValidationException" + arn = ddb.create_backup(TableName=name, BackupName="under-lower")["BackupDetails"]["BackupArn"] + _wait_backup_gone(ddb, arn) + _assert_no_manifest(d, name, arn) + reason = _failed_rows(d)[arn] + assert reason.startswith('1 item(s) cannot be restored; first: key {"pk":{"S":"big"},"sk":{"N":"1"}}: item size '), reason + assert reason.endswith(" bytes exceeds the limit of 65536 bytes"), reason + assert "#618" not in reason, reason + lines = _restored_item_lines(server.log_text(), arn) + assert len(lines) == 1 and "exceeds the limit of 65536 bytes" in lines[0], lines + # The first backup, taken under the default limit, is untouched. + listed = [b["BackupName"] for b in ddb.list_backups(TableName=name)["BackupSummaries"]] + assert listed == ["under-default"], listed + finally: + server.stop() diff --git a/tests/test_backup_restore_fidelity.py b/tests/test_backup_restore_fidelity.py index 35032df5e..8d152b607 100644 --- a/tests/test_backup_restore_fidelity.py +++ b/tests/test_backup_restore_fidelity.py @@ -24,6 +24,7 @@ import boto3 import pytest +import requests from botocore.config import Config from botocore.exceptions import ClientError, ReadTimeoutError @@ -41,13 +42,98 @@ BACKUP_TIMEOUT_S = 900.0 RESTORE_TIMEOUT_S = 1800.0 +# Where the server under test writes its backups: `filesystem` or `s3` (a +# configured `[backup.store]`, under which CreateBackup is asynchronous and +# backs up vector tables) or `none` (the in-catalog create). The CI jobs set +# it beside EXTENDDB_TEST_ENDPOINT; the one test whose expected outcome +# depends on it fails when it is unset rather than accept either outcome. +BACKUP_STORE = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() or None +BACKUP_STORE_VALUES = ("filesystem", "s3", "none") + +# RestoreTableFromBackup of a backup written to a configured store answers +# ValidationException ("is in a backup store; this release restores +# in-catalog backups only") until the restore job of PR 9a lands. The cells +# that restore a backup this server created skip on such a server and come +# back with PR 9a; the in-catalog create's backups (`none`, and the service) +# restore as before. The marker is gated on the target as well as the +# variable, so a stale `EXTENDDB_TEST_BACKUP_STORE=filesystem` in the +# environment of a real-DynamoDB run (which does not require the variable) +# cannot skip the 24 cells against the service. +restores_a_backup_of_this_server = pytest.mark.skipif( + BACKUP_STORE in ("filesystem", "s3") and not _is_real_dynamodb(), + reason=( + f"EXTENDDB_TEST_BACKUP_STORE={BACKUP_STORE}: RestoreTableFromBackup of a backup " + "in the configured store is refused until the restore job lands (PR 9a)" + ), +) + + +@pytest.fixture(scope="session") +def backup_store_component(endpoint_url): + """The server's `backup_store` health component, read once from + `/health`, checked against EXTENDDB_TEST_BACKUP_STORE: a configured store + (`filesystem`, `s3`) must be reported with that kind and `healthy`, and + `none` must report no component. A variable the harness cannot observe + is not a parameter: a check that only reads it asserts nothing about the + server, and without this, `s3` against a server with a filesystem store + passed every cell.""" + if BACKUP_STORE not in BACKUP_STORE_VALUES: + pytest.fail( + f"EXTENDDB_TEST_BACKUP_STORE is {'unset' if BACKUP_STORE is None else repr(BACKUP_STORE)}; " + f"set it to one of {BACKUP_STORE_VALUES} to say where the server under test writes " + "its backups" + ) + assert endpoint_url, "the backup store component is ExtendDB's; no endpoint is set" + health = requests.get(f"{endpoint_url.rstrip('/')}/health", verify=False, timeout=10).json() + component = health.get("components", {}).get("backup_store") + if BACKUP_STORE == "none": + assert component is None, ( + f"EXTENDDB_TEST_BACKUP_STORE=none but the server reports a backup store: {component}" + ) + else: + assert component is not None, ( + f"EXTENDDB_TEST_BACKUP_STORE={BACKUP_STORE} but the server reports no backup store: " + f"{health}" + ) + assert component.get("kind") == BACKUP_STORE, ( + f"EXTENDDB_TEST_BACKUP_STORE={BACKUP_STORE} but the server's store is {component}" + ) + assert component.get("status") == "healthy", ( + f"the server's {BACKUP_STORE} store is not healthy: {component}" + ) + print(f"backup store under test: {component if component else 'none'}") + return component + + +def wait_for_backup_available(client, backup_arn: str, timeout: float = BACKUP_TIMEOUT_S) -> None: + """Poll DescribeBackup until the backup is AVAILABLE. + + The service creates a backup asynchronously; so does ExtendDB with a + configured `[backup.store]` (CreateBackup answers CREATING and the job + uploads). A backup still CREATING refuses DeleteBackup (BackupInUse) and + restore, so every caller that goes on to delete, restore or read the + size of a backup it just created waits here first. Any other status is a + failure: a backup that fails is BackupNotFound, never FAILED, on the wire. + """ + deadline = time.monotonic() + timeout + while True: + status = client.describe_backup(BackupArn=backup_arn)["BackupDescription"][ + "BackupDetails" + ]["BackupStatus"] + if status == "AVAILABLE": + return + assert status == "CREATING", f"backup {backup_arn} is {status}" + assert time.monotonic() < deadline, f"backup {backup_arn} not AVAILABLE in time" + time.sleep(_poll_interval()) + def _create_backup(client, table_name: str) -> str: """CreateBackup, then wait for the backup to be AVAILABLE. The service creates a backup asynchronously and refuses to restore one - that is still CREATING; ExtendDB returns it AVAILABLE. A just-created - table can also refuse CreateBackup for a short while, which is retried. + that is still CREATING; so does ExtendDB with a configured store. A + just-created table can also refuse CreateBackup for a short while, which + is retried. """ deadline = time.monotonic() + 60 while True: @@ -62,26 +148,25 @@ def _create_backup(client, table_name: str) -> str: if not retryable or time.monotonic() >= deadline: raise time.sleep(0.5) - deadline = time.monotonic() + BACKUP_TIMEOUT_S try: - while True: - status = client.describe_backup(BackupArn=arn)["BackupDescription"][ - "BackupDetails" - ]["BackupStatus"] - if status == "AVAILABLE": - return arn - assert status == "CREATING", f"backup {arn} is {status}" - assert time.monotonic() < deadline, f"backup {arn} not AVAILABLE in time" - time.sleep(_poll_interval()) + wait_for_backup_available(client, arn) + return arn except BaseException: # The caller never learns the ARN, so the backup is removed here. - try: - client.delete_backup(BackupArn=arn) - except ClientError: - pass + _delete_backup_when_available(client, arn) raise +def _delete_backup_when_available(client, backup_arn: str) -> None: + """DeleteBackup once the backup is no longer CREATING; a backup that is + gone (failed or deleted) is left as it is.""" + try: + wait_for_backup_available(client, backup_arn) + client.delete_backup(BackupArn=backup_arn) + except (ClientError, AssertionError): + pass + + def _backup_creation_time(client, backup_arn: str): return client.describe_backup(BackupArn=backup_arn)["BackupDescription"][ "BackupDetails" @@ -256,7 +341,11 @@ def _index_items(client, table: str, index: str, hash_attr: str, values: list) - def _cleanup(client, backup_arn: str | None, *tables: str) -> None: - """Delete tables and the backup; each step runs even if an earlier one fails.""" + """Delete tables and the backup; each step runs even if an earlier one fails. + + A backup still CREATING (the asynchronous create) is waited for first: + DeleteBackup of one is BackupInUseException. + """ if tables: try: _drop(client, tables[0]) @@ -265,6 +354,7 @@ def _cleanup(client, backup_arn: str | None, *tables: str) -> None: return if backup_arn: try: + wait_for_backup_available(client, backup_arn) client.delete_backup(BackupArn=backup_arn) except client.exceptions.BackupNotFoundException: pass @@ -338,11 +428,13 @@ def _round_trip(client, sort_kind: str | None, hash_kind: str = "S") -> None: _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server @pytest.mark.parametrize("hash_kind", ["S", "N", "B"]) def test_restore_hash_only_table(dynamodb_client, hash_kind): _round_trip(dynamodb_client, None, hash_kind) +@restores_a_backup_of_this_server @pytest.mark.parametrize("sort_kind", ["S", "N", "B"]) def test_restore_composite_key_table(dynamodb_client, sort_kind): _round_trip(dynamodb_client, sort_kind) @@ -359,6 +451,7 @@ def _index_by_name(indexes: list[dict]) -> dict: return {i["IndexName"]: i for i in indexes} +@restores_a_backup_of_this_server def test_restore_preserves_indexes_and_provisioned_throughput(dynamodb_client): """GSIs and LSIs, their key schemas, projections, and throughput, the table's provisioned throughput, and every index's contents come back.""" @@ -546,6 +639,7 @@ def ranked(table: str) -> list[str]: _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_client): """A backup taken after an on-demand switch does not restore stale capacity.""" client = dynamodb_client @@ -620,6 +714,7 @@ def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_cli _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): client = dynamodb_client source = f"restore-override-ppr-{uuid.uuid4().hex[:10]}" @@ -658,6 +753,7 @@ def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): client = dynamodb_client source = f"restore-override-prov-{uuid.uuid4().hex[:10]}" @@ -692,6 +788,7 @@ def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server def test_provisioned_restore_override_requiring_gsi_override_leaves_no_target( dynamodb_client, ): @@ -934,6 +1031,7 @@ def large_backup(dynamodb_client): _cleanup(client, backup_arn, source) +@restores_a_backup_of_this_server def test_restore_response_arrives_before_its_copy_finishes(dynamodb_client, large_backup): """The CREATING response reaches a client with a 10 s read timeout and no retries before the copy finishes, and the table then reaches ACTIVE with @@ -996,6 +1094,7 @@ def _table_appears(client, name: str, timeout: float) -> bool: time.sleep(_poll_interval() * 5) +@restores_a_backup_of_this_server def test_restore_survives_a_client_that_disconnects_before_the_response( dynamodb_client, large_backup ): @@ -1051,6 +1150,7 @@ def test_restore_survives_a_client_that_disconnects_before_the_response( _cleanup(dynamodb_client, None, target) +@restores_a_backup_of_this_server def test_first_read_after_active_succeeds(dynamodb_client, large_backup): """A read on the restore target while it is CREATING is ResourceNotFoundException, and the first read after DescribeTable reports @@ -1107,6 +1207,7 @@ def test_first_read_after_active_succeeds(dynamodb_client, large_backup): _cleanup(client, None, target) +@restores_a_backup_of_this_server def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, large_backup): """CreateBackup on a table that is still CREATING is TableNotFoundException "Table not found: " (measured on DynamoDB), @@ -1136,6 +1237,7 @@ def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, l _cleanup(client, None, target) +@restores_a_backup_of_this_server def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused( dynamodb_client, large_backup ): @@ -1178,6 +1280,7 @@ def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_r _cleanup(client, None, target, other) +@restores_a_backup_of_this_server def test_target_and_backup_are_held_while_the_copy_runs(dynamodb_client, large_backup): """DeleteTable of a CREATING restore target is ResourceInUseException, and (ExtendDB) DeleteBackup of its source is BackupInUseException. @@ -1628,6 +1731,7 @@ def test_backup_arn_shape_and_account_are_checked(dynamodb_client): _cleanup(client, backup_arn, source) +@restores_a_backup_of_this_server def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dynamodb_client): client = dynamodb_client source = f"restore-noop-{uuid.uuid4().hex[:10]}" @@ -1678,6 +1782,7 @@ def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dy _cleanup(client, backup_arn, plain, restored, source) +@restores_a_backup_of_this_server @pytest.mark.parametrize("billing", ["PROVISIONED", "PAY_PER_REQUEST"]) def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_client, billing): """GlobalSecondaryIndexOverride and LocalSecondaryIndexOverride built from @@ -1848,6 +1953,7 @@ def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_clien _cleanup(client, backup_arn, changed, restored, source) +@restores_a_backup_of_this_server def test_on_demand_throughput_override_is_applied(dynamodb_client): """OnDemandThroughputOverride is the restored table's on-demand throughput, equal to the backup's or changed (measured on DynamoDB). @@ -1903,8 +2009,23 @@ def test_on_demand_throughput_override_is_applied(dynamodb_client): @extenddb_only -def test_create_backup_of_a_vector_table_is_refused(dynamodb_client): - """Refused at CreateBackup, not at restore time.""" +def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_store_component): + """CreateBackup of a table with vector indexes: the outcome of the + server's configured path, read from EXTENDDB_TEST_BACKUP_STORE and + checked against the server's `/health` (`backup_store_component`). + + `none` (no `[backup.store]`, or the in-catalog create): refused at + CreateBackup with the ValidationException main gives, and no backup is + created. `filesystem` or `s3`: the store-backed create backs the table + up (the manifest records its vector indexes): CreateBackup answers + CREATING, DescribeBackup reaches AVAILABLE with the table's features, + ListBackups lists it, and RestoreTableFromBackup of it answers the + ValidationException naming the store, which PR 9a's restore job + replaces with the restore (its refusal then narrows to a target without + vector support). The test fails rather than accept either outcome when + the variable is unset (the fixture fails first), and prints which branch + ran. + """ from test_vector_index_query_scan import _error_message, _signed_post client = dynamodb_client @@ -1913,9 +2034,19 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client): "CreateTable", { "TableName": name, - "AttributeDefinitions": [{"AttributeName": "pk", "AttributeType": "S"}], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "g", "AttributeType": "S"}, + ], "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}], "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [ + { + "IndexName": "by-g", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"}, + } + ], "VectorIndexes": [ { "IndexName": "vidx", @@ -1930,25 +2061,66 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client): if resp.status_code == 400 and "not supported" in _error_message(resp): pytest.skip("target backend does not support vector indexes") assert resp.status_code == 200, resp.text + backup_arn = None try: wait_for_active(client, name, timeout=180) vector = _signed_post("DescribeTable", {"TableName": name}).json()["Table"] if not vector.get("VectorIndexes"): pytest.skip("target accepted CreateTable but did not create the vector index") + client.put_item( + TableName=name, + Item={"pk": {"S": "a"}, "g": {"S": "g1"}, "emb": {"L": [{"N": "0.1"}] * 4}}, + ) + if BACKUP_STORE == "none": + print("branch: the in-catalog create refuses a vector table") + # A plain call, so a create that is accepted against expectation + # leaves its ARN for the cleanup rather than a backup nobody + # deletes. + try: + accepted = client.create_backup(TableName=name, BackupName=f"{name}-bkp") + except ClientError as e: + error = e.response["Error"] + else: + backup_arn = accepted["BackupDetails"]["BackupArn"] + pytest.fail( + f"the in-catalog create accepted a vector table: {accepted['BackupDetails']}" + ) + assert error["Code"] == "ValidationException" + assert error["Message"] == ( + f"table {name} has 1 vector index(es); restoring a table with vector " + "indexes is not supported by this storage backend" + ) + listed = client.list_backups(TableName=name)["BackupSummaries"] + assert listed == [], listed + return + print(f"branch: the {BACKUP_STORE} store backs a vector table up") + details = client.create_backup(TableName=name, BackupName=f"{name}-bkp")["BackupDetails"] + backup_arn = details["BackupArn"] + assert details["BackupStatus"] == "CREATING", details + wait_for_backup_available(client, backup_arn) + desc = client.describe_backup(BackupArn=backup_arn)["BackupDescription"] + assert desc["BackupDetails"]["BackupStatus"] == "AVAILABLE", desc + assert desc["SourceTableDetails"]["TableName"] == name + assert desc["SourceTableDetails"]["ItemCount"] == 1, desc["SourceTableDetails"] + gsis = desc["SourceTableFeatureDetails"]["GlobalSecondaryIndexes"] + assert [g["IndexName"] for g in gsis] == ["by-g"], desc["SourceTableFeatureDetails"] + listed = client.list_backups(TableName=name)["BackupSummaries"] + assert [b["BackupArn"] for b in listed] == [backup_arn], listed + # Until PR 9a's restore job, the restore of a backup in the store is + # refused; PR 9a replaces this assertion with the round trip. with pytest.raises(ClientError) as exc_info: - client.create_backup(TableName=name, BackupName=f"{name}-bkp") + client.restore_table_from_backup(TargetTableName=f"{name}-r", BackupArn=backup_arn) error = exc_info.value.response["Error"] - assert error["Code"] == "ValidationException" + assert error["Code"] == "ValidationException", error assert error["Message"] == ( - f"table {name} has 1 vector index(es); restoring a table with vector " - "indexes is not supported by this storage backend" - ) - listed = client.list_backups(TableName=name)["BackupSummaries"] - assert listed == [], listed + f"Backup {backup_arn} is in a backup store; this release restores in-catalog " + "backups only" + ), error finally: - _cleanup(client, None, name) + _cleanup(client, backup_arn, name) +@restores_a_backup_of_this_server def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_client): """Once a restored table is ACTIVE, no description-returning response carries RestoreSummary: UpdateTable's no more than DescribeTable's. @@ -1981,6 +2153,7 @@ def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_cl _cleanup(client, backup_arn, restored, source) +@restores_a_backup_of_this_server @extenddb_only def test_a_provisioned_gsi_without_throughput_restores_with_the_table_throughput( dynamodb_client, @@ -2349,6 +2522,7 @@ def test_create_backup_accepts_the_table_arn(dynamodb_client): _cleanup(client, backup_arn, source) +@restores_a_backup_of_this_server def test_table_class_is_not_restored(dynamodb_client): """A STANDARD_INFREQUENT_ACCESS source restores as STANDARD (measured on DynamoDB); DescribeBackup still reports nothing of the diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py index 174c7216e..4494144ce 100644 --- a/tests/test_backup_store_config.py +++ b/tests/test_backup_store_config.py @@ -10,11 +10,11 @@ with a filesystem store, `checking` while an S3 store's first check is in flight, unhealthy (status `degraded`) while an S3 endpoint refuses connections or stalls, and healthy again once it answers, without a restart; -- the restore and delete of a store-backed backup refused with a - ValidationException naming the store's kind and the problem's category and - not its root while the store is checking or unhealthy; CreateBackup served - throughout (this release's create writes the in-catalog format, which - touches no store); +- the restore and delete of a store-backed backup, and CreateBackup (which + writes to the store), refused with a ValidationException naming the + store's kind and the problem's category and not its root while the store + is checking or unhealthy, and CreateBackup writing a backup under the + store's root once it is healthy; - a stalled S3 endpoint holding the listener for no longer than the server's own startup, and each reconciler cycle for no longer than the check's bound; - the configuration errors that stop startup with the key, path or bucket in @@ -46,6 +46,7 @@ import json import os +import re import shutil import signal import socket @@ -184,6 +185,18 @@ def _client(port: int, creds: dict): ) +def _wait_backup_available(ddb, arn: str, timeout: float = 60.0) -> None: + """Poll DescribeBackup to AVAILABLE; any other status fails.""" + deadline = time.monotonic() + timeout + while True: + status = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] + if status == "AVAILABLE": + return + assert status == "CREATING", f"backup {arn} is {status}" + assert time.monotonic() < deadline, f"backup {arn} not AVAILABLE in {timeout} s" + time.sleep(0.1) + + def _create_table(ddb, name: str) -> None: ddb.create_table( TableName=name, @@ -441,14 +454,38 @@ def test_default_root_is_created_private_and_reported_healthy(self, sqlite_deplo configured = f" (configured as {root})" if canonical != root else "" assert f" backup store: filesystem {canonical}{configured} (the default beside the database file)" in log, log assert f"backup store: filesystem {canonical} (store id" in log, log - # CreateBackup takes the in-catalog path and says the store is configured. + # CreateBackup writes to the store: CREATING answered, the job's + # objects under the root, the row AVAILABLE once the manifest is + # written, and no deprecation warning. creds = _provision(d.port, d.cert, ADMIN_PASSWORD) ddb = _client(d.port, creds) _create_table(ddb, "Music") + ddb.put_item(TableName="Music", Item={"pk": {"S": "a"}, "n": {"N": "1"}}) details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") - assert "writing the in-catalog backup format; the configured filesystem backup store" in server.log_text() - assert "configure [backup.store]" not in server.log_text() + assert details["BackupStatus"] == "CREATING", details + arn = details["BackupArn"] + _wait_backup_available(ddb, arn) + backup_id = arn.rsplit("/", 1)[1] + prefix = os.path.join(root, "123456789012", "Music", backup_id) + assert sorted(os.listdir(prefix)) == [ + "data", "extenddb-manifest.json", "extenddb-manifest.sha256", + "manifest-files.json", "manifest-summary.json", + ], os.listdir(prefix) + assert (os.stat(os.path.join(prefix, "extenddb-manifest.json")).st_mode & 0o777) == 0o600 + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["backup_arn"] == arn and manifest["source_backend"] == "sqlite", manifest + described = ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"] + assert described["BackupSizeBytes"] == manifest["backup_size_bytes"] > 0, described + log = server.log_text() + assert f"backup {arn}: started, " in log and "in the table as last counted by the stats pass, key-ordered" in log, log + # The completion line carries the backup's own count, and its + # bytes count the documents beside the data files. + written = re.search(rf"backup {re.escape(arn)}: AVAILABLE, (\d+) items, (\d+) bytes written in", log) + assert written and int(written.group(1)) == 1, log + assert int(written.group(2)) > manifest["backup_size_bytes"], log + assert "in-catalog backup format" not in log, log + assert "configure [backup.store]" not in log, log finally: server.stop() @@ -474,8 +511,9 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql """`sqlite-memory`'s shape (`[storage.sqlite] path = ":memory:"`, the default of that build) on the SQLite build: with `[backup.store.filesystem] path` set the store is configured and healthy, its identity file is - written, and a backup created there (in-catalog in this release, the - log naming the configured store) restores with its item; without + written, and a backup created there is written to the store (CREATING, + then AVAILABLE with its manifest under the root; its restore is + refused naming the store until the restore job lands); without `[backup.store]` there is no default root (no database file to put it beside), no component, and CreateBackup writes a legacy backup with the warning naming `[backup.store]`.""" @@ -499,19 +537,17 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql ddb = _client(d.port, creds) _create_table(ddb, "Music") ddb.put_item(TableName="Music", Item={"pk": {"S": "a"}, "v": {"N": "1"}}) - arn = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"]["BackupArn"] - deadline = time.monotonic() + 60 - while ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] != "AVAILABLE": - assert time.monotonic() < deadline, "the backup did not become AVAILABLE" - time.sleep(0.1) - assert "the configured filesystem backup store" in server.log_text(), server.log_text() - ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) - deadline = time.monotonic() + 60 - while ddb.describe_table(TableName="Restored")["Table"]["TableStatus"] != "ACTIVE": - assert time.monotonic() < deadline, "the restore did not finish" - time.sleep(0.1) - item = ddb.get_item(TableName="Restored", Key={"pk": {"S": "a"}})["Item"] - assert item == {"pk": {"S": "a"}, "v": {"N": "1"}}, item + details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] + assert details["BackupStatus"] == "CREATING", details + arn = details["BackupArn"] + _wait_backup_available(ddb, arn) + prefix = os.path.join(root, "123456789012", "Music", arn.rsplit("/", 1)[1]) + assert os.path.isfile(os.path.join(prefix, "extenddb-manifest.json")), os.listdir(root) + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "ValidationException", refused.value.response + assert "is in a backup store" in refused.value.response["Error"]["Message"], refused.value.response + assert "configure [backup.store]" not in server.log_text(), server.log_text() finally: server.stop() @@ -626,10 +662,13 @@ def test_an_unreachable_s3_endpoint_starts_the_server_unhealthy_and_recovers( creds = _provision(d.port, d.cert, ADMIN_PASSWORD) ddb = _client(d.port, creds) _create_table(ddb, "Music") - # The in-catalog create touches no store: it runs on an unhealthy - # one, as it does on a server with none. - details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + # CreateBackup writes to the store, so it is refused while the + # store is unhealthy, naming the kind and the category, not the + # bucket; no row is created. + with pytest.raises(ClientError) as refused: + ddb.create_backup(TableName="Music", BackupName="nightly") + _assert_store_refusal(refused.value, "s3", bucket, category="unreachable") + assert ddb.list_backups(TableName="Music")["BackupSummaries"] == [] log = server.log_text() assert "is unhealthy: the store could not be reached" in log, log assert bucket in log, "the operator's line names the bucket" @@ -641,11 +680,25 @@ def test_an_unreachable_s3_endpoint_starts_the_server_unhealthy_and_recovers( ) assert health["status"] == "healthy" assert "is healthy again" in server.log_text() - details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") - # The identity file was created on the first passing check. + # The identity file was created on the first passing check, and + # CreateBackup now writes a backup into the bucket. body = _s3_client().get_object(Bucket=bucket, Key="extenddb-store.json")["Body"].read() assert json.loads(body)["backend"] == "sqlite" + ddb.put_item(TableName="Music", Item={"pk": {"S": "a"}, "n": {"N": "1"}}) + details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] + assert details["BackupStatus"] == "CREATING", details + arn = details["BackupArn"] + _wait_backup_available(ddb, arn) + keys = sorted( + o["Key"].rsplit("/", 1)[1] + for o in _s3_client().list_objects_v2( + Bucket=bucket, Prefix=f"123456789012/Music/{arn.rsplit('/', 1)[1]}/" + )["Contents"] + ) + assert keys == [ + "000001.json.gz", "extenddb-manifest.json", "extenddb-manifest.sha256", + "manifest-files.json", "manifest-summary.json", + ], keys # And degrades again once the endpoint is gone. forwarder.stop() @@ -654,8 +707,9 @@ def test_an_unreachable_s3_endpoint_starts_the_server_unhealthy_and_recovers( d.port, lambda h: h["components"]["backup_store"]["status"] == "unhealthy", timeout=30, server=server ) assert health["status"] == "degraded" - details = ddb.create_backup(TableName="Music", BackupName="again")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE"), "the in-catalog create is not gated" + with pytest.raises(ClientError) as refused: + ddb.create_backup(TableName="Music", BackupName="again") + _assert_store_refusal(refused.value, "s3", bucket, category="unreachable") finally: if forwarder is not None: forwarder.stop() @@ -665,9 +719,9 @@ def test_a_stalled_s3_endpoint_holds_neither_the_listener_nor_the_reconciler(sel """An endpoint that accepts connections and never answers. Checked: the server binds and serves before the store's first check, which runs after the listener is up (`/health` answers well within the SDK's - attempt timeout with the component `checking`); the in-catalog - CreateBackup is served meanwhile; the restore of a seeded store-backed - row is refused naming the pending check; the check is abandoned at its + attempt timeout with the component `checking`); CreateBackup and the + restore of a seeded store-backed row are refused naming the pending + check; the check is abandoned at its 30 s bound, the component turns `unhealthy` (`unreachable`) without a restart and the refusal names that; and the reconciler keeps cycling while the endpoint stalls (the next cycle's check opens a new @@ -720,14 +774,16 @@ def accept_and_hold(): log = server.log_text() assert "listening on" in log and "run after the listener is up" in log, log assert "backup store: s3" in log and "tarpit-bucket" in log, log - # Requests are served while the check hangs: the in-catalog create - # runs, and the seeded row's restore is refused naming the pending - # check and the store's kind, not the bucket. + # Requests are served while the check hangs: the table is + # created, and CreateBackup and the seeded row's restore are + # refused naming the pending check and the store's kind, not the + # bucket. creds = _provision(d.port, d.cert, ADMIN_PASSWORD) ddb = _client(d.port, creds) _create_table(ddb, "Music") - details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + with pytest.raises(ClientError) as refused: + ddb.create_backup(TableName="Music", BackupName="nightly") + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="first check is pending") with pytest.raises(ClientError) as refused: ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="first check is pending") @@ -759,8 +815,9 @@ def accept_and_hold(): assert accepted_at[1] - first < 45.0, f"the second connection came {accepted_at[1] - first:.1f} s after the first" assert "is unhealthy" in log and log.count("is unhealthy: the store could not be reached") == 1, \ "the transition is logged once" - details = ddb.create_backup(TableName="Music", BackupName="nightly2")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + with pytest.raises(ClientError) as refused: + ddb.create_backup(TableName="Music", BackupName="nightly2") + _assert_store_refusal(refused.value, "s3", "tarpit-bucket", category="unreachable") finally: server.stop() tarpit.close() @@ -814,10 +871,11 @@ def test_a_default_root_with_rows_is_unhealthy_rather_than_unconfigured(self, sq with pytest.raises(ClientError) as refused: ddb.delete_backup(BackupArn=arn) _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") - # The in-catalog create is served on the unhealthy store. + # CreateBackup is refused on the unhealthy store too. _create_table(ddb, "Music") - details = ddb.create_backup(TableName="Music", BackupName="nightly")["BackupDetails"] - assert details["BackupStatus"] in ("CREATING", "AVAILABLE") + with pytest.raises(ClientError) as refused: + ddb.create_backup(TableName="Music", BackupName="nightly") + _assert_store_refusal(refused.value, "filesystem", root, category="manifest probe failed") # The manifest put back: the next cycle recovers the store. for level in (os.path.dirname(os.path.dirname(manifest_dir)), os.path.dirname(manifest_dir), manifest_dir): if not os.path.isdir(level): @@ -900,8 +958,8 @@ def test_a_catalog_re_created_after_destroy_warns_about_the_old_default_root(sel """`destroy` then `init --overwrite` on the same directory: the default root's identity file names the old catalog. The server starts with the store unconfigured and a warning naming the file (not degraded, not a - startup error), CreateBackup keeps the in-catalog path, and `destroy`'s - output said to remove the file.""" + startup error), CreateBackup takes the in-catalog path with its + deprecation warning, and `destroy`'s output said to remove the file.""" d = sqlite_deployment server = _Server(d.config, str(tmp_path / "serve.log")) try: diff --git a/tests/test_mongodb_backup_restore.py b/tests/test_mongodb_backup_restore.py index c68c34cdb..adee9b5ac 100644 --- a/tests/test_mongodb_backup_restore.py +++ b/tests/test_mongodb_backup_restore.py @@ -26,6 +26,17 @@ def mongodb_container() -> str: container = os.environ.get("EXTENDDB_TEST_MONGODB_CONTAINER", "").strip() if not container: pytest.skip("requires devtools/run-mongodb-tests") + # These cells read the in-catalog create's documents (`backups`) and + # restore its backups at once; on a server with a `[backup.store]` + # CreateBackup is the store-backed job (CREATING, then AVAILABLE in the + # store) and RestoreTableFromBackup of its backups is refused until the + # restore job lands (PR 9a), which brings these cells back there. + store = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() + if store in ("filesystem", "s3"): + pytest.skip( + f"EXTENDDB_TEST_BACKUP_STORE={store}: these cells read and restore the " + "in-catalog create's backups; the store-backed restore is PR 9a's" + ) return container From 0773f03aa52a638ec24a73340c74166b33b2af31 Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Wed, 4 Nov 2026 11:16:14 +0000 Subject: [PATCH 3/3] feat(backup): RestoreTableFromBackup job and cross-backend matrix With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account
    `, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df3: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee Signed-off-by: Scott Robinson --- .github/workflows/integration-mongodb.yml | 14 +- .github/workflows/integration.yml | 45 +- .github/workflows/matrix-nightly.yml | 471 + CONTRIBUTING.md | 60 +- crates/app/src/cmd_backup.rs | 294 +- crates/app/src/lib.rs | 2 +- crates/config/src/backup.rs | 12 +- crates/core/src/backup_format/data_file.rs | 8 + crates/core/src/backup_format/manifest.rs | 7 + crates/core/src/metrics/collector.rs | 37 +- crates/core/src/metrics/collector_tests.rs | 58 + crates/core/src/metrics/types.rs | 45 +- crates/engine/Cargo.toml | 6 +- crates/engine/src/backup.rs | 116 +- crates/engine/src/backup/job.rs | 91 +- crates/engine/src/backup/job/failure.rs | 51 +- crates/engine/src/backup/job/heartbeat.rs | 140 +- crates/engine/src/backup/maintenance.rs | 204 +- crates/engine/src/backup/reconciler.rs | 1087 ++- crates/engine/src/backup/restore.rs | 1473 +++ crates/engine/src/backup/restore/download.rs | 320 + crates/engine/src/backup/restore/load.rs | 285 + crates/engine/src/backup/restore/spool.rs | 516 ++ crates/engine/src/backup/runtime.rs | 128 +- crates/engine/src/backup/tests.rs | 7975 ++++++++++++++++- crates/engine/src/lib.rs | 6 +- crates/server/Cargo.toml | 12 +- crates/server/src/backup_store.rs | 31 + crates/server/src/metrics_endpoint.rs | 3 + crates/server/src/serve.rs | 3 +- crates/storage-cassandra/Cargo.toml | 6 +- .../storage-cassandra/src/backup_catalog.rs | 572 +- crates/storage-cassandra/src/backup_engine.rs | 333 +- crates/storage-cassandra/src/create_table.rs | 64 +- .../src/data/put_get_item.rs | 97 + crates/storage-cassandra/src/lib.rs | 1 + .../storage-cassandra/src/migrations/mod.rs | 5 + .../storage-cassandra/src/migrations/tests.rs | 1 + .../storage-cassandra/src/restore_engine.rs | 599 ++ crates/storage-cassandra/src/test_hooks.rs | 175 +- .../tests/direct/backup_engine.rs | 2498 +++++- crates/storage-mongodb/Cargo.toml | 7 +- crates/storage-mongodb/src/backup_catalog.rs | 435 +- crates/storage-mongodb/src/backup_engine.rs | 233 +- .../storage-mongodb/src/backup_live_tests.rs | 1085 ++- crates/storage-mongodb/src/lib.rs | 1 + crates/storage-mongodb/src/migrations/mod.rs | 5 + .../storage-mongodb/src/migrations/tests.rs | 3 +- crates/storage-mongodb/src/restore_engine.rs | 372 + crates/storage-mongodb/src/snapshot_engine.rs | 132 +- crates/storage-mongodb/src/table_engine.rs | 39 +- crates/storage-mongodb/src/ttl_worker.rs | 16 +- .../storage-mongodb/tests/migrations_fresh.rs | 18 +- .../storage-mongodb/tests/migrations_locks.rs | 2 +- .../tests/migrations_released.rs | 2 +- .../data_migrations/0007_restore_progress.sql | 18 + crates/storage-postgres/src/backup_catalog.rs | 484 +- crates/storage-postgres/src/backup_engine.rs | 456 +- crates/storage-postgres/src/create_table.rs | 169 +- crates/storage-postgres/src/lib.rs | 6 + crates/storage-postgres/src/migrations/mod.rs | 16 +- crates/storage-postgres/src/restore_engine.rs | 438 + crates/storage-postgres/tests/restore_job.rs | 1797 ++++ crates/storage-sqlite/src/backup.rs | 48 +- crates/storage-sqlite/src/backup_catalog.rs | 381 +- crates/storage-sqlite/src/create_table.rs | 200 +- .../storage-sqlite/src/data/vector_index.rs | 2 +- crates/storage-sqlite/src/lib.rs | 27 +- crates/storage-sqlite/src/migrations/mod.rs | 4 + .../src/migrations/released_tests.rs | 8 +- crates/storage-sqlite/src/migrations/tests.rs | 9 +- crates/storage-sqlite/src/restore_engine.rs | 562 ++ .../storage-sqlite/src/restore_job_tests.rs | 1537 ++++ .../storage-sqlite/tests/backup_job_panic.rs | 4 +- .../tests/restore_job_resume.rs | 491 + crates/storage/src/backup_catalog.rs | 266 +- crates/storage/src/backup_definition.rs | 70 + crates/storage/src/lib.rs | 10 +- crates/storage/src/migrations/registry.rs | 9 + crates/storage/src/restore_engine.rs | 169 + devtools/backup-kill-points.py | 95 +- devtools/backup-restore-matrix | 974 ++ devtools/run-mongodb-tests | 126 +- docs/design/15-backup-restore-bench-pr9.md | 310 + docs/design/15-backup-restore.md | 136 +- docs/design/README.md | 1 + docs/differences-from-dynamodb.md | 19 +- docs/dynamodb-limits.md | 4 +- docs/manuals/03-usage-guide.md | 2 +- docs/manuals/05-admin-guide.md | 21 +- docs/manuals/06-developer-test-guide.md | 65 +- docs/manuals/07-upgrade-manual.md | 8 +- docs/schema/postgres.txt | 6 + requirements-matrix.txt | 9 + tests/rust/src/backup_restore.rs | 82 +- tests/rust/src/restore_active_completeness.rs | 39 +- tests/test_auth_permissions.py | 16 +- tests/test_backup_job.py | 891 +- tests/test_backup_restore_fidelity.py | 106 +- tests/test_backup_store_config.py | 46 +- tests/test_mongodb_backup_restore.py | 274 +- 101 files changed, 28837 insertions(+), 1775 deletions(-) create mode 100644 .github/workflows/matrix-nightly.yml create mode 100644 crates/engine/src/backup/restore.rs create mode 100644 crates/engine/src/backup/restore/download.rs create mode 100644 crates/engine/src/backup/restore/load.rs create mode 100644 crates/engine/src/backup/restore/spool.rs create mode 100644 crates/storage-cassandra/src/restore_engine.rs create mode 100644 crates/storage-mongodb/src/restore_engine.rs create mode 100644 crates/storage-postgres/data_migrations/0007_restore_progress.sql create mode 100644 crates/storage-postgres/src/restore_engine.rs create mode 100644 crates/storage-postgres/tests/restore_job.rs create mode 100644 crates/storage-sqlite/src/restore_engine.rs create mode 100644 crates/storage-sqlite/src/restore_job_tests.rs create mode 100644 crates/storage-sqlite/tests/restore_job_resume.rs create mode 100644 crates/storage/src/restore_engine.rs create mode 100755 devtools/backup-restore-matrix create mode 100644 docs/design/15-backup-restore-bench-pr9.md create mode 100644 requirements-matrix.txt diff --git a/.github/workflows/integration-mongodb.yml b/.github/workflows/integration-mongodb.yml index 1b5f84c77..2f439e0f1 100644 --- a/.github/workflows/integration-mongodb.yml +++ b/.github/workflows/integration-mongodb.yml @@ -54,9 +54,11 @@ jobs: run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB pytest suite env: - # run-mongodb-tests initializes the server without [backup.store]: - # CreateBackup is the in-catalog create. - EXTENDDB_TEST_BACKUP_STORE: none + # run-mongodb-tests gives the server a filesystem [backup.store] + # under its output directory: CreateBackup is the store-backed job + # and the restore cells (tests/test_mongodb_backup_restore.py) + # rewrite prefixes under that root. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-mongodb-tests -- --pytest --comprehensive --parallel # SnapshotEngine consistency check against a disposable replica set. The @@ -134,9 +136,9 @@ jobs: run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB rust integration suite env: - # run-mongodb-tests initializes the server without [backup.store]: - # the Rust backup tests restore the in-catalog create's backups. - EXTENDDB_TEST_BACKUP_STORE: none + # run-mongodb-tests gives the server a filesystem [backup.store]: + # the Rust backup tests restore the store-backed job's backups. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-mongodb-tests --rust-live -- --rust --rust-integration # The store-backed CreateBackup on MongoDB: a server with a filesystem diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index 4ea07acce..124e64e50 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -28,7 +28,10 @@ jobs: # rather than "latest" because ubuntu-latest migrates to 26.04 in Oct–Nov # 2026 and this compiles Rust; move to 26 deliberately. runs-on: extenddb_ubuntu-2404_4-core - timeout-minutes: 40 + # The job's own suites run close to 40 minutes; the backup store and job + # suites it ends with (their own servers, the restore cells among them) + # add about ten, so the bound leaves room for a slow runner. + timeout-minutes: 55 services: postgres: # pgvector's own image, which is postgres:16 plus the extension. The @@ -172,14 +175,34 @@ jobs: EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + # moto for the job suite's store-outage cell, the one cell that + # exercises `spool_path` on PostgreSQL (an S3 download into the spool, + # the retry under a cut, the spool entry removed); without the endpoint + # the cell skips. Same environment as the s3 job's. + - name: Start S3 mock + run: | + python3 -m venv "$RUNNER_TEMP/moto" + "$RUNNER_TEMP/moto/bin/pip" install -r requirements-moto.txt + nohup "$RUNNER_TEMP/moto/bin/moto_server" -p 9000 > moto.log 2>&1 & + for _ in $(seq 1 30); do + if curl -sf http://127.0.0.1:9000/ > /dev/null; then exit 0; fi + sleep 1 + done + echo "moto_server did not start"; cat moto.log; exit 1 + # The backup store and job suites start servers of their own, so they # are excluded from run-tests' line and run by name. - name: Run the backup store and job suites env: EXTENDDB_TEST_PG_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 + EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9000 run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider + - name: Print the S3 mock's log + if: failure() + run: cat moto.log || true + # run-tests restarts the server daemonized to apply the import/export # config, and a daemonized server logs to syslog, not the job log. Without # this dump, the storage error behind a client-visible 500 leaves no @@ -190,7 +213,10 @@ jobs: run-integration-sqlite: runs-on: extenddb_ubuntu-2404_4-core - timeout-minutes: 40 + # The job's own suites run close to 40 minutes; the backup store and job + # suites it ends with (their own servers, the restore cells among them) + # add about ten, so the bound leaves room for a slow runner. + timeout-minutes: 55 steps: - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 @@ -326,9 +352,11 @@ jobs: # with EXTENDDB_TEST_BACKUP_STORE=s3, and the fidelity suite's health check # confirms the server's store is the S3 one; the store configuration # suite's S3 cells need the same endpoint. The job suite - # (tests/test_backup_job.py) starts servers of its own with the SQLite - # build's defaulted filesystem root, as on the SQLite job: its deployment - # fixture has no S3 branch, so it covers the job and not this store. + # (tests/test_backup_job.py) starts servers of its own: most of its cells + # on the SQLite build's defaulted filesystem root, as on the SQLite job, + # and the store-outage cell on an S3 store at the moto endpoint, which it + # cuts mid-download and restores, so this job runs one restore from an S3 + # store. run-integration-sqlite-s3: runs-on: extenddb_ubuntu-2404_4-core timeout-minutes: 40 @@ -423,7 +451,9 @@ jobs: run: devtools/run-tests --extenddb --pytest --parallel --filter "backup" # The store configuration suite's S3 cells against moto; the job suite - # against the build's defaulted filesystem root (the job comment above). + # against the build's defaulted filesystem root, except the store-outage + # cell, which runs against moto through EXTENDDB_TEST_S3_ENDPOINT (the + # job comment above). - name: Run the store configuration suite's S3 cells and the job suite env: EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9000 @@ -772,7 +802,8 @@ jobs: # A file-backed SQLite server without [backup.store] has the defaulted # filesystem root beside its database file, so CreateBackup is the # asynchronous, store-backed one: the backup tests wait for AVAILABLE, - # delete what they create, and expect the restore's refusal. + # delete what they create, and restore the store-backed backups + # (the round trip). EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --rust-integration --release diff --git a/.github/workflows/matrix-nightly.yml b/.github/workflows/matrix-nightly.yml new file mode 100644 index 000000000..b7951da97 --- /dev/null +++ b/.github/workflows/matrix-nightly.yml @@ -0,0 +1,471 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +name: Cross-backend restore matrix (nightly) + +# The cross-backend restore matrix (design PR 9b): a backup taken on one +# backend restores, through the store, into another. Each cell starts a source +# and a target server from the per-backend binaries, copies the source's backup +# prefix under a root the target owns, seeds the target's `backup_index` row as +# `backup sync --adopt-store` will (PR 10), restores and compares. The cells are +# heavy (two servers each, four backends, PostgreSQL + a MongoDB replica set + +# Cassandra) so they run nightly rather than on every pull request. + +on: + schedule: + - cron: "0 7 * * *" + workflow_dispatch: + +permissions: + contents: read + +# One run at a time: the cells drop and re-create fixed MongoDB database and +# Cassandra keyspace names, so two runs on one runner would destroy each +# other's catalogs. A second run queues behind the first. +concurrency: + group: ${{ github.workflow }} + cancel-in-progress: false + +jobs: + matrix: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 120 + services: + postgres: + image: pgvector/pgvector:pg16 + env: + POSTGRES_PASSWORD: devpass + ports: + - 127.0.0.1:55432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 10 + cassandra: + image: cassandra:4.1 + ports: + - 127.0.0.1:9042:9042 + env: + CASSANDRA_CLUSTER_NAME: extenddb-ci + HEAP_NEWSIZE: 128M + MAX_HEAP_SIZE: 1024M + options: >- + --health-cmd "cqlsh -e 'SELECT release_version FROM system.local'" + --health-interval 15s + --health-timeout 10s + --health-retries 20 + --health-start-period 60s + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + # Only runs on main save, so other runs restore main's cache; a + # scheduled run's ref is the default branch, so the nightly saves. + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + # requirements-matrix.txt adds the MongoDB and Cassandra drivers the + # script's seed and teardown use on those backends. + run: pip install -r requirements.txt -r requirements-matrix.txt + + - name: Install the PostgreSQL client + # The matrix script seeds and tears down every PostgreSQL source and + # target through the host's psql; the runner image is not assumed to + # carry one. + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends postgresql-client + + - name: Check the host tools the matrix script calls + run: | + psql --version + python3 -c "import pymongo, cassandra; print(pymongo.version, cassandra.__version__)" + + - name: Configure Cassandra authentication + run: | + docker exec ${{ job.services.cassandra.id }} sed -i \ + 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + /etc/cassandra/cassandra.yaml + docker restart ${{ job.services.cassandra.id }} + for i in $(seq 1 90); do + if docker exec ${{ job.services.cassandra.id }} \ + cqlsh -u cassandra -p cassandra \ + -e 'SELECT release_version FROM system.local' >/dev/null 2>&1; then + echo "Cassandra ready with PasswordAuthenticator after ${i}s" + exit 0 + fi + sleep 2 + done + echo "Cassandra did not become ready" >&2 + exit 1 + + - name: Start a MongoDB replica set + # GitHub `services:` cannot express a replica set (--replSet + initiate + # + wait-for-PRIMARY), so start it directly, as devtools/run-mongodb-tests + # does. Published on loopback only; no authentication. + run: | + docker run -d --name matrix-mongo -p 127.0.0.1:27117:27017 mongo:7 \ + mongod --replSet rs0 --bind_ip_all + for i in $(seq 1 30); do + docker exec matrix-mongo mongosh --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null \ + | grep -qx 1 && break + sleep 1 + done + docker exec matrix-mongo mongosh --quiet --eval \ + 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]})' >/dev/null 2>&1 || true + for i in $(seq 1 30); do + state=$(docker exec matrix-mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' 2>/dev/null | tail -1) + [ "$state" = PRIMARY ] && { echo "replica set PRIMARY after ${i}s"; exit 0; } + sleep 1 + done + echo "MongoDB replica set did not elect a PRIMARY" >&2 + exit 1 + + - name: Build the four backend binaries + # One target directory, four sequential builds (cargo rebuilds only + # the feature-dependent crates between them), each binary copied to + # the path the script's --bin defaults use + # (target/rel-/release/extenddb). Every `init` the script + # runs writes the TLS material under ~/.extenddb/tls when absent, so + # nothing has to generate it here. + run: | + for b in postgres sqlite mongodb cassandra; do + cargo build --release -p extenddb --no-default-features --features "$b" + mkdir -p "target/rel-$b/release" + cp target/release/extenddb "target/rel-$b/release/extenddb" + done + + - name: Run the matrix + run: | + set -o pipefail + mkdir -p matrix-logs + rc=0 + plain="postgres sqlite mongodb" + targets="postgres sqlite mongodb cassandra" + for s in $plain; do + for t in $targets; do + echo "=== $s -> $t ===" + python3 devtools/backup-restore-matrix --source "$s" --target "$t" \ + --work "matrix-logs/$s-$t" || rc=1 + done + done + # The duplicate-key cell into every target, from a PostgreSQL source. + for t in $targets; do + echo "=== postgres -> $t --duplicate-key ===" + python3 devtools/backup-restore-matrix --source postgres --target "$t" --duplicate-key \ + --work "matrix-logs/postgres-$t-dup" || rc=1 + done + # The vector cells: into PostgreSQL and SQLite the vector index is + # restored and answers; into MongoDB and Cassandra the manifest is + # refused before any table. + for s in postgres sqlite; do + for t in postgres sqlite mongodb cassandra; do + echo "=== $s -> $t --vector ===" + python3 devtools/backup-restore-matrix --source "$s" --target "$t" --vector \ + --work "matrix-logs/$s-$t-vector" || rc=1 + done + done + exit $rc + + - name: Upload the matrix logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: matrix-logs + path: matrix-logs + if-no-files-found: ignore + + - name: Stop the MongoDB replica set + if: always() + run: docker rm -f matrix-mongo >/dev/null 2>&1 || true + + # The restore pause points, driven through devtools/backup-kill-points.py + # (committed by the CreateBackup PR; the restore half of the harness): + # `restore --bin --work ` pauses a + # restore at the point, kills the server, restarts it and asserts the + # outcome. No other job builds `backup-test-hooks`, so the restore kill + # points have no CI coverage without these jobs, one per backend, each + # starting its backend as the matrix job does. The points a backend has are + # the script's `RESTORE_POINTS`; the two second-reach runs put the kill + # after an earlier batch's progress is recorded. Every outcome is ACTIVE + # with the exact base and GSI counts, except the tampered-file point + # (`restore_between_failed_and_removal`), whose target is removed. + restore-kill-points-sqlite: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 90 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + run: pip install -r requirements.txt + - name: Build the SQLite binary with the backup test hooks + # The pause points inside the backends' restore writes compile only + # under `backup-test-hooks`; the harness drives this binary. + run: cargo build --release -p extenddb --no-default-features --features sqlite,extenddb-app/backup-test-hooks + - name: Run the SQLite restore kill points + run: | + set -o pipefail + rc=0 + for point in restore_after_download restore_after_file_commit \ + restore_sqlite_mid_batch restore_between_failed_and_removal sigterm; do + echo "=== sqlite restore kill point: $point ===" + python3 devtools/backup-kill-points.py restore sqlite "$point" \ + --bin target/release/extenddb --work "kill-point-logs/sqlite-$point" || rc=1 + done + exit $rc + - name: Upload the kill-point logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: kill-point-logs-sqlite + path: kill-point-logs + if-no-files-found: ignore + + restore-kill-points-postgres: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 90 + services: + postgres: + image: pgvector/pgvector:pg16 + env: + POSTGRES_PASSWORD: devpass + ports: + - 127.0.0.1:55432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 10 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + run: pip install -r requirements.txt + - name: Install the PostgreSQL client + # The script creates, probes (`pg_terminate_backend`, the vector + # relation check) and drops its PostgreSQL databases through the + # host's psql. + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends postgresql-client + psql --version + - name: Build the PostgreSQL binary with the backup test hooks + run: cargo build --release -p extenddb --no-default-features --features postgres,extenddb-app/backup-test-hooks + - name: Run the PostgreSQL restore kill points + # `pg_terminate_backend` is not a pause point of its own: the claim + # session is terminated while the job is paused after its first + # file's commit. `--vector` on the tampered-file point asserts the + # removed target's vector relation gone (the service image carries + # pgvector). + run: | + set -o pipefail + rc=0 + for point in restore_after_download restore_after_file_commit \ + restore_postgres_before_progress_delete restore_between_failed_and_removal \ + restore_fenced_out pg_terminate_backend sigterm; do + echo "=== postgres restore kill point: $point ===" + python3 devtools/backup-kill-points.py restore postgres "$point" \ + --bin target/release/extenddb --work "kill-point-logs/postgres-$point" || rc=1 + done + echo "=== postgres restore kill point: restore_between_failed_and_removal --vector ===" + python3 devtools/backup-kill-points.py restore postgres restore_between_failed_and_removal \ + --vector --bin target/release/extenddb \ + --work kill-point-logs/postgres-restore_between_failed_and_removal-vector || rc=1 + exit $rc + - name: Upload the kill-point logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: kill-point-logs-postgres + path: kill-point-logs + if-no-files-found: ignore + + restore-kill-points-mongodb: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 90 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + # requirements-matrix.txt adds pymongo, which the script's setup and + # teardown use to drop the fixed databases. + run: | + pip install -r requirements.txt -r requirements-matrix.txt + python3 -c "import pymongo; print(pymongo.version)" + - name: Start a MongoDB replica set + # As the matrix job does: `services:` cannot express a replica set. + # Published on loopback only; no authentication. + run: | + docker run -d --name kill-point-mongo -p 127.0.0.1:27117:27017 mongo:7 \ + mongod --replSet rs0 --bind_ip_all + for i in $(seq 1 30); do + docker exec kill-point-mongo mongosh --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null \ + | grep -qx 1 && break + sleep 1 + done + docker exec kill-point-mongo mongosh --quiet --eval \ + 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]})' >/dev/null 2>&1 || true + for i in $(seq 1 30); do + state=$(docker exec kill-point-mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' 2>/dev/null | tail -1) + [ "$state" = PRIMARY ] && { echo "replica set PRIMARY after ${i}s"; exit 0; } + sleep 1 + done + echo "MongoDB replica set did not elect a PRIMARY" >&2 + exit 1 + - name: Build the MongoDB binary with the backup test hooks + run: cargo build --release -p extenddb --no-default-features --features mongodb,extenddb-app/backup-test-hooks + - name: Run the MongoDB restore kill points + # The second reach of `restore_mongodb_after_insert_many` kills after + # an earlier batch's progress committed (24,000 items in three files). + run: | + set -o pipefail + rc=0 + for point in restore_after_download restore_after_file_commit \ + restore_mongodb_after_insert_many restore_mongodb_before_worker_flip \ + restore_between_failed_and_removal sigterm; do + echo "=== mongodb restore kill point: $point ===" + python3 devtools/backup-kill-points.py restore mongodb "$point" \ + --bin target/release/extenddb --work "kill-point-logs/mongodb-$point" || rc=1 + done + echo "=== mongodb restore kill point: restore_mongodb_after_insert_many --second-reach ===" + python3 devtools/backup-kill-points.py restore mongodb restore_mongodb_after_insert_many \ + --second-reach --items 24000 --bin target/release/extenddb \ + --work kill-point-logs/mongodb-restore_mongodb_after_insert_many-second || rc=1 + exit $rc + - name: Upload the kill-point logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: kill-point-logs-mongodb + path: kill-point-logs + if-no-files-found: ignore + - name: Stop the MongoDB replica set + if: always() + run: docker rm -f kill-point-mongo >/dev/null 2>&1 || true + + restore-kill-points-cassandra: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 90 + services: + cassandra: + image: cassandra:4.1 + ports: + - 127.0.0.1:9042:9042 + env: + CASSANDRA_CLUSTER_NAME: extenddb-ci + HEAP_NEWSIZE: 128M + MAX_HEAP_SIZE: 1024M + options: >- + --health-cmd "cqlsh -e 'SELECT release_version FROM system.local'" + --health-interval 15s + --health-timeout 10s + --health-retries 20 + --health-start-period 60s + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + - uses: dtolnay/rust-toolchain@683b588c487a69d01f6c6ddf40f036ff682cc2c8 # master + with: + toolchain: stable + - uses: Swatinem/rust-cache@d1527c8f67ef9af357290475d4af4b0fab8f12e0 # v2.9.3 + with: + cache-on-failure: true + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.11" + - name: Install Python dependencies + # requirements-matrix.txt adds cassandra-driver, which the script's + # teardown uses to drop the keyspaces. + run: | + pip install -r requirements.txt -r requirements-matrix.txt + python3 -c "import cassandra; print(cassandra.__version__)" + - name: Configure Cassandra authentication + # The script's `init` and config use cassandra/cassandra, as the + # matrix job's do. + run: | + docker exec ${{ job.services.cassandra.id }} sed -i \ + 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + /etc/cassandra/cassandra.yaml + docker restart ${{ job.services.cassandra.id }} + for i in $(seq 1 90); do + if docker exec ${{ job.services.cassandra.id }} \ + cqlsh -u cassandra -p cassandra \ + -e 'SELECT release_version FROM system.local' >/dev/null 2>&1; then + echo "Cassandra ready with PasswordAuthenticator after ${i}s" + exit 0 + fi + sleep 2 + done + echo "Cassandra did not become ready" >&2 + exit 1 + - name: Build the Cassandra hooks binary and the SQLite source binary + # Cassandra has no SnapshotEngine, so its CreateBackup is in-catalog + # and no backup taken on it restores through the store-backed job: the + # script takes each point's backup on a SQLite source server + # (`--source-bin`, a plain SQLite build) and adopts it into the + # Cassandra target's catalog, as the matrix does. One target + # directory, two sequential builds. + run: | + cargo build --release -p extenddb --no-default-features --features sqlite + mkdir -p target/rel-sqlite/release + cp target/release/extenddb target/rel-sqlite/release/extenddb + cargo build --release -p extenddb --no-default-features --features cassandra,extenddb-app/backup-test-hooks + - name: Run the Cassandra restore kill points + # The first reach of `restore_cassandra_before_progress` leaves the + # first batch's items with no progress recorded and resumes with + # upserts; the second reach kills after the first batch's progress. + run: | + set -o pipefail + rc=0 + for point in restore_after_download restore_after_file_commit \ + restore_cassandra_before_progress restore_between_failed_and_removal sigterm; do + echo "=== cassandra restore kill point: $point ===" + python3 devtools/backup-kill-points.py restore cassandra "$point" \ + --bin target/release/extenddb --source-bin target/rel-sqlite/release/extenddb \ + --work "kill-point-logs/cassandra-$point" || rc=1 + done + echo "=== cassandra restore kill point: restore_cassandra_before_progress --second-reach ===" + python3 devtools/backup-kill-points.py restore cassandra restore_cassandra_before_progress \ + --second-reach --bin target/release/extenddb --source-bin target/rel-sqlite/release/extenddb \ + --work kill-point-logs/cassandra-restore_cassandra_before_progress-second || rc=1 + exit $rc + - name: Upload the kill-point logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: kill-point-logs-cassandra + path: kill-point-logs + if-no-files-found: ignore diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a3f1e2c99..e5c071e16 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,8 +50,25 @@ cargo test --workspace # Integration tests (requires a running extenddb server; the variable says # where that server writes its backups, see the developer test guide) EXTENDDB_TEST_BACKUP_STORE=filesystem devtools/run-tests --extenddb --pytest + +# The backup store and job suites start servers of their own from the +# release build named by EXTENDDB_BINARY, so they run by name (the S3 cells +# need a moto server at EXTENDDB_TEST_S3_ENDPOINT; see the developer test +# guide for the PostgreSQL and moto variables) +EXTENDDB_BINARY=target/release/extenddb python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v ``` +The MongoDB backup cells that read the store's root and the server's log +(`EXTENDDB_TEST_BACKUP_ROOT`, `EXTENDDB_TEST_SERVER_LOG`) run through +`devtools/run-mongodb-tests`, which exports both; its legacy LSI fallback +cell needs the runner's `--backup-store none`. The cross-backend restore +matrix is `devtools/backup-restore-matrix` (one cell per invocation, from the +four per-backend release binaries; `pip install -r requirements-matrix.txt` +for its MongoDB and Cassandra cells) and runs nightly from +`.github/workflows/matrix-nightly.yml`. The developer test guide +(`docs/manuals/06-developer-test-guide.md`, "Test Runner Script") has a +paragraph on each. + ### Code Style All contributions must pass: @@ -342,8 +359,8 @@ SQLite, PostgreSQL, Cassandra and MongoDB migrate through this framework. ### Example: a nullable column on `tables` -Migration 7, `table_note` (7 being the next free id at the time of writing; -migration 6 is `backup_catalog_index`), adds a nullable `note TEXT` column to +Migration 8, `table_note` (8 being the next free id at the time of writing; +migration 7 is `restore_progress`), adds a nullable `note TEXT` column to the catalog's `tables` table on SQLite and PostgreSQL. In `crates/storage/src/migrations/registry.rs`, at the end of the @@ -351,18 +368,18 @@ In `crates/storage/src/migrations/registry.rs`, at the end of the ```rust /// Adds a nullable `note` column to the `tables` catalog table. - TableNote = 7, "table_note", + TableNote = 8, "table_note", "add a nullable note column to the tables catalog"; ``` PostgreSQL has `ADD COLUMN IF NOT EXISTS`, so its body is a file, -`crates/storage-postgres/migrations/0007_table_note.sql`: +`crates/storage-postgres/migrations/0008_table_note.sql`: ```sql -- Copyright 2026 ExtendDB contributors -- SPDX-License-Identifier: Apache-2.0 -- --- Migration 7, table_note, PostgreSQL catalog database: a nullable note +-- Migration 8, table_note, PostgreSQL catalog database: a nullable note -- column on tables. IF NOT EXISTS makes a second run change nothing. ALTER TABLE tables ADD COLUMN IF NOT EXISTS note TEXT; @@ -377,7 +394,7 @@ In `crates/storage-postgres/src/migrations/mod.rs`, the arm in `migration()`: bodies: vec![file_body!( StoreKind::Catalog, "migrations", - "0007_table_note.sql" + "0008_table_note.sql" )], }, ``` @@ -390,19 +407,20 @@ and the id in the transactional arm of `runs_in()`: | MigrationId::SortKeyDefinitions | MigrationId::DropContinuousBackups | MigrationId::BackupCatalogIndex + | MigrationId::RestoreProgress | MigrationId::TableNote => RunsIn::Transaction, ``` SQLite has no `ADD COLUMN IF NOT EXISTS`, and a file with a plain `ADD COLUMN` fails when it runs a second time, so the SQLite body is code that checks for the column first. -`crates/storage-sqlite/src/migrations/m0007_table_note.rs`: +`crates/storage-sqlite/src/migrations/m0008_table_note.rs`: ```rust // Copyright 2026 ExtendDB contributors // SPDX-License-Identifier: Apache-2.0 -//! Migration 7, `table_note`, SQLite: a nullable `note` column on `tables`. +//! Migration 8, `table_note`, SQLite: a nullable `note` column on `tables`. //! //! SQLite has no `ADD COLUMN IF NOT EXISTS`, so the body reads the table's //! columns first and adds the column only when it is missing. A second run @@ -414,10 +432,10 @@ use extenddb_storage::migrations::{CodeBody, Done, MigrationError}; use super::SqliteStores; /// This file, whose SHA-256 the checksum test pins. -pub(crate) const SOURCE: &str = include_str!("m0007_table_note.rs"); +pub(crate) const SOURCE: &str = include_str!("m0008_table_note.rs"); /// Recorded as this migration's checksum. Changes only with a new migration. -const FINGERPRINT: &str = "0007_table_note/sqlite/1"; +const FINGERPRINT: &str = "0008_table_note/sqlite/1"; pub(crate) struct TableNote; @@ -456,7 +474,7 @@ beside the others: ```rust #[rustfmt::skip] #[allow(clippy::all, deprecated)] -mod m0007_table_note; +mod m0008_table_note; ``` and the arm in `migration()`: @@ -467,8 +485,8 @@ and the arm in `migration()`: convergent: true, bodies: vec![StoreBody::code( StoreKind::Catalog, - &m0007_table_note::TableNote, - m0007_table_note::SOURCE, + &m0008_table_note::TableNote, + m0008_table_note::SOURCE, )], }, ``` @@ -480,8 +498,8 @@ rows of `tables`: ```rust #[tokio::test] -async fn migration_7_adds_the_note_column_to_tables() { - let db = temp_db("m7"); +async fn migration_8_adds_the_note_column_to_tables() { + let db = temp_db("m8"); load_dump(&db, DEV_0_1_14).await; run(&db).await.expect("migrate"); let mut text = dump( @@ -491,12 +509,12 @@ async fn migration_7_adds_the_note_column_to_tables() { ) .await; text.push_str(&dump(&db, "SELECT table_name, note FROM tables ORDER BY table_name").await); - golden("0007_table_note", &text); + golden("0008_table_note", &text); } ``` -`EXTENDDB_BLESS_EXPECTED=1 cargo test -p extenddb-storage-sqlite --lib migration_7` -writes `crates/storage-sqlite/tests/fixtures/golden/0007_table_note.txt`, one +`EXTENDDB_BLESS_EXPECTED=1 cargo test -p extenddb-storage-sqlite --lib migration_8` +writes `crates/storage-sqlite/tests/fixtures/golden/0008_table_note.txt`, one tab-separated line per row under each query: ``` @@ -511,15 +529,15 @@ The PostgreSQL body is a file, so it has no golden test. The example declares `Kind::Breaking`. A nullable column can be additive by the rule in step 4, but declaring it additive needs the old-binary job, which this example does not add. As a breaking migration it makes a 1.x binary -without migration 7 refuse a catalog that has it, and a rollback restores the +without migration 8 refuse a catalog that has it, and a rollback restores the backup. The same PR changes the code that reads and writes `tables.note` in each backend's catalog store, and commits the `docs/schema/sqlite.txt` and `docs/schema/postgres.txt` that the integration jobs regenerate. It changes no per-table DDL, because `tables` is a catalog table, and adds no `RELEASED` -entry. `extenddb migrate --list` then shows migration 7, and on a catalog at -migration 6 `extenddb migrate` lists migration 7 as pending and exits 3. +entry. `extenddb migrate --list` then shows migration 8, and on a catalog at +migration 7 `extenddb migrate` lists migration 8 as pending and exits 3. ### Running the migration tests diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs index 0f1ae6f4f..2ddc308b2 100644 --- a/crates/app/src/cmd_backup.rs +++ b/crates/app/src/cmd_backup.rs @@ -71,8 +71,11 @@ //! while the command holds the file, so before the wait the command runs //! the reconciler's duties itself with a zero stale window, as the holder //! of the maintenance claim (the cycle takes its held branch and does not -//! read the claim back through that window): a legacy restore -//! the stopped server left RUNNING is failed and its target removed, a +//! read the claim back through that window): a restore the stopped server +//! left RUNNING is failed and its target removed, a store-backed one too, +//! which the held branch fails rather than resumes (a record with no +//! recorded progress over a target that holds items is failed +//! `target_not_empty` and its target left for `remove-target`), a //! target a FAILED restore left CREATING is removed, an in-catalog CREATING //! backup it left is removed, and a DELETING row is finished; the wait and //! the CREATING-row removal then also use a zero window, so a CREATING or @@ -85,17 +88,18 @@ //! exclusion. //! //! Every command opens the catalog through the backend's factory, under its -//! own name (`extenddb backup list`, `extenddb backup purge`), and none runs -//! the startup recovery a server runs (`ServerComponentsOptions::startup_recovery` -//! off): that work belongs to the servers the command runs beside, and a -//! command that ran it could exit with it half done. On -//! PostgreSQL, MongoDB and Cassandra the command holds a serving -//! registration for its run, as a server does, so a migration waits for it, -//! and the registration carries the command's name, which the migration's -//! wait prints (PostgreSQL's session `application_name`, MongoDB's -//! `extenddb_servers` document, Cassandra's `extenddb_servers` row). The -//! command releases the registration when it ends, whatever the outcome, so -//! a migration does not wait for a row or document to expire. A catalog the +//! own name (`extenddb backup list`, `extenddb backup purge`, `extenddb +//! backup remove-target`), and none runs the startup recovery a server runs +//! (`ServerComponentsOptions::startup_recovery` off): that work belongs to +//! the servers the command runs beside, and a command that ran it could exit +//! with it half done. On PostgreSQL, MongoDB and Cassandra the command holds +//! a serving registration for its run, as a server does, so a migration +//! waits for it, and the registration carries the command's name, which the +//! migration's wait prints (PostgreSQL's session `application_name`, +//! MongoDB's `extenddb_servers` document, Cassandra's `extenddb_servers` +//! row). The command releases the registration when it ends, whatever the +//! outcome, so a migration does not wait for a row or document to expire. A +//! catalog the //! factory refuses (a migration running, the catalog at another migration //! than this binary's) ends the command with the message and the exit code //! `serve` gives for it: 6 while a migration runs, 3 for a catalog behind @@ -109,6 +113,24 @@ //! `extenddb destroy` removes the databases and leaves the store untouched: //! the store is the copy that survives destroy, and this command is the one //! that deletes it. +//! +//! `backup remove-target --account
    ` removes a restore target +//! DeleteTable refuses (a CREATING table) with its restore record, under the +//! maintenance claim like `purge`: a FAILED record's target whatever the +//! reason (`target_not_empty` included, the one state the reconciler leaves +//! for this command: on MongoDB and Cassandra a crash before the first +//! progress record, the whole first data file on MongoDB and the first +//! 10,000 items or 32 MiB on Cassandra, ends in it), a RUNNING record's +//! target whose owner is dead (no heartbeat within `stale_job_after_secs`; +//! on SQLite under the serve lock every owner is dead), and a legacy +//! record's target the same way. It refuses a RUNNING record whose owner is +//! live, a DONE record and a table no record names ("not a restore target"; +//! an ACTIVE table is one or the other). It needs no store: it opens the +//! catalog as `serve` does (on SQLite under the serve lock) whether or not +//! `[backup.store]` is configured, prints the claim it took and the one line +//! of what it removed (`extenddb_engine::maintenance::remove_target` states +//! each step), and logs the removal under `extenddb::audit` with the +//! account, the table and the backup's ARN. use std::sync::Arc; use std::time::{Duration, Instant}; @@ -116,12 +138,12 @@ use std::time::{Duration, Instant}; use clap::{Args, Subcommand}; use extenddb_config as config; use extenddb_engine::maintenance::{ - Interrupt, Stop, Wait, WaitOutcome, purge_rows, wait_for_quiet, + Interrupt, RemoveTarget, Stop, Wait, WaitOutcome, purge_rows, remove_target, wait_for_quiet, }; use extenddb_engine::{BackupRuntime, ConfiguredStore, HeldClaim, StoreHealth, StoreProblem}; use extenddb_storage::StorageEngine; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, StoreIdentity, + BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, JobStatus, StoreIdentity, }; use extenddb_storage::error::StorageError; use extenddb_storage::hooks::ServingRegistrar; @@ -157,6 +179,17 @@ enum BackupCommand { #[arg(long)] confirm: bool, }, + /// Remove a restore target DeleteTable refuses (a CREATING table) and its restore record: + /// the target of a FAILED restore (target_not_empty included) or of a RUNNING restore whose + /// owner is dead. Takes the maintenance claim first. Refuses a RUNNING restore with a live + /// owner, a DONE restore and a table no restore record names. + RemoveTarget { + /// The account the table belongs to + #[arg(long)] + account: String, + /// The table's name + table: String, + }, } /// How often the waiting purge re-reads the rows. @@ -168,6 +201,7 @@ const WAIT_POLL: Duration = Duration::from_secs(2); /// carries this name, which the migration prints while it waits. const LIST: &str = "extenddb backup list"; const PURGE: &str = "extenddb backup purge"; +const REMOVE_TARGET: &str = "extenddb backup remove-target"; pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> { if !std::path::Path::new(&args.config).exists() { @@ -184,10 +218,31 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> // `migrate --yes` to wait out. During the purge the same handle ends the // wait at once and the deletion loop before its next row. let interrupt = listen_for_signals(); + if let BackupCommand::RemoveTarget { account, table } = &args.command { + // The catalog as `serve` opens it; the store is not needed, so it is + // described and never checked, and the default root and an identity + // file are not created. + let (catalog, _) = open_catalog( + &app_config, + &args.config, + release, + REMOVE_TARGET, + Mode::WriteCatalogOnly, + ) + .await?; + if let Some(signal) = interrupt.signalled() { + catalog.close().await; + anyhow::bail!("{signal} received while opening the catalog; nothing was changed"); + } + let done = remove_restore_target(&catalog, &app_config, account, table).await; + catalog.close().await; + return done; + } let (command, mode) = match args.command { BackupCommand::List { .. } => (LIST, Mode::Read), BackupCommand::Purge { confirm: false } => (PURGE, Mode::Read), BackupCommand::Purge { confirm: true } => (PURGE, Mode::Write), + BackupCommand::RemoveTarget { .. } => unreachable!("handled above"), }; let opening = open(&app_config, &args.config, release, command, mode).await?; if let Some(signal) = interrupt.signalled() { @@ -233,6 +288,7 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> (BackupCommand::Purge { confirm: true }, Opening::NoStore(reason)) => { anyhow::bail!("{reason}") } + (BackupCommand::RemoveTarget { .. }, _) => unreachable!("handled above"), } } @@ -256,55 +312,46 @@ enum Mode { /// and which the command must not exit with half done; the store's /// check may write the identity file of an empty root. Write, + /// As `serve` opens it, for a command that needs no store + /// (`remove-target`): the store is described as `Read` leaves it, so + /// neither the default root nor an identity file is created. + WriteCatalogOnly, } -/// The storage engine, the backup runtime with the configured store, the -/// store's identity as the command read it, and the serving registration -/// the command holds for its run. -struct Opened { +/// The open catalog: the storage engine, the backup runtime (with the +/// configured store when there is one), the backend's name, and the serving +/// registration the command holds for its run. +struct Catalog { storage: Arc, runtime: Arc, backend: String, - identity: Identity, registration: Option>, } -/// The store's identity as the command read it. -enum Identity { - /// The check passed, or the file was read and agrees with the rows. - Known(StoreIdentity), - /// The root has no identity file and the catalog no rows for it: a - /// fresh root nothing has written to. - Empty, - /// The check failed (a listing still runs; a purge refuses). - Problem(StoreProblem), -} - -impl Opened { +impl Catalog { /// End the command's use of the catalog: release its serving /// registration. async fn close(self) { release_registration(self.registration).await; } - - fn store(&self) -> &ConfiguredStore { - self.runtime - .configured_store() - .expect("open returns only with a configured store") - } } -/// Open the catalog as `command`. A catalog the factory refuses (on -/// PostgreSQL: a migration running, or the catalog at another migration than -/// this binary's) is an [`extenddb_server::CatalogRefusal`], with the -/// message and exit code `serve` gives for the same status. -async fn open( +/// Open the catalog as `command` in `mode` and describe the configured +/// store without checking it; the second member is why no store is +/// configured, when none is. A catalog the factory refuses (a migration +/// running, or the catalog at another migration than this binary's) is an +/// [`extenddb_server::CatalogRefusal`], with the message and exit code +/// `serve` gives for the same status. +async fn open_catalog( app_config: &config::AppConfig, config_path: &str, release: &'static str, command: &'static str, mode: Mode, -) -> anyhow::Result { +) -> anyhow::Result<( + Catalog, + Option, +)> { let mut options = extenddb_storage::server_components::ServerComponentsOptions::default(); options.release = release; options.max_concurrent_jobs = app_config.backup.max_concurrent_jobs; @@ -337,7 +384,7 @@ async fn open( Err(e) => anyhow::bail!("cannot open the catalog: {e}"), }; // The registration is the runtime hooks', which the command does not - // otherwise keep: held here, it lasts until `Opened::close`. + // otherwise keep: held here, it lasts until `Catalog::close`. let registration = components .runtime_hooks .as_ref() @@ -350,7 +397,9 @@ async fn open( &backend, &*storage, match mode { - Mode::Read => extenddb_server::backup_store::Caller::ReadOnlyCommand, + Mode::Read | Mode::WriteCatalogOnly => { + extenddb_server::backup_store::Caller::ReadOnlyCommand + } Mode::Write => extenddb_server::backup_store::Caller::Command, }, ) @@ -362,17 +411,69 @@ async fn open( return Err(e); } }; - let runtime = Arc::new(setup.runtime); - let Some(store) = runtime.configured_store() else { + Ok(( + Catalog { + storage, + runtime: Arc::new(setup.runtime), + backend, + registration, + }, + setup.not_configured, + )) +} + +/// The catalog with the configured store and the store's identity as the +/// command read it. +struct Opened { + catalog: Catalog, + identity: Identity, +} + +/// The store's identity as the command read it. +enum Identity { + /// The check passed, or the file was read and agrees with the rows. + Known(StoreIdentity), + /// The root has no identity file and the catalog no rows for it: a + /// fresh root nothing has written to. + Empty, + /// The check failed (a listing still runs; a purge refuses). + Problem(StoreProblem), +} + +impl Opened { + /// End the command's use of the catalog: release its serving + /// registration. + async fn close(self) { + self.catalog.close().await; + } + + fn store(&self) -> &ConfiguredStore { + self.catalog + .runtime + .configured_store() + .expect("open returns only with a configured store") + } +} + +async fn open( + app_config: &config::AppConfig, + config_path: &str, + release: &'static str, + command: &'static str, + mode: Mode, +) -> anyhow::Result { + let (catalog, not_configured) = + open_catalog(app_config, config_path, release, command, mode).await?; + let Some(store) = catalog.runtime.configured_store() else { // Why there is none, as `configure_described` found it: the section // absent or disabled, the default root not yet created, or the // default root failed its check with no rows for it. - let why = setup - .not_configured - .expect("a runtime without a store carries the reason"); - release_registration(registration).await; + let why = not_configured.expect("a runtime without a store carries the reason"); + catalog.close().await; return Ok(Opening::NoStore(why.to_string())); }; + let storage = &catalog.storage; + let backend = &catalog.backend; // The server's startup leaves an S3 store's check to its reconciler; // the command runs it here. A read leaves the root as it finds it: the // identity file is read, never created, under the store's check bound as @@ -380,13 +481,15 @@ async fn open( // or a mount that has stopped answering costs the command the bound and // not the store's retries. let identity = match (mode, store.health()) { - (Mode::Read, _) => match extenddb_engine::read_identity_bounded(store, &*storage).await { - Ok(Some(identity)) => Identity::Known(identity), - Ok(None) => Identity::Empty, - Err(problem) => Identity::Problem(problem), - }, + (Mode::Read | Mode::WriteCatalogOnly, _) => { + match extenddb_engine::read_identity_bounded(store, &**storage).await { + Ok(Some(identity)) => Identity::Known(identity), + Ok(None) => Identity::Empty, + Err(problem) => Identity::Problem(problem), + } + } (Mode::Write, StoreHealth::Checking) => { - let outcome = extenddb_engine::check_configured_store(store, &*storage, &backend).await; + let outcome = extenddb_engine::check_configured_store(store, &**storage, backend).await; store.record_check(outcome.clone()); match outcome { Ok(identity) => Identity::Known(identity), @@ -401,13 +504,7 @@ async fn open( }, (Mode::Write, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem), }; - Ok(Opening::Store(Opened { - storage, - runtime, - backend, - identity, - registration, - })) + Ok(Opening::Store(Opened { catalog, identity })) } /// How long a command waits for its registration's release: a row or @@ -478,6 +575,7 @@ async fn rows_of_root( ) -> anyhow::Result<(Vec, Vec)> { let store = opened.store(); let mut ids = opened + .catalog .storage .read_store_ids(store.kind, &store.root) .await @@ -494,7 +592,7 @@ async fn rows_of_root( store_root: store.root.clone(), store_id: id.clone(), }; - rows.extend(rows_of(&*opened.storage, &identity, status).await?); + rows.extend(rows_of(&*opened.catalog.storage, &identity, status).await?); } Ok((ids, rows)) } @@ -603,7 +701,7 @@ async fn purge( mut interrupt: Interrupt, ) -> anyhow::Result<()> { let backup = &app_config.backup; - let storage = &*opened.storage; + let storage = &*opened.catalog.storage; let configured = opened.store(); let identity = match &opened.identity { Identity::Known(identity) => identity, @@ -650,7 +748,7 @@ async fn purge( // owner of any row is dead whatever its heartbeat says, so the window a // live owner is told apart by is zero. On the other backends purge runs // beside live servers, whose jobs heartbeat within `stale_after`. - let owner_window = if opened.backend == "sqlite" { + let owner_window = if opened.catalog.backend == "sqlite" { resolve_dead_owners(opened).await; Duration::ZERO } else { @@ -825,10 +923,10 @@ async fn signal_name() -> &'static str { /// wait decides. async fn resolve_dead_owners(opened: &Opened) { match extenddb_engine::reconciler::cycle( - &*opened.storage, - &opened.runtime, + &opened.catalog.storage, + &opened.catalog.runtime, Duration::ZERO, - &opened.backend, + &opened.catalog.backend, &mut extenddb_engine::reconciler::ReconcilerState::for_maintenance_holder(), ) .await @@ -882,6 +980,60 @@ async fn release(claim: HeldClaim) { } } +/// `backup remove-target --account
    ` (module doc). +async fn remove_restore_target( + catalog: &Catalog, + app_config: &config::AppConfig, + account_id: &str, + table_name: &str, +) -> anyhow::Result<()> { + let stale_after = Duration::from_secs(app_config.backup.stale_job_after_secs); + let holder = format!( + "extenddb backup remove-target on {} (pid {})", + extenddb_server::backup_store::hostname(), + std::process::id() + ); + println!("=== extenddb backup remove-target ==="); + println!("Taking the maintenance claim as {holder:?}."); + // On SQLite the command holds the file alone, so every job owner is + // dead whatever its heartbeat says; on the other backends a RUNNING + // record whose heartbeat is within `stale_after` has a live owner. + let owner_window = if catalog.backend == "sqlite" { + Duration::ZERO + } else { + stale_after + }; + let removed = remove_target( + &*catalog.storage, + &RemoveTarget { + account_id: account_id.to_owned(), + table_name: table_name.to_owned(), + holder, + stale_after, + owner_window, + heartbeat_interval: JOB_HEARTBEAT_INTERVAL, + }, + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + let state = match (removed.status, removed.failure_reason.as_deref()) { + (JobStatus::Running, _) => "RUNNING with a dead owner".to_owned(), + (status, Some(reason)) => format!("{status} ({reason})"), + (status, None) => status.to_string(), + }; + let kind = match removed.kind { + extenddb_storage::backup_catalog::RestoreKind::Manifest => "store-backed", + extenddb_storage::backup_catalog::RestoreKind::Legacy => "in-catalog", + }; + println!( + "removed the restore target {table_name} (account {account_id}, table id {}) and its \ + record: its {kind} restore from {} was {state}", + removed.table_id, removed.backup_arn + ); + println!("The maintenance claim is released."); + Ok(()) +} + #[cfg(test)] mod tests { use std::sync::Arc; diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index 43151f03e..30870c3bb 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -69,7 +69,7 @@ enum Command { Init(cmd_init::InitArgs), /// Tear down a extenddb deployment (the databases; the backup store is left untouched) Destroy(cmd_destroy::DestroyArgs), - /// List or purge the backups of the configured backup store + /// List or purge the backups of the configured backup store, or remove a restore target Backup(cmd_backup::BackupArgs), /// Validate a extenddb deployment Verify(cmd_verify::VerifyArgs), diff --git a/crates/config/src/backup.rs b/crates/config/src/backup.rs index 7ff107f90..098acd631 100644 --- a/crates/config/src/backup.rs +++ b/crates/config/src/backup.rs @@ -77,10 +77,18 @@ //! either format stay readable, listable, restorable and deletable. //! //! The spool. Restores from an S3 store download to `spool_path`, created -//! `0700` with files `0600`. On file-backed SQLite it defaults to +//! `0700` with files `0600`: one entry `restore-
    /` per running +//! restore, holding the manifest, its sidecar and each data file with a +//! `.verified` marker once its digests matched, so the spool costs the +//! compressed size of the backups being restored at once; the job removes +//! its entry when it ends, a server stopped by SIGTERM keeps it for the +//! resume, and the reconciler removes entries of restores no longer RUNNING +//! with this host's `spool_host`. On file-backed SQLite it defaults to //! `backup-spool/` beside the database file; on every other backend an S3 //! store needs `spool_path` and startup refuses without it (a filesystem -//! store needs no spool: files are verified in place). A `spool_path` that +//! store needs no spool: files are hashed in place and read again for the +//! load, so a filesystem store's `get` errors, which are local I/O errors, +//! are not retried as an S3 store's transport errors are). A `spool_path` that //! contains or is contained in a filesystem store root is refused (the store //! would list and delete it, and a sync would read its first component as an //! account id), and so is one that is or contains the database file's diff --git a/crates/core/src/backup_format/data_file.rs b/crates/core/src/backup_format/data_file.rs index bb5e48033..70a226541 100644 --- a/crates/core/src/backup_format/data_file.rs +++ b/crates/core/src/backup_format/data_file.rs @@ -965,6 +965,14 @@ impl DataFileDecoder { self.reader.get_ref().get_ref().hasher.is_some() } + /// The uncompressed bytes decoded so far: every line consumed, its + /// newline included. The difference across one [`Self::next_item`] is + /// that item's line length, the unit a reader bounds its memory in. + #[must_use] + pub fn bytes_decoded(&self) -> u64 { + self.bytes_decoded + } + /// The error for a failed read: [`FormatError::CompressedSizeMismatch`] /// when the source ran past its recorded size, otherwise /// [`FormatError::Decode`]. diff --git a/crates/core/src/backup_format/manifest.rs b/crates/core/src/backup_format/manifest.rs index 28658df85..5afa2afd9 100644 --- a/crates/core/src/backup_format/manifest.rs +++ b/crates/core/src/backup_format/manifest.rs @@ -588,6 +588,13 @@ impl DataFileReader { } checked } + + /// The uncompressed bytes decoded so far + /// ([`DataFileDecoder::bytes_decoded`]). + #[must_use] + pub fn bytes_decoded(&self) -> u64 { + self.decoder.bytes_decoded() + } } impl Iterator for DataFileReader { diff --git a/crates/core/src/metrics/collector.rs b/crates/core/src/metrics/collector.rs index 8975e2675..f6a3e6f68 100755 --- a/crates/core/src/metrics/collector.rs +++ b/crates/core/src/metrics/collector.rs @@ -385,13 +385,44 @@ impl MetricsCollector { self.record(MetricName::BackupSnapshotAge, age_secs, None, None, None); } - /// Record a failed store call of a backup job or of the backup - /// reconciler; `op` names the call (the job's `put` and - /// `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`). + /// Record a failed store call of a backup job, a restore job or the + /// backup reconciler; `op` names the call (a backup job's `put` and + /// `delete_prefix`, a restore's `get`, the reconciler's `get`, `list` + /// and `delete_prefix`). pub fn record_backup_store_error(&self, op: &str) { self.record(MetricName::BackupStoreErrors, 1.0, None, None, Some(op)); } + /// Record the end of a store-backed restore job: one `RestoreJobs` under + /// the outcome and its duration in microseconds under the same + /// dimension, as [`Self::record_backup_job`] records a backup's. The job + /// measures the duration from its registration to the end of its last + /// catalog or store call. + pub fn record_restore_job(&self, outcome: super::types::BackupJobOutcome, duration_us: f64) { + let outcome = outcome.to_string(); + self.record(MetricName::RestoreJobs, 1.0, None, None, Some(&outcome)); + self.record( + MetricName::RestoreJobDuration, + duration_us, + None, + None, + Some(&outcome), + ); + } + + /// Record bytes a restore job read from the store, as each download + /// arrived. + #[allow(clippy::cast_precision_loss)] // byte counts of one backup + pub fn record_restore_bytes_downloaded(&self, bytes: u64) { + self.record( + MetricName::RestoreBytesDownloaded, + bytes as f64, + None, + None, + None, + ); + } + /// Record per-segment latency breakdown for a request. pub fn record_segments( &self, diff --git a/crates/core/src/metrics/collector_tests.rs b/crates/core/src/metrics/collector_tests.rs index 385b07e83..1683aa25f 100755 --- a/crates/core/src/metrics/collector_tests.rs +++ b/crates/core/src/metrics/collector_tests.rs @@ -232,3 +232,61 @@ fn backup_job_recorders_land_under_the_outcome_and_the_store_call() { assert_eq!(outcome.to_string(), text); } } + +/// The restore job's recorders land as the backup job's do: one +/// `RestoreJobs` and one `RestoreJobDuration` per job under the outcome, +/// the bytes undimensioned, a `get` failure under `BackupStoreErrors`; and +/// the names render as the `/metrics` endpoint prints them. +#[test] +fn restore_job_recorders_land_under_the_outcome_and_render_their_names() { + use crate::metrics::types::{BackupJobOutcome, QuerySource}; + + let c = MetricsCollector::new(); + c.record_restore_job(BackupJobOutcome::Succeeded, 2_000.0); + c.record_restore_job(BackupJobOutcome::Cancelled, 500.0); + c.record_restore_bytes_downloaded(8192); + c.record_backup_store_error("get"); + + let by = |metric: MetricName| { + let mut snaps = c.query(&MetricsQuery { + metric: Some(metric), + window: Some(TimeWindow::AllTime), + ..Default::default() + }); + snaps.sort_by(|a, b| format!("{:?}", a.dimensions).cmp(&format!("{:?}", b.dimensions))); + snaps + }; + let jobs = by(MetricName::RestoreJobs); + let outcomes: Vec = jobs.iter().map(|s| format!("{:?}", s.dimensions)).collect(); + assert_eq!( + outcomes, + [ + "[Operation(\"cancelled\")]".to_owned(), + "[Operation(\"succeeded\")]".to_owned() + ] + ); + assert!(jobs.iter().all(|s| s.count == 1 && s.sum == 1.0)); + let durations = by(MetricName::RestoreJobDuration); + assert_eq!(durations.len(), 2); + assert_eq!(durations.iter().map(|s| s.sum).sum::(), 2_500.0); + let bytes = by(MetricName::RestoreBytesDownloaded); + assert_eq!(bytes.len(), 1); + assert!(bytes[0].dimensions.is_empty()); + assert_eq!(bytes[0].sum, 8192.0); + let errors = by(MetricName::BackupStoreErrors); + assert_eq!( + format!("{:?}", errors[0].dimensions), + "[Operation(\"get\")]" + ); + assert!(by(MetricName::BackupJobs).is_empty()); + assert_eq!(MetricName::RestoreJobs.to_string(), "RestoreJobs"); + assert_eq!( + MetricName::RestoreJobDuration.to_string(), + "RestoreJobDuration" + ); + assert_eq!( + MetricName::RestoreBytesDownloaded.to_string(), + "RestoreBytesDownloaded" + ); + assert_eq!(QuerySource::RestoreJob.to_string(), "RestoreJob"); +} diff --git a/crates/core/src/metrics/types.rs b/crates/core/src/metrics/types.rs index 3d1b747c6..0030ce1b2 100755 --- a/crates/core/src/metrics/types.rs +++ b/crates/core/src/metrics/types.rs @@ -75,16 +75,36 @@ pub enum MetricName { /// Seconds between a running backup job's snapshot instant and now /// (gauge), recorded at each confirmed heartbeat. BackupSnapshotAge, - /// Store operations a backup job or the backup reconciler saw fail; the - /// operation dimension names the store call (`put` by the job's - /// uploads, `delete_prefix` by its failure path and the reconciler's - /// deleting duties, `get` and `list` by the reconciler's manifest probe - /// and deleting duties). + /// Store operations a backup or restore job or the backup reconciler + /// saw fail; the operation dimension names the store call (`put` by a + /// backup job's uploads, `delete_prefix` by its failure path and the + /// reconciler's deleting duties, `get` by a restore's reads of the + /// manifest and its sidecar, in the request and in a resume, by a + /// restore job's downloads and by the reconciler's manifest probe, and + /// `list` by the reconciler's manifest probe and deleting duties). BackupStoreErrors, + /// Store-backed restore jobs ended, one per job; the operation dimension + /// carries the outcome ([`BackupJobOutcome`]: `succeeded` for a DONE + /// record, `failed` for a FAILED one or a job that stopped without a + /// status write of its own, `cancelled` for one the shutdown stopped, + /// whose record stays RUNNING for a resume). Named as `BackupJobs` is + /// (the `/metrics` name; a converter's would be + /// `restore_jobs_total{outcome}`). + RestoreJobs, + /// Duration of a store-backed restore job in microseconds, from its + /// registration to the end of its last catalog or store call, with the + /// outcome as the operation dimension. + RestoreJobDuration, + /// Bytes of each data-file download attempt of a restore job as they + /// arrived, verified or not, so an attempt the job retried counts its + /// bytes too; the manifest and sidecar reads, and a filesystem store's + /// second read of each file for the load, are not counted. + RestoreBytesDownloaded, } -/// How a store-backed backup job ended, the operation dimension of -/// [`MetricName::BackupJobs`] and [`MetricName::BackupJobDuration`]. +/// How a store-backed backup or restore job ended, the operation dimension +/// of [`MetricName::BackupJobs`], [`MetricName::BackupJobDuration`], +/// [`MetricName::RestoreJobs`] and [`MetricName::RestoreJobDuration`]. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum BackupJobOutcome { /// The manifest was written and the row is AVAILABLE. @@ -92,8 +112,8 @@ pub enum BackupJobOutcome { /// The row is FAILED, or the job stopped without a status write of its /// own (its claim lost). Failed, - /// The server's shutdown stopped the job; the row is FAILED with reason - /// `shutdown`. + /// The server's shutdown stopped the job; a backup row is FAILED with + /// reason `shutdown`, a restore record stays RUNNING for a resume. Cancelled, } @@ -141,6 +161,9 @@ impl std::fmt::Display for MetricName { Self::BackupBytesUploaded => f.write_str("BackupBytesUploaded"), Self::BackupSnapshotAge => f.write_str("BackupSnapshotAge"), Self::BackupStoreErrors => f.write_str("BackupStoreErrors"), + Self::RestoreJobs => f.write_str("RestoreJobs"), + Self::RestoreJobDuration => f.write_str("RestoreJobDuration"), + Self::RestoreBytesDownloaded => f.write_str("RestoreBytesDownloaded"), } } } @@ -189,6 +212,9 @@ pub enum QuerySource { /// A store-backed backup job (the worker metrics are recorded once per /// job, at its end). BackupJob, + /// A store-backed restore job (the worker metrics are recorded once per + /// job, at its end; a job the shutdown stopped records none). + RestoreJob, /// Management CLI commands. Management, } @@ -211,6 +237,7 @@ impl std::fmt::Display for QuerySource { Self::LoginAttemptCleanup => f.write_str("LoginAttemptCleanup"), Self::BackupReconciler => f.write_str("BackupReconciler"), Self::BackupJob => f.write_str("BackupJob"), + Self::RestoreJob => f.write_str("RestoreJob"), Self::Management => f.write_str("Management"), } } diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml index 6800acfdd..1efd942e8 100755 --- a/crates/engine/Cargo.toml +++ b/crates/engine/Cargo.toml @@ -27,8 +27,10 @@ hmac = "0.12" sha2 = "0.10" hex = "0.4" tokio = { workspace = true } -# `TaskTracker` for the backup jobs the server drains at shutdown. -tokio-util = { workspace = true, features = ["rt"] } +# `TaskTracker` for the backup jobs the server drains at shutdown; +# `StreamReader` and `SyncIoBridge` for the restore's in-place decode of a +# store object on the blocking pool. +tokio-util = { workspace = true, features = ["rt", "io", "io-util"] } bytes = { workspace = true } futures = { workspace = true } diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs index 32d7b4615..9d01ada6f 100755 --- a/crates/engine/src/backup.rs +++ b/crates/engine/src/backup.rs @@ -29,14 +29,16 @@ use extenddb_storage::backup_catalog::{ }; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, LegacyBackupReader, MetadataEngine, RestoreRequest, - RestoreTableOverrides, SnapshotEngine, TableEngine, + BackupCatalog, BackupRestoreSource, DataEngine, LegacyBackupReader, MetadataEngine, + RestoreEngine, RestoreRequest, RestoreSource, RestoreTableOverrides, SnapshotEngine, + TableEngine, }; use futures::TryStreamExt; use serde_json::{Value, json}; pub use job::KillPoint; pub use maintenance::HeldClaim; +pub use restore::{OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, TARGET_NOT_EMPTY}; pub use runtime::{ BackupRuntime, ConfiguredStore, JobSettings, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, StoreHealth, StoreHealthReport, check_configured_store, read_identity_bounded, @@ -1505,13 +1507,21 @@ async fn restore_table_from_backup_detached( after: F, ) -> Result where - S: BackupCatalog + LegacyBackupReader + TableEngine + ?Sized + Send + Sync + 'static, + S: BackupCatalog + + LegacyBackupReader + + TableEngine + + DataEngine + + RestoreEngine + + ?Sized + + Send + + Sync + + 'static, F: FnOnce(Arc, String) -> Fut + Send + 'static, Fut: std::future::Future + Send, { tokio::spawn(async move { let (target_table_name, response) = - restore_table_from_backup(&*storage, &runtime, &account_id, ®ion, &limits, &body) + restore_table_from_backup(&storage, &runtime, &account_id, ®ion, &limits, &body) .await?; after(account_id, target_table_name).await; Ok(response) @@ -1546,15 +1556,20 @@ where /// rest of its checks ([`check_catalog_row_for_restore`]: a backup in use is /// `BackupInUseException`, a row of another store a `ValidationException`, /// a row of the configured store while it is unhealthy one naming the -/// store's kind, and an AVAILABLE row of the healthy store the -/// `ValidationException` naming the store, since this release restores -/// in-catalog backups only); (6) refuse what no -/// backend restores faithfully (multi-part key, vector indexes); (7) check -/// the throughput override against the backup's billing mode, build the -/// target's `CreateTableInput`, and apply the index and on-demand throughput -/// overrides to it, refusing what the service refuses ([`overrides`]); -/// (8) validate that input as CreateTable does; (9) refuse while a -/// maintenance command holds the catalog's claim +/// store's kind), and an AVAILABLE row of the healthy configured store is +/// restored from the store by the restore job ([`restore::request`]: the +/// permit (when none is freed within the runtime's wait, +/// `LimitExceededException`), the manifest and its checks, the vector +/// refusal of a target without vector support, the overrides, the detached +/// registration whose answer the request awaits; the maintenance claim is +/// checked by the job once its record exists, the design's 5.3 step 4); the +/// steps below are the legacy path's: (6) refuse what no backend restores +/// faithfully from an in-catalog backup (multi-part key, vector indexes); +/// (7) check the throughput override against the backup's billing mode, +/// build the target's `CreateTableInput`, and apply the index and on-demand +/// throughput overrides to it, refusing what the service refuses +/// ([`overrides`]); (8) validate that input as CreateTable does; (9) refuse +/// while a maintenance command holds the catalog's claim /// ([`refuse_while_maintenance_held`]), then take the job permit, last among /// the checks (`LimitExceededException` for either; the permit when none is /// freed within the runtime's wait), and ask the backend to create and @@ -1562,14 +1577,25 @@ where /// `TableAlreadyExistsException`, a name being restored into /// `TableInUseException` (the backend checks the name again, fenced, under /// the registration), a backup being restored from `BackupInUseException`. -async fn restore_table_from_backup( - storage: &S, +async fn restore_table_from_backup( + storage: &Arc, runtime: &BackupRuntime, account_id: &str, region: &str, limits: &LimitsConfig, body: &Value, -) -> Result<(String, Value), DynamoDbError> { +) -> Result<(String, Value), DynamoDbError> +where + S: BackupCatalog + + LegacyBackupReader + + TableEngine + + DataEngine + + RestoreEngine + + ?Sized + + Send + + Sync + + 'static, +{ // 1. The overrides, parsed and shape-checked: no storage I/O, so a // malformed request is refused before a missing backup can take // precedence over the request's own error. @@ -1585,10 +1611,11 @@ async fn restore_table_from_backup Result<(), Dyna /// store other than the configured one is a `ValidationException` naming no /// root, and a row of the configured store while that store is unhealthy or /// its first check pending is a `ValidationException` naming the store's -/// kind ([`BackupRuntime::require_healthy_store`]). An AVAILABLE row of a -/// healthy configured store is a backup in a store, whose restore this -/// release does not run. -async fn check_catalog_row_for_restore( +/// kind ([`BackupRuntime::require_healthy_store`]). An AVAILABLE row of the +/// healthy configured store passes, and that store is returned: the restore +/// reads the backup from it ([`restore::request`]). +async fn check_catalog_row_for_restore<'r, S: BackupCatalog + ?Sized>( storage: &S, - runtime: &BackupRuntime, + runtime: &'r BackupRuntime, arn: &BackupArn, row: &BackupRow, -) -> Result<(), DynamoDbError> { +) -> Result<&'r Arc, DynamoDbError> { let arn_text = arn.to_arn_string(); if let Some(table) = storage .backup_in_use(&BackupKey::from_arn(arn)) @@ -1687,10 +1727,7 @@ async fn check_catalog_row_for_restore( if !runtime.is_configured_store(&row.store) { return Err(backup_in_another_store(&arn_text)); } - runtime.require_healthy_store()?; - Err(DynamoDbError::ValidationException(format!( - "Backup {arn_text} is in a backup store; this release restores in-catalog backups only" - ))) + runtime.require_healthy_store() } /// Step 5 of a restore, the target name under the account, read through @@ -1735,7 +1772,9 @@ async fn check_restore_target_name( } } -/// The RestoreTableFromBackup response body for the backend's description. +/// The RestoreTableFromBackup response body for the backend's description, +/// on both paths (`source` is the legacy reader's source or the manifest's +/// three members). /// /// The service returns the target CREATING with `RestoreSummary` /// { `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` = the backup's @@ -1746,7 +1785,7 @@ async fn check_restore_target_name( /// ground-truth runs of 2026-08-24). fn restore_response( mut desc: TableDescription, - source: &BackupRestoreSource, + source: &RestoreSource, on_demand: bool, ) -> Result { desc.table_status = TableStatus::Creating; @@ -1769,9 +1808,9 @@ fn restore_response( } // The readiness invariant is applied on every path that hands a - // description to a client. A restored table has no vector indexes (the - // restore refuses them), so this holds trivially, but the invariant claims - // to cover every such path. + // description to a client: a legacy restore's target has no vector + // indexes (the reader refuses them); a store-backed restore's may, and + // the backend reports them as its CreateTable does. desc.validate_vector_index_readiness()?; // Same rules as every other path that emits a table description; the // target is CREATING, so the summary set above is kept. @@ -1868,6 +1907,7 @@ pub mod maintenance; mod overrides; mod pitr; pub mod reconciler; +mod restore; mod runtime; pub mod store_identity; #[cfg(test)] diff --git a/crates/engine/src/backup/job.rs b/crates/engine/src/backup/job.rs index c0084d5a7..90cdd7e5a 100644 --- a/crates/engine/src/backup/job.rs +++ b/crates/engine/src/backup/job.rs @@ -62,6 +62,8 @@ //! //! Kill points ([`kill_point`]) pause the job for the binary-level crash //! tests under the `backup-test-hooks` feature and are no-ops without it. +//! The restore job ([`super::restore`]) shares this module's environment +//! ([`JobEnv`]), its kill points, its heartbeat and its status writes. pub(crate) mod failure; pub(crate) mod heartbeat; @@ -91,19 +93,24 @@ use extenddb_storage::{ use tokio::sync::oneshot; use tokio::time::Instant; -use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings}; +use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings, Spool}; use super::{create_backup_storage_err_to_dynamo, storage_err_to_dynamo}; use failure::{FailurePath, Written}; -use heartbeat::HeartbeatHandle; +use heartbeat::{HeartbeatHandle, HeartbeatMetric}; use writer::{DataFiles, Pipeline, Stop}; +/// The reason a restore record carries when the maintenance claim was held +/// as the job started (the design's 5.3 step 4; the backup job checks the +/// claim before its row exists and writes no such row). +pub(crate) const MAINTENANCE: &str = "maintenance"; + /// The reason a row carries when the server's shutdown stopped the job. pub(crate) const SHUTDOWN: &str = "shutdown"; /// What the runtime hands a job: the store as the runtime holds it (its /// health and identity, read before every put), its identity as the row -/// records it and its manifest names, the settings, the collector, the -/// shutdown token and the source names. +/// records it and its manifest names, the restore spool, the settings, the +/// collector, the shutdown token and the source names. #[derive(Clone)] pub(crate) struct JobEnv { pub configured: Arc, @@ -111,6 +118,9 @@ pub(crate) struct JobEnv { pub identity: StoreIdentity, pub bucket: String, pub key_prefix: String, + /// The restore spool of an S3 store; `None` for a filesystem store, + /// whose files a restore reads in place. + pub spool: Option, pub settings: JobSettings, pub metrics: Option>, pub shutdown: CancellationToken, @@ -137,6 +147,7 @@ impl JobEnv { identity, bucket: store.bucket.clone(), key_prefix: store.key_prefix.clone(), + spool: store.spool.clone(), settings: runtime.job_settings(), metrics: runtime.metrics().cloned(), shutdown: runtime.shutdown(), @@ -363,7 +374,7 @@ async fn fail_without_store_access(claim: &dyn JobClaim, arn: &BackupArn, reason completed: true, ..StatusFields::default() }; - match failure::write_status(claim, JobStatus::Failed, &fields).await { + match failure::write_status(claim, JobStatus::Creating, JobStatus::Failed, &fields).await { Written::Matched => {} Written::NotMatched(why) | Written::Error(why) => tracing::error!( "backup {}: the row could not be marked FAILED ({reason}): {why}", @@ -394,8 +405,13 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { let heartbeat = HeartbeatHandle::start( Arc::clone(&claim), env.settings.heartbeat_interval, - env.metrics.clone(), - created_at, + match env.metrics.clone() { + Some(metrics) => HeartbeatMetric::BackupSnapshotAge { + metrics, + created_at, + }, + None => HeartbeatMetric::None, + }, claimed_at, ); tracing::info!( @@ -560,7 +576,8 @@ async fn write_documents( source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), completed: true, }; - match failure::write_status(&**claim, JobStatus::Available, &fields).await { + match failure::write_status(&**claim, JobStatus::Creating, JobStatus::Available, &fields).await + { Written::Matched => Ok(bytes_uploaded), Written::NotMatched(why) | Written::Error(why) => Err(Stop::ClaimLost(format!( "the AVAILABLE write did not settle: {why}" @@ -597,11 +614,17 @@ pub(crate) async fn put_document(p: &Pipeline<'_>, name: &str, bytes: &[u8]) -> } } -/// The points at which a job pauses under `backup-test-hooks` when -/// `EXTENDDB_BACKUP_PAUSE` names one, until the file -/// `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists (or for good, for a test -/// that kills the server there). Each is logged as `backup job reached kill -/// point `. +/// The points at which a backup or restore job pauses under +/// `backup-test-hooks` when `EXTENDDB_BACKUP_PAUSE` names one, until the +/// file `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists (or for good, for a +/// test that kills the server there). Each is logged as `job reached kill +/// point ` (the backend's own points log `restore job reached kill +/// point `; the harness matches `reached kill point `). This +/// enum is the engine's list: the points inside a backend's own writes are +/// paused by the backend through +/// `extenddb_storage::backup_catalog::restore_pause_point` under +/// `extenddb-storage/test-hooks`, and each backend's own point is named +/// beside `restore_pause_point`, once. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum KillPoint { /// After the put of the first data file returned, before the next file @@ -611,6 +634,22 @@ pub enum KillPoint { BeforeManifest, /// After the manifest's put returned, before the AVAILABLE write. AfterManifest, + /// Restore: after every data file downloaded and verified, before the + /// first `bulk_load`. + RestoreAfterDownload, + /// Restore: after the first file's `bulk_load` returned, before the + /// next file. + RestoreAfterFileCommit, + /// Restore: after a `finish_restore` that left the ACTIVE flip to the + /// backend's index backfill returned, before the job ends (MongoDB). + RestoreMongodbBeforeWorkerFlip, + /// Restore: after the FAILED write matched, before + /// `remove_restore_target`. + RestoreBetweenFailedAndRemoval, + /// Restore: before the DONE write, with the job's heartbeat stopped + /// first, so a reconciler takes the record over while the job pauses + /// and the job's write then matches no record. + RestoreFencedOut, } impl KillPoint { @@ -621,10 +660,30 @@ impl KillPoint { Self::AfterDataFile => "after_data_file", Self::BeforeManifest => "before_manifest", Self::AfterManifest => "after_manifest", + Self::RestoreAfterDownload => "restore_after_download", + Self::RestoreAfterFileCommit => "restore_after_file_commit", + Self::RestoreMongodbBeforeWorkerFlip => "restore_mongodb_before_worker_flip", + Self::RestoreBetweenFailedAndRemoval => "restore_between_failed_and_removal", + Self::RestoreFencedOut => "restore_fenced_out", } } } +/// Whether the environment names `point` as the pause point (so a caller +/// can do what the pause needs first, as the fenced-out point stops the +/// heartbeat); `false` without `backup-test-hooks`. +pub(crate) fn kill_point_named(point: KillPoint) -> bool { + #[cfg(feature = "backup-test-hooks")] + { + std::env::var("EXTENDDB_BACKUP_PAUSE").ok().as_deref() == Some(point.name()) + } + #[cfg(not(feature = "backup-test-hooks"))] + { + let _ = point; + false + } +} + /// Pause at `point` if the environment names it (type doc); a ready future /// without `backup-test-hooks`. #[cfg_attr(not(feature = "backup-test-hooks"), allow(clippy::unused_async))] @@ -636,7 +695,7 @@ pub(crate) async fn kill_point(point: KillPoint, shutdown: &CancellationToken) { } let release = std::env::var("EXTENDDB_BACKUP_PAUSE_RELEASE").ok(); tracing::warn!( - "backup job reached kill point {}; paused until {} exists", + "job reached kill point {}; paused until {} exists", point.name(), release .as_deref() @@ -646,12 +705,12 @@ pub(crate) async fn kill_point(point: KillPoint, shutdown: &CancellationToken) { if let Some(release) = &release && std::path::Path::new(release).exists() { - tracing::warn!("backup job released from kill point {}", point.name()); + tracing::warn!("job released from kill point {}", point.name()); return; } if shutdown.is_cancelled() { tracing::warn!( - "backup job at kill point {} released by the shutdown", + "job at kill point {} released by the shutdown", point.name() ); return; diff --git a/crates/engine/src/backup/job/failure.rs b/crates/engine/src/backup/job/failure.rs index 398ab7880..c0a2eba2d 100644 --- a/crates/engine/src/backup/job/failure.rs +++ b/crates/engine/src/backup/job/failure.rs @@ -5,15 +5,17 @@ //! path. //! //! Every status write goes through [`write_status`]: the backend writes -//! conditioned on the claim's generation and on the row being CREATING, and -//! this module gives the three outcomes one meaning. `Matched` is the -//! write. `NotMatched` means the row is no longer this claim's (gone, -//! another generation, or not CREATING) and the caller makes no further -//! store or table access of any kind. `Unknown` (the connection failed -//! after the statement was sent) is followed by a re-read of the row, not -//! treated as a failure: a row that shows the write landed is `Matched`; a -//! row still CREATING under this generation did not take the write, which -//! is sent once more, at most; anything else is `NotMatched`. Each unknown +//! conditioned on the claim's generation and on the record being in the +//! status the job holds it in (`from`: CREATING for a backup row, RUNNING +//! for a restore record), and this module gives the three outcomes one +//! meaning. `Matched` is the write. `NotMatched` means the record is no +//! longer this claim's (gone, another generation, or not in `from`) and the +//! caller makes no further store or table access of any kind. `Unknown` +//! (the connection failed after the statement was sent) is followed by a +//! re-read of the record, not treated as a failure: a record that shows the +//! write landed is `Matched`; one still in `from` under this generation did +//! not take the write, which is sent once more, at most; anything else is +//! `NotMatched`. Each unknown //! outcome has its re-read, the second write's included, so a second //! unknown outcome whose write landed is `Matched` (the row is in the //! target status under this generation) and one whose write was lost is @@ -28,11 +30,13 @@ //! job stops, the session's end frees the advisory lock, and the row is the //! reconciler's (AVAILABLE from a complete manifest, FAILED otherwise). //! -//! The failure path ([`fail_row`]) is the FAILED write followed, only when -//! it matched, by `delete_prefix` of the backup's prefix; the holder of the -//! claim owns the prefix, so a job whose FAILED write did not match never -//! deletes. A deletion that fails leaves the prefix to the reconciler, -//! which deletes a FAILED row's prefix once its heartbeat is stale. +//! The backup job's failure path ([`fail_row`]) is the FAILED write +//! followed, only when it matched, by `delete_prefix` of the backup's +//! prefix; the holder of the claim owns the prefix, so a job whose FAILED +//! write did not match never deletes. A deletion that fails leaves the +//! prefix to the reconciler, which deletes a FAILED row's prefix once its +//! heartbeat is stale. The restore job's failure path, FAILED then the +//! target's removal under the same gate, is `super::super::restore::fail_restore`. use extenddb_backup_store::BackupStore; use extenddb_core::metrics::MetricsCollector; @@ -51,23 +55,24 @@ pub(crate) enum Written { Error(String), } -/// Move the row from CREATING to `to` through `claim` (module doc). +/// Move the record from `from` to `to` through `claim` (module doc). pub(crate) async fn write_status( claim: &dyn JobClaim, + from: JobStatus, to: JobStatus, fields: &StatusFields, ) -> Written { for attempt in 0..2 { - let outcome = match claim.set_status(JobStatus::Creating, to, fields).await { + let outcome = match claim.set_status(from, to, fields).await { Ok(outcome) => outcome, Err(e) => return Written::Error(format!("the {to} write failed: {e}")), }; match outcome { WriteOutcome::Matched => return Written::Matched, WriteOutcome::NotMatched => { - return Written::NotMatched( - "the row is no longer CREATING under this claim's generation".to_owned(), - ); + return Written::NotMatched(format!( + "the record is no longer {from} under this claim's generation" + )); } WriteOutcome::Unknown => {} } @@ -89,14 +94,14 @@ pub(crate) async fn write_status( if row.status == to { return Written::Matched; } - if row.status != JobStatus::Creating || attempt == 1 { + if row.status != from || attempt == 1 { return Written::NotMatched(format!( - "the row is {} after a {to} write whose outcome was unknown", + "the record is {} after a {to} write whose outcome was unknown", row.status )); } tracing::warn!( - "the {to} write's outcome was unknown and the row is still CREATING under this \ + "the {to} write's outcome was unknown and the record is still {from} under this \ claim; writing once more" ); } @@ -129,7 +134,7 @@ pub(crate) async fn fail_row( completed: true, ..StatusFields::default() }; - match write_status(claim, JobStatus::Failed, &fields).await { + match write_status(claim, JobStatus::Creating, JobStatus::Failed, &fields).await { Written::Matched => {} Written::NotMatched(why) | Written::Error(why) => return FailurePath::NoStoreAccess(why), } diff --git a/crates/engine/src/backup/job/heartbeat.rs b/crates/engine/src/backup/job/heartbeat.rs index 6395f707d..41bdb72d5 100644 --- a/crates/engine/src/backup/job/heartbeat.rs +++ b/crates/engine/src/backup/job/heartbeat.rs @@ -14,13 +14,24 @@ //! old (one for the timer to fire, one for the beat to return, the bound //! the timer itself enforces), or a beat is in flight (the last paragraph), //! and no loss has been raised. The claim's -//! insert stands for the first confirmed beat: the check measures from the -//! instant the job took before `claim_backup`, so the maintenance check -//! that runs between the insert and the handle's start (one catalog query -//! with no bound of its own) counts against the two periods, and a job -//! whose first put comes within two periods of the insert needs no beat of -//! its own; `stale_job_after_secs` is at least three periods, so a row that -//! passes the check is never one a reconciler may take over. +//! insert stands for the first confirmed beat, and the two jobs hand its +//! instant in by their own convention: the backup job takes it before +//! `claim_backup` (the row's `heartbeat_at` is stamped inside the insert, so +//! the instant is at or before the beat and the maintenance check that runs +//! between the insert and the handle's start, one catalog query with no +//! bound of its own, counts against the two periods); the restore job takes +//! it after its first beat returned confirmed, a beat it sends itself right +//! after `claim_restore` returned (the record's `heartbeat_at` as the +//! registration stamped it can be old: on PostgreSQL `now()` is the +//! transaction's start, so a registration stalled on a lock commits a stale +//! beat; the first beat refreshes it on the catalog's clock), and after +//! `claim_restore_dead` returned for a resume (the takeover stamped +//! `heartbeat_at` inside that write). Either way the instant is at or after +//! the beat it stands for, by a write's own duration, and a slow +//! registration does not end the job at its first check. A job +//! whose first write comes within two periods of its instant needs no beat +//! of its own; `stale_job_after_secs` is at least three periods, so a row +//! that passes the check is never one a reconciler may take over. //! //! A process stopped and resumed (SIGSTOP, a paused VM) meets the same //! rule. On PostgreSQL its claim session keeps the advisory lock while it @@ -33,18 +44,21 @@ //! outlives its one-period bound, or when a put comes before the next beat //! and finds the last confirmed beat more than two periods old. In that //! case its claim closes, the lock is freed and the reconciler resolves the -//! row (AVAILABLE from a complete manifest, FAILED otherwise). On MongoDB -//! and Cassandra the row may have been taken over during the stop: a put in +//! row (AVAILABLE from a complete manifest, FAILED otherwise) or resumes a +//! store-backed restore from its last committed batch. On MongoDB and +//! Cassandra the row may have been taken over during the stop: a put in //! flight at the stop may still land (the reconciler's FAILED-prefix retry //! deletes it), and the job's next beat and status write match no row. //! //! The one-period bound is the backend's to place //! ([`JobClaim::heartbeat_within`]): on SQLite the beat is an `UPDATE` //! under the engine write lock, and the bound starts once the beat holds -//! that lock, so a writer holding it past a period (a GSI backfill) delays -//! the beat and does not end the job. The re-check before the manifest -//! ([`HeartbeatHandle::recheck`]) is the same call, and a confirmation it -//! gets counts as a beat for the pre-put check. +//! that lock, so a writer holding it past a period (a restore's write +//! batches, item writes, a GSI backfill, the TTL sweep, a table's drop) +//! delays the beat and does not end the job; the SQLite restore engine's +//! module doc gives the batch times the bench record measured. The re-check +//! before the manifest ([`HeartbeatHandle::recheck`]) is the same call, and +//! a confirmation it gets counts as a beat for the pre-put check. //! //! The pre-put check has a third clause for the same case: a beat that is //! in flight and has not failed confirms the claim, however old the last @@ -143,17 +157,27 @@ fn beat_failure(outcome: Option>, bound: Duratio } } +/// What a confirmed beat records, if anything: a backup job records the +/// snapshot's age, a restore job records nothing per beat. +pub(crate) enum HeartbeatMetric { + /// `BackupSnapshotAge`, seconds since `created_at`, into `metrics`. + BackupSnapshotAge { + metrics: Arc, + created_at: f64, + }, + /// No per-beat metric. + None, +} + impl HeartbeatHandle { /// Start the timer on `claim`, beating every `period` from one period - /// after now; `claimed_at`, the instant of the claim's insert, stands - /// for the first confirmed beat. With `metrics`, each confirmed beat - /// records the snapshot's age (`BackupSnapshotAge`), seconds since - /// `created_at`. + /// after now; `claimed_at`, the instant of the claim's insert as the + /// job's convention takes it (module doc), stands for the first + /// confirmed beat. Each confirmed beat records `metric`. pub(crate) fn start( claim: Arc, period: Duration, - metrics: Option>, - created_at: f64, + metric: HeartbeatMetric, claimed_at: Instant, ) -> Self { let (tx, last_confirmed) = watch::channel(claimed_at); @@ -180,13 +204,21 @@ impl HeartbeatHandle { }; if let Some(Ok(Heartbeat::Confirmed)) = outcome { let _ = confirmed.send(Instant::now()); - if let Some(metrics) = &metrics { - metrics.record_backup_snapshot_age(seconds_since(created_at)); + if let HeartbeatMetric::BackupSnapshotAge { + metrics, + created_at, + } = &metric + { + metrics.record_backup_snapshot_age(seconds_since(*created_at)); } continue; } - *reason.lock().unwrap_or_else(PoisonError::into_inner) = - Some(beat_failure(outcome, period)); + let why = beat_failure(outcome, period); + *reason.lock().unwrap_or_else(PoisonError::into_inner) = Some(why.clone()); + tracing::warn!( + "the job's heartbeat raised the loss of its claim; the job stops at its \ + next check: {why}" + ); lost.cancel(); return; } @@ -219,6 +251,19 @@ impl HeartbeatHandle { .unwrap_or_else(|| "the claim is lost".to_owned()) } + /// Why the pre-put check failed, for the job's stop line: the timer's + /// own reason when it raised the loss ([`Self::reason`]), else the + /// second clause, that no beat confirmed the claim within two periods, + /// naming `step`, what the job was about to do ("to load X", "for the + /// DONE write"). + pub(crate) fn why_not_confirmed(&self, step: &str) -> String { + if self.lost.is_cancelled() { + self.reason() + } else { + format!("no heartbeat confirmed the claim recently enough {step}") + } + } + /// The pre-put check (module doc): no loss raised, and either the last /// confirmed beat at most two periods old or a beat in flight (parked on /// SQLite's write lock behind a long writer; within one period of its @@ -226,9 +271,20 @@ impl HeartbeatHandle { /// timer task that stopped running without raising the loss; the third /// never outlives its beat. pub(crate) fn confirms_the_claim(&self) -> bool { - !self.lost.is_cancelled() - && (self.last_confirmed.borrow().elapsed() <= 2 * self.period - || in_flight_confirms(&self.in_flight, self.parks, self.period)) + self.claim_check().confirms_the_claim() + } + + /// The pre-put check as a value a blocking thread can hold (the + /// restore's decoder checks it at every chunk boundary): the same three + /// clauses read from the same watch, token and flag. + pub(crate) fn claim_check(&self) -> ClaimCheck { + ClaimCheck { + last_confirmed: self.last_confirmed.clone(), + lost: self.lost.clone(), + in_flight: Arc::clone(&self.in_flight), + parks: self.parks, + period: self.period, + } } /// One beat outside the timer, under the period's bound (the claim @@ -248,11 +304,39 @@ impl HeartbeatHandle { } /// End the timer without raising the loss: a timer task that stopped - /// running, for the test of the pre-put check's second clause. - #[cfg(test)] + /// running, for the test of the pre-put check's second clause and for + /// the restore's fenced-out kill point, which lets a reconciler take the + /// record over while the job pauses. pub(crate) fn stop_timer(&self) { self.timer.abort(); } + + /// A handle that ends the timer as [`Self::stop_timer`] does, for a test + /// that holds no reference to the job's handle. + #[cfg(test)] + pub(crate) fn timer_abort_handle(&self) -> tokio::task::AbortHandle { + self.timer.abort_handle() + } +} + +/// [`HeartbeatHandle::confirms_the_claim`] detached from the handle. +#[derive(Clone)] +pub(crate) struct ClaimCheck { + last_confirmed: watch::Receiver, + lost: CancellationToken, + in_flight: Arc>>, + parks: bool, + period: Duration, +} + +impl ClaimCheck { + /// No loss raised, and either the last confirmed beat at most two + /// periods old or a beat in flight within its bound (module doc). + pub(crate) fn confirms_the_claim(&self) -> bool { + !self.lost.is_cancelled() + && (self.last_confirmed.borrow().elapsed() <= 2 * self.period + || in_flight_confirms(&self.in_flight, self.parks, self.period)) + } } impl Drop for HeartbeatHandle { diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs index dd95800da..5536c36fb 100644 --- a/crates/engine/src/backup/maintenance.rs +++ b/crates/engine/src/backup/maintenance.rs @@ -2,7 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 //! The holder's side of the catalog-level maintenance claim, and the steps a -//! maintenance command (`extenddb backup purge`) runs under it. +//! maintenance command (`extenddb backup purge`, `extenddb backup +//! remove-target`) runs under it. //! //! [`HeldClaim`] is the handle the command keeps from the take to the //! release, with the heartbeat running on a timer for the handle's whole @@ -25,6 +26,23 @@ //! operator's Ctrl-C ends the wait at once and the deletion loop before its //! next row, never inside one (the row's own steps are the idempotent //! sequence a second deleter finishes). +//! +//! [`remove_target`] is the whole of +//! `extenddb backup remove-target --account
    `: the claim +//! taken and held for its duration, the table's +//! restore record read, the refusals, the removal through +//! `remove_restore_target`, the release. It is the one command that removes +//! a restore target DeleteTable refuses (a CREATING table): a FAILED +//! record's target, whatever the reason (the reconciler leaves a +//! `target_not_empty` one for this command; on MongoDB and Cassandra a +//! crash before the first progress record, the whole first data file on +//! MongoDB and the first 10,000 items or 32 MiB on Cassandra, ends in that +//! state), and a RUNNING record's target whose owner is dead (the backend's +//! `list_dead_restore_jobs` names it under the owner window: no heartbeat +//! within the window on the catalog's clock); a legacy record's target the +//! same way. It refuses a RUNNING record whose owner is live, a DONE record +//! (the table is ACTIVE, or on MongoDB awaiting its index backfill) and a +//! table no restore record names, the latter two as "not a restore target". use std::sync::{Arc, Mutex, PoisonError}; use std::time::Duration; @@ -32,9 +50,10 @@ use std::time::Duration; use extenddb_backup_store::BackupStore; use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE}; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, StaleClock, StoreIdentity, - is_backup_not_found, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreJobRow, RestoreKind, + StaleClock, StoreIdentity, is_backup_not_found, }; +use extenddb_storage::error::StorageError; use extenddb_storage::{BackupCatalog, MaintenanceClaim}; use futures::TryStreamExt; use tokio::sync::watch; @@ -105,7 +124,7 @@ impl HeldClaim { /// /// The backend's, when the release could not be written, or the bound; /// the claim then expires by the stale rule. - pub async fn release(mut self) -> Result<(), extenddb_storage::error::StorageError> { + pub async fn release(mut self) -> Result<(), StorageError> { self.timer.abort(); // The abort lands at the task's next await; a heartbeat already sent // finishes first, so the release never overlaps it on the claim's @@ -475,6 +494,183 @@ pub async fn purge_rows( outcome } +/// What `extenddb backup remove-target` is asked to do. +#[derive(Debug, Clone)] +pub struct RemoveTarget { + /// The account the table belongs to. + pub account_id: String, + /// The table's name. + pub table_name: String, + /// The maintenance claim's holder label (the command and its host). + pub holder: String, + /// `stale_job_after_secs`: the claim's stale window. + pub stale_after: Duration, + /// The window `list_dead_restore_jobs` is asked with: a RUNNING record + /// the backend does not list as dead under it (a heartbeat within the + /// window on the catalog's clock) has a live owner and is refused; zero + /// where every owner is known dead (SQLite under the serve lock, where + /// a zero window lists every RUNNING record). + pub owner_window: Duration, + /// The claim's heartbeat period (the jobs', `JOB_HEARTBEAT_INTERVAL`). + pub heartbeat_interval: Duration, +} + +/// What [`remove_target`] removed: the record as it was read, for the +/// command's line. +#[derive(Debug, Clone, PartialEq)] +pub struct RemovedTarget { + /// The target's table id. + pub table_id: String, + /// The backup the restore read. + pub backup_arn: String, + /// The record's kind. + pub kind: RestoreKind, + /// The record's status when it was read: FAILED, or RUNNING with a + /// dead owner. + pub status: JobStatus, + /// The FAILED record's reason. + pub failure_reason: Option, +} + +/// `extenddb backup remove-target` (module doc): under the maintenance +/// claim, taken here and released before the return, remove the CREATING +/// target of the restore record that names `request`'s table and the record +/// with it, through `remove_restore_target`. +/// +/// # Errors +/// +/// The text for the operator: the claim held by a live holder; a table no +/// record names, a DONE record, or a target no longer a CREATING restore +/// target with the record's id ("not a restore target"); a CREATING restore +/// target whose owner's lock is still held (the record unchanged after the +/// backend refused the removal: wait for the owner or stop it); a RUNNING +/// record the backend does not list as dead under the owner window; the +/// claim lost before the removal; a catalog call failing. +pub async fn remove_target( + storage: &S, + request: &RemoveTarget, +) -> Result { + let claim = storage + .maintenance_claim(&request.holder, request.stale_after) + .await + .map_err(|e| match e { + StorageError::AlreadyExists(m) => format!( + "{m}; one maintenance command runs at a time, and a claim whose holder died \ + expires after stale_job_after_secs ({} s)", + request.stale_after.as_secs() + ), + other => format!("cannot take the maintenance claim: {other}"), + })?; + let claim = HeldClaim::start(claim, request.heartbeat_interval); + let outcome = remove_target_under(storage, request, &claim).await; + if let Err(e) = claim.release().await { + tracing::warn!("the maintenance claim could not be released; it expires on its own: {e}"); + } + outcome +} + +/// The steps of [`remove_target`] between the take and the release. +async fn remove_target_under( + storage: &S, + request: &RemoveTarget, + claim: &HeldClaim, +) -> Result { + let what = format!("{} (account {})", request.table_name, request.account_id); + let records: Vec = storage + .list_restore_jobs(None) + .await + .map_err(|e| format!("cannot read the restore job records: {e}"))?; + let record = records + .into_iter() + .find(|r| { + r.target.account_id == request.account_id && r.target.table_name == request.table_name + }) + .ok_or_else(|| format!("{what} is not a restore target: no restore record names it"))?; + match record.status { + JobStatus::Done => { + return Err(format!( + "{what} is not a restore target: its restore from {} is DONE (the table is \ + ACTIVE, or awaits its index backfill)", + record.source_backup_arn + )); + } + JobStatus::Running => { + // The owner is dead only if the backend's own listing says so: + // `list_dead_restore_jobs` compares the heartbeat with the + // catalog's clock, the one the owner stamps, so a skew of this + // host's clock can neither refuse a dead owner nor remove under + // a live one. A zero window (SQLite under the serve lock) lists + // every RUNNING record, as it does for the purge's wait. + let dead = storage + .list_dead_restore_jobs(request.owner_window) + .await + .map_err(|e| format!("cannot list the dead restore jobs: {e}"))? + .iter() + .any(|r| r.target.table_id == record.target.table_id); + if !dead { + return Err(format!( + "the restore into {what} from {} is RUNNING with a live owner (a heartbeat \ + within stale_job_after_secs); stop it or wait for it", + record.source_backup_arn + )); + } + } + JobStatus::Failed | JobStatus::Creating | JobStatus::Available => {} + } + if let Some(reason) = claim.lost() { + return Err(format!("stopped before the removal: {reason}")); + } + let removed = storage + .remove_restore_target_exclusive(&record.target) + .await + .map_err(|e| format!("cannot remove the restore target {what}: {e}"))?; + if !removed { + // The backend refused without saying which of its two conditions + // failed. The record tells them apart: the target and its record go + // together, and the ACTIVE flip writes DONE, so a record unchanged + // since the listing names a table that is still its CREATING restore + // target, and the refusal was the owner's lock (PostgreSQL's + // per-session advisory lock, held by a frozen owner's session). + let unchanged = storage + .read_restore_job(&record.target) + .await + .map_err(|e| format!("cannot re-read the restore record of {what}: {e}"))? + .is_some_and(|now| { + now.status == record.status && now.owner_generation == record.owner_generation + }); + if unchanged { + return Err(format!( + "the restore into {what} from {} is a CREATING restore target (table id {}) \ + whose owner's lock is still held: its owner's session is alive though its \ + heartbeat is stale; wait for the owner to stop, or stop it, and run the \ + command again", + record.source_backup_arn, record.target.table_id + )); + } + return Err(format!( + "{what} is not a restore target: it is no longer a CREATING restore target with \ + table id {}", + record.target.table_id + )); + } + tracing::info!( + target: "extenddb::audit", + action = "backup remove-target", + account_id = %record.target.account_id, + table_name = %record.target.table_name, + table_id = %record.target.table_id, + backup_arn = %record.source_backup_arn, + "removed the restore target and its record" + ); + Ok(RemovedTarget { + table_id: record.target.table_id, + backup_arn: record.source_backup_arn, + kind: record.kind, + status: record.status, + failure_reason: record.failure_reason, + }) +} + #[cfg(test)] mod tests { use extenddb_storage::error::StorageError; diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs index 9c1b8faef..05eb212db 100644 --- a/crates/engine/src/backup/reconciler.rs +++ b/crates/engine/src/backup/reconciler.rs @@ -24,47 +24,92 @@ //! //! The duties, in the order a cycle runs them: //! -//! - A RUNNING restore job record whose `heartbeat_at` is older than -//! `stale_job_after` (on the catalog's clock) has a dead owner. One of kind -//! `legacy` is marked FAILED (`fail_restore_job`, conditioned on the -//! `owner_generation` and `heartbeat_at` the listing read, so a heartbeat -//! that lands after the listing keeps the record its owner's) and its -//! target removed through +//! - With a store configured: the store's check runs again +//! ([`super::check_configured_store`]: `validate` while it has not passed, +//! then the identity check and manifest probe) and sets the store's +//! health, so a store whose checks run after the listener is up, or that +//! failed at startup or later, is healthy once a check passes, and one +//! that fails is refused from the rest of this cycle on: every duty below +//! that reads the store's health reads the health this cycle's check left. +//! The check runs under the store's bound +//! ([`super::ConfiguredStore::check_timeout`]: 5 s for a filesystem root, +//! 30 s for a bucket), so a store that has stopped answering costs each +//! cycle that bound and nothing more: the duties run every `interval` plus +//! the bound, the check is recorded as a transport failure, and a +//! filesystem check leaves one blocking thread parked on the mount until +//! it answers, after which no check starts until that call returns (the +//! runtime module's in-flight rule), so a mount that stays stalled parks +//! one thread and not one per cycle. The check is also selected against +//! the shutdown token, so a cycle that is checking when the server stops +//! returns at once with no further duty and the drain finishes. +//! - Over the records `list_dead_restore_jobs` returns: a RUNNING restore +//! job record whose `heartbeat_at` is older than `stale_job_after` (on +//! the catalog's clock) has a dead owner. One of kind `legacy` is marked +//! FAILED (`fail_restore_job`, conditioned on the `owner_generation` and +//! `heartbeat_at` the listing read, so a heartbeat that lands after the +//! listing keeps the record its owner's) and its target removed through //! `remove_restore_target`, never resumed: an in-catalog copy has no data -//! files or progress to resume from. One of kind `manifest` is left for the -//! duty that resumes it, and logged once per record for as long as this -//! process runs: the set of logged records ([`ReconcilerState`]) belongs to -//! the caller of [`run`] (`serve`), so a worker restart does not log a -//! record again. -//! - A FAILED restore job record whose table is still a CREATING restore -//! target, the FAILED-target duty: the FAILED write landed (the job's own, -//! or an earlier cycle's) and the target's removal did not (an error -//! between the two statements, or the process stopped between them). The -//! backend lists it once the record is stale, and the target is removed -//! through `remove_restore_target` with no second FAILED write. +//! files or progress to resume from. One of kind `manifest` is resumed (`super::restore::resume`), +//! in this order: once the server's shutdown has begun (the jobs' token is +//! cancelled) the record is left as it is, with no takeover and no FAILED +//! write, for the next instance's reconciler (a resume spawned on the way +//! out would stop at its first check and leave the record with a fresh +//! heartbeat, a second stale window for the next instance to wait); once +//! this worker's own token has fired with the jobs' token intact (the +//! server stopping its workers while it holds no serving registration) it +//! is left the same way, for the next set's reconciler, since a resume is +//! a job and the workers' stop would not stop it; +//! while the maintenance claim is held it is failed and its target +//! removed like a legacy one ([`OWNER_DEAD`]; a resume is a job, which +//! the claim excludes), except a record with no recorded progress at all +//! (no `file_index`, no progress row) whose target already holds items, +//! which is failed [`super::restore::TARGET_NOT_EMPTY`] through the same +//! unclaimed write and its target left for the operator, as below (the +//! command's wait is satisfied either way: the record is no longer +//! RUNNING); a record of a store other than the +//! configured one (no store configured included) is failed naming +//! `backup sync --move-store`, its target removed by the FAILED-target +//! duty next; while the configured store is unhealthy, or not yet checked, +//! the record is skipped (one line per cycle, naming which); with no free +//! job permit it is skipped to the next cycle; then the backend's +//! `claim_restore_dead` is the test and the takeover (a fresh generation, +//! this host's `spool_host` and a fresh `heartbeat_at`, so a second taker +//! finds the record live at once), and a record it does not hand over has +//! a live owner; the instant the takeover returned is the resumed job's +//! first confirmed beat (the record's `heartbeat_at` was stamped inside +//! the write, so the instant is at or after the beat it stands for by the +//! write's own duration, and the reads that follow count against the +//! heartbeat's window rather than widen it); the takeover after this +//! instance's fifth of one record ([`RESTORE_TAKEOVERS`], counted in +//! [`ReconcilerState`]) fails the record [`TAKEOVERS_EXHAUSTED`] and +//! removes its target, since a record whose every resume ends RUNNING +//! again (a read or the count that fails every time) would otherwise be +//! retaken every stale window; under the claim, a record with no recorded progress at all +//! (no `file_index`, no progress row) whose target already holds more +//! items than the backup's row counts is FAILED +//! [`super::restore::TARGET_NOT_EMPTY`] and its target left for the +//! operator (`extenddb backup remove-target --account
    `), +//! since a catalog restored +//! from a dump can name a restore that finished and was written to since; +//! a target holding no more than the total is a first data file applied +//! in part before its progress was recorded (Cassandra), and the resume +//! re-applies that file with upserts (`exceeds_backup_total`); +//! otherwise the job is spawned through the runtime's tracker and runs +//! from the recorded progress. A FAILED restore job record whose table is +//! still a CREATING restore target, the FAILED-target duty: the FAILED +//! write landed (the job's own, or an earlier cycle's) and the target's +//! removal did not; the backend lists it once the record is stale, and the +//! target is removed through `remove_restore_target` with no second +//! FAILED write, except a record whose reason is `target_not_empty`, +//! whose target is the operator's and is logged with the command. //! - An in-catalog CREATING backup row whose create is dead (reported by the //! backend's `abandoned_creating_backups`) is removed through the reader's //! `delete`, which takes its items with it, unless the backend removed it //! in the call that reported it (`AbandonedLegacyBackup::removed`), when //! the reconciler only logs it. -//! - With a store configured: the store's check runs again -//! ([`super::check_configured_store`]: `validate` while it has not passed, -//! then the identity check and manifest probe) and sets the store's -//! health, so a store whose checks run after the listener is up, or that -//! failed at startup or later, is healthy once a check passes, and one -//! that fails is refused from the next cycle on. The check runs under the -//! store's bound ([`super::ConfiguredStore::check_timeout`]: 5 s for a -//! filesystem root, 30 s for a bucket), so a store that has stopped -//! answering costs each cycle that bound and nothing more: the duties -//! above run every `interval` plus the bound, the check is recorded as a -//! transport failure, and a filesystem check leaves one blocking thread -//! parked on the mount until it answers, after which no check starts -//! until that call returns (the runtime module's in-flight rule), so a -//! mount that stays stalled parks one thread and not one per cycle. The -//! check is also selected against the shutdown token, so a cycle that is -//! checking when the server stops returns at once and the drain finishes. -//! While the store is unhealthy the reconciler acts on no row of the -//! store. While it is healthy, on the rows of the configured store: +//! - With a store configured: while the store is unhealthy the reconciler +//! acts on no row of the store. While it is healthy, on the rows of the +//! configured store: //! - Each DELETING row (a DeleteBackup whose deletion did not finish) is //! finished (`finish_backup_delete` in the engine's backup module): the //! manifest, then the prefix, then the row to DELETED. @@ -101,7 +146,33 @@ //! (the completed clock) is removed. Every age is the database's //! comparison; the engine holds no clock. //! - Not while the maintenance claim is held, on every backend: the -//! backend's `sweep_restore_registrations` (a no-op but on Cassandra). +//! backend's `sweep_restore_registrations` (a no-op but on Cassandra); +//! the progress-row sweep (`sweep_restore_progress` with the table ids of +//! the RUNNING restore targets: PostgreSQL's `restore_progress` rows of +//! any other table go, the other backends keep progress on the record and +//! have nothing to sweep); and, with a spool configured, the spool sweep +//! (`super::restore::spool::sweep_entries`: entries whose table id is +//! not a RUNNING restore with this host's `spool_host`). Both sweeps read +//! their candidates before the records: the progress-row sweep's +//! candidates are the backend's, the spool's are listed +//! (`super::restore::spool::list_entries`) before `list_restore_jobs`, +//! so a restore whose registration commits between the two reads is +//! never a candidate of either. A backend that cannot list the records +//! yet logs once per cycle and the two sweeps are skipped. +//! +//! The worker runs in the server's restartable worker set: while the server +//! holds no serving registration (PostgreSQL, MongoDB and Cassandra) it is +//! stopped with the other workers and a new one started once the server +//! registers again, as `extenddb_server`'s `registration` module doc says; +//! a cycle that is running at the stop finishes or is aborted, and spawns +//! no resume after its token fired (above). Each start builds a fresh +//! [`ReconcilerState`], and its first cycle runs at once; no duty depends +//! on what an earlier set's cycles did. The backup and restore jobs, +//! resumes included, run on the jobs' token, a child of the server's +//! shutdown token, so the workers' stop does not stop them: a job that is +//! running when the registration is lost goes on writing the catalog and +//! the store until it ends or the server stops, beside a migration that +//! may already have taken the catalog (GitHub issue #755 is the gate). //! //! Every duty acts only on rows of the configured store (kind, root and //! id), the legacy duty excepted. A cycle that fails logs and waits for the @@ -126,15 +197,20 @@ use extenddb_core::backup_format::{ }; use extenddb_core::metrics::{MetricsCollector, QuerySource}; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, JobStatus, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, + BackupRow, BackupStatus, JobStatus, RestoreJobRow, RestoreKind, StaleClock, StatusFields, + StoreIdentity, }; use extenddb_storage::error::StorageError; -use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader, sleep_or_shutdown}; +use extenddb_storage::{ + BackupCatalog, CancellationToken, JobPermit, LegacyBackupReader, RestoreEngine, + sleep_or_shutdown, +}; use futures::TryStreamExt; -use super::BackupRuntime; +use super::job::JobEnv; use super::job::failure::{Written, write_status}; +use super::restore::{self, OTHER_STORE, TARGET_NOT_EMPTY}; +use super::{BackupRuntime, ConfiguredStore}; /// The two durations a reconciler reads from `[backup]`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -153,6 +229,20 @@ pub const OWNER_DEAD: &str = "owner dead: no heartbeat within stale_job_after_se pub const OWNER_DEAD_NO_MANIFEST: &str = "owner dead: no heartbeat within stale_job_after_secs; no complete manifest"; +/// How many times one instance resumes a dead store-backed restore record +/// before it fails the record instead: a resume that keeps ending with the +/// record RUNNING (a progress read or the final count that fails every time, +/// a load that stops `LeftRunning`) would otherwise be retaken every stale +/// window for as long as the server runs. The count is this instance's, +/// kept in [`ReconcilerState`]; a restart starts it over. +pub const RESTORE_TAKEOVERS: u32 = 5; + +/// The reason written on a RUNNING store-backed restore record that this +/// instance took over [`RESTORE_TAKEOVERS`] times without the restore +/// ending; the target is removed as for [`OWNER_DEAD`]. +pub const TAKEOVERS_EXHAUSTED: &str = + "owner dead: taken over 5 times by one instance without the restore ending"; + /// What one cycle did. #[derive(Debug, Default, Clone, PartialEq, Eq)] pub struct CycleReport { @@ -167,11 +257,32 @@ pub struct CycleReport { /// this cycle wrote the FAILED or found it), left to their current owner, /// by table name. pub targets_left_to_owner: Vec, - /// Manifest restore jobs with a dead owner, left for their resumer. - pub manifest_restores_seen: Vec, - /// Those of `manifest_restores_seen` this cycle logged: the ones no - /// earlier cycle of this process had seen. - pub manifest_restores_logged: Vec, + /// Manifest restore jobs with a dead owner this cycle took over and + /// spawned a resume of, by table name. + pub manifest_restores_resumed: Vec, + /// Manifest restore jobs with a dead owner this cycle marked FAILED and + /// left their target to the FAILED-target duty or the operator, by table + /// name with the reason: another store, or `target_not_empty`. + pub manifest_restores_failed: Vec<(String, String)>, + /// Manifest restore jobs with a dead owner this cycle marked FAILED + /// `OWNER_DEAD` under the maintenance claim and removed the target of, + /// as a legacy record is, by table name. + pub manifest_restores_failed_and_removed: Vec, + /// Manifest restore jobs with a dead owner skipped to the next cycle: + /// no job permit was free, by table name. + pub restores_skipped_no_permit: Vec, + /// Manifest restore jobs with a dead owner skipped because the + /// configured store is unhealthy, by table name. + pub restores_skipped_store_unhealthy: Vec, + /// FAILED `target_not_empty` records whose target is left for the + /// operator, by table name. + pub targets_left_for_operator: Vec, + /// Progress rows of tables that are not RUNNING restore targets removed + /// (PostgreSQL's `restore_progress`). + pub progress_rows_swept: u64, + /// Spool entries of restores not RUNNING with this host's `spool_host` + /// removed, by table id. + pub spool_entries_swept: Vec, /// In-catalog CREATING backup rows removed, by ARN. pub legacy_backups_removed: Vec, /// The configured store's health after this cycle's check; `None` with @@ -203,6 +314,11 @@ impl CycleReport { pub fn acted(&self) -> bool { !(self.legacy_restores_removed.is_empty() && self.failed_targets_removed.is_empty() + && self.manifest_restores_resumed.is_empty() + && self.manifest_restores_failed.is_empty() + && self.manifest_restores_failed_and_removed.is_empty() + && self.progress_rows_swept == 0 + && self.spool_entries_swept.is_empty() && self.legacy_backups_removed.is_empty() && self.deletions_finished.is_empty() && self.creating_resolved.is_empty() @@ -213,22 +329,24 @@ impl CycleReport { } } -/// What a reconciler carries from one cycle to the next: the manifest -/// records it has already logged, so a record it cannot act on is reported -/// once rather than every cycle (the caller of [`run`] owns it and hands the -/// same state to every worker it starts, so the set outlives a worker -/// restart: `serve` restarts its workers each time it takes the serving -/// registration again), the worker's shutdown token, which the -/// store's check is selected against, and, for a maintenance command's own -/// cycle, that its caller holds the maintenance claim. +/// What a reconciler carries from one cycle to the next: the worker's +/// shutdown token, which the store's check is selected against (the caller +/// of [`run`] owns the state and hands the same one to every worker it +/// starts, `serve` restarting its workers each time it takes the serving +/// registration again, and the token is set each cycle), and, for a +/// maintenance command's own cycle, that its caller holds the maintenance +/// claim. #[derive(Debug, Default)] pub struct ReconcilerState { - logged_manifest_restores: HashSet, shutdown: Option, /// `Some(true)` when the caller holds the maintenance claim: the cycle /// takes the held branch and does not read the claim. `None` reads it /// through `check_maintenance_claim`. maintenance_held: Option, + /// How many times this instance took over each store-backed restore + /// record, by target table id, for the bound [`RESTORE_TAKEOVERS`]; an + /// entry is removed when the record is failed by the bound. + restore_takeovers: std::collections::HashMap, } impl ReconcilerState { @@ -269,7 +387,7 @@ pub async fn run( state: Arc>, shutdown: CancellationToken, ) where - S: BackupCatalog + LegacyBackupReader + ?Sized, + S: BackupCatalog + LegacyBackupReader + RestoreEngine + ?Sized + Send + Sync + 'static, { loop { let started = std::time::Instant::now(); @@ -279,7 +397,7 @@ pub async fn run( // this worker's, so it is set here, each cycle. state.shutdown = Some(shutdown.clone()); cycle( - &*storage, + &storage, &runtime, settings.stale_job_after, &backend, @@ -294,12 +412,18 @@ pub async fn run( metrics.record_worker_success(QuerySource::BackupReconciler, cycle_us); if report.acted() { tracing::info!( - "backup reconciler: failed and removed {} dead legacy restore(s), removed \ - the target(s) of {} FAILED restore record(s) and {} abandoned in-catalog \ - backup(s), finished {} deletion(s), resolved {} dead CREATING row(s), \ - deleted {} FAILED prefix(es), swept {} DELETED prefix(es), removed {} \ - tombstone(s), swept the registrations of {} backup(s)", + "backup reconciler: failed and removed {} dead legacy restore(s), resumed \ + {}, failed {} and failed and removed {} (under the maintenance claim) \ + dead store-backed restore(s), removed the target(s) of {} FAILED restore \ + record(s) and {} abandoned in-catalog backup(s), finished {} \ + deletion(s), resolved {} dead CREATING row(s), deleted {} FAILED \ + prefix(es), swept {} DELETED prefix(es), removed {} tombstone(s), swept \ + the registrations of {} backup(s), {} progress row(s) and {} spool \ + entry(ies)", report.legacy_restores_removed.len(), + report.manifest_restores_resumed.len(), + report.manifest_restores_failed.len(), + report.manifest_restores_failed_and_removed.len(), report.failed_targets_removed.len(), report.legacy_backups_removed.len(), report.deletions_finished.len(), @@ -307,7 +431,9 @@ pub async fn run( report.failed_prefixes_deleted.len(), report.deleted_prefixes_swept.len(), report.tombstones_removed.len(), - report.registrations_swept.len() + report.registrations_swept.len(), + report.progress_rows_swept, + report.spool_entries_swept.len() ); } } @@ -332,14 +458,14 @@ pub async fn run( /// A catalog read that fails ends the cycle; a failed write on one row is /// logged and the cycle goes on to the next row. pub async fn cycle( - storage: &S, + storage: &Arc, runtime: &BackupRuntime, stale_job_after: Duration, backend: &str, state: &mut ReconcilerState, ) -> Result where - S: BackupCatalog + LegacyBackupReader + ?Sized, + S: BackupCatalog + LegacyBackupReader + RestoreEngine + ?Sized + Send + Sync + 'static, { let maintenance_held = match state.maintenance_held { Some(held) => held, @@ -356,13 +482,86 @@ where ); } + // The store's check first, so every duty of this cycle that reads the + // store's health reads what this cycle's check left. + if let Some(store) = runtime.configured_store() { + let check = super::check_configured_store(store, &**storage, backend); + let outcome = match &state.shutdown { + Some(shutdown) => tokio::select! { + outcome = check => Some(outcome), + () = shutdown.cancelled() => None, + }, + None => Some(check.await), + }; + match outcome { + Some(outcome) => { + // The defaulted SQLite root's rule, applied to the first + // check when the startup checks did not finish and left the + // store checking: a failure with no rows for the root is the + // warning the startup check would have given, and the store + // is left unconfigured rather than unhealthy. + if store.is_defaulted() + && store.health() == super::StoreHealth::Checking + && let Err(problem) = &outcome + && !matches!(problem, super::StoreProblem::Catalog(_)) + && storage + .read_store_ids(store.kind, &store.root) + .await + .is_ok_and(|ids| ids.is_empty()) + { + tracing::warn!( + "backup store: the default root {} failed its first check ({problem}); \ + the catalog has no backup rows for it, so the store is unconfigured and \ + CreateBackup writes in-catalog backups. Fix the root (owned by this \ + user, mode 0700, under directories others cannot write to, its \ + extenddb-store.json naming this catalog), configure [backup.store] to \ + choose another, or set kind = \"disabled\" to silence this", + store.root + ); + store.set_aside(); + return Ok(report); + } + store.record_check(outcome); + } + None => { + tracing::info!( + "backup store ({}) check was stopped by shutdown; its health is left as the \ + last check left it", + store.kind + ); + report.store_healthy = Some(store.is_healthy()); + return Ok(report); + } + } + report.store_healthy = Some(store.is_healthy()); + } + + let mut store_not_ready_logged = false; for job in storage.list_dead_restore_jobs(stale_job_after).await? { match (job.status, job.kind) { + // A `target_not_empty` target is the operator's. + (JobStatus::Failed, _) if job.failure_reason.as_deref() == Some(TARGET_NOT_EMPTY) => { + tracing::warn!( + "the restore into {} (account {}) from {} is FAILED {TARGET_NOT_EMPTY}: its \ + target ({}) holds items no record of the restore accounts for and is left \ + for the operator; inspect it at the storage level and remove it with \ + extenddb backup remove-target --account {} {}", + job.target.table_name, + job.target.account_id, + job.source_backup_arn, + job.target.table_id, + job.target.account_id, + job.target.table_name + ); + report + .targets_left_for_operator + .push(job.target.table_name.clone()); + } // The FAILED-target duty, whatever the record's kind: the job is // over and only its target is left. (JobStatus::Failed, _) => { match remove_restore_target( - storage, + &**storage, &job, "was FAILED when the reconciler reached it", ) @@ -377,7 +576,7 @@ where Removal::Failed => {} } } - (_, RestoreKind::Legacy) => match remove_dead_legacy_restore(storage, &job).await { + (_, RestoreKind::Legacy) => match remove_dead_legacy_restore(&**storage, &job).await { Some(Removal::Removed) => report .legacy_restores_removed .push(job.target.table_name.clone()), @@ -387,21 +586,47 @@ where Some(Removal::Failed) | None => {} }, (_, RestoreKind::Manifest) => { - if state.logged_manifest_restores.insert(job.target.clone()) { - tracing::info!( - "restore of {} into {} (account {}) has no live owner; a manifest \ - restore is resumed by a later release's reconciler (reported once)", - job.source_backup_arn, - job.target.table_name, - job.target.account_id - ); - report - .manifest_restores_logged - .push(job.target.table_name.clone()); + let name = job.target.table_name.clone(); + let stopping = state + .shutdown + .as_ref() + .is_some_and(CancellationToken::is_cancelled); + match resume_dead_manifest_restore( + storage, + runtime, + stale_job_after, + &job, + report.maintenance_held, + stopping, + &mut state.restore_takeovers, + ) + .await + { + Resumed::Spawned => report.manifest_restores_resumed.push(name), + Resumed::Failed(reason) => report.manifest_restores_failed.push((name, reason)), + Resumed::FailedAndRemoved(removal) => match removal { + Removal::Removed => report.manifest_restores_failed_and_removed.push(name), + Removal::LeftToOwner => report.targets_left_to_owner.push(name), + Removal::Failed => {} + }, + Resumed::NoPermit => report.restores_skipped_no_permit.push(name), + Resumed::StoreUnhealthy => { + if !store_not_ready_logged { + store_not_ready_logged = true; + let state = + match runtime.configured_store().map(ConfiguredStore::health) { + Some(super::StoreHealth::Checking) => "not yet checked", + _ => "unhealthy", + }; + tracing::warn!( + "the configured backup store is {state}; dead store-backed \ + restores are not resumed this cycle" + ); + } + report.restores_skipped_store_unhealthy.push(name); + } + Resumed::LeftToOwner | Resumed::Error => {} } - report - .manifest_restores_seen - .push(job.target.table_name.clone()); } } } @@ -437,81 +662,566 @@ where } } - if let Some(store) = runtime.configured_store() { - let check = super::check_configured_store(store, storage, backend); - let outcome = match &state.shutdown { - Some(shutdown) => tokio::select! { - outcome = check => Some(outcome), - () = shutdown.cancelled() => None, - }, - None => Some(check.await), + if let Some(store) = runtime.configured_store() + && let Some(identity) = store.identity().filter(|_| store.is_healthy()) + { + let calls = StoreCalls { + store: &*store.store, + metrics: runtime.metrics().map(|m| &**m), }; - match outcome { - Some(outcome) => { - // The defaulted SQLite root's rule, applied to the first - // check when the startup checks did not finish and left the - // store checking: a failure with no rows for the root is the - // warning the startup check would have given, and the store - // is left unconfigured rather than unhealthy. - if store.is_defaulted() - && store.health() == super::StoreHealth::Checking - && let Err(problem) = &outcome - && !matches!(problem, super::StoreProblem::Catalog(_)) - && storage - .read_store_ids(store.kind, &store.root) - .await - .is_ok_and(|ids| ids.is_empty()) + report.deletions_finished = finish_deleting_rows(&**storage, store, &identity).await?; + (report.creating_resolved, report.creating_left_to_owner) = + resolve_dead_creating_rows(&**storage, calls, &identity, stale_job_after).await?; + if !report.maintenance_held { + let retention = runtime.job_settings().failed_row_retention; + report.failed_prefixes_deleted = + delete_stale_failed_prefixes(&**storage, calls, &identity, stale_job_after).await?; + report.deleted_prefixes_swept = + sweep_deleted_prefixes(&**storage, calls, &identity).await?; + report.tombstones_removed = + remove_tombstones(&**storage, calls, &identity, retention).await?; + } + } + + if !report.maintenance_held { + report.registrations_swept = storage.sweep_restore_registrations().await?; + (report.progress_rows_swept, report.spool_entries_swept) = + sweep_restore_leftovers(&**storage, runtime.configured_store()).await; + } + Ok(report) +} + +/// What the manifest arm did with one dead RUNNING record. +enum Resumed { + /// Taken over and the job spawned. + Spawned, + /// FAILED with the reason, the target left to the FAILED-target duty + /// (another store) or to the operator (`target_not_empty`). + Failed(String), + /// FAILED `OWNER_DEAD` and the target removed (or found moved, or the + /// removal failed), as a legacy record is, under the maintenance claim. + FailedAndRemoved(Removal), + /// No job permit was free; the next cycle retries. + NoPermit, + /// The configured store is unhealthy; the next cycle retries. + StoreUnhealthy, + /// The backend did not hand the record over: its owner is alive, or + /// another reconciler took it; or the server is shutting down and the + /// record is left for the next reconciler. + LeftToOwner, + /// A catalog or store call failed; logged, the next cycle retries. + Error, +} + +/// The manifest arm (module doc). `takeovers` counts this instance's +/// takeovers of each record for the bound [`RESTORE_TAKEOVERS`]. +async fn resume_dead_manifest_restore( + storage: &Arc, + runtime: &BackupRuntime, + stale_job_after: Duration, + job: &RestoreJobRow, + maintenance_held: bool, + stopping: bool, + takeovers: &mut std::collections::HashMap, +) -> Resumed +where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let target = &job.target; + let what = format!( + "the restore of {} into {} (account {})", + job.source_backup_arn, target.table_name, target.account_id + ); + // Once the server's shutdown has begun the jobs' token is cancelled: a + // takeover now would spawn a resume that stops at its first check and + // leaves the record with this instance's fresh heartbeat, so the next + // instance waits a second stale window for it; a FAILED write on the + // way out is no better. The record is left as it is for the next + // reconciler to meet. The same holds once this reconciler's own token + // fires while the jobs' token does not (`stopping`: the server stops its + // workers while it holds no serving registration and starts a new set + // once it holds one): a resume is a job, on the jobs' token, which the + // workers' stop does not stop, so the record is left for the next set's + // reconciler. + if runtime.shutdown().is_cancelled() { + tracing::info!( + "{what} has no live owner and the server is shutting down; it is left for the next \ + reconciler" + ); + return Resumed::LeftToOwner; + } + if stopping { + tracing::info!( + "{what} has no live owner and the reconciler is stopping; it is left for the next \ + reconciler" + ); + return Resumed::LeftToOwner; + } + if maintenance_held { + tracing::info!( + "{what} has no live owner and a maintenance claim is held; it is failed rather than \ + resumed" + ); + // The `target_not_empty` guard first, as on the takeover arm: a + // record with no progress over a target that holds items is the + // operator's, and `OWNER_DEAD` with the target's removal would drop + // what the record does not account for. + match unaccounted_items(&**storage, job).await { + Ok(Some(count)) => { + return match storage + .fail_restore_job( + target, + job.owner_generation, + job.heartbeat_at, + TARGET_NOT_EMPTY, + ) + .await { - tracing::warn!( - "backup store: the default root {} failed its first check ({problem}); \ - the catalog has no backup rows for it, so the store is unconfigured and \ - CreateBackup writes in-catalog backups. Fix the root (owned by this \ - user, mode 0700, under directories others cannot write to, its \ - extenddb-store.json naming this catalog), configure [backup.store] to \ - choose another, or set kind = \"disabled\" to silence this", - store.root + Ok(true) => { + tracing::error!( + "{what} recorded no progress and its target ({}) holds {count} \ + item(s); the record is FAILED {TARGET_NOT_EMPTY} and the target is \ + left for the operator: extenddb backup remove-target --account {} {}", + target.table_id, + target.account_id, + target.table_name + ); + Resumed::Failed(TARGET_NOT_EMPTY.to_owned()) + } + Ok(false) => Resumed::LeftToOwner, + Err(e) => { + tracing::error!( + "{what}: the FAILED write failed; the next cycle retries: {e}" + ); + Resumed::Error + } + }; + } + Ok(None) => {} + Err(e) => { + tracing::error!( + "{what}: its progress or its target could not be read; the next cycle \ + retries: {e}" + ); + return Resumed::Error; + } + } + return match remove_dead_legacy_restore(&**storage, job).await { + Some(removal) => Resumed::FailedAndRemoved(removal), + None => Resumed::LeftToOwner, + }; + } + let store = runtime.configured_store_arc(); + // An instance with no configured store does not own any store-backed + // record: it leaves every one alone for the instance that holds its + // store, rather than failing records it cannot read (an instance without + // `[backup.store]` had failed every dead store-backed record it listed + // `OTHER_STORE`). Only an instance that has a store, and whose store is + // not the record's, fails the record OTHER_STORE. + if store.is_none() { + tracing::info!( + "{what} has no live owner and this instance has no configured store; it is left for \ + the instance that holds the record's store" + ); + return Resumed::LeftToOwner; + } + let of_this_store = job + .store + .as_ref() + .is_some_and(|row_store| runtime.is_configured_store(row_store)); + if !of_this_store { + tracing::error!( + "{what} has no live owner and reads a store other than the configured one; the \ + record is FAILED and its target removed once the record is stale: {OTHER_STORE}" + ); + // Conditioned on the claim the listing read, as the legacy arm's + // write is: a heartbeat that landed since keeps the record its + // owner's (`Ok(false)`). + return match storage + .fail_restore_job(target, job.owner_generation, job.heartbeat_at, OTHER_STORE) + .await + { + Ok(true) => Resumed::Failed(OTHER_STORE.to_owned()), + Ok(false) => Resumed::LeftToOwner, + Err(e) => { + tracing::error!("{what} could not be marked FAILED; the next cycle retries: {e}"); + Resumed::Error + } + }; + } + let Some(store) = store.filter(|store| store.is_healthy()) else { + return Resumed::StoreUnhealthy; + }; + let env = match JobEnv::new(runtime, store) { + Ok(env) => env, + Err(_) => return Resumed::StoreUnhealthy, + }; + let Some(permit) = JobPermit::try_acquire(runtime.slots()) else { + tracing::info!("{what} has no live owner; no job permit is free, the next cycle retries"); + return Resumed::NoPermit; + }; + let spool_host = env.spool.as_ref().map(|s| s.spool_host.clone()); + let claim = match storage + .claim_restore_dead(target, stale_job_after, spool_host.as_deref()) + .await + { + Ok(Some(claim)) => claim, + Ok(None) => return Resumed::LeftToOwner, + Err(e) => { + tracing::error!( + "could not test whether the owner of {what} is dead; the next cycle retries: {e}" + ); + return Resumed::Error; + } + }; + // The resumed job's first confirmed beat (module doc): the takeover + // stamped `heartbeat_at` before it returned. + let claimed_at = tokio::time::Instant::now(); + // The cycle read the maintenance claim once at its start; a command may + // have taken it since. Re-read it now the record is this instance's, as + // the request's `start` re-checks after its registration: held, the + // resume is left for the maintenance arm of a later cycle rather than + // run beside a command, and the record keeps this takeover's fresh + // heartbeat. + match storage.check_maintenance_claim(stale_job_after).await { + Ok(false) => {} + Ok(true) => { + tracing::info!( + "{what} was taken over but a maintenance claim is now held; the resume is left \ + for the next cycle" + ); + return Resumed::LeftToOwner; + } + Err(e) => { + tracing::error!( + "{what} was taken over but the maintenance claim could not be re-read; the next \ + cycle retries: {e}" + ); + return Resumed::Error; + } + } + // The bound on this instance's takeovers of the record + // (`RESTORE_TAKEOVERS`): the takeover after the fifth fails the record + // and removes its target as the arm below does for a target with no + // tables, since a record whose every resume ends RUNNING again would + // otherwise be retaken every stale window for as long as the server runs. + let taken = takeovers.entry(target.table_id.clone()).or_insert(0); + *taken = taken.saturating_add(1); + if *taken > RESTORE_TAKEOVERS { + takeovers.remove(&target.table_id); + let fields = StatusFields { + failure_reason: Some(TAKEOVERS_EXHAUSTED.to_owned()), + completed: true, + ..StatusFields::default() + }; + return match write_status(&*claim, JobStatus::Running, JobStatus::Failed, &fields).await { + Written::Matched => { + tracing::error!( + "{what} has no live owner and was taken over {RESTORE_TAKEOVERS} times by \ + this instance without ending; the record is FAILED and the target ({}) \ + removed", + target.table_id + ); + Resumed::FailedAndRemoved( + remove_restore_target(&**storage, job, "was marked FAILED by the reconciler") + .await, + ) + } + Written::NotMatched(why) => { + tracing::info!("{what} was no longer this cycle's ({why}); left to its owner"); + Resumed::LeftToOwner + } + Written::Error(e) => { + tracing::error!("{what}: the FAILED write failed; the next cycle retries: {e}"); + Resumed::Error + } + }; + } + // The progress under the claim: the record as it is now and, where the + // backend keeps them apart, its progress rows. + let (record, progress_rows) = match ( + storage.read_restore_job(target).await, + storage.read_restore_progress(target).await, + ) { + (Ok(Some(record)), Ok(rows)) => (record, rows), + (Ok(None), _) => { + tracing::info!("{what} was gone when its progress was read; left to whoever moved it"); + return Resumed::LeftToOwner; + } + (Err(e), _) | (_, Err(e)) => { + tracing::error!( + "{what} was taken over but its progress could not be read; the next cycle \ + retries once the record is stale again: {e}" + ); + return Resumed::Error; + } + }; + // The target's data tables must exist: every backend commits the record + // before the data DDL (the trait doc of `restore_target_ready`), so an + // owner killed in between leaves a RUNNING record over a target with no + // table, or on MongoDB with its index metadata missing. Such a record is + // never resumable, and the count below would fail on the missing table + // and leave the takeover's fresh heartbeat, so the record would be + // retaken every stale window and never fail. It is failed and its target + // removed instead, under the claim this takeover holds. + match storage.restore_target_ready(target).await { + Ok(true) => {} + Ok(false) => { + let fields = StatusFields { + failure_reason: Some(OWNER_DEAD.to_owned()), + completed: true, + ..StatusFields::default() + }; + return match write_status(&*claim, JobStatus::Running, JobStatus::Failed, &fields).await + { + Written::Matched => { + tracing::error!( + "{what} has no live owner and its target ({}) has no data tables (its \ + owner died between the registration and the data DDL); the record is \ + FAILED and the target removed", + target.table_id ); - store.set_aside(); - return Ok(report); + Resumed::FailedAndRemoved( + remove_restore_target( + &**storage, + job, + "was marked FAILED by the reconciler", + ) + .await, + ) } - store.record_check(outcome); + Written::NotMatched(why) => { + tracing::info!("{what} was no longer this cycle's ({why}); left to its owner"); + Resumed::LeftToOwner + } + Written::Error(e) => { + tracing::error!("{what}: the FAILED write failed; the next cycle retries: {e}"); + Resumed::Error + } + }; + } + Err(e) => { + tracing::error!( + "{what} was taken over but whether its target has data tables could not be read; \ + the next cycle retries: {e}" + ); + return Resumed::Error; + } + } + if has_no_progress(&record, &progress_rows) { + let count = match storage.count_restore_target(target).await { + Ok(count) => count, + Err(e) => { + tracing::error!( + "{what} was taken over but its target could not be counted; the next cycle \ + retries once the record is stale again: {e}" + ); + return Resumed::Error; } - None => { - tracing::info!( - "backup store ({}) check was stopped by shutdown; its health is left as the \ - last check left it", - store.kind + }; + let unaccounted = match exceeds_backup_total(&**storage, job, count).await { + Ok(unaccounted) => unaccounted, + Err(e) => { + tracing::error!( + "{what} was taken over but its backup's row could not be read; the next \ + cycle retries once the record is stale again: {e}" ); - report.store_healthy = Some(store.is_healthy()); - return Ok(report); + return Resumed::Error; } + }; + if count > 0 && !unaccounted { + tracing::info!( + "{what} recorded no progress and its target ({}) holds {count} item(s), no more \ + than the backup's total: a first data file applied in part before its progress \ + was recorded; the resume re-applies it with upserts", + target.table_id + ); } - report.store_healthy = Some(store.is_healthy()); - if let Some(identity) = store.identity().filter(|_| store.is_healthy()) { - let calls = StoreCalls { - store: &*store.store, - metrics: runtime.metrics().map(|m| &**m), + if unaccounted { + let fields = StatusFields { + failure_reason: Some(TARGET_NOT_EMPTY.to_owned()), + completed: true, + ..StatusFields::default() + }; + return match write_status(&*claim, JobStatus::Running, JobStatus::Failed, &fields).await + { + Written::Matched => { + tracing::error!( + "{what} recorded no progress and its target ({}) holds {count} item(s); \ + the record is FAILED {TARGET_NOT_EMPTY} and the target is left for the \ + operator: extenddb backup remove-target --account {} {}", + target.table_id, + target.account_id, + target.table_name + ); + Resumed::Failed(TARGET_NOT_EMPTY.to_owned()) + } + Written::NotMatched(why) => { + tracing::info!("{what} was no longer this cycle's ({why}); left to its owner"); + Resumed::LeftToOwner + } + Written::Error(e) => { + tracing::error!("{what}: the FAILED write failed; the next cycle retries: {e}"); + Resumed::Error + } }; - report.deletions_finished = finish_deleting_rows(storage, store, &identity).await?; - (report.creating_resolved, report.creating_left_to_owner) = - resolve_dead_creating_rows(storage, calls, &identity, stale_job_after).await?; - if !report.maintenance_held { - let retention = runtime.job_settings().failed_row_retention; - report.failed_prefixes_deleted = - delete_stale_failed_prefixes(storage, calls, &identity, stale_job_after) - .await?; - report.deleted_prefixes_swept = - sweep_deleted_prefixes(storage, calls, &identity).await?; - report.tombstones_removed = - remove_tombstones(storage, calls, &identity, retention).await?; - } } } + // The record's progress as recorded; the job's own start line names the + // first file it opens. + tracing::info!( + "{what} has no live owner; taken over by this instance and resumed with {}", + match (record.file_index, record.items_applied) { + (Some(file), Some(items)) => + format!("{items} items of data file {file} recorded as applied"), + (Some(file), None) => format!("data file {file} recorded as applied"), + (None, _) if !progress_rows.is_empty() => + format!("{} data file(s) recorded as applied", progress_rows.len()), + (None, _) => "nothing recorded as applied".to_owned(), + } + ); + let limits = (&**runtime.limits()).into(); + runtime.spawn_job(restore::resume( + Arc::clone(storage), + env, + permit, + claim, + restore::DeadRecord { + row: record, + recorded_spool_host: job.spool_host.clone(), + progress_rows, + claimed_at, + }, + limits, + )); + Resumed::Spawned +} - if !report.maintenance_held { - report.registrations_swept = storage.sweep_restore_registrations().await?; +/// Whether a RUNNING record has no progress recorded at all: no +/// `file_index` on the record and no progress row beside it. +fn has_no_progress(record: &RestoreJobRow, progress_rows: &[(u32, u64)]) -> bool { + record.file_index.is_none() && progress_rows.is_empty() +} + +/// The `target_not_empty` test as the maintenance-held arm runs it, with +/// no claim on the record: the record and its progress rows read, and when +/// [`has_no_progress`] the target counted; `Some(count)` when the target +/// holds items no record accounts for ([`exceeds_backup_total`]), `None` +/// when progress is recorded, the target is empty or holds no more than the +/// backup's total, or the record is gone (the caller's write then finds it +/// so). +async fn unaccounted_items( + storage: &S, + job: &RestoreJobRow, +) -> Result, StorageError> { + let target = &job.target; + let Some(record) = storage.read_restore_job(target).await? else { + return Ok(None); + }; + let progress_rows = storage.read_restore_progress(target).await?; + if !has_no_progress(&record, &progress_rows) { + return Ok(None); } - Ok(report) + let count = storage.count_restore_target(target).await?; + Ok(exceeds_backup_total(storage, job, count) + .await? + .then_some(count)) +} + +/// The `target_not_empty` rule's test: whether `count` items in a target +/// with no recorded progress are more than the backup holds (its row's +/// `item_count`). At or below the total they are a first data file applied +/// in part before its progress was recorded (Cassandra records a batch's +/// progress after the batch's writes, so a crash inside the first batch +/// leaves this state), and the resume re-applies the first file with +/// `reapplying` set, every backend's load being idempotent for a file +/// resumed that way. Above the total the target holds items the backup does +/// not account for (a catalog restored from a dump naming a restore that +/// finished and was written to since), the operator's. A count of zero is +/// never unaccounted; a backup row this catalog does not hold, or an ARN +/// this build cannot parse, leaves every item unaccounted. +async fn exceeds_backup_total( + storage: &S, + job: &RestoreJobRow, + count: u64, +) -> Result { + if count == 0 { + return Ok(false); + } + let Ok(arn) = BackupArn::parse(&job.source_backup_arn) else { + return Ok(true); + }; + let total = storage + .read_backup_row(&arn) + .await? + .map(|row| u64::try_from(row.item_count).unwrap_or(0)); + Ok(total.is_none_or(|total| count > total)) +} + +/// The progress-row and spool sweeps (module doc). The spool directory is +/// listed first, then the RUNNING records, and only entries of that listing +/// that no record of this host names are removed, so a restore whose +/// registration commits and whose entry opens between the two reads is +/// never a candidate (its entry is not in the listing, or its record is in +/// the records). A listing the backend cannot make is logged and skips +/// both. +async fn sweep_restore_leftovers( + storage: &S, + store: Option<&ConfiguredStore>, +) -> (u64, Vec) { + let spool = store.and_then(|store| store.spool.as_ref()); + let entries = match spool.map(|spool| restore::spool::list_entries(&spool.path)) { + None => Vec::new(), + Some(Ok(entries)) => entries, + Some(Err(e)) => { + tracing::error!( + "the restore spool {} could not be listed; the spool sweep is skipped this \ + cycle: {e}", + spool + .map(|s| s.path.display().to_string()) + .unwrap_or_default() + ); + Vec::new() + } + }; + let running = match storage.list_restore_jobs(Some(JobStatus::Running)).await { + Ok(rows) => rows, + Err(e) => { + tracing::warn!( + "the restore job records could not be listed; the progress-row and spool sweeps \ + are skipped this cycle: {e}" + ); + return (0, Vec::new()); + } + }; + let running_ids: Vec = running.iter().map(|r| r.target.table_id.clone()).collect(); + let progress_rows = match storage.sweep_restore_progress(&running_ids).await { + Ok(n) => { + if n > 0 { + tracing::info!( + "removed {n} restore progress row(s) of tables that are not RUNNING restore \ + targets" + ); + } + n + } + Err(e) => { + tracing::error!( + "the restore progress rows could not be swept; the next cycle retries: {e}" + ); + 0 + } + }; + let Some(spool) = spool else { + return (progress_rows, Vec::new()); + }; + let this_host: HashSet = running + .iter() + .filter(|r| r.spool_host.as_deref() == Some(spool.spool_host.as_str())) + .map(|r| r.target.table_id.clone()) + .collect(); + let swept = restore::spool::sweep_entries(entries, &this_host); + (progress_rows, swept) } /// The duty for CREATING rows of the configured store whose owner is dead @@ -584,7 +1294,7 @@ async fn resolve_dead_creating_rows( continue; } }; - match write_status(&*claim, to, &fields).await { + match write_status(&*claim, JobStatus::Creating, to, &fields).await { Written::Matched => { tracing::info!( "backup {} (account {}): its owner was dead; the row is now {to}", @@ -634,21 +1344,11 @@ impl StoreCalls<'_> { e } - /// One object's bytes, `None` when absent. - async fn read_object(self, key: &str) -> Result>, StoreError> { - let body = match self.store.get(key).await { - Ok(body) => body, - Err(StoreError::NotFound) => return Ok(None), - Err(e) => return Err(self.failed("get", e)), - }; - let bytes = body - .try_fold(Vec::new(), |mut acc, chunk| async move { - acc.extend_from_slice(&chunk); - Ok(acc) - }) + /// One object's bytes within `bound` ([`read_object`]). + async fn read_object(self, key: &str, bound: u64) -> Result { + read_object(self.store, key, bound) .await - .map_err(|e| self.failed("get", e))?; - Ok(Some(bytes)) + .map_err(|e| self.failed("get", e)) } /// The prefix's objects, key to size. @@ -694,22 +1394,38 @@ enum Probe { async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result { let prefix = arn.store_prefix(); let manifest = match store - .read_object(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")) + .read_object( + &format!("{prefix}{EXTENDDB_MANIFEST_FILE}"), + MANIFEST_READ_BOUND, + ) .await? { - Some(bytes) => bytes, - None => return Ok(Probe::Incomplete("no manifest".to_owned())), + ReadObject::Bytes(bytes) => bytes, + ReadObject::Absent => return Ok(Probe::Incomplete("no manifest".to_owned())), + ReadObject::Oversize => { + return Ok(Probe::Incomplete(format!( + "a manifest over {MANIFEST_READ_BOUND} bytes" + ))); + } }; let sidecar = match store - .read_object(&format!("{prefix}{MANIFEST_CHECKSUM_FILE}")) + .read_object( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + SIDECAR_READ_BOUND, + ) .await? { - Some(bytes) => bytes, - None => { + ReadObject::Bytes(bytes) => bytes, + ReadObject::Absent => { return Ok(Probe::Incomplete( "a manifest without its sidecar".to_owned(), )); } + ReadObject::Oversize => { + return Ok(Probe::Incomplete(format!( + "a sidecar over {SIDECAR_READ_BOUND} bytes" + ))); + } }; if let Err(e) = verify_manifest_checksum(&manifest, &sidecar) { return Ok(Probe::Incomplete(format!( @@ -766,6 +1482,51 @@ async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result), + /// No object under the key. + Absent, + /// The object has more bytes than the bound; the read stopped there and + /// collected nothing. + Oversize, +} + +/// One object's bytes, collected only while they stay within `bound` +/// (the body is read in chunks and dropped once the count passes it). The +/// restore's request and resume read the manifest and sidecar through it +/// too, each with its own bound, so a prefix an attacker rewrote cannot make +/// a request hold an object of any size in memory. +pub(super) async fn read_object( + store: &dyn BackupStore, + key: &str, + bound: u64, +) -> Result { + let mut body = match store.get(key).await { + Ok(body) => body, + Err(StoreError::NotFound) => return Ok(ReadObject::Absent), + Err(e) => return Err(e), + }; + let mut bytes = Vec::new(); + while let Some(chunk) = body.try_next().await? { + if (bytes.len() as u64).saturating_add(chunk.len() as u64) > bound { + return Ok(ReadObject::Oversize); + } + bytes.extend_from_slice(&chunk); + } + Ok(ReadObject::Bytes(bytes)) +} + /// The FAILED-prefix duty (module doc): FAILED rows whose heartbeat is older /// than `stale_job_after`, their prefix deleted while it lists objects. async fn delete_stale_failed_prefixes( diff --git a/crates/engine/src/backup/restore.rs b/crates/engine/src/backup/restore.rs new file mode 100644 index 000000000..23d774482 --- /dev/null +++ b/crates/engine/src/backup/restore.rs @@ -0,0 +1,1473 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The store-backed RestoreTableFromBackup: the request's manifest branch +//! and the restore job (the design's 5.3). +//! +//! The request ([`request`]) runs once `check_catalog_row_for_restore` has +//! accepted the backup's row: it takes the job permit, reads the sidecar and +//! the manifest from the store (a missing one is `BackupNotFoundException` +//! and an error line naming `backup sync --prune-missing`; each read +//! collects at most its bound, [`super::reconciler::SIDECAR_READ_BOUND`] and +//! [`super::reconciler::MANIFEST_READ_BOUND`], and an object over it is a +//! `ValidationException` naming a corrupt prefix, as a bad sidecar is), +//! verifies the +//! sidecar, parses the manifest (`ExtenddbManifest::from_json_bytes`, whose +//! own gates are the `format_version` and the identity rule that the ARN, +//! `source_table.table_name` and `source_table.table_arn` name one table), +//! checks that it names the row's backup, refuses a non-empty +//! `vector_indexes` into a backend without vector support with the text the +//! legacy reader uses, builds the target's `CreateTableInput` from the +//! manifest with the overrides applied, validates it as CreateTable does, +//! resolves the key schema the validator holds every item to, and spawns +//! the detached task ([`start`]) whose answer it awaits for at most +//! [`RESTORE_REGISTRATION_TIMEOUT`]; past that it answers +//! `InternalServerError` while the task goes on (the target then appears +//! through DescribeTable). The task registers the target and its RUNNING +//! record in one write (`claim_restore`, under the backup row's lock), sends +//! the job's first beat through the claim (the registration's own +//! `heartbeat_at` stamp can be old: the design's 4(e) states each backend's +//! clock, PostgreSQL's `now()` being the transaction's start, so a +//! registration stalled on the `backup_index` lock or the DDL commits a +//! stale beat that the reconciler under a maintenance claim and +//! `remove-target`, which judge the owner by the heartbeat alone, would +//! read as dead; the beat refreshes it at the registration's end on every +//! backend, and a beat that finds the claim lost or fails ends the task as +//! a lost claim before the reply, the record failed as the maintenance +//! refusal fails it and the request answered `InternalServerError`), then +//! checks the maintenance claim: held, the record is FAILED `maintenance`, +//! the target removed, and the request refused with +//! `LimitExceededException` (a job end: `RestoreJobs{failed}` and the worker +//! error are recorded); otherwise the request is answered CREATING and the +//! job runs ([`run`]). The heartbeat's first confirmed beat is that first +//! beat, and the instant it returned is what the pre-write check +//! (`heartbeat.rs`) counts the two periods from, so the registration's own +//! duration, however long, does not end the job at its first check. A +//! resumed job's instant is the takeover's return, taken by the reconciler +//! beside `claim_restore_dead` and passed in (the takeover stamped +//! `heartbeat_at` inside its write), so the reads between the takeover and +//! the job's start count against the window rather than widen it. +//! +//! The job: step 3, every data file downloaded and verified +//! ([`download`]: into the spool entry on an S3 store, hashed in place on a +//! filesystem store, the manifest and sidecar spooled first for a resume, +//! each as a data file is: `.partial`, synced, renamed; a resume that read +//! them from the spool does not write them again); +//! step 5, each file decoded, validated and written through one +//! `bulk_load` ([`load`]), skipping the files and items a resume has +//! recorded; the count check (`count_restore_target` against the manifest's +//! total, a shortfall being a key the backup holds twice); step 6, +//! `finish_restore` (DONE and ACTIVE, or DONE with the index backfill +//! pending on MongoDB). A `finish_restore` that returns an error is a +//! conditional write whose outcome is unknown, handled as `write_status` +//! handles one: the record is re-read through the claim; DONE under this +//! generation means the write landed and the restore is reported complete +//! with a WARN naming the error; RUNNING under this generation means it did +//! not, and the write is sent once more, at most, its own error re-read the +//! same way; anything else is the DONE write that matched no record. The +//! heartbeat runs on a timer for the job's life +//! and is checked before every download attempt, every `bulk_load`, the +//! count and the flip; a lost claim ends the job with no catalog or table +//! access. Step 7 ([`fail_restore`]): FAILED with the reason, then +//! `remove_restore_target` only when that write matched (the claim holder +//! owns the table); the spool entry is removed on every end but the +//! shutdown's and the ends at which the job learnt the record is another's +//! (a lost claim, a DONE or FAILED write that matched no record): a +//! same-host successor may be reading that entry, so it is left to the +//! reconciler's sweep, which lists the spool before it reads the records and +//! removes the entries it listed that no RUNNING record of this host names. +//! The shutdown token stops the job between download attempts, +//! files and chunks with no status write: the record stays RUNNING with its +//! spool and progress for a resume ([`resume`], the reconciler's entry). +//! +//! Each job records `RestoreJobs` and `RestoreJobDuration` under its +//! outcome, `RestoreBytesDownloaded` as the files arrive, +//! `BackupStoreErrors{get}` on every failed `get` of a restore job (the +//! request's and a resume's two document reads and each download attempt +//! that the store failed, whatever the store's error), and the worker +//! metrics under `QuerySource::RestoreJob` (`WorkerLastSuccess` on DONE, +//! `WorkerErrorCount` on FAILED or a job that ended without its write; a +//! stopped job records neither); one info line at start and end, one error +//! line per failure. + +pub(crate) mod download; +pub(crate) mod load; +pub(crate) mod spool; + +use std::sync::Arc; +use std::time::Duration; + +use extenddb_core::backup_format::{ + BackupArn, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, ItemOrder, MANIFEST_CHECKSUM_FILE, + verify_manifest_checksum, +}; +use extenddb_core::error::DynamoDbError; +use extenddb_core::limits::LimitsConfig; +use extenddb_core::metrics::{BackupJobOutcome, QuerySource}; +use extenddb_core::types::{BillingMode, CreateTableInput, TableDescription}; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits}; +use extenddb_storage::backup_catalog::{ + BackupKey, Heartbeat, JobStatus, RestoreJobRow, RestoreRegistrationRow, RestoreTargetKey, + StatusFields, UNOWNED_GENERATION, billing_mode_name, +}; +use extenddb_storage::backup_definition::{restore_input_from_manifest, vector_index_refusal}; +use extenddb_storage::{ + BackupCatalog, CancellationToken, DataEngine, Finished, JobClaim, JobPermit, RestoreEngine, + RestoreSource, RestoreTableOverrides, +}; +use serde_json::Value; +use tokio::sync::oneshot; +use tokio::time::Instant; + +use super::job::failure::{Written, write_status}; +use super::job::heartbeat::{HeartbeatHandle, HeartbeatMetric}; +use super::job::{JobEnv, KillPoint, MAINTENANCE, kill_point, kill_point_named}; +use super::overrides::IndexOverrides; +use super::reconciler::{MANIFEST_READ_BOUND, ReadObject, SIDECAR_READ_BOUND}; +use super::runtime::{BackupRuntime, ConfiguredStore}; +use super::{ + backup_not_found, maintenance_held, restore_response, restore_storage_err_to_dynamo, + storage_err_to_dynamo, validate_override_for_source, +}; +use download::{Dest, Downloader, Stop}; +use load::{FileLoad, Source}; + +/// How long the request waits for the detached task to register the target +/// before it answers `InternalServerError` and leaves the task to finish +/// (against a catalog that has stopped handing out connections). One value +/// on every backend. +pub const RESTORE_REGISTRATION_TIMEOUT: Duration = Duration::from_secs(60); + +/// The reason a resumed restore's record carries when no progress was +/// recorded and the target already holds items (a catalog restored from a +/// dump can name a restore that finished and was written to since): the +/// target is left for the operator (`extenddb backup remove-target`). +pub const TARGET_NOT_EMPTY: &str = "target_not_empty"; + +/// The reason the reconciler writes on a RUNNING record whose store is not +/// the configured one. +pub const OTHER_STORE: &str = "the restore reads a store other than the configured one; backup sync --move-store re-points it"; + +/// What the request resolved before it spawned the task. +pub(crate) struct RestoreRequestParts { + pub account_id: String, + pub arn: BackupArn, + pub backup: BackupKey, + pub manifest: Arc, + pub manifest_bytes: Vec, + pub sidecar_bytes: Vec, + pub input: CreateTableInput, + /// The key schema the validator holds every item to. + pub key_schema: RestorableKeySchema, + pub limits: RestorableLimits, + /// This instance's in-flight entry for the registration, which the task + /// ends with the registration's outcome; `None` for a caller outside + /// the request path. + pub registration: Option, +} + +/// The one-shot the task answers the request through. +pub(crate) type Reply = oneshot::Sender>; + +/// The request's manifest branch (module doc), from the permit to the +/// response body, for a row `check_catalog_row_for_restore` accepted whose +/// store is the healthy configured one. +#[allow(clippy::too_many_arguments)] +pub(super) async fn request( + storage: &Arc, + runtime: &BackupRuntime, + store: &Arc, + account_id: &str, + limits: &LimitsConfig, + target_table_name: &str, + arn: &BackupArn, + overrides: &RestoreTableOverrides, + index_overrides: &IndexOverrides, +) -> Result<(String, Value), DynamoDbError> +where + S: BackupCatalog + DataEngine + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let arn_text = arn.to_arn_string(); + let env = JobEnv::new(runtime, store)?; + // The permit, last among the checks that need no store access. + let permit = runtime.take_permit().await?; + + // The sidecar and the manifest; a missing one behind an AVAILABLE row is + // BackupNotFound and the operator's line; one over its bound is a + // prefix nothing this server wrote produces. + let prefix = arn.store_prefix(); + let read = async |name: &str, bound: u64| -> Result, DynamoDbError> { + match super::reconciler::read_object(&*env.store, &format!("{prefix}{name}"), bound).await { + Ok(ReadObject::Bytes(bytes)) => Ok(bytes), + Ok(ReadObject::Absent) => { + tracing::error!( + "backup {arn_text} (account {account_id}): its row is AVAILABLE but the store \ + holds no {name} under its prefix; backup sync --prune-missing marks the row" + ); + Err(backup_not_found(&arn_text)) + } + Ok(ReadObject::Oversize) => Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a {name} over {bound} bytes, which no backup this server \ + wrote produces; the backup's prefix is corrupt" + ))), + Err(e) => { + if let Some(metrics) = runtime.metrics() { + metrics.record_backup_store_error("get"); + } + tracing::error!("backup {arn_text}: reading {name} from the store failed: {e}"); + Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )) + } + } + }; + let sidecar_bytes = read(MANIFEST_CHECKSUM_FILE, SIDECAR_READ_BOUND).await?; + let manifest_bytes = read(EXTENDDB_MANIFEST_FILE, MANIFEST_READ_BOUND).await?; + verify_manifest_checksum(&manifest_bytes, &sidecar_bytes).map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose checksum sidecar does not match: {e}" + )) + })?; + // The parser ran the version gate and the identity rule (the ARN's + // table is `source_table.table_name`); what is left is that the + // manifest's backup is the row's. + let manifest = ExtenddbManifest::from_json_bytes(&manifest_bytes).map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest this server cannot restore from: {e}" + )) + })?; + match manifest.backup_arn() { + Ok(named) if named == *arn => {} + Ok(named) => { + return Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest that names another backup, {}", + named.to_arn_string() + ))); + } + Err(e) => { + return Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose backup ARN is refused: {e}" + ))); + } + } + + // Step 2: the one feature a target can lack. + let source_table = &manifest.source_table; + if crate::vector_gate::ensure_create_table_supported( + source_table.vector_indexes.as_ref(), + storage.as_vector_search(), + ) + .is_err() + { + let count = source_table.vector_indexes.as_ref().map_or(0, Vec::len); + return Err(storage_err_to_dynamo(vector_index_refusal( + &format!("backup {arn_text}"), + count, + ))); + } + + // The target's input, with the overrides, validated as CreateTable does. + validate_override_for_source(overrides, billing_mode_name(source_table.billing_mode))?; + let mut input = restore_input_from_manifest(source_table, target_table_name, overrides); + index_overrides.apply(&mut input, overrides.billing_mode)?; + extenddb_core::validation::validate_create_table(&input, limits)?; + // The key schema the validator holds every item to: a pure function of + // the manifest, resolved here with step 2's refusals rather than after + // the client was told the target is CREATING. + let key_schema = RestorableKeySchema::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose key schema cannot be checked: {e}" + )) + })?; + let on_demand = input.billing_mode == Some(BillingMode::PayPerRequest); + let source = RestoreSource { + backup_arn: arn_text.clone(), + source_table_arn: source_table.table_arn.clone(), + backup_creation_date_time: manifest.backup_creation_date_time, + }; + + // Step 4 in the detached task; the request waits for its answer. A + // registration of this instance already in flight for the name is met + // here: the same backup's is this request's own, retried by a client + // whose read timeout (botocore's default is this bound, 60 s) ended its + // first attempt while the registration waited on the `backup_index` + // lock or the DDL, and the retry waits for that registration's answer + // rather than registering a second target or being refused the name; + // another backup's is `TableInUseException`, as a committed registration + // answers it. + let guard = match runtime.begin_restore_registration(account_id, target_table_name, &arn_text) { + Ok(guard) => guard, + Err(in_flight) => { + drop(permit); + if in_flight.backup_arn != arn_text { + return Err(restore_storage_err_to_dynamo( + extenddb_storage::error::StorageError::TableBeingRestored { + table_name: target_table_name.to_owned(), + backup_arn: in_flight.backup_arn, + }, + )); + } + tracing::info!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): a registration of this request is in flight; this request waits \ + for its answer" + ); + let desc = + await_registration(in_flight, &arn_text, target_table_name, account_id).await?; + let response = restore_response(desc, &source, on_demand)?; + return Ok((target_table_name.to_owned(), response)); + } + }; + let parts = RestoreRequestParts { + account_id: account_id.to_owned(), + arn: arn.clone(), + backup: BackupKey::from_arn(arn), + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits: limits.into(), + registration: Some(guard), + }; + let (reply, answer) = oneshot::channel(); + runtime.spawn_job(start(Arc::clone(storage), env, permit, parts, reply)); + let desc = match tokio::time::timeout(RESTORE_REGISTRATION_TIMEOUT, answer).await { + Ok(Ok(answer)) => answer?, + Ok(Err(_)) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the job task ended without answering" + ); + return Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )); + } + Err(_) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the registration did not answer within {} s; the task goes on \ + and the target, if registered, appears through DescribeTable", + RESTORE_REGISTRATION_TIMEOUT.as_secs() + ); + return Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )); + } + }; + let response = restore_response(desc, &source, on_demand)?; + Ok((target_table_name.to_owned(), response)) +} + +/// A retry's wait on its own registration in flight (module doc): the +/// registration's outcome, for at most [`RESTORE_REGISTRATION_TIMEOUT`] +/// from this request's arrival; a registration that returned no outcome +/// (its task ended) or none within the bound is `InternalServerError`, as +/// the registering request's own bound is. +async fn await_registration( + mut in_flight: super::runtime::InFlightRegistration, + arn_text: &str, + target_table_name: &str, + account_id: &str, +) -> Result { + let internal = || DynamoDbError::InternalServerError("Internal server error".to_owned()); + let outcome = async { + loop { + if let Some(outcome) = in_flight.outcome.borrow_and_update().clone() { + return Some(outcome); + } + if in_flight.outcome.changed().await.is_err() { + return in_flight.outcome.borrow().clone(); + } + } + }; + match tokio::time::timeout(RESTORE_REGISTRATION_TIMEOUT, outcome).await { + Ok(Some(outcome)) => outcome, + Ok(None) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the registration this request waited on ended without answering" + ); + Err(internal()) + } + Err(_) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the registration this request waited on did not answer within {} \ + s; it goes on and the target, if registered, appears through DescribeTable", + RESTORE_REGISTRATION_TIMEOUT.as_secs() + ); + Err(internal()) + } + } +} + +/// The detached task: the registration, the maintenance check, the reply, +/// then the job. +pub(crate) async fn start( + storage: Arc, + env: JobEnv, + permit: JobPermit, + parts: RestoreRequestParts, + reply: Reply, +) where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let started = Instant::now(); + let RestoreRequestParts { + account_id, + arn, + backup, + manifest, + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits, + registration, + } = parts; + let arn_text = arn.to_arn_string(); + let target_table_name = input.table_name.clone(); + let spool_host = env.spool.as_ref().map(|s| s.spool_host.clone()); + let registration_row = RestoreRegistrationRow { + backup, + source_table_arn: manifest.source_table.table_arn.clone(), + restore_date_time: manifest.backup_creation_date_time, + store: env.identity.clone(), + spool_host: spool_host.clone(), + owner_generation: UNOWNED_GENERATION, + }; + let registered = storage + .claim_restore(&account_id, input, ®istration_row) + .await; + // The in-flight entry ends with the reply, for the retries waiting on + // it: the description, or the error the request itself is answered. + let (desc, claim) = match registered { + Ok(registered) => registered, + Err(e) => { + drop(permit); + let error = restore_storage_err_to_dynamo(e); + if let Some(registration) = registration { + registration.finish(Err(error.clone())); + } + if let Err(Err(error)) = reply.send(Err(error)) { + // The request has answered `InternalServerError` at its bound + // and told the operator the target appears if registered; + // this line is the one trace of why it will not. + tracing::error!( + "restore of {arn_text} into {target_table_name} (account {account_id}): the \ + registration failed after the request had stopped waiting for it; no target \ + was created: {error}" + ); + } + return; + } + }; + // The job's first beat, before the reply: the record's `heartbeat_at` as + // the registration stamped it is the catalog's clock at some point of + // the registration (on PostgreSQL `now()` is the transaction's start, so + // a registration stalled on the `backup_index` lock or the DDL commits a + // stale beat), and the two readers that judge the owner by the heartbeat + // alone (the reconciler under a maintenance claim, `remove-target`) + // would find a live job dead until its timer's first beat. A confirmed + // beat refreshes it at the registration's end on every backend, and its + // return is the instant the heartbeat's two-period rule counts from. A + // beat that finds the claim lost, or fails, ends the task as a lost + // claim before the reply: the record is failed as the maintenance + // refusal fails it (the write matches nothing when the claim is + // another's) and the request is answered `InternalServerError`. + let first_beat = match claim.heartbeat().await { + Ok(Heartbeat::Confirmed) => Ok(Instant::now()), + Ok(Heartbeat::Lost) => Err("the claim was lost before the job started".to_owned()), + Err(e) => Err(format!( + "the claim was lost before the job started: its first heartbeat failed: {e}" + )), + }; + let claim: Arc = Arc::from(claim); + let target = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The maintenance claim, once the record exists so the command's wait + // sees it: held, the record is FAILED `maintenance`, the target removed + // and the request refused. + let (claimed_at, refused) = match first_beat { + Err(reason) => ( + started, + Some(( + reason, + DynamoDbError::InternalServerError("Internal server error".to_owned()), + )), + ), + Ok(beat) => ( + beat, + match storage + .check_maintenance_claim(env.settings.stale_job_after) + .await + { + Ok(false) => None, + Ok(true) => Some((MAINTENANCE.to_owned(), maintenance_held())), + Err(e) => Some(( + format!("the maintenance claim could not be read: {e}"), + storage_err_to_dynamo(e), + )), + }, + ), + }; + if let Some((reason, error)) = refused { + let path = fail_restore(&*storage, &*claim, &target, &reason, None, &env.shutdown).await; + tracing::error!( + "restore of {arn_text} into {target_table_name} (account {account_id}): FAILED \ + ({reason}) before it started; {path}" + ); + // A FAILED record is a job end: the outcome and the worker error. + if let Some(metrics) = env.metrics.as_deref() { + #[allow(clippy::cast_precision_loss)] + let duration_us = started.elapsed().as_micros() as f64; + metrics.record_restore_job(BackupJobOutcome::Failed, duration_us); + metrics.record_worker_error(QuerySource::RestoreJob); + } + drop(permit); + if let Some(registration) = registration { + registration.finish(Err(error.clone())); + } + let _ = reply.send(Err(error)); + return; + } + // A request whose client went away still has its job. + if let Some(registration) = registration { + registration.finish(Ok(desc.clone())); + } + let _ = reply.send(Ok(desc)); + let job = RestoreJob { + storage, + claim, + target, + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress: None, + recorded_spool_host: spool_host, + documents_from_spool: false, + }; + run(job).await; + drop(permit); +} + +/// Per file, what an earlier run recorded as applied; `None` for a file +/// with no record. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct Progress { + recorded: Vec>, +} + +impl Progress { + /// From a dead owner's record and, on PostgreSQL, its progress rows: + /// every file before the record's `file_index` is complete, `file_index` + /// itself has `items_applied` of its items (all of them when the backend + /// records per file), and each progress row names its file's count. + pub(crate) fn from_records( + row: &RestoreJobRow, + progress_rows: &[(u32, u64)], + manifest: &ExtenddbManifest, + ) -> Self { + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + let mut recorded = vec![None; counts.len()]; + if let Some(last) = row.file_index { + let last = last as usize; + for (i, slot) in recorded.iter_mut().enumerate().take(last.min(counts.len())) { + *slot = Some(counts[i]); + } + if let Some(slot) = recorded.get_mut(last) { + *slot = Some(row.items_applied.unwrap_or(counts[last])); + } + } + for (index, applied) in progress_rows { + if let Some(slot) = recorded.get_mut(*index as usize) { + *slot = Some((*applied).max(slot.unwrap_or(0))); + } + } + Self { recorded } + } + + /// The first file not complete, for the start line. + fn first_open(&self, counts: &[u64]) -> Option<(usize, u64)> { + self.recorded + .iter() + .enumerate() + .find_map(|(i, recorded)| match recorded { + Some(n) if *n >= counts[i] => None, + Some(n) => Some((i, *n)), + None => Some((i, 0)), + }) + } + + /// Per file, the items recorded as applied; `None` for a file with no + /// record. + #[cfg(test)] + pub(crate) fn recorded(&self) -> &[Option] { + &self.recorded + } + + /// [`Self::first_open`] against `manifest`'s counts. + #[cfg(test)] + pub(crate) fn first_open_in(&self, manifest: &ExtenddbManifest) -> Option<(usize, u64)> { + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + self.first_open(&counts) + } +} + +/// The job once its record exists. +pub(crate) struct RestoreJob { + storage: Arc, + claim: Arc, + target: RestoreTargetKey, + arn: BackupArn, + manifest: Arc, + manifest_bytes: Vec, + sidecar_bytes: Vec, + key_schema: RestorableKeySchema, + limits: RestorableLimits, + env: JobEnv, + started: Instant, + /// The instant the heartbeat's two-period rule counts from: the return + /// of the job's first beat, sent right after the registration, for a + /// fresh job; the takeover's return for a resume (the takeover stamped + /// the record's `heartbeat_at` inside its write). + claimed_at: Instant, + /// A resumed job's recorded progress; `None` for a fresh one. + progress: Option, + /// The `spool_host` the record carried before this job's claim: the + /// registration's own for a fresh job, the dead owner's for a resume. + /// A spool entry of this table id is this job's to reuse only when it + /// is this host's. + recorded_spool_host: Option, + /// Whether the manifest and sidecar in hand were read from this host's + /// spool entry (a resume), in which case the job does not write them + /// there again. + documents_from_spool: bool, +} + +/// The job (module doc), from the download to the final write. +pub(crate) async fn run(job: RestoreJob) -> BackupJobOutcome +where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let RestoreJob { + storage, + claim, + target, + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress, + recorded_spool_host, + documents_from_spool, + } = job; + let arn_text = arn.to_arn_string(); + let name = format!( + "restore of {arn_text} into {} (account {})", + target.table_name, target.account_id + ); + let metrics = env.metrics.as_deref(); + let heartbeat = HeartbeatHandle::start( + Arc::clone(&claim), + env.settings.heartbeat_interval, + HeartbeatMetric::None, + claimed_at, + ); + #[cfg(test)] + test_support::register_timer(&target.table_id, &heartbeat); + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + let total_items: u64 = counts.iter().sum(); + let resumed_from = progress.as_ref().and_then(|p| p.first_open(&counts)); + // The files a resume's progress records as complete: neither + // downloaded nor loaded again. + let complete: Vec = counts + .iter() + .enumerate() + .map(|(i, count)| { + progress + .as_ref() + .and_then(|p| p.recorded[i]) + .is_some_and(|n| n >= *count) + }) + .collect(); + tracing::info!( + "{name}: started, {} data file(s), {total_items} items, {}{}", + manifest.data_files.len(), + match manifest.item_order { + ItemOrder::KeyOrdered => "key-ordered", + ItemOrder::Unordered => "unordered", + }, + match (&progress, resumed_from) { + (None, _) => String::new(), + (Some(_), Some((file, items))) => + format!("; resumed from data file {file} after {items} of its items"), + (Some(_), None) => "; resumed with every file recorded".to_owned(), + } + ); + let prefix = arn.store_prefix(); + let mut entry = None; + let mut bytes_downloaded = 0; + let mut applied_total = 0; + let end: Result = async { + // Step 3. The entry is the job's from the open, so a document write + // that fails (the disk full, a name taken by a directory) reaches + // the failure path with the entry in hand and it is removed as any + // other failed end's is. + if let Some(spool) = &env.spool { + let opened = spool::Entry::open( + &spool.path, + &target.table_id, + recorded_spool_host.as_deref(), + &spool.spool_host, + ) + .map_err(|e| Stop::Fail(format!("opening the spool entry: {e}")))?; + let opened = entry.insert(opened); + if !documents_from_spool { + for (doc, bytes) in [ + (EXTENDDB_MANIFEST_FILE, &manifest_bytes), + (MANIFEST_CHECKSUM_FILE, &sidecar_bytes), + ] { + opened + .write_document(doc, bytes) + .await + .map_err(|e| Stop::Fail(format!("writing {doc} to the spool: {e}")))?; + } + } + } + let dest = entry.as_ref().map_or(Dest::InPlace, Dest::Spool); + let downloader = Downloader { + store: &*env.store, + prefix: &prefix, + heartbeat: &heartbeat, + shutdown: &env.shutdown, + metrics, + }; + bytes_downloaded = download::download_all(&downloader, &manifest, &dest, &complete).await?; + kill_point(KillPoint::RestoreAfterDownload, &env.shutdown).await; + + // Step 5. + #[cfg(not(test))] + let chunk_bytes = load::LOAD_CHUNK_BYTES; + #[cfg(test)] + let chunk_bytes = test_support::chunk_bytes_for(&target.table_id); + let mut first_loaded = true; + let mut reapplying = progress.is_some(); + for (i, file) in manifest.data_files.iter().enumerate() { + let recorded = progress.as_ref().and_then(|p| p.recorded[i]); + let start_at = match recorded { + Some(n) if n >= file.item_count => continue, + Some(n) => n, + None => 0, + }; + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed(&format!("to load {}", file.key)), + )); + } + let file_index = + u32::try_from(i).map_err(|_| Stop::Fail("too many data files".to_owned()))?; + let source = match &entry { + Some(entry) => Source::Spool(entry.file_path(file_index)), + None => Source::InPlace { + store: Arc::clone(&env.store), + key: format!("{prefix}{}", file.key), + }, + }; + let applied = load::load_file( + &*storage, + &target, + &*claim, + FileLoad { + manifest: &manifest, + entry: file, + file_index, + start_at, + reapplying, + key_schema: &key_schema, + limits, + shutdown: &env.shutdown, + heartbeat: &heartbeat, + chunk_bytes, + }, + source, + ) + .await?; + applied_total += applied; + // Only the first file a resume meets may have been applied in + // part without a record. + reapplying = false; + if first_loaded { + first_loaded = false; + kill_point(KillPoint::RestoreAfterFileCommit, &env.shutdown).await; + } + } + + // The count: every restore ends with it, so a backend that kept one + // of two rows with one key, and a key-range check that missed, are + // both caught. A count that fails is retried; one that keeps + // failing stops the job with the record RUNNING, since the load is + // complete and nothing of it is in doubt (on Cassandra the count is a + // full `COUNT(*)`, which a node can time out), and the reconciler + // resumes the job once the record is stale. + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed("to count the target"), + )); + } + let count = count_target(&*storage, &target, &heartbeat, &env.shutdown).await?; + if count != total_items { + return Err(Stop::Fail(format!( + "restored {count} items, the manifest lists {total_items}: {}", + if count < total_items { + "the backup holds a key twice" + } else { + "the target holds items the backup does not account for" + } + ))); + } + + // Step 6. The fenced-out kill point stops the heartbeat first, so a + // reconciler can take the record over while the job pauses, and the + // pre-write check is skipped so the conditional write itself is what + // the test observes. + let fenced = kill_point_named(KillPoint::RestoreFencedOut); + if fenced { + heartbeat.stop_timer(); + } + kill_point(KillPoint::RestoreFencedOut, &env.shutdown).await; + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !fenced && !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed("for the DONE write"), + )); + } + // The restored table's ItemCount and TableSizeBytes are 0 at ACTIVE, + // as the service reports them (measured: 0 and 0 on 15 restored + // tables, one of them 10 minutes after it went ACTIVE); the + // statistics pass rewrites them later where the backend has one. + finish(&*storage, &*claim, &target, 0, 0).await + } + .await; + let elapsed = started.elapsed().as_secs_f64(); + let (outcome, remove_entry) = match end { + Ok(Ended::Written(Finished::Active)) => { + tracing::info!( + "{name}: ACTIVE, {total_items} items ({applied_total} applied by this run), \ + {bytes_downloaded} bytes downloaded, in {elapsed:.1} s" + ); + (BackupJobOutcome::Succeeded, true) + } + Ok(Ended::Written(Finished::Pending)) => { + tracing::info!( + "{name}: DONE, {total_items} items ({applied_total} applied by this run), \ + {bytes_downloaded} bytes downloaded, in {elapsed:.1} s; index backfill pending, \ + the backend's worker flips the table ACTIVE" + ); + kill_point(KillPoint::RestoreMongodbBeforeWorkerFlip, &env.shutdown).await; + (BackupJobOutcome::Succeeded, true) + } + // The write landed and its answer did not come back: the table is + // ACTIVE, or awaits the backend's worker, which this job did not see. + Ok(Ended::DoneOnReRead) => { + tracing::info!( + "{name}: complete (DONE), {total_items} items ({applied_total} applied by this \ + run), {bytes_downloaded} bytes downloaded, in {elapsed:.1} s; the DONE write's \ + answer was lost and the record was re-read DONE under this job's generation (the \ + table is ACTIVE, or the backend's worker flips it)" + ); + (BackupJobOutcome::Succeeded, true) + } + // The record is another's: its owner may be reading this host's + // entry, so the entry is left to the sweep. + Ok(Ended::Written(Finished::NotMatched(why))) => { + tracing::error!( + "{name}: the DONE write matched no record ({why}); the record is not this job's \ + to write and no table access was made" + ); + (BackupJobOutcome::Failed, false) + } + Err(Stop::ClaimLost(why)) => { + tracing::error!( + "{name}: stopped with no further catalog or table access: {why}; the record is \ + left to its owner or to the reconciler" + ); + (BackupJobOutcome::Failed, false) + } + // The load is complete and a read after it kept failing: no FAILED + // write, the record stays RUNNING with its progress for a resume. + Err(Stop::LeftRunning(why)) => { + tracing::error!( + "{name}: stopped with the record RUNNING and no FAILED written: {why}; the \ + reconciler resumes it once the record is stale" + ); + (BackupJobOutcome::Failed, false) + } + Err(Stop::Cancelled) => { + tracing::info!( + "{name}: stopped by the server's shutdown; the record stays RUNNING with its \ + spool and its progress for a resume" + ); + (BackupJobOutcome::Cancelled, false) + } + Err(Stop::Fail(reason)) => { + let path = fail_restore( + &*storage, + &*claim, + &target, + &reason, + entry.as_ref(), + &env.shutdown, + ) + .await; + tracing::error!("{name}: FAILED ({reason}); {path}"); + // `fail_restore` removed the entry, or left it to the sweep. + (BackupJobOutcome::Failed, false) + } + }; + if remove_entry && let Some(entry) = &entry { + remove_spool_entry(entry).await; + } + // After the failure path too, so a FAILED job's duration covers its + // FAILED write and the target's removal. + let duration = started.elapsed(); + drop(heartbeat); + if let Some(metrics) = metrics { + #[allow(clippy::cast_precision_loss)] + let duration_us = duration.as_micros() as f64; + metrics.record_restore_job(outcome, duration_us); + match outcome { + BackupJobOutcome::Succeeded => { + metrics.record_worker_success(QuerySource::RestoreJob, duration_us); + } + BackupJobOutcome::Failed => metrics.record_worker_error(QuerySource::RestoreJob), + // A clean shutdown is not a worker error. + BackupJobOutcome::Cancelled => {} + } + } + outcome +} + +/// How many times the final count is attempted before the job stops with +/// the record RUNNING. +const COUNT_ATTEMPTS: u32 = 3; + +/// The final count (module doc): `count_restore_target`, retried after a +/// short wait on an error, [`COUNT_ATTEMPTS`] times in all, the shutdown +/// token and the heartbeat read before each attempt; an error on the last +/// attempt is [`Stop::LeftRunning`], never a FAILED write, since every item +/// is loaded and recorded and the reconciler's resume counts again. +async fn count_target( + storage: &S, + target: &RestoreTargetKey, + heartbeat: &HeartbeatHandle, + shutdown: &CancellationToken, +) -> Result { + for attempt in 1..=COUNT_ATTEMPTS { + if attempt > 1 { + if shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed("to count the target again"), + )); + } + } + match storage.count_restore_target(target).await { + Ok(count) => return Ok(count), + Err(e) if attempt == COUNT_ATTEMPTS => { + return Err(Stop::LeftRunning(format!( + "counting the target's items failed {COUNT_ATTEMPTS} times, the last: {e}" + ))); + } + Err(e) => { + let wait = Duration::from_secs(u64::from(attempt)); + tracing::warn!( + "counting the target's items failed (attempt {attempt} of \ + {COUNT_ATTEMPTS}); retried in {} s: {e}", + wait.as_secs() + ); + tokio::select! { + biased; + () = shutdown.cancelled() => return Err(Stop::Cancelled), + () = heartbeat.lost().cancelled() => return Err(Stop::ClaimLost(heartbeat.reason())), + () = tokio::time::sleep(wait) => {} + } + } + } + } + unreachable!("the attempt loop returns on its last attempt") +} + +/// What step 6 settled to. +enum Ended { + /// The backend's own answer to the DONE write. + Written(Finished), + /// The DONE write's answer was lost and the record re-read DONE under + /// this generation: the restore is complete; whether the table is ACTIVE + /// or awaits the backend's worker (MongoDB) is a fact the backend holds + /// and this job did not see. + DoneOnReRead, +} + +/// Step 6's write (module doc): `finish_restore`, with an error from it +/// handled as a conditional write whose outcome is unknown. The record is +/// re-read through the claim: DONE under this generation is the write +/// landed ([`Ended::DoneOnReRead`], with a WARN naming the error), RUNNING +/// under this generation is the write not taken, sent once more at most; +/// another generation, another status, a failed re-read, or a second error +/// after a RUNNING re-read is the DONE write that matched no record, after +/// which the job makes no table access. +async fn finish( + storage: &S, + claim: &dyn JobClaim, + target: &RestoreTargetKey, + item_count: u64, + size_bytes: u64, +) -> Result { + let not_matched = |why: String| Ok(Ended::Written(Finished::NotMatched(why))); + for attempt in 0..2 { + let error = match storage + .finish_restore(target, claim, item_count, size_bytes) + .await + { + Ok(finished) => return Ok(Ended::Written(finished)), + Err(e) => e, + }; + let row = match claim.read().await { + Ok(row) => row, + Err(e) => { + return not_matched(format!( + "the DONE write failed ({error}) and the record could not be re-read: {e}" + )); + } + }; + if row.owner_generation != claim.generation() { + return not_matched(format!( + "the DONE write failed ({error}) and the record carries generation {} (this \ + claim's is {})", + row.owner_generation, + claim.generation() + )); + } + if row.status == JobStatus::Done { + tracing::warn!( + "the DONE write returned an error ({error}) and the record is DONE under this \ + job's generation: the write landed and the restore is complete" + ); + return Ok(Ended::DoneOnReRead); + } + if row.status != JobStatus::Running { + return not_matched(format!( + "the DONE write failed ({error}) and the record is {} under this claim", + row.status + )); + } + if attempt == 1 { + return not_matched(format!( + "the DONE write failed twice ({error}) and the record is still RUNNING under \ + this claim; the reconciler resumes it once the record is stale" + )); + } + tracing::warn!( + "the DONE write returned an error ({error}) and the record is still RUNNING under \ + this claim; writing once more" + ); + } + unreachable!("the loop returns on its second attempt") +} + +/// The end of the restore's failure path. +#[derive(Debug)] +pub(crate) enum RestoreFailurePath { + /// FAILED written and the target removed. + Removed, + /// FAILED written; the target was no longer a CREATING restore target + /// with this id and belongs to whoever moved it. + LeftToOwner, + /// FAILED written; the removal failed and the reconciler's FAILED-target + /// duty removes the target once the record is stale. + RemoveFailed(String), + /// FAILED not written by this claim (the record is another's, or the + /// claim connection failed): no table access was made. + NoTableAccess(String), +} + +impl std::fmt::Display for RestoreFailurePath { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Removed => f.write_str("the target was removed"), + Self::LeftToOwner => f.write_str( + "the target was no longer a CREATING restore target with this id and is left to \ + its owner", + ), + Self::RemoveFailed(e) => write!( + f, + "the target could not be removed and the reconciler removes it once the record \ + is stale: {e}" + ), + Self::NoTableAccess(why) => write!( + f, + "the record is not this job's to write ({why}); no table access was made" + ), + } + } +} + +/// Step 7 (module doc): FAILED with `reason` through `claim`, then +/// `remove_restore_target` only when that write matched; the spool entry, +/// when there is one, removed when the write matched and left to the sweep +/// when it did not (the record is another's, whose job may be reading the +/// entry). +pub(crate) async fn fail_restore( + storage: &S, + claim: &dyn JobClaim, + target: &RestoreTargetKey, + reason: &str, + entry: Option<&spool::Entry>, + shutdown: &CancellationToken, +) -> RestoreFailurePath { + let fields = StatusFields { + failure_reason: Some(reason.to_owned()), + completed: true, + ..StatusFields::default() + }; + let path = match write_status(claim, JobStatus::Running, JobStatus::Failed, &fields).await { + Written::Matched => { + kill_point(KillPoint::RestoreBetweenFailedAndRemoval, shutdown).await; + match storage.remove_restore_target(target).await { + Ok(true) => RestoreFailurePath::Removed, + Ok(false) => RestoreFailurePath::LeftToOwner, + Err(e) => RestoreFailurePath::RemoveFailed(e.to_string()), + } + } + Written::NotMatched(why) | Written::Error(why) => RestoreFailurePath::NoTableAccess(why), + }; + if let (Some(entry), false) = (entry, matches!(path, RestoreFailurePath::NoTableAccess(_))) { + remove_spool_entry(entry).await; + } + path +} + +async fn remove_spool_entry(entry: &spool::Entry) { + if let Err(e) = entry.remove().await { + tracing::error!( + "the spool entry {} could not be removed; the reconciler's sweep retries: {e}", + entry.dir().display() + ); + } +} + +/// A dead owner's record as the reconciler hands it to [`resume`]. +pub(crate) struct DeadRecord { + /// The record as read under the takeover's claim. + pub row: RestoreJobRow, + /// The `spool_host` the record carried before the takeover (which wrote + /// this host's): whose spool entry, if any, holds the files. + pub recorded_spool_host: Option, + /// The progress rows a backend keeps apart from the record. + pub progress_rows: Vec<(u32, u64)>, + /// The instant `claim_restore_dead` returned: the resumed job's first + /// confirmed beat (module doc). + pub claimed_at: Instant, +} + +/// The reconciler's entry: a dead owner's RUNNING record taken over +/// (`claim`), resumed from its progress (the record and, on PostgreSQL, its +/// progress rows). The manifest and sidecar are read from this host's spool +/// entry when the record named this host before the takeover and both are +/// there and verify, else from the store; either way they are verified and +/// must name the record's backup. A manifest that cannot be read or is +/// refused fails the restore (step 7), and this host's entry, when the +/// record named this host, goes with it. The first confirmed beat is +/// `record.claimed_at`, the takeover's return. +pub(crate) async fn resume( + storage: Arc, + env: JobEnv, + permit: JobPermit, + claim: Box, + record: DeadRecord, + limits: RestorableLimits, +) where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let started = Instant::now(); + let claim: Arc = Arc::from(claim); + let DeadRecord { + row, + recorded_spool_host, + progress_rows, + claimed_at, + } = record; + let target = row.target.clone(); + let name = format!( + "restore of {} into {} (account {})", + row.source_backup_arn, target.table_name, target.account_id + ); + // This host's entry, when the record named this host before the + // takeover: the spooled documents are read from it, and a resume that + // fails before it runs removes it with the failure path, as the job + // removes its entry at every failed end. An entry that cannot be opened + // is left to the sweep and the store's documents are read. + let entry = env + .spool + .as_ref() + .filter(|spool| recorded_spool_host.as_deref() == Some(spool.spool_host.as_str())) + .and_then(|spool| { + spool::Entry::open( + &spool.path, + &target.table_id, + recorded_spool_host.as_deref(), + &spool.spool_host, + ) + .ok() + }); + let prepared = prepare_resume(&env, &row, entry.as_ref(), &progress_rows).await; + let (arn, manifest, manifest_bytes, sidecar_bytes, key_schema, progress, from_spool) = + match prepared { + Ok(prepared) => prepared, + Err(reason) => { + let path = fail_restore( + &*storage, + &*claim, + &target, + &reason, + entry.as_ref(), + &env.shutdown, + ) + .await; + tracing::error!("{name}: FAILED on resume ({reason}); {path}"); + if let Some(metrics) = env.metrics.as_deref() { + metrics.record_worker_error(QuerySource::RestoreJob); + #[allow(clippy::cast_precision_loss)] + let duration_us = started.elapsed().as_micros() as f64; + metrics.record_restore_job(BackupJobOutcome::Failed, duration_us); + } + drop(permit); + return; + } + }; + tracing::info!( + "{name}: resumed by this instance; the manifest was read from {}", + if from_spool { + "this host's spool" + } else { + "the store" + } + ); + let job = RestoreJob { + storage, + claim, + target, + arn, + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress: Some(progress), + recorded_spool_host, + documents_from_spool: from_spool, + }; + run(job).await; + drop(permit); +} + +/// What a resume reads before it runs: the manifest and sidecar (from this +/// host's spool entry when there is one and the spooled pair verifies, else +/// the store), verified, with the key schema and the progress, and whether +/// the pair came from the spool; the `Err` is the FAILED reason. +type Prepared = ( + BackupArn, + ExtenddbManifest, + Vec, + Vec, + RestorableKeySchema, + Progress, + bool, +); + +/// The manifest of a (manifest, sidecar) pair that verifies and names +/// `arn`; the `Err` says what failed. +fn verified_manifest( + manifest_bytes: &[u8], + sidecar_bytes: &[u8], + arn: &BackupArn, +) -> Result { + verify_manifest_checksum(manifest_bytes, sidecar_bytes) + .map_err(|e| format!("the manifest's checksum sidecar does not match: {e}"))?; + let manifest = ExtenddbManifest::from_json_bytes(manifest_bytes) + .map_err(|e| format!("the manifest is refused: {e}"))?; + match manifest.backup_arn() { + Ok(named) if named == *arn => Ok(manifest), + Ok(named) => Err(format!( + "the manifest names another backup, {}", + named.to_arn_string() + )), + Err(e) => Err(format!("the manifest's backup ARN is refused: {e}")), + } +} + +async fn prepare_resume( + env: &JobEnv, + row: &RestoreJobRow, + entry: Option<&spool::Entry>, + progress_rows: &[(u32, u64)], +) -> Result { + let target = &row.target; + let arn = BackupArn::parse(&row.source_backup_arn) + .map_err(|e| format!("the record's backup ARN is refused: {e}"))?; + let prefix = arn.store_prefix(); + let spooled = entry.and_then(|entry| { + let manifest = entry.read_document(EXTENDDB_MANIFEST_FILE).ok()??; + let sidecar = entry.read_document(MANIFEST_CHECKSUM_FILE).ok()??; + Some((manifest, sidecar)) + }); + // The spooled pair, when it verifies; a pair a crash left short, or + // that names another backup, is replaced by the store's. + let spooled = spooled.and_then(|(manifest_bytes, sidecar_bytes)| { + match verified_manifest(&manifest_bytes, &sidecar_bytes, &arn) { + Ok(manifest) => Some((manifest, manifest_bytes, sidecar_bytes)), + Err(e) => { + tracing::warn!( + "the spooled manifest and sidecar of the restore into {} (account {}) do not \ + verify ({e}); the store's are read instead", + target.table_name, + target.account_id + ); + None + } + } + }); + let from_spool = spooled.is_some(); + let (manifest, manifest_bytes, sidecar_bytes) = match spooled { + Some(documents) => documents, + None => { + let read = async |name: &str, bound: u64| -> Result, String> { + match super::reconciler::read_object(&*env.store, &format!("{prefix}{name}"), bound) + .await + { + Ok(ReadObject::Bytes(bytes)) => Ok(bytes), + Ok(ReadObject::Absent) => { + Err(format!("the store holds no {name} for the backup")) + } + Ok(ReadObject::Oversize) => Err(format!( + "the store's {name} for the backup is over {bound} bytes, which no \ + backup this server wrote produces" + )), + Err(e) => { + if let Some(metrics) = env.metrics.as_deref() { + metrics.record_backup_store_error("get"); + } + Err(format!("reading {name} from the store: {e}")) + } + } + }; + let manifest_bytes = read(EXTENDDB_MANIFEST_FILE, MANIFEST_READ_BOUND).await?; + let sidecar_bytes = read(MANIFEST_CHECKSUM_FILE, SIDECAR_READ_BOUND).await?; + let manifest = verified_manifest(&manifest_bytes, &sidecar_bytes, &arn)?; + (manifest, manifest_bytes, sidecar_bytes) + } + }; + let key_schema = RestorableKeySchema::resolve( + &manifest.source_table.key_schema, + &manifest.source_table.attribute_definitions, + ) + .map_err(|e| format!("the key schema cannot be checked: {e}"))?; + let progress = Progress::from_records(row, progress_rows, &manifest); + Ok(( + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + progress, + from_spool, + )) +} + +/// What a test can do to a running job by its target table id: end its +/// heartbeat timer without raising the loss (`HeartbeatHandle::stop_timer`) +/// while the job is held at a gate of the storage double, and give its +/// loads a smaller chunk byte bound than [`load::LOAD_CHUNK_BYTES`]. Tests +/// that use it give their target a name of their own; the maps are +/// process-wide. +#[cfg(test)] +pub(crate) mod test_support { + use std::collections::HashMap; + use std::sync::{LazyLock, Mutex, PoisonError}; + + use super::HeartbeatHandle; + + static TIMERS: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + static CHUNK_BYTES: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + + /// Give the loads of the job into `table_id` a chunk byte bound of + /// `bytes`. + pub(crate) fn set_chunk_bytes(table_id: &str, bytes: u64) { + CHUNK_BYTES + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(table_id.to_owned(), bytes); + } + + pub(super) fn chunk_bytes_for(table_id: &str) -> u64 { + CHUNK_BYTES + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(table_id) + .copied() + .unwrap_or(super::load::LOAD_CHUNK_BYTES) + } + + pub(super) fn register_timer(table_id: &str, heartbeat: &HeartbeatHandle) { + TIMERS + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(table_id.to_owned(), heartbeat.timer_abort_handle()); + } + + /// End the heartbeat timer of the running job into `table_id`. + pub(crate) fn stop_timer(table_id: &str) { + let handle = TIMERS + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(table_id) + .cloned() + .expect("a job into the table is running"); + handle.abort(); + } +} diff --git a/crates/engine/src/backup/restore/download.rs b/crates/engine/src/backup/restore/download.rs new file mode 100644 index 000000000..7cb42273f --- /dev/null +++ b/crates/engine/src/backup/restore/download.rs @@ -0,0 +1,320 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Step 3 of a restore: every data file read from the store and compared +//! with the manifest's `sha256`, `md5` and `bytes` before any file is +//! loaded. +//! +//! On an S3 store each file is written to the job's spool entry as it +//! arrives ([`Dest::Spool`]), and committed with its `.verified` marker once +//! the digests match; a resume skips the files an earlier run marked (the +//! load hashes the spooled bytes again as it decodes them, so a marker is a +//! reason not to download, not a reason not to check). On a +//! filesystem store ([`Dest::InPlace`]) the file is hashed as the store +//! streams it and nothing is written: the load reads the same file through +//! a second `get`, hashing it again, so a file replaced between the two +//! passes fails there. +//! +//! A `get` that fails as a transport error, at the call or while the body +//! arrives, is retried with a backoff of `min(2^n s, 60 s)` plus jitter for +//! as long as the job has been downloading for less than +//! [`DOWNLOAD_RETRY_BOUND`], counted from the first download attempt of the +//! job (one clock for every file, so a store that flaps through a backup of +//! many files costs the restore the bound once, not once per file), the +//! record RUNNING and heartbeated meanwhile; the shutdown token and the +//! heartbeat's loss end the wait. Every other failure fails the restore at +//! once: a digest or size mismatch (the file is not the backup's), an absent +//! file (a prefix changed under an AVAILABLE row), a permission or key +//! refusal, a local I/O error (the filesystem store's `Io`, or the spool's +//! disk), or any other store error. The filesystem store reports a mount +//! that stopped answering as `Io`, so the retry covers S3 stores only. +//! Every `get` the store fails, retried or not, is one `BackupStoreErrors{get}`; +//! a mismatch and the spool's own errors are not the store's. + +use std::time::Duration; + +use extenddb_backup_store::{BackupStore, StoreError}; +use extenddb_core::backup_format::{CompressedHasher, DataFileEntry, ExtenddbManifest}; +use extenddb_core::metrics::MetricsCollector; +use extenddb_storage::CancellationToken; +use futures::TryStreamExt; +use tokio::io::AsyncWriteExt; +use tokio::time::Instant; + +use super::super::job::heartbeat::HeartbeatHandle; +use super::spool::Entry; + +/// How long the job's downloads are retried over transport errors, from +/// the job's first download attempt, before the restore fails. +pub(crate) const DOWNLOAD_RETRY_BOUND: Duration = Duration::from_secs(15 * 60); + +/// The longest wait between two attempts. +const MAX_BACKOFF: Duration = Duration::from_secs(60); + +/// Where a verified file goes. +pub(crate) enum Dest<'a> { + /// Hashed as it streams, written nowhere: the load reads the store + /// again (a filesystem store). + InPlace, + /// Written to the job's spool entry and marked `.verified`. + Spool(&'a Entry), +} + +/// Why the download stopped. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Stop { + /// The restore fails with this reason. + Fail(String), + /// The shutdown token fired. + Cancelled, + /// The heartbeat no longer confirms the claim. + ClaimLost(String), + /// The job stops with the record RUNNING and no FAILED written (the + /// final count kept failing after every item was loaded and recorded); + /// the reconciler resumes it once the record is stale. + LeftRunning(String), +} + +/// What one download runs with. +pub(crate) struct Downloader<'a> { + pub store: &'a dyn BackupStore, + /// The backup's store prefix, which every key is under. + pub prefix: &'a str, + pub heartbeat: &'a HeartbeatHandle, + pub shutdown: &'a CancellationToken, + pub metrics: Option<&'a MetricsCollector>, +} + +/// Download and verify every data file of `manifest` into `dest` (module +/// doc), except the files `complete` marks (a resume's progress records +/// them applied whole, so they are neither downloaded nor loaded) and the +/// files the spool holds verified with the manifest's digest. Returns the +/// bytes read from the store over every attempt. +pub(crate) async fn download_all( + d: &Downloader<'_>, + manifest: &ExtenddbManifest, + dest: &Dest<'_>, + complete: &[bool], +) -> Result { + // The retry bound's clock: one for the job, started here. + let started = Instant::now(); + let mut bytes = 0; + for (index, entry) in manifest.data_files.iter().enumerate() { + if complete.get(index).copied().unwrap_or(false) { + continue; + } + let index = + u32::try_from(index).map_err(|_| Stop::Fail("too many data files".to_owned()))?; + if let Dest::Spool(spool) = dest + && spool.is_verified(index, &entry.sha256) + { + continue; + } + bytes += fetch_verified(d, entry, index, dest, started).await?; + } + Ok(bytes) +} + +/// One file: attempts until the file verifies, a non-transport failure, the +/// retry bound (measured from `started`, the job's first attempt), the +/// shutdown or a lost claim. +async fn fetch_verified( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + dest: &Dest<'_>, + started: Instant, +) -> Result { + let key = format!("{}{}", d.prefix, entry.key); + let mut bytes = 0; + for attempt in 0u32.. { + if d.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !d.heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + d.heartbeat + .why_not_confirmed(&format!("to download {}", entry.key)), + )); + } + let outcome = tokio::select! { + biased; + () = d.shutdown.cancelled() => return Err(Stop::Cancelled), + () = d.heartbeat.lost().cancelled() => return Err(Stop::ClaimLost(d.heartbeat.reason())), + outcome = attempt_once(d, entry, index, &key, dest) => outcome, + }; + bytes += outcome.bytes; + if let Some(metrics) = d.metrics + && outcome.bytes > 0 + { + metrics.record_restore_bytes_downloaded(outcome.bytes); + } + let transport = match outcome.result { + Ok(()) => return Ok(bytes), + Err(Attempt::Fail(reason)) => return Err(Stop::Fail(reason)), + Err(Attempt::StoreFailed(reason)) => { + // The store's refusal counts as its error, as a transport + // failure does; the restore then fails at once. + if let Some(metrics) = d.metrics { + metrics.record_backup_store_error("get"); + } + return Err(Stop::Fail(reason)); + } + Err(Attempt::Transport(e)) => e, + }; + if let Some(metrics) = d.metrics { + metrics.record_backup_store_error("get"); + } + let elapsed = started.elapsed(); + if elapsed >= DOWNLOAD_RETRY_BOUND { + return Err(Stop::Fail(format!( + "downloading {}: {transport} (the job has retried its downloads for {} s; the \ + bound is {} s)", + entry.key, + elapsed.as_secs(), + DOWNLOAD_RETRY_BOUND.as_secs() + ))); + } + let wait = backoff(attempt); + tracing::warn!( + "downloading {key}: {transport}; attempt {} retried in {:.1} s", + attempt + 2, + wait.as_secs_f64() + ); + tokio::select! { + biased; + () = d.shutdown.cancelled() => return Err(Stop::Cancelled), + () = d.heartbeat.lost().cancelled() => return Err(Stop::ClaimLost(d.heartbeat.reason())), + () = tokio::time::sleep(wait) => {} + } + } + unreachable!("the attempt loop returns") +} + +/// `min(2^attempt s, 60 s)` plus a jitter of up to one second. +fn backoff(attempt: u32) -> Duration { + let base = Duration::from_secs(1u64 << attempt.min(6)); + let jitter_ms = u64::from(uuid::Uuid::new_v4().as_u128() as u32) % 1000; + base.min(MAX_BACKOFF) + Duration::from_millis(jitter_ms) +} + +/// How one attempt ended. +enum Attempt { + /// The restore fails with this reason; no retry, and not the store's + /// error (a digest or size mismatch, the spool's disk). + Fail(String), + /// The store failed the `get` with an error that is not retried (an + /// absent file, a refusal, an I/O error, any other); the restore fails + /// with this reason and the error is counted as the store's. + StoreFailed(String), + /// Retried within the bound. + Transport(StoreError), +} + +struct AttemptOutcome { + result: Result<(), Attempt>, + /// Bytes this attempt read from the store. + bytes: u64, +} + +/// One `get` of the file, hashed and compared, written to the spool when +/// `dest` is one. +async fn attempt_once( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + key: &str, + dest: &Dest<'_>, +) -> AttemptOutcome { + let mut bytes = 0; + let result = attempt_inner(d, entry, index, key, dest, &mut bytes).await; + AttemptOutcome { result, bytes } +} + +async fn attempt_inner( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + key: &str, + dest: &Dest<'_>, + bytes: &mut u64, +) -> Result<(), Attempt> { + let mut body = match d.store.get(key).await { + Ok(body) => body, + Err(e @ StoreError::Transport { .. }) => return Err(Attempt::Transport(e)), + Err(StoreError::NotFound) => { + return Err(Attempt::StoreFailed(format!( + "data file {} is not in the store", + entry.key + ))); + } + Err(e) => { + return Err(Attempt::StoreFailed(format!( + "reading data file {} from the store: {e}", + entry.key + ))); + } + }; + let mut file = match dest { + Dest::InPlace => None, + Dest::Spool(spool) => Some(spool.begin_file(index).await.map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?), + }; + let mut hasher = CompressedHasher::new(); + loop { + let chunk = match body.try_next().await { + Ok(Some(chunk)) => chunk, + Ok(None) => break, + Err(e @ StoreError::Transport { .. }) => return Err(Attempt::Transport(e)), + Err(e) => { + return Err(Attempt::StoreFailed(format!( + "reading data file {} from the store: {e}", + entry.key + ))); + } + }; + *bytes += chunk.len() as u64; + hasher.update(&chunk); + if let Some(file) = &mut file { + file.write_all(&chunk).await.map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?; + } + } + entry + .expectations() + .verify_digest(&hasher.finish()) + .map_err(|e| { + Attempt::Fail(format!( + "data file {} does not match the manifest: {e}", + entry.key + )) + })?; + if let (Some(file), Dest::Spool(spool)) = (file, dest) { + spool + .commit_file(index, file, &entry.sha256) + .await + .map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_backoff_doubles_to_a_minute_with_jitter() { + for (attempt, base) in [(0, 1), (1, 2), (3, 8), (5, 32), (6, 64), (20, 64)] { + let wait = backoff(attempt); + let base = Duration::from_secs(base).min(MAX_BACKOFF); + assert!( + wait >= base && wait < base + Duration::from_secs(1), + "{attempt}: {wait:?}" + ); + } + } +} diff --git a/crates/engine/src/backup/restore/load.rs b/crates/engine/src/backup/restore/load.rs new file mode 100644 index 000000000..3e5de40d4 --- /dev/null +++ b/crates/engine/src/backup/restore/load.rs @@ -0,0 +1,285 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Step 5 of a restore: one verified data file decoded, validated and +//! handed to the backend's `bulk_load`. +//! +//! The decoder is synchronous (`std::io::Read`), so it runs on the blocking +//! pool: over the spooled file, or, on a filesystem store, over a +//! second `get` of the store bridged to `Read`, in both cases through +//! `ExtenddbManifest::reader`, which hashes the bytes again as it decodes +//! them: a spooled file's `.verified` marker says the download matched the +//! manifest, not that the file in the spool still does (a resume on the +//! same host reads what the spool holds now). On both paths the +//! items are handed to `bulk_load` as they decode and the digest is checked +//! at the end of the stream, so the items of a file replaced between the two +//! passes reach the backend (and, on a backend that commits per batch, the +//! CREATING target) before the end-of-stream check fails the file and the +//! failure path removes the target; the store root's and the spool's +//! owner-only modes are what keep the file from changing. Each item past the resume's `start_at` is +//! validated against the target's limits and key schema +//! (`validate_restorable_item`), then collected into chunks of at most +//! [`LOAD_CHUNK_ITEMS`] items or [`LOAD_CHUNK_BYTES`] of line text (the +//! bytes the decoder read for the items, the unit the backup side budgets +//! in), whichever comes first, and sent over a channel of two chunks to the +//! async side, which flattens them into the item stream of one `bulk_load` +//! call per file. The memory a file's load holds is therefore the decoded +//! items of at most `LOAD_CHUNK_BYTES` plus one line of line text per chunk +//! (the line that crosses the bound is kept in the chunk it was read for) at +//! four chunk positions (two in the channel, the one being filled, the one +//! being drained), the decoder's line (at most `MAX_LINE_BYTES`) and the +//! deserializer's intermediate value for it, and the backend's own batch. +//! +//! A decode or validation failure, the shutdown token and the heartbeat +//! (its loss, or a timer that no longer confirms the claim, both read at +//! every chunk boundary) end the stream with an `Err` item, which the +//! backend answers by failing the file (the design: nothing of a failed file +//! is recorded where the backend writes a file in one transaction); the +//! reason is kept here so the job reports it rather than the backend's +//! wrapping of it. A decoder that stops without ending the stream itself (a +//! panic on the blocking thread) fails the file too: a clean end of stream +//! the backend saw is not taken for a complete file. The join of the +//! decoder's thread after `bulk_load` returned has no bound of its own: on +//! the in-place path the thread reads the store through the bridge, and a +//! filesystem mount that stopped answering holds it as it holds the store's +//! own `get`, until the mount answers. + +use std::io::Read; +use std::path::PathBuf; +use std::sync::{Arc, Mutex, PoisonError}; + +use extenddb_backup_store::BackupStore; +use extenddb_core::backup_format::{DataFileEntry, ExtenddbManifest}; +use extenddb_core::types::Item; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits, validate_restorable_item}; +use extenddb_storage::backup_catalog::RestoreTargetKey; +use extenddb_storage::error::StorageError; +use extenddb_storage::{BulkLoadFile, CancellationToken, JobClaim, RestoreEngine}; +use futures::{StreamExt, TryStreamExt}; +use tokio::sync::mpsc; + +use super::super::job::heartbeat::{ClaimCheck, HeartbeatHandle}; +pub(crate) use super::download::Stop; + +/// Items per chunk handed to the backend, at most. +pub(crate) const LOAD_CHUNK_ITEMS: usize = 1000; +/// Line bytes per chunk (the data file's text the decoder read for the +/// chunk's items), at most; a chunk closes at the first item that takes it +/// to this. +pub(crate) const LOAD_CHUNK_BYTES: u64 = 32 * 1024 * 1024; +/// Chunks the channel between the decoder and the backend holds. +const CHUNK_CHANNEL: usize = 2; + +/// Where the file's bytes are read from. +pub(crate) enum Source { + /// The spooled, verified file. + Spool(PathBuf), + /// The store again (a filesystem store, read in place). + InPlace { + store: Arc, + key: String, + }, +} + +/// One file's load. +pub(crate) struct FileLoad<'a> { + pub manifest: &'a Arc, + pub entry: &'a DataFileEntry, + pub file_index: u32, + /// Items of the file an earlier run applied, decoded and skipped. + pub start_at: u64, + pub reapplying: bool, + pub key_schema: &'a RestorableKeySchema, + pub limits: RestorableLimits, + pub shutdown: &'a CancellationToken, + pub heartbeat: &'a HeartbeatHandle, + /// The chunk's byte bound: [`LOAD_CHUNK_BYTES`], or a smaller bound a + /// test set for its job. + pub chunk_bytes: u64, +} + +/// Decode `source` from `load.start_at` on, validate every item, and write +/// the file through `storage.bulk_load` (module doc). Returns the items the +/// backend applied. +pub(crate) async fn load_file( + storage: &S, + target: &RestoreTargetKey, + claim: &dyn JobClaim, + load: FileLoad<'_>, + source: Source, +) -> Result { + let key = load.entry.key.clone(); + let read: Box = match source { + Source::Spool(path) => { + let file = std::fs::File::open(&path).map_err(|e| { + Stop::Fail(format!( + "opening data file {key} in the spool ({}): {e}", + path.display() + )) + })?; + Box::new(file) + } + Source::InPlace { + store, + key: store_key, + } => { + let body = store + .get(&store_key) + .await + .map_err(|e| Stop::Fail(format!("reading data file {key} from the store: {e}")))?; + let reader = tokio_util::io::StreamReader::new(body.map_err(std::io::Error::other)); + let bridge = tokio_util::io::SyncIoBridge::new_with_handle( + reader, + tokio::runtime::Handle::current(), + ); + Box::new(bridge) + } + }; + let stop: Arc>> = Arc::new(Mutex::new(None)); + let (tx, rx) = mpsc::channel::, StorageError>>(CHUNK_CHANNEL); + let feeder = Feeder { + manifest: Arc::clone(load.manifest), + entry: load.entry.clone(), + start_at: load.start_at, + key_schema: load.key_schema.clone(), + limits: load.limits, + shutdown: load.shutdown.clone(), + claim: load.heartbeat.claim_check(), + chunk_bytes: load.chunk_bytes, + stop: Arc::clone(&stop), + tx, + }; + let decoder = tokio::task::spawn_blocking(move || feeder.run(read)); + let items = futures::stream::unfold(rx, |mut rx| async move { + rx.recv().await.map(|chunk| (chunk, rx)) + }) + .flat_map(|chunk| match chunk { + Ok(items) => futures::stream::iter(items.into_iter().map(Ok)).boxed(), + Err(e) => futures::stream::once(async move { Err(e) }).boxed(), + }) + .boxed(); + let file = BulkLoadFile { + file_index: load.file_index, + start_at: load.start_at, + reapplying: load.reapplying, + }; + let result = tokio::select! { + biased; + () = load.heartbeat.lost().cancelled() => Err(None), + result = storage.bulk_load(target, claim, file, items) => result.map_err(Some), + }; + // The stream is dropped with the call, so a decoder blocked on the + // channel returns; its thread is joined before the file is reported. A + // decoder that did not return (it panicked) dropped the channel without + // an `Err` item, so the backend saw a clean end of a short stream: the + // file fails here, where a backend that writes per file rolls it back. + if decoder.await.is_err() { + return Err(Stop::Fail(format!( + "data file {key}: the decoder stopped unexpectedly" + ))); + } + let stopped = stop.lock().unwrap_or_else(PoisonError::into_inner).take(); + match (result, stopped) { + (Ok(applied), None) => Ok(applied), + (Ok(_), Some(stop)) | (Err(_), Some(stop)) => Err(stop), + (Err(None), None) => Err(Stop::ClaimLost(load.heartbeat.reason())), + (Err(Some(e)), None) => Err(Stop::Fail(format!("data file {key}: {e}"))), + } +} + +/// The blocking side: the decoder, the validator and the chunking. +struct Feeder { + manifest: Arc, + entry: DataFileEntry, + start_at: u64, + key_schema: RestorableKeySchema, + limits: RestorableLimits, + shutdown: CancellationToken, + /// The heartbeat's pre-write check, read at every chunk boundary: the + /// loss raised, or a timer that no longer confirms the claim. + claim: ClaimCheck, + chunk_bytes: u64, + stop: Arc>>, + tx: mpsc::Sender, StorageError>>, +} + +impl Feeder { + fn run(self, read: Box) { + let key = self.entry.key.clone(); + let mut reader = match self.manifest.reader(&self.entry, read) { + Ok(reader) => reader, + Err(e) => { + self.end(Stop::Fail(format!("data file {key}: {e}"))); + return; + } + }; + let mut index: u64 = 0; + let mut chunk = Vec::new(); + // The line bytes of the items in `chunk`, from the decoder's count. + let mut chunk_bytes: u64 = 0; + let mut decoded_before = reader.bytes_decoded(); + loop { + match reader.next_item() { + Ok(Some(item)) => { + let line_bytes = reader.bytes_decoded() - decoded_before; + decoded_before = reader.bytes_decoded(); + index += 1; + if index <= self.start_at { + continue; + } + if let Err(e) = validate_restorable_item(&item, &self.key_schema, &self.limits) + { + self.end(Stop::Fail(format!( + "data file {key}: item cannot be restored: {e}" + ))); + return; + } + chunk_bytes += line_bytes; + chunk.push(item); + if chunk.len() >= LOAD_CHUNK_ITEMS || chunk_bytes >= self.chunk_bytes { + if self.shutdown.is_cancelled() { + self.end(Stop::Cancelled); + return; + } + if !self.claim.confirms_the_claim() { + self.end(Stop::ClaimLost( + "the claim was lost while the file was being decoded".to_owned(), + )); + return; + } + if self + .tx + .blocking_send(Ok(std::mem::take(&mut chunk))) + .is_err() + { + // The backend ended the call; nothing more to feed. + return; + } + chunk_bytes = 0; + } + } + Ok(None) => { + if !chunk.is_empty() { + let _ = self.tx.blocking_send(Ok(chunk)); + } + return; + } + Err(e) => { + self.end(Stop::Fail(format!("data file {key}: {e}"))); + return; + } + } + } + } + + /// Record why the stream ends and end it with an `Err` item. + fn end(&self, stop: Stop) { + let text = match &stop { + Stop::Fail(reason) => reason.clone(), + Stop::Cancelled => "the restore was stopped by the server's shutdown".to_owned(), + Stop::ClaimLost(why) | Stop::LeftRunning(why) => why.clone(), + }; + *self.stop.lock().unwrap_or_else(PoisonError::into_inner) = Some(stop); + let _ = self.tx.blocking_send(Err(StorageError::Internal(text))); + } +} diff --git a/crates/engine/src/backup/restore/spool.rs b/crates/engine/src/backup/restore/spool.rs new file mode 100644 index 000000000..d13730864 --- /dev/null +++ b/crates/engine/src/backup/restore/spool.rs @@ -0,0 +1,516 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The restore spool of an S3 store: where a restore job downloads the +//! backup's files before it loads any of them. +//! +//! One restore owns one entry, `restore-
    /` under the spool +//! directory the server took at startup (`backup_store::take_spool`: owner +//! only, locked). The entry is created `0700` and every file in it `0600` +//! (the database and its sidecars are `0600`; the spool holds the same +//! data). A data file is written as `.json.gz.partial`, synced, +//! renamed to `.json.gz`, and only then marked `.json.gz.verified`, +//! after the job has compared its digests with the manifest's; the marker +//! holds the file's sha256, and a file counts as verified only while the +//! marker names the digest the manifest in hand records for it, so a +//! replaced manifest does not have an old entry's files accepted. A +//! `.partial`, an unmarked file or a marker naming another digest is one a +//! crash or another manifest left, and the next run of the job removes it +//! and downloads the file again. The manifest and its sidecar are written to +//! the entry too, the same way (`.partial`, synced, renamed), so a +//! crash during the write leaves the previous document or none and never a +//! truncated one, and a resume on the same host reads them from the spool +//! and verifies them again (a pair that fails verification is replaced by +//! the store's). An entry is this host's only while the record's +//! `spool_host` is this host's: a successor on another host finds no entry +//! of its own, and one whose own old entry survives from an earlier run of +//! the job (the record moved to another host and back) removes it before +//! use ([`Entry::open`]). The job removes its entry when it ends by any +//! path but the server's shutdown, which keeps it for a resume, and a job +//! that learnt the record is another's (its claim lost, its final write +//! matching no record), which leaves it to the sweep; the reconciler's sweep +//! ([`list_entries`], then [`sweep_entries`]) removes the entries it listed +//! before it read the records whose job is not RUNNING with this host's +//! `spool_host`, so an entry opened after the listing is never a candidate. + +use std::collections::HashSet; +use std::io; +use std::path::{Path, PathBuf}; + +use tokio::io::AsyncWriteExt; + +/// The prefix of an entry's directory name, followed by the table id. +const ENTRY_PREFIX: &str = "restore-"; +/// The suffix of a file being downloaded. +const PARTIAL: &str = ".partial"; +/// The suffix of the marker of a file whose digests the job verified. +const VERIFIED: &str = ".verified"; + +/// One restore's directory in the spool. +pub(crate) struct Entry { + dir: PathBuf, + /// Make the rename into place fail, so a test can see what a crash + /// there leaves. + #[cfg(test)] + pub(crate) fail_rename: std::sync::atomic::AtomicBool, +} + +impl Entry { + /// The entry of the restore into table `table_id`, created `0700` when + /// it does not exist. An existing entry is a resume's and is kept when + /// `record_spool_host`, the `spool_host` the record carried before this + /// job's claim, is `this_host`; otherwise the files in it were verified + /// by a run whose record has since been another host's, and the entry + /// is removed and created again, empty. + pub(crate) fn open( + spool: &Path, + table_id: &str, + record_spool_host: Option<&str>, + this_host: &str, + ) -> io::Result { + let dir = spool.join(format!("{ENTRY_PREFIX}{table_id}")); + if dir.is_dir() && record_spool_host != Some(this_host) { + tracing::info!( + "the spool entry {} is left from a run whose record moved to another host; it is \ + removed and the files downloaded again", + dir.display() + ); + std::fs::remove_dir_all(&dir)?; + } + if !dir.is_dir() { + let mut builder = std::fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + match builder.create(&dir) { + Ok(()) => {} + Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {} + Err(e) => return Err(e), + } + } + Ok(Self { + dir, + #[cfg(test)] + fail_rename: std::sync::atomic::AtomicBool::new(false), + }) + } + + /// The entry's directory. + pub(crate) fn dir(&self) -> &Path { + &self.dir + } + + /// The path of data file `index` once downloaded and renamed. + pub(crate) fn file_path(&self, index: u32) -> PathBuf { + self.dir.join(format!("{index:06}.json.gz")) + } + + fn partial_path(&self, index: u32) -> PathBuf { + let mut name = self.file_path(index).into_os_string(); + name.push(PARTIAL); + PathBuf::from(name) + } + + fn verified_path(&self, index: u32) -> PathBuf { + let mut name = self.file_path(index).into_os_string(); + name.push(VERIFIED); + PathBuf::from(name) + } + + /// Whether data file `index` is in the entry with a `.verified` marker + /// naming `sha256`, the manifest's digest of the file: a file an earlier + /// run of the job downloaded and verified against the same manifest. + pub(crate) fn is_verified(&self, index: u32, sha256: &str) -> bool { + self.file_path(index).is_file() + && std::fs::read(self.verified_path(index)).is_ok_and(|marker| { + marker.strip_suffix(b"\n").unwrap_or(&marker) == sha256.as_bytes() + }) + } + + /// Write a document (the manifest, the sidecar) as `name` in the entry + /// the way a data file is written: `.partial`, `0600`, synced, + /// then renamed over `name` and the directory synced. An existing + /// document stands until the rename, so a crash during the write leaves + /// it whole or leaves the new one, never a truncated file. + pub(crate) async fn write_document(&self, name: &str, bytes: &[u8]) -> io::Result<()> { + let path = self.dir.join(name); + let mut partial = path.clone().into_os_string(); + partial.push(PARTIAL); + let partial = PathBuf::from(partial); + let mut file = open_private(&partial).await?; + file.write_all(bytes).await?; + file.sync_all().await?; + drop(file); + #[cfg(test)] + if self.fail_rename.load(std::sync::atomic::Ordering::Acquire) { + return Err(io::Error::other("the process stopped at the rename")); + } + tokio::fs::rename(&partial, &path).await?; + sync_dir(&self.dir).await + } + + /// The bytes of document `name`, `None` when the entry has no such file. + pub(crate) fn read_document(&self, name: &str) -> io::Result>> { + match std::fs::read(self.dir.join(name)) { + Ok(bytes) => Ok(Some(bytes)), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e), + } + } + + /// Start downloading data file `index`: whatever an earlier attempt + /// left (a `.partial`, an unmarked file, a stale marker) is removed, and + /// the `.partial` is created `0600` for the caller to write. + pub(crate) async fn begin_file(&self, index: u32) -> io::Result { + for path in [ + self.verified_path(index), + self.file_path(index), + self.partial_path(index), + ] { + remove_if_present(&path).await?; + } + open_private(&self.partial_path(index)).await + } + + /// Finish data file `index` after its digests matched: sync the + /// `.partial`, rename it into place, then create and sync the + /// `.verified` marker holding `sha256`, and sync the directory, in that + /// order, so the marker exists only beside a complete file. + pub(crate) async fn commit_file( + &self, + index: u32, + mut file: tokio::fs::File, + sha256: &str, + ) -> io::Result<()> { + file.flush().await?; + file.sync_all().await?; + drop(file); + self.rename_into_place(index).await?; + let mut marker = open_private(&self.verified_path(index)).await?; + marker.write_all(sha256.as_bytes()).await?; + marker.write_all(b"\n").await?; + marker.sync_all().await?; + drop(marker); + sync_dir(&self.dir).await + } + + /// The `.partial` renamed to the file's name. + async fn rename_into_place(&self, index: u32) -> io::Result<()> { + #[cfg(test)] + if self.fail_rename.load(std::sync::atomic::Ordering::Acquire) { + return Err(io::Error::other("the process stopped at the rename")); + } + tokio::fs::rename(self.partial_path(index), self.file_path(index)).await + } + + /// Remove the entry and everything in it; an entry already gone is not + /// an error. + pub(crate) async fn remove(&self) -> io::Result<()> { + match tokio::fs::remove_dir_all(&self.dir).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e), + } + } +} + +/// Create or truncate `path` with mode `0600`. +async fn open_private(path: &Path) -> io::Result { + let mut options = tokio::fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + options.mode(0o600); + } + options.open(path).await +} + +async fn remove_if_present(path: &Path) -> io::Result<()> { + match tokio::fs::remove_file(path).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e), + } +} + +/// Sync a directory so the entries it gained or lost are durable. +async fn sync_dir(dir: &Path) -> io::Result<()> { + tokio::fs::File::open(dir).await?.sync_all().await +} + +/// The table id of an entry's directory name, `None` for any other name in +/// the spool (the `.lock` file, a name an operator put there). +fn entry_table_id(name: &str) -> Option<&str> { + let id = name.strip_prefix(ENTRY_PREFIX)?; + (!id.is_empty() + && id + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_')) + .then_some(id) +} + +/// The entries under `spool`, as `(table id, path)`: the reconciler's +/// sweep lists them before it reads the RUNNING records, so an entry a job +/// opens after this listing (its record committed after the listing too) +/// is never a candidate of the sweep that follows. Other names in the spool +/// (the `.lock` file, a name an operator put there, a file under an +/// entry-shaped name) are not listed. +pub(crate) fn list_entries(spool: &Path) -> io::Result> { + let mut entries = Vec::new(); + for dir_entry in std::fs::read_dir(spool)? { + let dir_entry = dir_entry?; + let name = dir_entry.file_name(); + let Some(table_id) = name.to_str().and_then(entry_table_id) else { + continue; + }; + if !dir_entry.file_type()?.is_dir() { + continue; + } + entries.push((table_id.to_owned(), dir_entry.path())); + } + Ok(entries) +} + +/// The reconciler's sweep: remove every entry of `entries` (a +/// [`list_entries`] listing made before the records were read) whose table +/// id is not in `running` (the RUNNING restore targets whose `spool_host` is +/// this host's), and return the ids removed. An entry gone by the time of +/// its removal (its own job removed it between the listing and now) is done, +/// not an error, and not counted; one that cannot be removed is logged and +/// left for the next cycle. +pub(crate) fn sweep_entries( + entries: Vec<(String, PathBuf)>, + running: &HashSet, +) -> Vec { + let mut removed = Vec::new(); + for (table_id, path) in entries { + if running.contains(&table_id) { + continue; + } + match std::fs::remove_dir_all(&path) { + Ok(()) => { + tracing::info!( + "removed the spool entry of the restore into table id {table_id}: no \ + RUNNING restore of this host's spool names it" + ); + removed.push(table_id); + } + Err(e) if e.kind() == io::ErrorKind::NotFound => {} + Err(e) => tracing::error!( + "could not remove the spool entry of table id {table_id}; the next cycle \ + retries: {e}" + ), + } + } + removed +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + + /// A scratch directory of this test's own, removed when dropped. + pub(crate) struct Scratch(PathBuf); + + impl Scratch { + pub(crate) fn new() -> Self { + let dir = std::env::temp_dir().join(format!( + "extenddb-engine-test-{}", + uuid::Uuid::new_v4().simple() + )); + std::fs::create_dir(&dir).unwrap(); + Self(dir) + } + + pub(crate) fn path(&self) -> &Path { + &self.0 + } + } + + impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } + } + + #[cfg(unix)] + fn mode(path: &Path) -> u32 { + use std::os::unix::fs::PermissionsExt; + std::fs::metadata(path).unwrap().permissions().mode() & 0o777 + } + + /// The entry is `0700`, its files `0600`; a committed file has its + /// marker naming the digest, a `.partial` has none, a marker naming + /// another digest does not verify the file, and `begin_file` removes + /// what an earlier attempt left. + #[tokio::test] + async fn an_entry_is_private_and_a_marker_follows_the_rename() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-1", Some("host-a"), "host-a").unwrap(); + assert_eq!(mode(entry.dir()), 0o700); + entry + .write_document("extenddb-manifest.json", b"{}") + .await + .unwrap(); + assert_eq!(mode(&entry.dir().join("extenddb-manifest.json")), 0o600); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"{}".to_vec()) + ); + assert_eq!(entry.read_document("missing").unwrap(), None); + + let mut file = entry.begin_file(3).await.unwrap(); + file.write_all(b"bytes").await.unwrap(); + assert_eq!(mode(&entry.dir().join("000003.json.gz.partial")), 0o600); + assert!(!entry.is_verified(3, "abc")); + entry.commit_file(3, file, "abc").await.unwrap(); + assert!(entry.is_verified(3, "abc")); + assert!( + !entry.is_verified(3, "abd"), + "the marker binds the digest, not the index" + ); + assert_eq!( + std::fs::read(entry.dir().join("000003.json.gz.verified")).unwrap(), + b"abc\n" + ); + assert_eq!(mode(&entry.file_path(3)), 0o600); + assert_eq!(mode(&entry.dir().join("000003.json.gz.verified")), 0o600); + assert!(!entry.dir().join("000003.json.gz.partial").exists()); + + // A second attempt removes the file and its marker before it writes. + let file = entry.begin_file(3).await.unwrap(); + assert!(!entry.is_verified(3, "abc")); + assert!(!entry.file_path(3).exists()); + drop(file); + // Reopening an existing entry of this host's record keeps it; one + // whose record was another host's is emptied. + let again = Entry::open(spool.path(), "tid-1", Some("host-a"), "host-a").unwrap(); + assert!(again.dir().join("000003.json.gz.partial").exists()); + let moved = Entry::open(spool.path(), "tid-1", Some("host-b"), "host-a").unwrap(); + assert!(moved.dir().is_dir()); + assert_eq!(mode(moved.dir()), 0o700); + assert!(!moved.dir().join("000003.json.gz.partial").exists()); + assert!(!moved.dir().join("extenddb-manifest.json").exists()); + let unknown = Entry::open(spool.path(), "tid-1", None, "host-a"); + assert!( + unknown.is_ok(), + "a record with no spool_host has no entry to keep" + ); + again.remove().await.unwrap(); + assert!(!again.dir().exists()); + again.remove().await.unwrap(); + } + + /// A crash at the rename leaves a `.partial` and no marker: the file is + /// never taken for verified, so the marker is written only once the + /// rename has landed. + #[tokio::test] + async fn the_marker_is_absent_until_the_file_is_renamed() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-2", Some("host-a"), "host-a").unwrap(); + let mut file = entry.begin_file(0).await.unwrap(); + file.write_all(b"bytes").await.unwrap(); + entry + .fail_rename + .store(true, std::sync::atomic::Ordering::Release); + let err = entry.commit_file(0, file, "abc").await.unwrap_err(); + assert!(err.to_string().contains("at the rename"), "{err}"); + assert!(entry.dir().join("000000.json.gz.partial").exists()); + assert!(!entry.file_path(0).exists()); + assert!(!entry.dir().join("000000.json.gz.verified").exists()); + assert!(!entry.is_verified(0, "abc")); + } + + /// The sweep removes the entries of restores not RUNNING on this host + /// and leaves a live entry and every other name; an entry listed and + /// then removed by its own job before the sweep reached it is not an + /// error and not counted. + #[test] + fn the_sweep_removes_only_dead_entries() { + let spool = Scratch::new(); + for name in [ + "restore-live", + "restore-dead-1", + "restore-gone", + "other-dir", + ] { + std::fs::create_dir(spool.path().join(name)).unwrap(); + } + std::fs::write(spool.path().join("restore-dead-1/000000.json.gz"), b"x").unwrap(); + std::fs::write(spool.path().join(".lock"), b"").unwrap(); + // A file, not a directory, under an entry-shaped name is left. + std::fs::write(spool.path().join("restore-file"), b"").unwrap(); + let running: HashSet = ["live".to_owned()].into_iter().collect(); + let mut listed = list_entries(spool.path()).unwrap(); + listed.sort(); + assert_eq!( + listed.iter().map(|(id, _)| id.as_str()).collect::>(), + ["dead-1", "gone", "live"], + "entries only, in whatever order the directory gives" + ); + // Between the listing and the sweep, the job of `gone` removed its + // entry. + std::fs::remove_dir(spool.path().join("restore-gone")).unwrap(); + let mut removed = sweep_entries(listed, &running); + removed.sort(); + assert_eq!(removed, ["dead-1"]); + let mut left: Vec = std::fs::read_dir(spool.path()) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + left.sort(); + assert_eq!(left, [".lock", "other-dir", "restore-file", "restore-live"]); + assert_eq!(entry_table_id("restore-"), None); + assert_eq!(entry_table_id("restore-a/b"), None); + assert_eq!(entry_table_id("restore-ab_1-2"), Some("ab_1-2")); + } + + /// A document is written beside the one it replaces and renamed over + /// it: a write that stops at the rename leaves the earlier document + /// whole and a `.partial` beside it; a write that completes leaves the + /// new document and no `.partial`. + #[tokio::test] + async fn a_document_replaces_the_earlier_one_at_the_rename() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-3", Some("host-a"), "host-a").unwrap(); + entry + .write_document("extenddb-manifest.json", b"first") + .await + .unwrap(); + entry + .fail_rename + .store(true, std::sync::atomic::Ordering::Release); + let err = entry + .write_document("extenddb-manifest.json", b"second") + .await + .unwrap_err(); + assert!(err.to_string().contains("at the rename"), "{err}"); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"first".to_vec()), + "the earlier document stands until the rename" + ); + assert_eq!( + std::fs::read(entry.dir().join("extenddb-manifest.json.partial")).unwrap(), + b"second" + ); + assert_eq!( + mode(&entry.dir().join("extenddb-manifest.json.partial")), + 0o600 + ); + entry + .fail_rename + .store(false, std::sync::atomic::Ordering::Release); + entry + .write_document("extenddb-manifest.json", b"third") + .await + .unwrap(); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"third".to_vec()) + ); + assert!(!entry.dir().join("extenddb-manifest.json.partial").exists()); + } +} diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs index 7d660355a..f29b468c7 100644 --- a/crates/engine/src/backup/runtime.rs +++ b/crates/engine/src/backup/runtime.rs @@ -42,13 +42,16 @@ //! thread costs nothing at exit: the binary ends its runtime with a bounded //! shutdown that detaches it. +use std::collections::HashMap; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, RwLock}; use std::time::Duration; use extenddb_backup_store::{BackupStore, ByteStream, ObjectMeta, StoreError}; use extenddb_core::error::DynamoDbError; +use extenddb_core::limits::LimitsConfig; use extenddb_core::metrics::MetricsCollector; +use extenddb_core::types::TableDescription; use extenddb_storage::backup_catalog::{ JOB_HEARTBEAT_INTERVAL, JobBound, StoreIdentity, StoreKind, limit_exceeded_message, }; @@ -164,6 +167,61 @@ pub struct BackupRuntime { source_backend: String, /// The build's version, the manifest's `source_extenddb_version`. source_extenddb_version: String, + /// The instance's limits, which a restore the reconciler resumes + /// validates its items against (a request's restore reads them from + /// the operation context). + limits: Arc, + /// The store-backed restore registrations this instance has spawned and + /// not yet seen return, by (account, target table name): what a retry + /// of the same request meets while the first registration is still in + /// flight ([`Self::begin_restore_registration`]). + restore_registrations: Arc>>, +} + +/// The registration outcome a waiting retry reads: `None` until the +/// registration returns. +pub(crate) type RegistrationOutcome = Option>; + +/// A restore registration this instance has spawned and not yet seen +/// return, as a retry of the request meets it. +#[derive(Clone)] +pub(crate) struct InFlightRegistration { + /// The backup the registration restores from. + pub backup_arn: String, + /// The registration's outcome once it returns. + pub outcome: tokio::sync::watch::Receiver, +} + +/// The registering request's side of an [`InFlightRegistration`]: the +/// registration is in flight while this exists, and +/// [`RegistrationGuard::finish`] publishes its outcome and ends it. Dropped +/// without `finish`, it ends the entry with no outcome (the waiters read the +/// closed channel). +pub(crate) struct RegistrationGuard { + runtime_registrations: Arc>>, + key: (String, String), + outcome: tokio::sync::watch::Sender, +} + +impl RegistrationGuard { + /// Publish the registration's outcome to the retries waiting on it and + /// end the entry. + pub(crate) fn finish(self, outcome: Result) { + let _ = self.outcome.send(Some(outcome)); + self.runtime_registrations + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .remove(&self.key); + } +} + +impl Drop for RegistrationGuard { + fn drop(&mut self) { + self.runtime_registrations + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .remove(&self.key); + } } impl std::fmt::Debug for BackupRuntime { @@ -876,9 +934,59 @@ impl BackupRuntime { jobs: TaskTracker::new(), source_backend: "test".to_owned(), source_extenddb_version: env!("CARGO_PKG_VERSION").to_owned(), + limits: Arc::new(LimitsConfig::default()), + restore_registrations: Arc::default(), } } + /// Mark a store-backed restore registration of `backup_arn` into + /// (`account_id`, `table_name`) as in flight, or return the one already + /// in flight for that name (a retry of the request, or another backup's + /// registration into the same name). The registering request's task + /// ends the entry through [`RegistrationGuard::finish`]. + pub(crate) fn begin_restore_registration( + &self, + account_id: &str, + table_name: &str, + backup_arn: &str, + ) -> Result { + let key = (account_id.to_owned(), table_name.to_owned()); + let mut registrations = self + .restore_registrations + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + if let Some(in_flight) = registrations.get(&key) { + return Err(in_flight.clone()); + } + let (outcome, receiver) = tokio::sync::watch::channel(None); + registrations.insert( + key.clone(), + InFlightRegistration { + backup_arn: backup_arn.to_owned(), + outcome: receiver, + }, + ); + Ok(RegistrationGuard { + runtime_registrations: Arc::clone(&self.restore_registrations), + key, + outcome, + }) + } + + /// The same runtime with the instance's limits, which a resumed restore + /// validates its items against (the default is `LimitsConfig::default`). + #[must_use] + pub fn with_limits(mut self, limits: Arc) -> Self { + self.limits = limits; + self + } + + /// The instance's limits. + #[must_use] + pub fn limits(&self) -> &Arc { + &self.limits + } + /// The same runtime with `[backup] stale_job_after_secs` set, the window /// the jobs' maintenance-claim check reads with (the default is the /// configuration's default, 300 s). @@ -1011,16 +1119,17 @@ impl BackupRuntime { return 0; } tracing::info!( - "waiting up to {} s for {running} backup job(s) to end", + "waiting up to {} s for {running} backup or restore job(s) to end", bound.as_secs() ); if tokio::time::timeout(bound, self.jobs.wait()).await.is_err() { let left = self.jobs.len(); tracing::warn!( - "{left} backup job task(s) did not end within {} s (tasks spawned by \ - CreateBackup, counted whether or not a row exists for them yet); a row one \ - left CREATING or FAILED is resolved or swept by the reconciler once its \ - heartbeat is stale", + "{left} job task(s) did not end within {} s (tasks spawned by CreateBackup and \ + RestoreTableFromBackup, counted whether or not a row exists for them yet); a \ + backup job's row left CREATING or FAILED is resolved or swept by the \ + reconciler once its heartbeat is stale, a restore job's record is RUNNING \ + with its progress and the reconciler resumes it", bound.as_secs() ); return left; @@ -1072,6 +1181,15 @@ impl BackupRuntime { .filter(|store| !store.is_set_aside()) } + /// [`Self::configured_store`] as the `Arc` a job's environment holds + /// ([`super::job::JobEnv::new`]). + #[must_use] + pub(crate) fn configured_store_arc(&self) -> Option<&Arc> { + self.configured_store + .as_ref() + .filter(|store| !store.is_set_aside()) + } + /// The configured store, taken out of a runtime a command built only to /// reach it. #[must_use] diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index 3277407c7..8f85ddcf6 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -22,17 +22,18 @@ use extenddb_core::types::{ }; use extenddb_storage::backup_catalog::{ AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, - JobRow, JobStatus, RestoreJobRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, - StoreKind, TimeBounds, WriteOutcome, new_generation, + JobRow, JobStatus, RestoreJobRow, RestoreKind, RestoreRegistrationRow, RestoreTargetKey, + StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, MetadataEngine, - RestoreRequest, SnapshotEngine, TableEngine, TableSnapshot, + BackupCatalog, BackupRestoreSource, BoxedFuture, BulkLoadFile, DataEngine, Finished, JobPermit, + LegacyBackupReader, MetadataEngine, RestoreEngine, RestoreRequest, SnapshotEngine, TableEngine, + TableSnapshot, }; -use futures::TryStreamExt; use futures::stream::BoxStream; +use futures::{StreamExt, TryStreamExt}; use serde_json::{Value, json}; use super::*; @@ -53,6 +54,9 @@ fn source_arn() -> String { #[derive(Default)] struct Fake { + /// Cancelled by `list_dead_restore_jobs`: a token that fires in the + /// middle of a cycle, after its store check. + cancel_on_list_dead: Mutex>, sources: Mutex>, descriptions: Mutex>, summaries: Mutex>, @@ -182,6 +186,332 @@ struct Fake { takeover_lost_to_another: std::sync::atomic::AtomicBool, /// What `sweep_restore_registrations` answers. swept_registrations: Mutex>, + /// The store-backed restore records, by target table id: what + /// `claim_restore` writes and the restore's claims, `read_restore_job`, + /// `list_restore_jobs` and the reconciler read. + restore_jobs: Arc>>, + /// The restore targets' data, by table id: the key schema the duplicate + /// rule reads keys with, the items `bulk_load` applied, and the + /// statuses `finish_restore` flips. + restore_tables: Mutex>, + /// Every `claim_restore` call: the account and the input. + restore_claims: Mutex>, + /// When set, the next `claim_restore` fails with it instead of + /// registering (a name race the backend lost, a backup deleted under + /// the lock). + claim_restore_error: Mutex>, + /// Every `set_status` a restore claim received, in order. + restore_writes: Arc>>, + /// The table ids `claim_restore_dead` hands over (the backend's liveness + /// test modelled as a set). + restore_dead: Mutex>, + /// Every `claim_restore_dead` that handed a claim over: the table id and + /// the `spool_host` it was given. + restore_takeovers: Mutex)>>, + /// Every `bulk_load`: the file, and the items the stream yielded before + /// it ended or failed. + bulk_loads: Mutex>, + /// When set, every `bulk_load` waits for one permit before it reads its + /// stream, so a test can hold the job between files. + bulk_load_gate: Mutex>>, + /// When set, `bulk_load` takes one permit before each item it pulls + /// from its stream, so a test can hold the job at a known item of a + /// file; `bulk_load_items_seen` counts the items pulled so far. + bulk_load_item_gate: Mutex>>, + bulk_load_items_seen: std::sync::atomic::AtomicUsize, + /// When set, `count_restore_target` answers this instead of the table's + /// item count (a backend that kept one of two rows with one key). + count_override: Mutex>, + /// When set, `count_restore_target` waits for one permit before it + /// counts, so a test can hold the job after its last `bulk_load`. + count_gate: Mutex>>, + /// `count_restore_target` calls, counted before the gate. + count_calls: std::sync::atomic::AtomicUsize, + /// When set, `claim_restore` takes this long before it registers (a + /// registration held behind the catalog's lock or pool). + claim_restore_delay: Mutex>, + /// What `finish_restore` answers when its conditions hold: ACTIVE, or + /// DONE with the backfill pending (MongoDB). + finish_pending: std::sync::atomic::AtomicBool, + /// The progress rows a backend keeps apart from the record + /// (PostgreSQL's `restore_progress`), by table id. + progress_rows: Mutex>>, + /// The `running` lists `sweep_restore_progress` received. + progress_sweeps: Mutex>>, + /// Whether `as_vector_search` answers an engine: a target with vector + /// support. + vector_search: std::sync::atomic::AtomicBool, + /// Every `list_restore_jobs` call's filter. + restore_lists: Mutex>>, + /// When set, `list_restore_jobs` fails as a backend that has no restore + /// records yet (its `Unsupported` stub). + restore_list_error: std::sync::atomic::AtomicBool, + /// `bulk_load` calls begun, counted before the gate, so a test can tell + /// a job waiting at the gate. + bulk_loads_started: std::sync::atomic::AtomicUsize, + /// When set, run inside `list_restore_jobs` after it builds its answer, + /// that is, after the reconciler read the RUNNING records and before its + /// spool listing uses them: the moment a registration can land between + /// the sweep's two reads. + sweep_hook: Mutex>, + /// What the next `finish_restore` calls do instead of answering, in + /// order: fail after the flip landed, or fail without flipping. + finish_errors: Mutex>, + /// Every `finish_restore` call's `size_bytes`. + finish_sizes: Mutex>, + /// When set, `read_restore_job` takes this long to answer (a catalog + /// read held behind the pool). + read_restore_job_delay: Mutex>, + /// When set, `claim_restore_dead` hands the claim over and then removes + /// the record (gone between the takeover and the reads that follow). + takeover_removes_record: std::sync::atomic::AtomicBool, + /// The maintenance claims `maintenance_claim` handed out, as the test's + /// handles on them; `None` in the error slot makes the next call fail + /// as a claim a live holder has. + maintenance_holders: Mutex>>, + maintenance_claim_held_by: Arc>>, + /// When set, `claim_restore` stamps the record's `heartbeat_at` this + /// many seconds in the past: a registration that stalled on a lock, as + /// PostgreSQL's transaction-start `now()` stamps it. + registration_heartbeat_age: Mutex>, + /// Table ids whose data tables never came up: `restore_target_ready` + /// answers `false` for them, the state a crash between the + /// registration's commit and the data DDL leaves. + targets_without_tables: Mutex>, + /// When set, the next `maintenance_claim` hands out a holder whose + /// heartbeats report the claim lost at once: the HeldClaim's timer sees + /// it and `lost()` reports it before the removal. + next_claim_lost: Mutex, + /// When set, `list_restore_jobs` sleeps this long before it answers, so a + /// test can let the maintenance claim's heartbeat timer observe a lost + /// claim during the read that precedes the removal. + list_jobs_delay: Mutex>, + /// How many `count_restore_target` calls still fail with an error (a + /// `COUNT(*)` that times out), taken one per call. + count_errors: std::sync::atomic::AtomicU32, + /// When set, run inside `claim_restore_dead` once it has handed a claim + /// over, before the reconciler's second read of the maintenance claim: + /// the moment a command can take that claim during the cycle. + takeover_hook: Mutex>, + /// When set, `remove_restore_target_exclusive` answers `false` without + /// removing anything: the owner's lock (PostgreSQL's per-session + /// advisory lock) is still held by a frozen owner's session. + owner_lock_held: Mutex, +} + +/// A closure the `Fake` runs inside `sweep_restore_progress`. +type SweepHook = Box; + +/// What a `finish_restore` of the `Fake` does instead of answering. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum FinishError { + /// The flip landed (the record DONE, the table ACTIVE) and the call + /// then failed: PostgreSQL's progress delete after the commit. + AfterFlip, + /// The call failed before anything was written. + BeforeFlip, + /// The call failed before anything was written, and another claim took + /// the record over and wrote DONE under its own generation before the + /// re-read. + TakenAndDone, +} + +/// A restore target's data in the `Fake`. +#[derive(Debug, Clone)] +struct RestoreTable { + name: String, + key_attributes: Vec, + items: Vec, + active: bool, +} + +/// One `bulk_load` the `Fake` received. +#[derive(Debug, Clone, PartialEq)] +struct BulkLoadCall { + table_id: String, + file: BulkLoadFile, + /// The items the stream yielded. + items: Vec, + /// The text of the `Err` item that ended the stream, if one did. + stream_error: Option, +} + +/// A `Fake` claim on one restore record, by table id: the backends' +/// `heartbeat` (`status = RUNNING AND owner_generation = ?`), `set_status` +/// conditioned on the generation and `from`, and `read`. +struct FakeRestoreClaim { + records: Arc>>, + table_id: String, + generation: u64, + writes: Arc>>, + heartbeat_answer: Arc>, + heartbeats: Arc, + unknown_write: Arc>>, + status_write_error: Arc, +} + +impl JobClaim for FakeRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxedFuture<'_, Result> { + self.heartbeats + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let answer = *self.heartbeat_answer.lock().unwrap(); + let confirmed = match self.records.lock().unwrap().get_mut(&self.table_id) { + Some(record) + if answer == HeartbeatAnswer::Confirm + && record.owner_generation == self.generation + && record.status == JobStatus::Running => + { + record.heartbeat_at = seconds_since_epoch(); + true + } + _ => false, + }; + Box::pin(async move { + match answer { + HeartbeatAnswer::Confirm if confirmed => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::AlwaysConfirm => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::Confirm | HeartbeatAnswer::Lost => Ok(Heartbeat::Lost), + HeartbeatAnswer::Stall | HeartbeatAnswer::Park => futures::future::pending().await, + HeartbeatAnswer::Error => Err(StorageError::Internal( + "the claim connection failed".to_owned(), + )), + } + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxedFuture<'a, Result> { + self.writes.lock().unwrap().push(StatusWrite { + arn: self.table_id.clone(), + generation: self.generation, + from, + to, + fields: fields.clone(), + }); + if self + .status_write_error + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(async move { + Err(StorageError::Internal( + "claim session: connection reset".to_owned(), + )) + }); + } + let unknown = self.unknown_write.lock().unwrap().take(); + let apply = unknown != Some(UnknownWrite::Lost); + let mut outcome = WriteOutcome::NotMatched; + if let Some(record) = self.records.lock().unwrap().get_mut(&self.table_id) + && record.owner_generation == self.generation + && record.status == from + { + if unknown == Some(UnknownWrite::TakenAndWritten) { + record.owner_generation = new_generation(); + record.status = to; + } else if apply { + record.status = to; + if let Some(reason) = &fields.failure_reason { + record.failure_reason = Some(reason.clone()); + } + if fields.completed { + record.completed_at = Some(seconds_since_epoch()); + } + outcome = WriteOutcome::Matched; + } + } + if unknown.is_some() { + outcome = WriteOutcome::Unknown; + } + Box::pin(async move { Ok(outcome) }) + } + + fn read(&self) -> BoxedFuture<'_, Result> { + let found = self + .records + .lock() + .unwrap() + .get(&self.table_id) + .map(|record| JobRow { + status: record.status, + owner_generation: record.owner_generation, + heartbeat_at: record.heartbeat_at, + failure_reason: record.failure_reason.clone(), + }); + Box::pin(async move { + found.ok_or_else(|| StorageError::Internal("the record is gone".to_owned())) + }) + } +} + +impl Fake { + fn restore_claim_on(&self, table_id: &str, generation: u64) -> Box { + Box::new(FakeRestoreClaim { + records: Arc::clone(&self.restore_jobs), + table_id: table_id.to_owned(), + generation, + writes: Arc::clone(&self.restore_writes), + heartbeat_answer: Arc::clone(&self.heartbeat_answer), + heartbeats: Arc::clone(&self.heartbeats), + unknown_write: Arc::clone(&self.unknown_write), + status_write_error: Arc::clone(&self.status_write_error), + }) + } + + /// The key of `item` under `attributes`, for the duplicate rule. + fn item_key(attributes: &[String], item: &Item) -> String { + attributes + .iter() + .map(|a| serde_json::to_string(&item.get(a)).unwrap_or_default()) + .collect::>() + .join("|") + } + + /// The restore record of the target named `table`, if any. + fn restore_job(&self, table: &str) -> Option { + self.restore_jobs + .lock() + .unwrap() + .values() + .find(|r| r.target.table_name == table) + .cloned() + } + + /// The items a restore target holds, by table name. + fn restored_items(&self, table: &str) -> Vec { + self.restore_tables + .lock() + .unwrap() + .values() + .find(|t| t.name == table) + .map(|t| t.items.clone()) + .unwrap_or_default() + } + + /// The `bulk_load` calls received, as `(file_index, start_at, reapplying, + /// item count)`. + fn bulk_load_files(&self) -> Vec<(u32, u64, bool, usize)> { + self.bulk_loads + .lock() + .unwrap() + .iter() + .map(|c| { + ( + c.file.file_index, + c.file.start_at, + c.file.reapplying, + c.items.len(), + ) + }) + .collect() + } } /// A table snapshot the `Fake` serves. @@ -235,6 +565,10 @@ enum UnknownWrite { Applied, /// The write was lost with the connection. Lost, + /// The write was lost with the connection, and another claim took the + /// record over and wrote the same status under its own generation + /// before the re-read. + TakenAndWritten, } /// A `Fake` claim on one row. @@ -432,13 +766,16 @@ impl JobClaim for FakeClaim { } impl Fake { - fn with_source(source: BackupRestoreSource) -> Self { + /// A `Fake` serving `source` through the legacy reader, in the `Arc` the + /// restore request takes (its manifest branch spawns a task that owns a + /// clone). + fn with_source(source: BackupRestoreSource) -> Arc { let fake = Self::default(); fake.sources .lock() .unwrap() .insert(source.backup_arn.clone(), source); - fake + Arc::new(fake) } fn restores(&self) -> Vec { self.restores.lock().unwrap().clone() @@ -500,6 +837,27 @@ struct MemStore { /// error; every other call answers (the identity file's read among /// them, so the store stays healthy). fail_gets_under: Mutex>, + /// When set, the next `n` `get`s of a data file (a key under a `data/` + /// component) fail at the call with the error kind given; every other + /// call answers. + fail_gets: Mutex>, + /// When set, the next `get` of a data file yields this many bytes and + /// then fails as a transport error mid-body (a `GetObject` body that + /// stopped arriving). + fail_get_after_bytes: Mutex>, + /// When set, every `get` of a data file fails at the call as a transport + /// error while the clock reads before this instant (an outage of a + /// known length under a paused clock). + fail_data_gets_until: Mutex>, + /// When set, every `get` of a key ending in this text fails at the call + /// as a transport error (one object the store never serves). + fail_gets_of: Mutex>, + /// When set, the body of the `get` with this ordinal (1-based, over + /// every `get` the store saw) panics after its first chunk: a reader on + /// the blocking pool that dies mid-file. + panic_body_of_get: std::sync::atomic::AtomicUsize, + /// Every `get`'s key, in order, failed ones included. + gets: Mutex>, /// Called with the key of every `put` that stored its object, so a test /// can act at a point of the job (after the sidecar's put, before the /// claim re-check). @@ -534,6 +892,15 @@ struct MemStore { /// What `MemStore::on_put` runs after a stored `put`, with the object's key. type OnPut = Box; +/// How `MemStore::fail_gets_with` fails a data file's `get`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum GetFailure { + /// The SDK's transport shape, which the restore retries. + Transport, + /// A denial, which it does not. + PermissionDenied, +} + /// A transport error with the SDK's shape: a top link naming only the class /// and the cause one link below it. #[derive(Debug)] @@ -604,6 +971,21 @@ impl MemStore { .unwrap() .insert(key.to_owned(), body.to_vec()); } + /// The next `times` `get`s of a data file fail with `failure`. + fn fail_gets_with(&self, failure: GetFailure, times: usize) { + *self.fail_gets.lock().unwrap() = Some((failure, times)); + } + /// The next `get` of a data file stops after `n` bytes with a transport + /// error. + fn fail_get_after_bytes(&self, n: usize) { + *self.fail_get_after_bytes.lock().unwrap() = Some(n); + } + fn gets(&self) -> Vec { + self.gets.lock().unwrap().clone() + } + fn is_data_file(key: &str) -> bool { + key.contains("/data/") + } fn keys(&self) -> Vec { self.objects.lock().unwrap().keys().cloned().collect() } @@ -719,6 +1101,7 @@ impl extenddb_backup_store::BackupStore for MemStore { ) -> BoxedFuture<'a, Result> { Box::pin(async move { + self.gets.lock().unwrap().push(key.to_owned()); self.stall().await; self.check()?; if self @@ -732,6 +1115,44 @@ impl extenddb_backup_store::BackupStore for MemStore { source: Box::new(DispatchFailure(ConnectError("get refused".to_owned()))), }); } + let ordinal = self.gets.lock().unwrap().len(); + if Self::is_data_file(key) + && (self + .fail_data_gets_until + .lock() + .unwrap() + .is_some_and(|until| tokio::time::Instant::now() < until) + || self + .fail_gets_of + .lock() + .unwrap() + .as_deref() + .is_some_and(|suffix| key.ends_with(suffix))) + { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "connection reset by peer".to_owned(), + ))), + }); + } + if Self::is_data_file(key) { + let mut fail = self.fail_gets.lock().unwrap(); + if let Some((failure, times)) = *fail { + *fail = (times > 1).then_some((failure, times - 1)); + return Err(match failure { + GetFailure::Transport => extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "connection reset by peer".to_owned(), + ))), + }, + GetFailure::PermissionDenied => { + extenddb_backup_store::StoreError::PermissionDenied(format!( + "GetObject {key}" + )) + } + }); + } + } let bytes = self .objects .lock() @@ -739,6 +1160,33 @@ impl extenddb_backup_store::BackupStore for MemStore { .get(key) .cloned() .ok_or(extenddb_backup_store::StoreError::NotFound)?; + if Self::is_data_file(key) + && let Some(n) = self.fail_get_after_bytes.lock().unwrap().take() + { + let head = bytes[..n.min(bytes.len())].to_vec(); + let stream = futures::stream::iter(vec![ + Ok(bytes::Bytes::from(head)), + Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "the body stopped arriving".to_owned(), + ))), + }), + ]); + return Ok(Box::pin(stream) as extenddb_backup_store::ByteStream); + } + if ordinal + == self + .panic_body_of_get + .load(std::sync::atomic::Ordering::Acquire) + { + let head = bytes[..bytes.len().min(16)].to_vec(); + let stream = futures::stream::iter(vec![Ok(bytes::Bytes::from(head))]).chain( + futures::stream::once(async { + panic!("the body's reader died (a test fixture)") + }), + ); + return Ok(Box::pin(stream) as extenddb_backup_store::ByteStream); + } Ok(extenddb_backup_store::byte_stream_from(bytes)) }) } @@ -901,6 +1349,115 @@ fn row(arn: &str, status: BackupStatus) -> BackupRow { } } +impl Fake { + /// `claim_restore`'s registration once any delay has passed. + fn register_restore( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result<(TableDescription, Box), StorageError> { + let arn = ®istration.backup.backup_arn; + let available = self + .rows + .lock() + .unwrap() + .get(arn) + .is_some_and(|r| r.status == BackupStatus::Available); + if !available { + return Err(extenddb_storage::backup_catalog::backup_not_found(arn)); + } + let in_use = self.in_use.lock().unwrap().get(arn).cloned().or_else(|| { + let jobs = self.restore_jobs.lock().unwrap(); + self.restore_tables + .lock() + .unwrap() + .iter() + .filter(|(_, t)| !t.active) + .find(|(id, _)| jobs.get(*id).is_some_and(|j| j.source_backup_arn == *arn)) + .map(|(_, t)| t.name.clone()) + }); + if let Some(table) = in_use { + return Err(extenddb_storage::backup_catalog::backup_in_use(&table, arn)); + } + let mut tables = self.tables.lock().unwrap(); + if let Some(taken) = tables.get(&input.table_name) { + let err = match taken + .restore_summary + .as_ref() + .filter(|_| taken.table_status == TableStatus::Creating) + .and_then(|s| s.source_backup_arn.clone()) + { + Some(backup_arn) => StorageError::TableBeingRestored { + table_name: input.table_name.clone(), + backup_arn, + }, + None => StorageError::TableAlreadyExists(input.table_name.clone()), + }; + return Err(err); + } + let table_id = format!("tid-{}", input.table_name); + let generation = new_generation(); + let mut desc = described(&RestoreRequest { + backup_arn: arn.clone(), + source_table_arn: registration.source_table_arn.clone(), + restore_date_time: registration.restore_date_time, + input: input.clone(), + }); + desc.table_id = table_id.clone(); + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + tables.insert(input.table_name.clone(), desc.clone()); + drop(tables); + let target = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: input.table_name.clone(), + table_id: table_id.clone(), + }; + self.restore_jobs.lock().unwrap().insert( + table_id.clone(), + RestoreJobRow { + target, + source_backup_arn: arn.clone(), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: generation, + heartbeat_at: seconds_since_epoch() + - self + .registration_heartbeat_age + .lock() + .unwrap() + .unwrap_or(0.0), + store: Some(registration.store.clone()), + spool_host: registration.spool_host.clone(), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + self.restore_tables.lock().unwrap().insert( + table_id.clone(), + RestoreTable { + name: input.table_name.clone(), + key_attributes: input + .key_schema + .iter() + .map(|k| k.attribute_name.clone()) + .collect(), + items: Vec::new(), + active: false, + }, + ); + let claim = self.restore_claim_on(&table_id, generation); + Ok((desc, claim)) + } +} + impl BackupCatalog for Fake { fn catalog_id(&self) -> BoxedFuture<'_, Result> { if *self.catalog_id_error.lock().unwrap() { @@ -939,6 +1496,135 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(claim) }) } + /// The backends' registration: under the backup row's lock the row must + /// be AVAILABLE (else `Backup not found`) and not in use (else + /// `BackupInUse`), the name free (else `TableAlreadyExists`, or + /// `TableBeingRestored` for a CREATING restore target); then the target + /// CREATING with its `RestoreSummary` and the record RUNNING under a + /// fresh generation, `heartbeat_at` now, in one step. + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxedFuture<'a, Result<(TableDescription, Box), StorageError>> { + self.restore_claims + .lock() + .unwrap() + .push((account_id.to_owned(), input.clone())); + if let Some(e) = self.claim_restore_error.lock().unwrap().take() { + return Box::pin(async move { Err(e) }); + } + let delay = *self.claim_restore_delay.lock().unwrap(); + Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + self.register_restore(account_id, input, registration) + }) + } + + /// The takeover: a RUNNING manifest record whose table id is in the + /// `restore_dead` set gets a fresh generation, the claimant's + /// `spool_host` and a fresh `heartbeat_at` (so a second taker finds it + /// live at once); anything else is `None`. + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + _stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxedFuture<'a, Result>, StorageError>> { + let dead = self.restore_dead.lock().unwrap().contains(&key.table_id); + let mut records = self.restore_jobs.lock().unwrap(); + let claim = match records.get_mut(&key.table_id) { + Some(record) + if dead + && record.status == JobStatus::Running + && record.kind == RestoreKind::Manifest => + { + record.owner_generation = new_generation(); + record.spool_host = spool_host.map(str::to_owned); + record.heartbeat_at = seconds_since_epoch(); + let generation = record.owner_generation; + if self + .takeover_removes_record + .load(std::sync::atomic::Ordering::Acquire) + { + records.remove(&key.table_id); + } + drop(records); + self.restore_takeovers + .lock() + .unwrap() + .push((key.table_id.clone(), spool_host.map(str::to_owned))); + if let Some(hook) = self.takeover_hook.lock().unwrap().take() { + hook(self); + } + Some(self.restore_claim_on(&key.table_id, generation)) + } + _ => None, + }; + Box::pin(async move { Ok(claim) }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + let delay = *self.read_restore_job_delay.lock().unwrap(); + Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + Ok(self + .restore_jobs + .lock() + .unwrap() + .get(&key.table_id) + .cloned()) + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxedFuture<'a, Result, StorageError>> { + self.restore_lists.lock().unwrap().push(status); + if self + .restore_list_error + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(async { + Err(StorageError::Unsupported("not implemented".to_owned())) + }); + } + let mut rows: Vec = self + .restore_jobs + .lock() + .unwrap() + .values() + .filter(|r| status.is_none_or(|s| r.status == s)) + .cloned() + .collect(); + rows.sort_by(|a, b| a.target.table_id.cmp(&b.target.table_id)); + // The moment a registration can land between the sweep's two reads: + // after the reconciler read the RUNNING records and before its spool + // listing uses them. The reconciler lists the spool first, so the + // entry a registration opens after this read is not in that listing + // and is never a sweep candidate; the hook plants such a + // registration so the test fails if the two reads are swapped. + if let Some(hook) = self.sweep_hook.lock().unwrap().take() { + hook(self); + } + let delay = *self.list_jobs_delay.lock().unwrap(); + Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + Ok(rows) + }) + } + /// The takeover: a CREATING row whose ARN is in the `dead` set gets a /// fresh generation, `heartbeat_at` untouched; anything else is `None`. fn claim_dead_backup<'a>( @@ -1201,34 +1887,113 @@ impl BackupCatalog for Fake { Box::pin(futures::stream::iter(rows.into_iter().map(Ok))) } + /// The preset answer, or the name of a CREATING restore target the + /// `Fake` registered from the backup. fn backup_in_use<'a>( &'a self, key: &'a BackupKey, ) -> BoxedFuture<'a, Result, StorageError>> { - let found = self.in_use.lock().unwrap().get(&key.backup_arn).cloned(); + let found = self + .in_use + .lock() + .unwrap() + .get(&key.backup_arn) + .cloned() + .or_else(|| { + let jobs = self.restore_jobs.lock().unwrap(); + self.restore_tables + .lock() + .unwrap() + .iter() + .filter(|(_, t)| !t.active) + .find(|(id, _)| { + jobs.get(*id) + .is_some_and(|j| j.source_backup_arn == key.backup_arn) + }) + .map(|(_, t)| t.name.clone()) + }); Box::pin(async move { Ok(found) }) } + /// A target the `Fake` registered is removed with its record and its + /// progress rows when it is still CREATING; any other key answers the + /// preset `target_still_creating`. fn remove_restore_target<'a>( &'a self, key: &'a RestoreTargetKey, ) -> BoxedFuture<'a, Result> { self.removed_targets.lock().unwrap().push(key.clone()); - let answer = *self.target_still_creating.lock().unwrap(); + let mut targets = self.restore_tables.lock().unwrap(); + let answer = match targets.get(&key.table_id) { + Some(target) if !target.active && target.name == key.table_name => { + let target = targets.remove(&key.table_id).expect("present"); + self.tables.lock().unwrap().remove(&target.name); + self.restore_jobs.lock().unwrap().remove(&key.table_id); + self.progress_rows.lock().unwrap().remove(&key.table_id); + true + } + Some(_) => false, + // The table is no longer a CREATING restore target with this id: + // its record went with it (the flip wrote DONE, the removal took + // both), so a re-read finds it changed or gone. + None => { + let still = *self.target_still_creating.lock().unwrap(); + if !still { + self.restore_jobs.lock().unwrap().remove(&key.table_id); + } + still + } + }; Box::pin(async move { Ok(answer) }) } + /// The command's removal: refused, nothing removed, while a test holds + /// the owner's lock (`owner_lock_held`); otherwise `remove_restore_target`. + fn remove_restore_target_exclusive<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result> { + if *self.owner_lock_held.lock().unwrap() { + return Box::pin(async { Ok(false) }); + } + self.remove_restore_target(key) + } + + /// The preset `dead_jobs`, plus the `Fake`'s own records that the + /// backends' rule would list: RUNNING ones in the `restore_dead` set, and + /// FAILED ones whose target is still CREATING. fn list_dead_restore_jobs( &self, _stale_after: Duration, ) -> BoxedFuture<'_, Result, StorageError>> { - let jobs = self.dead_jobs.lock().unwrap().clone(); - Box::pin(async move { Ok(jobs) }) - } - - /// The RUNNING records of `dead_jobs` and every record of - /// `running_jobs`, by status, whatever their heartbeat. - fn list_running_restore_jobs( + if let Some(token) = self.cancel_on_list_dead.lock().unwrap().take() { + token.cancel(); + } + let mut jobs = self.dead_jobs.lock().unwrap().clone(); + let dead = self.restore_dead.lock().unwrap(); + let targets = self.restore_tables.lock().unwrap(); + let mut own: Vec = self + .restore_jobs + .lock() + .unwrap() + .values() + .filter(|r| match r.status { + JobStatus::Running => dead.contains(&r.target.table_id), + JobStatus::Failed => targets.get(&r.target.table_id).is_some_and(|t| !t.active), + _ => false, + }) + .cloned() + .collect(); + own.sort_by(|a, b| a.target.table_id.cmp(&b.target.table_id)); + jobs.extend(own); + Box::pin(async move { Ok(jobs) }) + } + + /// A RUNNING record of the `Fake`'s own is FAILED with the reason; any + /// other key answers the preset `record_still_running`. + /// The RUNNING records of `dead_jobs` and every record of + /// `running_jobs`, by status, whatever their heartbeat. + fn list_running_restore_jobs( &self, ) -> BoxedFuture<'_, Result, StorageError>> { let mut jobs: Vec = self @@ -1258,7 +2023,10 @@ impl BackupCatalog for Fake { .lock() .unwrap() .push((key.clone(), reason.to_owned())); - let current = self + // The fence: the write matches only the claim the listing read + // (`owner_generation`, `heartbeat_at`), a beat that landed since + // (`heartbeat_after_listing`) making it miss. + let listed = self .dead_jobs .lock() .unwrap() @@ -1266,10 +2034,27 @@ impl BackupCatalog for Fake { .find(|j| j.target == *key) .map(|j| (j.owner_generation, j.heartbeat_at)); let beat = self.heartbeat_after_listing.lock().unwrap().remove(key); - let answer = *self.record_still_running.lock().unwrap() - && current.is_some_and(|(generation, heartbeat)| { - generation == owner_generation && beat.unwrap_or(heartbeat) == heartbeat_at - }); + let fence_holds = |generation: u64, heartbeat: f64| { + generation == owner_generation && beat.unwrap_or(heartbeat) == heartbeat_at + }; + let answer = match self.restore_jobs.lock().unwrap().get_mut(&key.table_id) { + Some(record) if record.status == JobStatus::Running => { + if fence_holds(record.owner_generation, record.heartbeat_at) { + record.status = JobStatus::Failed; + record.failure_reason = Some(reason.to_owned()); + record.completed_at = Some(seconds_since_epoch()); + true + } else { + false + } + } + Some(_) => false, + None => { + *self.record_still_running.lock().unwrap() + && listed + .is_some_and(|(generation, heartbeat)| fence_holds(generation, heartbeat)) + } + }; Box::pin(async move { Ok(answer) }) } @@ -1291,12 +2076,33 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(ids) }) } + /// A claim on the `Fake`'s maintenance record: refused naming the holder + /// while `maintenance_claim_held_by` names one, else a `FakeHolder` the + /// test keeps a handle on. fn maintenance_claim<'a>( &'a self, - _holder: &'a str, + holder: &'a str, _stale_after: Duration, ) -> BoxedFuture<'a, Result, StorageError>> { - unused() + if let Some(live) = self.maintenance_claim_held_by.lock().unwrap().clone() { + return Box::pin(async move { + Err(StorageError::AlreadyExists(format!( + "the maintenance claim is held by {live:?}" + ))) + }); + } + let fake = Arc::new(FakeHolder::default()); + *fake.lost.lock().unwrap() = *self.next_claim_lost.lock().unwrap(); + self.maintenance_holders + .lock() + .unwrap() + .push(Arc::clone(&fake)); + *self.maintenance_claim_held_by.lock().unwrap() = Some(holder.to_owned()); + let claim: Box = Box::new(ReleasingForward( + Arc::clone(&self.maintenance_claim_held_by), + fake, + )); + Box::pin(async move { Ok(claim) }) } fn check_maintenance_claim( @@ -1322,6 +2128,427 @@ impl BackupCatalog for Fake { } } +impl RestoreEngine for Fake { + /// The backends' load: under the claim's generation and RUNNING, the + /// stream read whole; an `Err` item fails the file with nothing applied; + /// a key the target holds fails the file unless the file is + /// `reapplying`, where the duplicate is skipped; the progress recorded + /// on the record as SQLite records it (`file_index`, the running total). + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxedFuture<'a, Result> { + Box::pin(async move { + self.bulk_loads_started + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let gate = self.bulk_load_gate.lock().unwrap().clone(); + if let Some(gate) = gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + let mut call = BulkLoadCall { + table_id: target.table_id.clone(), + file, + items: Vec::new(), + stream_error: None, + }; + let mut items = items; + let mut failed = None; + let item_gate = self.bulk_load_item_gate.lock().unwrap().clone(); + loop { + if let Some(gate) = &item_gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + let Some(next) = items.next().await else { + break; + }; + match next { + Ok(item) => { + self.bulk_load_items_seen + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + call.items.push(item); + } + Err(e) => { + call.stream_error = Some(e.to_string()); + failed = Some(e); + break; + } + } + } + let streamed = std::mem::take(&mut call.items); + call.items.clone_from(&streamed); + self.bulk_loads.lock().unwrap().push(call); + if let Some(e) = failed { + return Err(e); + } + let owned = self + .restore_jobs + .lock() + .unwrap() + .get(&target.table_id) + .is_some_and(|r| { + r.owner_generation == claim.generation() && r.status == JobStatus::Running + }); + if !owned { + return Err(StorageError::Internal( + "the restore record is not RUNNING under this claim".to_owned(), + )); + } + let mut targets = self.restore_tables.lock().unwrap(); + let Some(table) = targets.get_mut(&target.table_id) else { + return Err(StorageError::TableNotFound(target.table_name.clone())); + }; + let mut keys: std::collections::HashSet = table + .items + .iter() + .map(|i| Fake::item_key(&table.key_attributes, i)) + .collect(); + let mut applied = 0; + for item in streamed { + let key = Fake::item_key(&table.key_attributes, &item); + if !keys.insert(key.clone()) { + if file.reapplying { + continue; + } + return Err(StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key \ + twice): {key}" + ))); + } + table.items.push(item); + applied += 1; + } + drop(targets); + if let Some(record) = self.restore_jobs.lock().unwrap().get_mut(&target.table_id) { + record.file_index = Some(file.file_index); + record.items_applied = Some(file.start_at + applied); + } + Ok(applied) + }) + } + + /// DONE and ACTIVE in one step under the claim's generation, RUNNING and + /// a CREATING target; `Pending` when the test says the backend's worker + /// flips the table. A queued `FinishError` makes the call fail, after + /// the flip landed or before anything was written. + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxedFuture<'a, Result> { + self.finish_sizes.lock().unwrap().push(size_bytes); + let error = self.finish_errors.lock().unwrap().pop_front(); + if error == Some(FinishError::TakenAndDone) + && let Some(record) = self.restore_jobs.lock().unwrap().get_mut(&target.table_id) + && record.owner_generation == claim.generation() + { + record.owner_generation = new_generation(); + record.status = JobStatus::Done; + } + if matches!( + error, + Some(FinishError::BeforeFlip | FinishError::TakenAndDone) + ) { + return Box::pin(async { + Err(StorageError::Internal( + "the claim connection failed before the DONE write".to_owned(), + )) + }); + } + let mut records = self.restore_jobs.lock().unwrap(); + let mut targets = self.restore_tables.lock().unwrap(); + let finished = match ( + records.get_mut(&target.table_id), + targets.get_mut(&target.table_id), + ) { + (Some(record), Some(table)) + if record.owner_generation == claim.generation() + && record.status == JobStatus::Running + && !table.active => + { + record.status = JobStatus::Done; + record.completed_at = Some(seconds_since_epoch()); + let pending = self + .finish_pending + .load(std::sync::atomic::Ordering::Acquire); + if !pending { + table.active = true; + if let Some(desc) = self.tables.lock().unwrap().get_mut(&table.name) { + desc.table_status = TableStatus::Active; + desc.item_count = i64::try_from(item_count).unwrap_or(i64::MAX); + for gsi in desc.global_secondary_indexes.iter_mut().flatten() { + gsi.index_status = "ACTIVE".to_owned(); + } + } + } + self.progress_rows.lock().unwrap().remove(&target.table_id); + if pending { + Finished::Pending + } else { + Finished::Active + } + } + (Some(record), _) => Finished::NotMatched(format!( + "the record is {} under generation {} (the claim's is {})", + record.status, + record.owner_generation, + claim.generation() + )), + (None, _) => Finished::NotMatched("the record is gone".to_owned()), + }; + Box::pin(async move { + if error == Some(FinishError::AfterFlip) { + return Err(StorageError::Internal( + "the data pool failed after the flip's commit".to_owned(), + )); + } + Ok(finished) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result> { + self.count_calls + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let gate = self.count_gate.lock().unwrap().clone(); + Box::pin(async move { + if let Some(gate) = gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + // Take one error from the budget, if any is left: a + // compare-exchange loop, since `fetch_update` is deprecated on + // current stable and its replacement is not on 1.88. + let mut left = self.count_errors.load(std::sync::atomic::Ordering::Acquire); + while let Some(next) = left.checked_sub(1) { + match self.count_errors.compare_exchange( + left, + next, + std::sync::atomic::Ordering::AcqRel, + std::sync::atomic::Ordering::Acquire, + ) { + Ok(_) => { + return Err(StorageError::Internal( + "count restore target: the read timed out".to_owned(), + )); + } + Err(now) => left = now, + } + } + match *self.count_override.lock().unwrap() { + Some(count) => Ok(count), + None => self + .restore_tables + .lock() + .unwrap() + .get(&key.table_id) + .map(|t| t.items.len() as u64) + .ok_or_else(|| StorageError::TableNotFound(key.table_name.clone())), + } + }) + } + + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + let rows = self + .progress_rows + .lock() + .unwrap() + .get(&key.table_id) + .cloned() + .unwrap_or_default(); + Box::pin(async move { Ok(rows) }) + } + + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result> { + // The record must name a restore target with this id, as the + // backends' catalog read requires; a target the test put in + // `targets_without_tables` is one whose data DDL never ran (the + // owner died between the registration's commit and the DDL). + let known = self + .restore_jobs + .lock() + .unwrap() + .contains_key(&key.table_id); + let ready = !self + .targets_without_tables + .lock() + .unwrap() + .contains(&key.table_id); + let table_name = key.table_name.clone(); + let table_id = key.table_id.clone(); + Box::pin(async move { + if known { + Ok(ready) + } else { + Err(StorageError::TableNotFound(format!( + "{table_name} is not a CREATING restore target with table id {table_id}" + ))) + } + }) + } + + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxedFuture<'a, Result> { + self.progress_sweeps.lock().unwrap().push(running.to_vec()); + let mut rows = self.progress_rows.lock().unwrap(); + let before = rows.len(); + rows.retain(|id, _| running.contains(id)); + let removed = (before - rows.len()) as u64; + Box::pin(async move { Ok(removed) }) + } +} + +/// A vector-search engine the `Fake` hands out when it stands for a target +/// with vector support; never searched. +struct CapableVectorSearch; + +impl extenddb_storage::VectorSearchEngine for CapableVectorSearch { + fn search_vectors( + &self, + _req: extenddb_storage::VectorSearch<'_>, + ) -> BoxedFuture<'_, extenddb_storage::VectorSearchResult> { + unused() + } +} + +static CAPABLE_VECTOR_SEARCH: CapableVectorSearch = CapableVectorSearch; + +/// The restore's vector gate reads `as_vector_search`; nothing else of the +/// data plane is used by these tests. +impl DataEngine for Fake { + fn put_item( + &self, + _: &TableKeyInfo, + _: Item, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn get_item( + &self, + _: &TableKeyInfo, + _: &Item, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn delete_item( + &self, + _: &TableKeyInfo, + _: &Item, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn update_item( + &self, + _: &TableKeyInfo, + _: &Item, + _: &[extenddb_core::expression::UpdateAction], + _: bool, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, extenddb_storage::ItemPairResult> { + unused() + } + + fn query( + &self, + _: &TableKeyInfo, + _: &extenddb_core::expression::KeyCondition, + _: &extenddb_core::expression::ExpressionMaps, + _: bool, + _: Option, + _: Option<&Item>, + _: Option<&str>, + ) -> BoxedFuture<'_, extenddb_storage::QueryResult> { + unused() + } + + fn scan( + &self, + _: &TableKeyInfo, + _: Option, + _: Option<&Item>, + _: Option, + _: Option, + _: Option<&str>, + ) -> BoxedFuture<'_, extenddb_storage::QueryResult> { + unused() + } + + fn scan_key_in_segment( + &self, + _: &TableKeyInfo, + _: &Item, + _: i64, + _: i64, + _: Option<&str>, + ) -> BoxedFuture<'_, Result> { + unused() + } + + fn as_vector_search(&self) -> Option<&dyn extenddb_storage::VectorSearchEngine> { + self.vector_search + .load(std::sync::atomic::Ordering::Acquire) + .then_some(&CAPABLE_VECTOR_SEARCH) + } + + fn transact_get_items( + &self, + _: &[extenddb_storage::TransactGetOp<'_>], + ) -> BoxedFuture<'_, Result>, StorageError>> { + unused() + } + + fn transact_write_items( + &self, + _: &[extenddb_storage::TransactWriteOp<'_>], + _: Option>, + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn cleanup_expired_idempotency_tokens( + &self, + _: i64, + ) -> BoxedFuture<'_, Result> { + unused() + } +} + impl SnapshotEngine for Fake { /// The fixture for the table, or `TableNotFound`; the description and /// the items are the fixture's, in its order, through the gate when it @@ -1820,13 +3047,13 @@ fn restore_body(extra: Value) -> Value { body } -async fn restore(fake: &Fake, body: Value) -> Result { +async fn restore(fake: &Arc, body: Value) -> Result { restore_on(fake, &runtime(), &body).await } /// `restore` on `runtime` rather than the tests' default one. async fn restore_on( - fake: &Fake, + fake: &Arc, runtime: &BackupRuntime, body: &Value, ) -> Result { @@ -2104,7 +3331,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // DescribeBackup: canonical not found, as written found. let fake = with_stored(); - let desc = describe_backup(&fake, ACCOUNT, REGION, &spellings) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "AVAILABLE"); @@ -2128,7 +3355,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // DeleteBackup: the full description, and the stored row is gone. let fake = with_stored(); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(deleted.backup_details.backup_status, "DELETED"); @@ -2152,7 +3379,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { .unwrap(); assert_eq!(fake.restores()[0].backup_arn, ours); assert_eq!(fake.source_reads(), 1); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(deleted.backup_details.backup_status, "DELETED"); @@ -2161,7 +3388,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // Neither spelling: BackupNotFoundException from the backend, after both // were tried; the server's own spelling is tried once. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); for (body, reads) in [(stored.clone(), 2), (ours.clone(), 1)] { let before = fake.source_reads(); let err = restore(&fake, restore_body(json!({ "BackupArn": body }))) @@ -2173,13 +3400,13 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { ); assert_eq!(fake.source_reads() - before, reads); assert!(matches!( - describe_backup(&fake, ACCOUNT, REGION, &own(&body)) + describe_backup(&*fake, ACCOUNT, REGION, &own(&body)) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) )); assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&body)) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&body)) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -2267,7 +3494,11 @@ async fn a_restore_request_dropped_while_the_backup_is_read_still_restores() { use std::task::{Context, Waker}; let gate = Arc::new(tokio::sync::Semaphore::new(0)); - let mut fake = Fake::with_source(source(Some(definition("PROVISIONED")))); + let mut fake = Fake::default(); + fake.sources + .lock() + .unwrap() + .insert(arn(ACCOUNT), source(Some(definition("PROVISIONED")))); fake.source_gate = Some(Arc::clone(&gate)); let fake = Arc::new(fake); @@ -2521,14 +3752,131 @@ fn lsi_entry() -> Value { } /// The target input of a restore of `src` with `body`'s overrides, or the -/// engine's refusal. +/// engine's refusal, through both paths: the legacy reader's, and the +/// manifest path over a store-backed backup whose `source_table` is `src`'s +/// definition (`manifest_from_source`). The two must agree (the same input, +/// or the same refusal), so one list of override cases binds both; the +/// manifest records the SSE *description*, not the specification a legacy +/// definition keeps, so that member is compared only when an override set +/// it. A source without a definition (pre-0.0.4) has no manifest form and +/// runs the legacy path alone. async fn restored_input( src: BackupRestoreSource, body: Value, ) -> Result { - let fake = Fake::with_source(src); - restore(&fake, restore_body(body)).await?; - Ok(fake.restores()[0].input.clone()) + let fake = Fake::with_source(src.clone()); + let legacy = restore(&fake, restore_body(body.clone())) + .await + .map(|_| fake.restores()[0].input.clone()); + if legacy.is_err() { + assert!( + fake.restores().is_empty(), + "the legacy path reached storage for a refused {body}" + ); + } + if src.definition.is_none() { + return legacy; + } + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let item: Item = src + .key_schema + .iter() + .map(|k| { + ( + k.attribute_name.clone(), + extenddb_core::types::AttributeValue::S("a".to_owned()), + ) + }) + .collect(); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + manifest_from_source(&src), + &[vec![item]], + false, + ); + let runtime = job_runtime(&mem); + let manifest = restore_on(&fake, &runtime, &restore_body(body.clone())) + .await + .map(|_| fake.restore_claims.lock().unwrap()[0].1.clone()); + wait_for_jobs(&runtime).await; + // The manifest records the SSE description, from which the restore + // derives the enabled KMS specification, so the member is compared when + // the source had SSE enabled or an override set it; a legacy + // definition's disabled specification has no manifest form. + let source_sse_enabled = src + .definition + .as_ref() + .and_then(|d| d.sse_specification.as_ref()) + .and_then(|sse| sse.get("Enabled")) + .and_then(Value::as_bool) + == Some(true); + match (&legacy, &manifest) { + (Ok(a), Ok(b)) => { + let mut a = a.clone(); + let mut b = b.clone(); + if body.get("SSESpecificationOverride").is_none() && !source_sse_enabled { + a.sse_specification = None; + b.sse_specification = None; + } + assert_eq!(a, b, "the two paths built different inputs for {body}"); + } + (Err(a), Err(b)) => { + assert_eq!( + format!("{a:?}"), + format!("{b:?}"), + "the two paths refused differently for {body}" + ); + assert!( + fake.restore_claims.lock().unwrap().is_empty(), + "the manifest path registered a target it then refused for {body}" + ); + } + (a, b) => panic!("the two paths disagree for {body}: legacy {a:?}, manifest {b:?}"), + } + legacy +} + +/// A manifest's `source_table` for a legacy source with a definition: the +/// same keys, indexes, billing and throughput, so the manifest path builds +/// the input the legacy path builds. +fn manifest_from_source(src: &BackupRestoreSource) -> SourceTableSchema { + let def = src.definition.as_ref().expect("a definition"); + let sse_description = def + .sse_specification + .as_ref() + .filter(|s| s.get("Enabled").and_then(Value::as_bool) == Some(true)) + .map(|_| extenddb_core::types::SseDescription { + status: "ENABLED".to_owned(), + sse_type: Some(extenddb_core::types::SseType::KMS), + kms_master_key_arn: None, + }); + SourceTableSchema { + table_name: src.source_table_name.clone(), + table_id: src.source_table_id.clone(), + table_arn: src.source_table_arn.clone(), + table_creation_date_time: src.table_creation_date_time, + key_schema: src.key_schema.clone(), + attribute_definitions: src.attribute_definitions.clone(), + billing_mode: if def.billing_mode == "PAY_PER_REQUEST" { + BillingMode::PayPerRequest + } else { + BillingMode::Provisioned + }, + provisioned_throughput: def.provisioned_throughput.clone(), + on_demand_throughput: def.on_demand_throughput.clone(), + table_class: def.table_class.clone(), + sse_description, + global_secondary_indexes: def.global_secondary_indexes.clone(), + local_secondary_indexes: def.local_secondary_indexes.clone(), + vector_indexes: None, + stream_specification: None, + time_to_live: None, + item_count: src.item_count, + table_size_bytes: src.table_size_bytes, + } } /// The attribute names `input` defines. @@ -2904,10 +4252,12 @@ async fn index_overrides_the_service_refuses_are_refused() { ), ]; for (what, body, expect) in cases { - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - let msg = validation_message(restore(&fake, restore_body(body)).await.unwrap_err()); + let msg = validation_message( + restored_input(source(Some(definition("PROVISIONED"))), body) + .await + .unwrap_err(), + ); assert!(msg.contains(expect), "{what}: {msg}"); - assert!(fake.restores().is_empty(), "{what}: reached storage"); } // A legacy backup restores no index, so any listed one is not in it; an @@ -2996,17 +4346,17 @@ async fn a_provisioned_restore_needs_each_gsis_throughput_overridden() { ), ]; for (what, def, body) in refused { - let fake = Fake::with_source(source(Some(def))); - let msg = validation_message(restore(&fake, restore_body(body)).await.unwrap_err()); + let msg = validation_message(restored_input(source(Some(def)), body).await.unwrap_err()); assert_eq!(msg, expected, "{what}"); - assert!(fake.restores().is_empty(), "{what}: reached storage"); } // The first GSI of two is the one named. - let fake = Fake::with_source(source(Some(two_gsis("PAY_PER_REQUEST")))); let msg = validation_message( - restore(&fake, restore_body(provisioned.clone())) - .await - .unwrap_err(), + restored_input( + source(Some(two_gsis("PAY_PER_REQUEST"))), + provisioned.clone(), + ) + .await + .unwrap_err(), ); assert_eq!(msg, expected); @@ -3206,19 +4556,32 @@ async fn table_class_is_not_restored() { assert_eq!(input.table_class, None); } +/// A source with SSE enabled restores with an enabled KMS specification on +/// both paths: the legacy definition carries the specification, the +/// manifest the description it was derived from. +#[tokio::test] +async fn an_enabled_sse_source_restores_sse_on_both_paths() { + let mut def = definition("PROVISIONED"); + def.sse_specification = Some(json!({ "Enabled": true, "SSEType": "KMS" })); + let input = restored_input(source(Some(def)), json!({})).await.unwrap(); + assert_eq!( + input.sse_specification, + Some(json!({ "Enabled": true, "SSEType": "KMS" })) + ); +} + #[tokio::test] async fn sse_override_replaces_the_recorded_specification() { let mut def = definition("PROVISIONED"); def.sse_specification = Some(json!({ "Enabled": false })); - let fake = Fake::with_source(source(Some(def))); let sse = json!({ "Enabled": true, "SSEType": "KMS", "KMSMasterKeyId": "alias/k" }); - restore( - &fake, - restore_body(json!({ "SSESpecificationOverride": sse })), + let input = restored_input( + source(Some(def)), + json!({ "SSESpecificationOverride": sse }), ) .await .unwrap(); - assert_eq!(fake.restores()[0].input.sse_specification, Some(sse)); + assert_eq!(input.sse_specification, Some(sse)); // A legacy backup takes the override too. let fake = Fake::with_source(source(None)); @@ -3243,14 +4606,10 @@ async fn sse_override_replaces_the_recorded_specification() { #[tokio::test] async fn billing_overrides_shape_the_restore_input() { // PROVISIONED -> PAY_PER_REQUEST drops table and GSI throughput. - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - let out = restore( - &fake, - restore_body(json!({ "BillingModeOverride": "PAY_PER_REQUEST" })), - ) - .await - .unwrap(); - let input = &fake.restores()[0].input; + let body = json!({ "BillingModeOverride": "PAY_PER_REQUEST" }); + let input = restored_input(source(Some(definition("PROVISIONED"))), body.clone()) + .await + .unwrap(); assert_eq!(input.billing_mode, Some(BillingMode::PayPerRequest)); assert!(input.provisioned_throughput.is_none()); assert!( @@ -3258,6 +4617,9 @@ async fn billing_overrides_shape_the_restore_input() { .provisioned_throughput .is_none() ); + // The response zeros the throughput of an on-demand table and index. + let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); + let out = restore(&fake, restore_body(body)).await.unwrap(); let td = &out["TableDescription"]; assert_eq!(td["ProvisionedThroughput"]["ReadCapacityUnits"], 0); assert_eq!( @@ -3266,16 +4628,14 @@ async fn billing_overrides_shape_the_restore_input() { ); // A throughput override on a PROVISIONED backup replaces the table's. - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - restore( - &fake, - restore_body(json!({ + let input = restored_input( + source(Some(definition("PROVISIONED"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 11, "WriteCapacityUnits": 12 }, - })), + }), ) .await .unwrap(); - let input = &fake.restores()[0].input; assert_eq!(input.provisioned_throughput, Some(throughput(11, 12))); assert_eq!( input.global_secondary_indexes.as_ref().unwrap()[0].provisioned_throughput, @@ -3283,28 +4643,25 @@ async fn billing_overrides_shape_the_restore_input() { ); // A throughput override on an on-demand backup without a billing override. - let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); - let err = restore( - &fake, - restore_body(json!({ + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 }, - })), + }), ) .await .unwrap_err(); assert!(validation_message(err).contains("can only be specified")); - assert!(fake.restores().is_empty()); } #[tokio::test] async fn provisioned_override_on_an_on_demand_gsi_has_the_services_text() { - let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); - let err = restore( - &fake, - restore_body(json!({ + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "BillingModeOverride": "PROVISIONED", "ProvisionedThroughputOverride": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, - })), + }), ) .await .unwrap_err(); @@ -3313,12 +4670,11 @@ async fn provisioned_override_on_an_on_demand_gsi_has_the_services_text() { "Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to \ PROVISIONED for index gsi1" ); - assert!(fake.restores().is_empty()); } #[tokio::test] async fn override_shape_is_refused_before_any_storage_read() { - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let err = restore( &fake, restore_body(json!({ "BillingModeOverride": "PROVISIONED" })), @@ -3327,6 +4683,15 @@ async fn override_shape_is_refused_before_any_storage_read() { .unwrap_err(); assert!(validation_message(err).contains("ProvisionedThroughputOverride must")); assert_eq!(fake.source_reads(), 0); + // The manifest path refuses the shape the same way, before any store or + // row read. + let err = restored_input( + source(Some(definition("PROVISIONED"))), + json!({ "BillingModeOverride": "PROVISIONED" }), + ) + .await + .unwrap_err(); + assert!(validation_message(err).contains("ProvisionedThroughputOverride must")); } // --- refusals and validation -------------------------------------------------- @@ -3341,23 +4706,6 @@ async fn multi_part_key_backups_are_refused_in_the_engine() { assert!(fake.restores().is_empty()); } -#[tokio::test] -async fn vector_index_backups_are_refused_in_the_engine() { - let mut def = definition("PAY_PER_REQUEST"); - def.vector_index_names.push("v".to_owned()); - let fake = Fake::with_source(source(Some(def))); - let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); - assert!(msg.contains("vector index"), "{msg}"); - assert!(fake.restores().is_empty()); - - let mut legacy = source(None); - legacy.legacy_vector_index_count = 1; - let fake = Fake::with_source(legacy); - let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); - assert!(msg.contains("vector index"), "{msg}"); - assert!(fake.restores().is_empty()); -} - #[tokio::test] async fn a_stored_definition_create_table_would_refuse_is_refused() { type Corrupt = fn(&mut BackupRestoreSource); @@ -3498,7 +4846,7 @@ async fn a_table_create_table_accepted_restores() { #[tokio::test] async fn a_missing_backup_is_backup_not_found() { - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); assert!(matches!( restore(&fake, restore_body(json!({}))).await.unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -3867,7 +5215,7 @@ async fn name_lengths_are_counted_in_characters_and_table_names_take_the_limits( ); let fake = fake_with_summaries(vec![]); list_backups( - &fake, + &*fake, ACCOUNT, REGION, &raised, @@ -3887,7 +5235,7 @@ async fn name_lengths_are_counted_in_characters_and_table_names_take_the_limits( assert!(msg.contains("length less than or equal to 255"), "{msg}"); let msg = validation_message( list_backups( - &fake, + &*fake, ACCOUNT, REGION, &limits(), @@ -4040,7 +5388,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let on_demand = def.billing_mode == "PAY_PER_REQUEST"; let fake = fake_with_backup("AVAILABLE", source(Some(def))); let described = describe(&fake).await; - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); let deleted = serde_json::to_value(deleted).unwrap(); @@ -4067,7 +5415,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { assert_eq!(deleted, expected); // The backup is gone. assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -4088,7 +5436,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let what = format!("{err:?}"); let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(err); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_or_else(|e| panic!("{what}: {e:?}")); assert_eq!(deleted.backup_details.backup_status, "DELETED", "{what}"); @@ -4109,7 +5457,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(StorageError::Connection("refused".to_owned())); assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), DynamoDbError::InternalServerError(_) @@ -4188,7 +5536,7 @@ fn backup_description(status: &str, billing: &str) -> BackupDescription { } } -fn fake_with_backup(status: &str, src: BackupRestoreSource) -> Fake { +fn fake_with_backup(status: &str, src: BackupRestoreSource) -> Arc { let billing = src.billing_mode.clone(); let fake = Fake::with_source(src); fake.descriptions @@ -4332,7 +5680,7 @@ async fn describe_backup_survives_a_definition_that_does_not_read() { let what = format!("{err:?}"); let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(err); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn_text)) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn_text)) .await .unwrap_or_else(|e| panic!("{what}: {e:?}")); assert_eq!(fake.source_reads(), 1, "{what}"); @@ -4358,7 +5706,7 @@ async fn describe_backup_survives_a_definition_that_does_not_read() { *fake.source_error.lock().unwrap() = Some(err); // The backup operations' mapping reports both as a 500 // (`storage_err_to_dynamo` in backup.rs), which SDKs retry. - match describe_backup(&fake, ACCOUNT, REGION, &own(&arn_text)).await { + match describe_backup(&*fake, ACCOUNT, REGION, &own(&arn_text)).await { Err(DynamoDbError::InternalServerError(_)) => {} other => panic!("{what} must reach the client as an error: {other:?}"), } @@ -4406,8 +5754,8 @@ fn summary(n: u64, millis: u64) -> BackupSummary { } } -fn fake_with_summaries(summaries: Vec) -> Fake { - let fake = Fake::default(); +fn fake_with_summaries(summaries: Vec) -> Arc { + let fake = Arc::new(Fake::default()); *fake.summaries.lock().unwrap() = summaries; fake } @@ -4851,8 +6199,8 @@ fn music_arn(account: &str, id: &str) -> String { format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/{id}") } -fn fake_with_row(status: BackupStatus) -> Fake { - let fake = Fake::default(); +fn fake_with_row(status: BackupStatus) -> Arc { + let fake = Arc::new(Fake::default()); fake.rows .lock() .unwrap() @@ -4874,7 +6222,7 @@ async fn describe_backup_reads_the_catalog_row_first() { arn(ACCOUNT), backup_description("AVAILABLE", "PAY_PER_REQUEST"), ); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_name, "music-nightly"); @@ -4906,13 +6254,13 @@ async fn describe_backup_reads_the_catalog_row_first() { #[tokio::test] async fn describe_backup_maps_the_row_status() { let fake = fake_with_row(BackupStatus::Creating); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "CREATING"); let fake = fake_with_row(BackupStatus::Deleting); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -4923,7 +6271,7 @@ async fn describe_backup_maps_the_row_status() { .lock() .unwrap() .insert(arn(ACCOUNT), backup_description("AVAILABLE", "PROVISIONED")); - let err = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -4942,7 +6290,7 @@ async fn describe_backup_of_another_accounts_id_is_not_found() { const OTHER: &str = "999999999999"; let fake = fake_with_row(BackupStatus::Available); let other = own_for(&music_arn(OTHER, "01791976431438-73d8d5bc"), OTHER); - let err = describe_backup(&fake, OTHER, REGION, &other) + let err = describe_backup(&*fake, OTHER, REGION, &other) .await .unwrap_err(); assert!( @@ -5129,7 +6477,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { for status in [BackupStatus::Deleted, BackupStatus::Failed] { let fake = fake_with_row(status); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5145,7 +6493,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { } let fake = fake_with_row(BackupStatus::Creating); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5162,7 +6510,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { // No store configured: every row is a row of another store. let fake = fake_with_row(BackupStatus::Available); let msg = validation_message( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), ); @@ -5185,7 +6533,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { .lock() .unwrap() .insert(arn(ACCOUNT), "Earlier".to_owned()); - let err = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -5206,7 +6554,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { let fake = fake_with_row(BackupStatus::Available); let desc = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5227,7 +6575,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { let fake = fake_with_row(BackupStatus::Deleting); let desc = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5264,7 +6612,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { .unwrap() .insert(arn(ACCOUNT), answer.clone()); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5298,7 +6646,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { .unwrap() .insert(arn(ACCOUNT), DeleteStart::Status(BackupStatus::Available)); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5406,7 +6754,7 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy .unwrap() .insert(legacy_arn.clone(), description); - let desc = describe_backup(&fake, ACCOUNT, REGION, &spellings) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_arn, legacy_arn); @@ -5431,7 +6779,7 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy assert_eq!(fake.restores().len(), 1); assert_eq!(fake.restores()[0].backup_arn, legacy_arn); - let desc = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let desc = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -5452,14 +6800,14 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy /// or not the row is of the configured store (the in-use check runs before /// the other-store check, as the delete's does); a row /// of another store is a ValidationException, with no store configured and -/// with a different one configured; and an AVAILABLE row of the -/// configured store is refused by this release. The runtime has no free -/// slot, so a permit taken anywhere before the row's refusal, even one -/// dropped again, would turn the refusal into LimitExceeded; the legacy -/// reader is never reached. +/// with a different one configured; and an AVAILABLE row of the healthy +/// configured store passes to the permit, the next check, with no store +/// read. The runtime has no free slot, so a permit taken anywhere before a +/// row's refusal, even one dropped again, would turn the refusal into +/// LimitExceeded; the legacy reader is never reached. #[tokio::test] async fn restore_over_a_row_refuses_before_the_permit() { - async fn check(fake: Fake, runtime: &BackupRuntime) -> DynamoDbError { + async fn check(fake: Arc, runtime: &BackupRuntime) -> DynamoDbError { assert_eq!(runtime.free_slots(), 0, "every slot is held"); let err = restore_table_from_backup( &fake, @@ -5479,11 +6827,12 @@ async fn restore_over_a_row_refuses_before_the_permit() { assert!(fake.restores().is_empty()); err } + let configured_mem = Arc::new(MemStore::default()); let configured = BackupRuntime::new(1) .with_store(ConfiguredStore::new( store().store_kind, store().store_root, - Arc::new(MemStore::default()) as Arc, + Arc::clone(&configured_mem) as Arc, None, Ok(store()), )) @@ -5531,8 +6880,29 @@ async fn restore_over_a_row_refuses_before_the_permit() { matches!(err, DynamoDbError::BackupInUseException(ref m) if m.contains("another table")), "the in-use check runs before the other-store check: {err:?}" ); - let msg = validation_message(check(fake_with_row(BackupStatus::Available), &configured).await); - assert!(msg.contains("in-catalog backups only"), "{msg}"); + // An AVAILABLE row of the healthy configured store passes the row checks: + // the permit is the next check, before any store read, so with every + // slot held the request is LimitExceeded and the store saw no get. + let fake = fake_with_row(BackupStatus::Available); + let err = restore_table_from_backup( + &fake, + &configured, + ACCOUNT, + REGION, + &LimitsConfig::default(), + &restore_body(json!({})), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(_)), + "the permit is the check after the row's: {err:?}" + ); + assert!( + configured_mem.gets().is_empty(), + "no store read before the permit" + ); + assert!(fake.restore_claims.lock().unwrap().is_empty()); let msg = validation_message(check(fake_with_row(BackupStatus::Available), &none).await); assert!(msg.contains("store other than"), "{msg}"); // A configured store that is not the row's: the row is of another store, @@ -5564,7 +6934,7 @@ async fn restore_over_a_row_refuses_before_the_permit() { #[tokio::test] async fn restore_takes_the_permit_last() { async fn restore_with( - fake: &Fake, + fake: &Arc, runtime: &BackupRuntime, ) -> Result<(String, Value), DynamoDbError> { let body = restore_body(json!({})); @@ -5602,7 +6972,7 @@ async fn restore_takes_the_permit_last() { // A refusal earlier in the order takes no slot: a missing backup. let runtime = BackupRuntime::new(1); - let empty = Fake::default(); + let empty = Arc::new(Fake::default()); let err = restore_with(&empty, &runtime).await.unwrap_err(); assert!( matches!(err, DynamoDbError::BackupNotFoundException(_)), @@ -5651,7 +7021,7 @@ async fn restore_checks_the_target_name_after_the_backup_and_before_the_permit() ..Default::default() } } - async fn check(fake: Fake, target: TableDescription) -> (Fake, DynamoDbError) { + async fn check(fake: Arc, target: TableDescription) -> (Arc, DynamoDbError) { let full = BackupRuntime::new(1).with_permit_wait(Duration::ZERO); let _held = full.take_permit().await.unwrap(); fake.tables @@ -5712,7 +7082,7 @@ async fn restore_checks_the_target_name_after_the_backup_and_before_the_permit() // The backup missing: BackupNotFound whatever the target's state, and // the name is never read. for (i, target) in targets.iter().enumerate() { - let (fake, err) = check(Fake::default(), target.clone()).await; + let (fake, err) = check(Arc::new(Fake::default()), target.clone()).await; assert!( matches!(err, DynamoDbError::BackupNotFoundException(ref m) if *m == format!("Backup not found: {}", arn(ACCOUNT))), @@ -6113,7 +7483,7 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting /// the claim is gone both run. #[tokio::test] async fn jobs_are_refused_while_a_maintenance_claim_is_held() { - let fake = Arc::new(Fake::with_source(source(Some(definition("PROVISIONED"))))); + let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); *fake.maintenance_held.lock().unwrap() = true; let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7)); let err = create_backup( @@ -6279,6 +7649,26 @@ fn held(holder: &Arc) -> maintenance::HeldClaim { ) } +/// The claim object the `Fake`'s `maintenance_claim` hands out: `Forward` +/// whose release also frees the `Fake`'s record, so a second command may +/// take the claim once the first released it. +struct ReleasingForward(Arc>>, Arc); + +impl extenddb_storage::MaintenanceClaim for ReleasingForward { + fn generation(&self) -> u64 { + self.1.generation() + } + fn heartbeat( + &self, + ) -> BoxedFuture<'_, Result> { + self.1.heartbeat() + } + fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> { + *self.0.lock().unwrap() = None; + self.1.release() + } +} + fn seconds_since_epoch() -> f64 { std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) @@ -6453,9 +7843,11 @@ async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() { /// restore record whose target is still CREATING (the reconciler's duty, /// not a running job) and a FAILED backup row whose owner is dead; busy at /// the limit, with the jobs named (a CREATING row and a FAILED row with a -/// live heartbeat, a RUNNING restore) and announced once; a signal, during -/// a sleep and before the first read; the catalog unreadable; the claim -/// lost. +/// live heartbeat, a RUNNING restore) and announced once; the ages read on +/// the catalog's clock, not this host's (a skewed catalog clock decides +/// busy and quiet against what this host's clock would say); a signal, +/// during a sleep and before the first read; the catalog unreadable; the +/// claim lost. #[tokio::test(start_paused = true)] async fn the_wait_reports_each_outcome_as_itself() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; @@ -6576,27 +7968,30 @@ async fn the_wait_reports_each_outcome_as_itself() { // --- the reconciler ------------------------------------------------------------- fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow { - RestoreJobRow { - target: RestoreTargetKey { + RestoreJobRow::reconciler_row( + RestoreTargetKey { account_id: ACCOUNT.to_owned(), table_name: name.to_owned(), table_id: format!("tid-{name}"), }, - source_backup_arn: arn(ACCOUNT), + arn(ACCOUNT), kind, - status: extenddb_storage::backup_catalog::JobStatus::Running, - owner_generation: 0, - heartbeat_at: 0.0, - } + extenddb_storage::backup_catalog::JobStatus::Running, + 0, + 0.0, + ) } /// A legacy restore whose owner is dead is marked FAILED and its target -/// removed; a manifest restore is left for its resumer; an abandoned -/// in-catalog CREATING backup is removed through the reader's delete. +/// removed; a manifest restore whose store is not the configured one is +/// left alone when this instance has no store configured (it is the job of +/// the instance that holds the record's store, not this one's to fail); +/// an abandoned in-catalog CREATING backup is removed through the reader's +/// delete. #[tokio::test] async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() { use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.target_still_creating.lock().unwrap() = true; *fake.record_still_running.lock().unwrap() = true; fake.dead_jobs.lock().unwrap().extend([ @@ -6628,12 +8023,31 @@ async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups assert!(!report.maintenance_held); assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]); assert!(report.failed_targets_removed.is_empty()); - assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]); - assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]); + assert!(report.manifest_restores_resumed.is_empty()); + assert!( + report.manifest_restores_failed.is_empty(), + "an instance with no configured store leaves a store-backed record alone: {:?}", + report.manifest_restores_failed + ); + assert!( + fake.failed_jobs.lock().unwrap().is_empty() + || fake + .failed_jobs + .lock() + .unwrap() + .iter() + .all(|(_, reason)| reason != OTHER_STORE), + "no OTHER_STORE write when this instance has no store: {:?}", + fake.failed_jobs.lock().unwrap() + ); assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]); let failed = fake.failed_jobs.lock().unwrap().clone(); - assert_eq!(failed.len(), 1); + assert_eq!( + failed.len(), + 1, + "only the legacy restore is failed: {failed:?}" + ); assert_eq!(failed[0].0.table_name, "LegacyTarget"); assert_eq!(failed[0].1, reconciler::OWNER_DEAD); assert_eq!( @@ -6648,7 +8062,7 @@ async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups #[tokio::test] async fn the_reconciler_leaves_a_target_whose_record_moved() { use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = false; *fake.target_still_creating.lock().unwrap() = true; fake.dead_jobs @@ -6679,7 +8093,7 @@ async fn the_reconciler_leaves_a_target_whose_record_moved() { #[tokio::test] async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_failed_write() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = true; *fake.target_still_creating.lock().unwrap() = false; let dead = dead_job(RestoreKind::Legacy, "LegacyTarget"); @@ -6894,6 +8308,86 @@ async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_wind assert_eq!(report.registrations_swept, [arn_n(9)]); } +/// The restore arm of a cycle run by the holder of the maintenance claim with +/// a zero dead-owner window (`backup purge` on SQLite): a RUNNING `manifest` +/// record whose owner is dead is FAILED `OWNER_DEAD` and its target removed, +/// as 4(f) asks of a held claim, and not taken over and resumed inside the +/// command. The command's claim was heartbeated a millisecond ago, which a +/// zero window read through the clock finds stale; a cycle that read it +/// would take the arm for a free claim and spawn a resume. +#[tokio::test] +async fn a_cycle_run_by_the_claim_holder_fails_a_dead_manifest_restore_rather_than_resuming_it() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "precondition: the stopped restore left its record RUNNING" + ); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + *fake.maintenance_beat_age.lock().unwrap() = Some(Duration::from_millis(1)); + // The restore's own request read the claim; the cycle must not. + fake.maintenance_checks.lock().unwrap().clear(); + + let command = job_runtime(&mem); + let report = reconciler::cycle( + &fake, + &command, + Duration::ZERO, + "test", + &mut reconciler::ReconcilerState::for_maintenance_holder(), + ) + .await + .unwrap(); + wait_for_jobs(&command).await; + assert!( + report.manifest_restores_resumed.is_empty(), + "the holder's cycle resumed a restore: {:?}", + report.manifest_restores_resumed + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!( + fake.failed_jobs + .lock() + .unwrap() + .iter() + .any(|(key, reason)| key.table_name == "MusicRestored" + && reason == reconciler::OWNER_DEAD), + "{:?}", + fake.failed_jobs.lock().unwrap() + ); + assert!(fake.maintenance_checks.lock().unwrap().is_empty()); +} + /// The worker runs a cycle at once (observed through the maintenance check, /// the cycle's first read), records the worker metrics the other workers /// record, then waits an interval, and stops at the shutdown token. @@ -7004,7 +8498,7 @@ async fn a_failed_reconciler_cycle_records_a_worker_error() { #[tokio::test] async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_failed_again() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.target_still_creating.lock().unwrap() = true; let mut failed = dead_job(RestoreKind::Legacy, "FailedTarget"); failed.status = JobStatus::Failed; @@ -7032,70 +8526,14 @@ async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_fa ["FailedTarget", "FailedManifestTarget"] ); assert!(report.legacy_restores_removed.is_empty()); - assert!(report.manifest_restores_seen.is_empty()); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(report.manifest_restores_failed.is_empty()); assert_eq!( fake.removed_targets.lock().unwrap().as_slice(), [failed.target, failed_manifest.target] ); } -/// A dead manifest record, which this release cannot act on, is reported -/// once per record for as long as the process runs, not every cycle. -#[tokio::test] -async fn a_dead_manifest_record_is_logged_once_per_record() { - use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); - fake.dead_jobs - .lock() - .unwrap() - .push(dead_job(RestoreKind::Manifest, "ManifestTarget")); - let mut state = reconciler::ReconcilerState::default(); - let first = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(first.manifest_restores_seen, ["ManifestTarget"]); - assert_eq!(first.manifest_restores_logged, ["ManifestTarget"]); - let second = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(second.manifest_restores_seen, ["ManifestTarget"]); - assert!( - second.manifest_restores_logged.is_empty(), - "the same record is not reported twice" - ); - fake.dead_jobs - .lock() - .unwrap() - .push(dead_job(RestoreKind::Manifest, "Another")); - let third = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(third.manifest_restores_logged, ["Another"]); - assert!( - fake.failed_jobs.lock().unwrap().is_empty() - && fake.removed_targets.lock().unwrap().is_empty(), - "a manifest record is never acted on by this release" - ); -} - /// The FAILED write for a dead legacy restore is conditioned on the /// `owner_generation` and `heartbeat_at` the listing read (design 4(e)): a /// heartbeat that lands between the listing and the write says the owner is @@ -7104,7 +8542,7 @@ async fn a_dead_manifest_record_is_logged_once_per_record() { #[tokio::test] async fn the_reconciler_leaves_a_record_whose_owner_heartbeated_after_the_listing() { use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.target_still_creating.lock().unwrap() = true; *fake.record_still_running.lock().unwrap() = true; let mut listed = dead_job(RestoreKind::Legacy, "LateBeat"); @@ -7183,8 +8621,8 @@ fn newer_arn(account: &str) -> String { format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/01791976999999-0badf00d") } -fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake { - let fake = Fake::default(); +fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Arc { + let fake = Arc::new(Fake::default()); for (arn, status) in rows { fake.rows .lock() @@ -7196,15 +8634,15 @@ fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake { /// A restore of a store-backed row checks the target name once the row is /// known to exist and AVAILABLE, before the backup's in-use rule and before -/// this release's refusal of the store-backed restore (the service's order: -/// a restore of a backup into its own CREATING target is -/// `TableInUseException`, measured): an existing table of the name is -/// `TableAlreadyExistsException`, a CREATING restore target -/// `TableInUseException` naming its source backup, and a free name reaches -/// the refusal naming the store. A row that is not AVAILABLE answers for -/// itself before the name is read. +/// the store is read (the service's order: a restore of a backup into its +/// own CREATING target is `TableInUseException`, measured): an existing +/// table of the name is `TableAlreadyExistsException`, a CREATING restore +/// target `TableInUseException` naming its source backup, and a free name +/// goes on to the manifest read (here `BackupNotFoundException`: the row is +/// AVAILABLE and the store holds nothing under its prefix). A row that is +/// not AVAILABLE answers for itself before the name is read. #[tokio::test] -async fn a_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusal() { +async fn a_store_backed_restore_checks_the_target_name_after_the_row_and_before_the_store() { let mem = Arc::new(MemStore::default()); let runtime = runtime_on(&mem, store()); let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Available)]); @@ -7248,14 +8686,15 @@ async fn a_store_backed_restore_checks_the_target_name_after_the_row_and_before_ ); fake.tables.lock().unwrap().remove("MusicRestored"); + assert!(mem.gets().is_empty(), "no store read before the name"); let err = restore_on(&fake, &runtime, &restore_body(json!({}))) .await .unwrap_err(); assert!( - matches!(err, DynamoDbError::ValidationException(ref m) - if m.contains("is in a backup store; this release restores in-catalog backups only")), + matches!(err, DynamoDbError::BackupNotFoundException(_)), "{err:?}" ); + assert!(!mem.gets().is_empty(), "the free name reached the store"); // A row that is not AVAILABLE: its own answer, the name unread. let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleted)]); @@ -7336,7 +8775,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { &identity_file("store-1", "another-catalog"), ); let fake = fake_with_row(BackupStatus::Available); - let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &*fake, "test") .await .unwrap_err(); assert_eq!( @@ -7351,7 +8790,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { // Restored: the check passes. mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}"); - let identity = store_identity::recheck(&mem, &store(), &fake, "test").await; + let identity = store_identity::recheck(&mem, &store(), &*fake, "test").await; assert_eq!(identity, Ok(())); } @@ -7387,7 +8826,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -7407,7 +8846,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -7424,7 +8863,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -7439,7 +8878,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -7472,7 +8911,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7489,7 +8928,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7503,7 +8942,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &empty, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7545,12 +8984,12 @@ async fn an_unreachable_store_is_unhealthy_and_recovers_on_recheck() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.set_down(true); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let err = store_identity::establish( &*mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7634,7 +9073,7 @@ async fn a_stalled_store_check_is_abandoned_at_its_bound() { let runtime = runtime_on(&mem, store()); let configured = runtime.configured_store().unwrap(); assert_eq!(configured.check_timeout(), LOCAL_CHECK_TIMEOUT); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = true; *fake.target_still_creating.lock().unwrap() = true; fake.dead_jobs @@ -8031,7 +9470,7 @@ async fn a_catalog_read_failure_leaves_the_health_unchanged() { ); // And through a cycle: a catalog that fails `catalog_id` leaves a // healthy store healthy. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.catalog_id_error.lock().unwrap() = true; let report = reconciler::cycle( &fake, @@ -8099,7 +9538,7 @@ async fn a_store_checked_after_startup_is_checking_until_the_first_check() { ); // The first check: `validate` fails on the unreachable store. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let report = reconciler::cycle( &fake, &runtime, @@ -8198,7 +9637,7 @@ async fn an_unhealthy_store_refuses_create_delete_and_restore_unless_bound_to_th fake.permits.lock().unwrap().clear(); // DeleteBackup of a row of the store. - let fake = Arc::new(fake_with_row(BackupStatus::Available)); + let fake = fake_with_row(BackupStatus::Available); let err = delete_backup(&*fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); @@ -8297,7 +9736,7 @@ async fn delete_backup_of_a_configured_store_row_deletes_manifest_prefix_then_ma (newer_arn(ACCOUNT), BackupStatus::Available), ]); let runtime = runtime_on(&mem, store()); - let desc = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -8338,7 +9777,7 @@ async fn a_failed_deletion_stays_deleting_and_the_reconciler_finishes_it() { let runtime = runtime_on(&mem, store()); mem.set_down(true); - let out = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let out = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!( @@ -8510,7 +9949,7 @@ async fn a_prefix_that_still_lists_an_object_is_left_deleting() { let store_handle = Refilling(MemStore::default()); let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]); let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap(); - let err = finish_backup_delete(&fake, &store_handle, &parsed) + let err = finish_backup_delete(&*fake, &store_handle, &parsed) .await .unwrap_err(); assert!(err.contains("still lists"), "{err}"); @@ -8529,7 +9968,7 @@ mod review_round_2 { Interrupt, PurgeOutcome, Stop, Wait, WaitOutcome, purge_rows, wait_for_quiet, }; - async fn cycle_on(fake: &Fake, runtime: &BackupRuntime) -> reconciler::CycleReport { + async fn cycle_on(fake: &Arc, runtime: &BackupRuntime) -> reconciler::CycleReport { reconciler::cycle( fake, runtime, @@ -8553,7 +9992,7 @@ mod review_round_2 { mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); *mem.validate_error.lock().unwrap() = Some("write probe: Input/output error (os error 5)".to_owned()); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let runtime = runtime_unhealthy( &mem, StoreProblem::Validate("write probe: Input/output error (os error 5)".to_owned()), @@ -8719,9 +10158,7 @@ mod review_round_2 { /// their own ValidationException, not LimitExceededException. #[tokio::test] async fn the_maintenance_refusal_comes_after_the_requests_own_checks() { - let fake = Arc::new(Fake::with_source(source(Some(definition( - "PAY_PER_REQUEST", - ))))); + let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); *fake.maintenance_held.lock().unwrap() = true; let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7)); @@ -8790,7 +10227,7 @@ mod review_round_2 { async fn a_recheck_catches_rows_with_another_id_a_lost_file_and_a_foreign_file() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let runtime = runtime_on(&mem, store()); let configured = runtime.configured_store().unwrap(); assert_eq!(cycle_on(&fake, &runtime).await.store_healthy, Some(true)); @@ -9017,7 +10454,7 @@ mod review_round_2 { STORE_IDENTITY_KEY, &identity_file("store-1", "other-catalog"), ); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let checking = || { ConfiguredStore::checking( store().store_kind, @@ -9085,8 +10522,8 @@ mod review_round_2 { let (first, second) = tokio::time::timeout( Duration::from_secs(120), futures::future::join( - delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))), - delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))), + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))), + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))), ), ) .await @@ -9157,8 +10594,8 @@ mod review_round_2 { mem.set_delay(Some(Duration::from_secs(3))); let started = tokio::time::Instant::now(); let (first, second) = futures::future::join( - delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))), - delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))), + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))), + delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&newer_arn(ACCOUNT))), ) .await; assert_eq!(first.unwrap().backup_details.backup_status, "DELETED"); @@ -10191,8 +11628,7 @@ async fn the_heartbeat_handle_confirms_the_claim_within_two_periods() { let handle = job::heartbeat::HeartbeatHandle::start( Arc::clone(&claim), period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); // Half a period past the tenth tick, so that tick's beat has confirmed. @@ -10222,8 +11658,7 @@ async fn the_heartbeat_handle_confirms_the_claim_within_two_periods() { let handle = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); tokio::time::sleep(period + period / 2).await; @@ -10406,6 +11841,17 @@ impl LogLines { .map(|(_, m)| m.clone()) .collect() } + + /// The messages logged at `level`, in order. + fn at_level(&self, level: tracing::Level) -> Vec { + self.0 + .lock() + .unwrap() + .iter() + .filter(|(l, _)| *l == level) + .map(|(_, m)| m.clone()) + .collect() + } } struct MessageVisitor<'a>(&'a mut String); @@ -10841,8 +12287,7 @@ async fn a_stopped_heartbeat_timer_starts_no_put() { let heartbeat = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); heartbeat.stop_timer(); @@ -11037,7 +12482,7 @@ fn seed_dead_creating_with_manifest(fake: &Fake, mem: &MemStore, complete: bool) } /// One cycle on `fake` and `mem`, the store healthy. -async fn cycle_on(fake: &Fake, runtime: &BackupRuntime) -> reconciler::CycleReport { +async fn cycle_on(fake: &Arc, runtime: &BackupRuntime) -> reconciler::CycleReport { reconciler::cycle( fake, runtime, @@ -11061,7 +12506,7 @@ async fn the_reconcilers_failed_store_calls_are_counted_by_call() { use extenddb_core::metrics::{MetricName, MetricsCollector}; let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let now = seconds_since_epoch(); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); let dead_prefix = BackupArn::parse(&dead).unwrap().store_prefix(); @@ -11130,7 +12575,7 @@ async fn the_reconcilers_failed_store_calls_are_counted_by_call() { async fn the_reconciler_resolves_a_dead_creating_row_with_a_manifest_to_available() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); let stats_pass = serde_json::json!({"TableName": "Music", "ItemCount": 0}); fake.rows @@ -11210,7 +12655,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() ] { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let mut dead = seed_dead_creating_with_manifest(&fake, &mem, shape != "short file"); let prefix = BackupArn::parse(&dead).unwrap().store_prefix(); let golden = stored_manifest(&mem, &dead); @@ -11323,7 +12768,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() // store answering, resolves it. let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); *mem.fail_gets_under.lock().unwrap() = Some(BackupArn::parse(&dead).unwrap().store_prefix()); let runtime = runtime_on(&mem, store()); @@ -11358,7 +12803,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() async fn a_dead_creating_row_whose_status_write_errors_is_left_for_the_next_cycle() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); fake.status_write_error .store(true, std::sync::atomic::Ordering::Release); @@ -11389,7 +12834,7 @@ async fn a_dead_creating_row_whose_status_write_errors_is_left_for_the_next_cycl async fn a_row_failed_by_the_cycle_loses_its_prefix_in_that_cycle() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, false); assert!(!objects_under(&mem, &dead).is_empty()); let runtime = runtime_on(&mem, store()); @@ -11419,7 +12864,7 @@ async fn a_row_failed_by_the_cycle_loses_its_prefix_in_that_cycle() { async fn a_dead_creating_row_taken_over_by_another_is_left_to_that_owner() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); fake.takeover_lost_to_another .store(true, std::sync::atomic::Ordering::Release); @@ -11454,7 +12899,7 @@ async fn the_reconciler_deletes_failed_and_deleted_prefixes_and_removes_tombston { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let now = seconds_since_epoch(); let stale_failed = arn_n(1); let fresh_failed = arn_n(2); @@ -11549,124 +12994,5883 @@ async fn the_reconciler_deletes_failed_and_deleted_prefixes_and_removes_tombston assert!(mem.deletes.lock().unwrap().is_empty()); assert_eq!(fake.rows.lock().unwrap().len(), 9); assert_eq!( - fake.rows.lock().unwrap()[&dead].status, - BackupStatus::Creating + fake.rows.lock().unwrap()[&dead].status, + BackupStatus::Creating + ); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + + // Under the maintenance claim, the store healthy again: the + // dead-CREATING duty runs, the deleting duties and the registrations + // sweep (which the backend has something for) do not. + fake.swept_registrations.lock().unwrap().push(arn_n(9)); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert!(report.maintenance_held); + assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + assert!(report.tombstones_removed.is_empty()); + assert!( + report.registrations_swept.is_empty(), + "the sweep does not run under the claim" + ); + assert!(mem.deletes.lock().unwrap().is_empty()); + assert_eq!(fake.rows.lock().unwrap().len(), 9); + + // An AVAILABLE row past the retention with an empty prefix: a tombstone + // is a FAILED or DELETED row and nothing else. It enters the catalog + // here, older than the row the cycle above resolved, because the + // store's check heads the newest AVAILABLE row's manifest and this row + // has none. + let old_available_with_an_empty_prefix = + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/01700000000000-00000000"); + let mut r = row(&old_available_with_an_empty_prefix, BackupStatus::Available); + r.heartbeat_at = now - 1000.0; + r.completed_at = Some(now - 1000.0); + fake.rows + .lock() + .unwrap() + .insert(old_available_with_an_empty_prefix.clone(), r); + + // The claim released: each duty on its clock. + *fake.maintenance_held.lock().unwrap() = false; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.failed_prefixes_deleted, + std::slice::from_ref(&stale_failed) + ); + assert_eq!( + report.deleted_prefixes_swept, + [ + deleted_with_objects.clone(), + old_tombstone_with_objects.clone() + ] + ); + // The tombstone duty runs after the two deleting duties, so a prefix + // they emptied this cycle counts as empty here. + let mut removed = report.tombstones_removed.clone(); + removed.sort(); + assert_eq!( + removed, + [ + stale_failed.clone(), + deleted_with_objects.clone(), + old_failed_tombstone.clone(), + old_deleted_tombstone.clone(), + old_tombstone_with_objects.clone(), + ] + ); + assert_eq!(report.registrations_swept, [arn_n(9)]); + assert!(report.acted()); + assert_eq!( + objects_under(&mem, &fresh_failed).len(), + 1, + "a FAILED row with a live heartbeat keeps its prefix" + ); + assert!(objects_under(&mem, &stale_failed).is_empty()); + assert!(objects_under(&mem, &deleted_with_objects).is_empty()); + assert_eq!( + objects_under(&mem, &old_failed_with_objects_and_a_live_heartbeat).len(), + 1, + "no duty empties the prefix of a FAILED row with a live heartbeat" + ); + { + let rows = fake.rows.lock().unwrap(); + assert!(rows.contains_key(&fresh_failed), "its heartbeat is live"); + assert!( + rows.contains_key(&young_tombstone), + "its retention has not passed" + ); + assert!( + rows.contains_key(&old_failed_with_objects_and_a_live_heartbeat), + "its prefix still lists an object" + ); + assert!( + rows.contains_key(&old_available_with_an_empty_prefix), + "AVAILABLE is not a tombstone status" + ); + assert_eq!( + rows[&old_available_with_an_empty_prefix].status, + BackupStatus::Available + ); + assert_eq!(rows.len(), 5, "the four kept and the resolved CREATING row"); + } + assert!( + !fake + .remove_attempts + .lock() + .unwrap() + .contains(&old_available_with_an_empty_prefix), + "the duty asks the catalog to remove FAILED and DELETED rows only; the backends would \ + refuse an AVAILABLE one, and the duty does not lean on that" + ); + + // The next cycle has nothing left to do. + let report = cycle_on(&fake, &runtime).await; + assert!(report.tombstones_removed.is_empty()); + assert!(report.deleted_prefixes_swept.is_empty()); + assert!(report.failed_prefixes_deleted.is_empty()); + assert!(report.creating_resolved.is_empty()); +} + +// --- the store-backed restore --------------------------------------------------- + +use extenddb_core::backup_format::{ + BackupDocuments, DataFileEntry, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, ItemOrder, KeyOrder, + KeyRangeTracker, MANIFEST_CHECKSUM_FILE, ManifestHeader, SnapshotDescriptor, SourceTableSchema, + data_file_key, encode_data_file, +}; +use extenddb_core::metrics::MetricName; +use extenddb_storage::snapshot::source_table_schema; + +/// The `Music` table's source schema as a manifest carries it. +fn music_source_table() -> SourceTableSchema { + source_table_schema(&music_table(), None).unwrap() +} + +/// `music_source_table` as an on-demand table (vector indexes need one). +fn on_demand_source_table() -> SourceTableSchema { + let mut source_table = music_source_table(); + source_table.billing_mode = BillingMode::PayPerRequest; + source_table.provisioned_throughput = None; + for gsi in &mut source_table.global_secondary_indexes { + gsi.provisioned_throughput = None; + } + source_table +} + +/// A backup of `source_table` under `mem` for `arn`, as the writer leaves +/// one: one data file per entry of `files`, encoded with the format's +/// encoder; key-ordered with the tracker's ranges when `key_ordered`; the +/// four documents built by `BackupDocuments::build`; and the row AVAILABLE +/// in `fake` with the manifest's `source_table`. Returns the manifest. +fn seed_manifest_backup( + fake: &Fake, + mem: &MemStore, + arn: &str, + mut source_table: SourceTableSchema, + files: &[Vec], + key_ordered: bool, +) -> ExtenddbManifest { + let parsed = BackupArn::parse(arn).unwrap(); + let prefix = parsed.store_prefix(); + let mut tracker = key_ordered.then(|| { + KeyRangeTracker::new( + &KeyOrder::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .unwrap(), + ) + }); + let mut entries = Vec::new(); + let mut total = 0; + for (i, items) in files.iter().enumerate() { + let (bytes, checksums) = encode_data_file(items.iter().cloned()).unwrap(); + let range = tracker.as_mut().map(|t| { + for item in items { + t.push(item).unwrap(); + } + t.next_file().expect("a file with items has a range") + }); + let key = data_file_key(u32::try_from(i + 1).unwrap()); + mem.insert(&format!("{prefix}{key}"), &bytes); + total += items.len(); + entries.push((DataFileEntry::from_checksums(key, &checksums, range), None)); + } + source_table.item_count = i64::try_from(total).unwrap(); + seed_manifest_documents(fake, mem, arn, source_table, entries, key_ordered) +} + +/// The documents of a backup whose data files are already under the prefix, +/// from `entries`, and the AVAILABLE row; the manifest as built. +fn seed_manifest_documents( + fake: &Fake, + mem: &MemStore, + arn: &str, + source_table: SourceTableSchema, + entries: Vec<(DataFileEntry, Option)>, + key_ordered: bool, +) -> ExtenddbManifest { + let parsed = BackupArn::parse(arn).unwrap(); + let prefix = parsed.store_prefix(); + let total: u64 = entries.iter().map(|(e, _)| e.item_count).sum(); + let docs = BackupDocuments::build( + ManifestHeader { + backup_arn: parsed, + backup_name: "nightly".to_owned(), + source_backend: "test".to_owned(), + source_extenddb_version: "0.0.0".to_owned(), + snapshot: SnapshotDescriptor { + kind: "test".to_owned(), + marker: String::new(), + }, + item_order: if key_ordered { + ItemOrder::KeyOrdered + } else { + ItemOrder::Unordered + }, + source_table, + }, + entries, + "", + &store().store_root, + ) + .unwrap(); + for (name, bytes) in docs.documents() { + mem.insert(&format!("{prefix}{name}"), bytes); + } + let manifest = docs.manifest().clone(); + let mut available = row(arn, BackupStatus::Available); + available.source_table_details_json = serde_json::to_value(&manifest.source_table).unwrap(); + available.size_bytes = manifest.backup_size_bytes; + available.item_count = i64::try_from(total).unwrap(); + fake.rows.lock().unwrap().insert(arn.to_owned(), available); + manifest +} + +/// `n` songs from `from`, in key order. +fn songs_from(from: usize, n: usize) -> Vec { + (from..from + n) + .map(|i| song(&format!("artist-{i:05}"), &format!("title-{i:05}"))) + .collect() +} + +/// A healthy runtime on `mem` with a spool at `spool` (an S3 store's shape), +/// the job settings and a collector. +fn spooled_runtime(mem: &Arc, spool: &std::path::Path) -> BackupRuntime { + runtime() + .with_store( + ConfiguredStore::new( + StoreKind::S3, + "bucket/prefix".to_owned(), + Arc::clone(mem) as Arc, + Some(Spool { + path: spool.to_path_buf(), + spool_host: "host-a:/var/spool".to_owned(), + }), + Ok(StoreIdentity { + store_kind: StoreKind::S3, + store_root: "bucket/prefix".to_owned(), + store_id: "store-1".to_owned(), + }), + ) + .with_manifest_names("bucket".to_owned(), "prefix/".to_owned()), + ) + .with_job_settings(job_settings()) + .with_metrics(Arc::new(extenddb_core::metrics::MetricsCollector::new())) +} + +/// The `s3_row` of `arn`: the row with the spooled runtime's identity. +fn s3_identity() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::S3, + store_root: "bucket/prefix".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// The `RestoreJobs` outcomes the collector drained, with their count. +fn restore_outcomes(runtime: &BackupRuntime) -> Vec<(String, f64)> { + let mut outcomes: Vec<(String, f64)> = runtime + .metrics() + .unwrap() + .drain(Duration::ZERO) + .into_iter() + .filter(|b| b.metric == MetricName::RestoreJobs) + .map(|b| (b.operation, b.sum)) + .collect(); + outcomes.sort_by(|a, b| a.0.cmp(&b.0)); + outcomes +} + +/// Every metric bucket the collector drained, as `(name, operation, sum)`. +fn drained(runtime: &BackupRuntime) -> Vec<(MetricName, String, f64)> { + runtime + .metrics() + .unwrap() + .drain(Duration::ZERO) + .into_iter() + .map(|b| (b.metric, b.operation, b.sum)) + .collect() +} + +/// Wait until `n` `bulk_load` calls have begun (a job waiting at the gate +/// counts). +async fn wait_for_bulk_loads_started(fake: &Fake, n: usize) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake + .bulk_loads_started + .load(std::sync::atomic::Ordering::Acquire) + < n + { + assert!(tokio::time::Instant::now() < deadline, "no bulk_load began"); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// The status writes the restore claims received, as `(from, to, reason)`. +fn restore_writes(fake: &Fake) -> Vec<(JobStatus, JobStatus, Option)> { + fake.restore_writes + .lock() + .unwrap() + .iter() + .map(|w| (w.from, w.to, w.fields.failure_reason.clone())) + .collect() +} + +/// The store keys `mem` saw `get` for, without the prefix. +fn gets_under(mem: &MemStore, arn: &str) -> Vec { + let prefix = BackupArn::parse(arn).unwrap().store_prefix(); + mem.gets() + .into_iter() + .filter_map(|k| k.strip_prefix(&prefix).map(str::to_owned)) + .collect() +} + +/// The happy path. The request: the row passes, the permit is taken, the +/// sidecar and manifest are read, the registration carries the manifest's +/// table (key schema, GSI, billing) and the row's store, the maintenance +/// claim is read once after it, and the response is CREATING with the +/// manifest's provenance in its `RestoreSummary`. The job: every file is +/// read once to hash it (in order), then once more to decode it, each file +/// goes to one `bulk_load` from item 0 with `reapplying` false, the items +/// come back whole and in order, the target is counted, the record is DONE +/// and the table ACTIVE with its GSI, no status write was needed, the slot +/// is free; `RestoreJobs{succeeded}`, `RestoreBytesDownloaded` of every +/// file's bytes, `WorkerLastSuccess{RestoreJob}`. +#[tokio::test] +async fn a_store_backed_restore_answers_creating_and_its_job_loads_the_backup() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 2), songs_from(5, 4)]; + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let table = &response["TableDescription"]; + assert_eq!(table["TableStatus"], "CREATING"); + assert_eq!(table["TableName"], "MusicRestored"); + assert_eq!(table["RestoreSummary"]["SourceBackupArn"], arn(ACCOUNT)); + assert_eq!(table["RestoreSummary"]["SourceTableArn"], source_arn()); + assert_eq!( + table["RestoreSummary"]["RestoreDateTime"], + json!(manifest.backup_creation_date_time) + ); + assert_eq!(table["RestoreSummary"]["RestoreInProgress"], true); + let claims = fake.restore_claims.lock().unwrap().clone(); + assert_eq!(claims.len(), 1); + assert_eq!(claims[0].0, ACCOUNT); + let input = &claims[0].1; + assert_eq!(input.table_name, "MusicRestored"); + assert_eq!(input.key_schema, music_table().key_schema); + assert_eq!(input.billing_mode, Some(BillingMode::Provisioned)); + assert_eq!(gsi_names(input), ["GenreIndex"]); + assert_eq!(input.vector_indexes, None); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.kind, RestoreKind::Manifest); + assert_eq!(record.store, Some(store())); + assert_eq!(record.spool_host, None, "a filesystem store has no spool"); + assert_eq!( + fake.maintenance_checks.lock().unwrap().as_slice(), + [job_settings().stale_job_after], + "the maintenance claim was read once, after the registration" + ); + + wait_for_jobs(&runtime).await; + assert_eq!( + fake.bulk_load_files(), + [(0, 0, false, 3), (1, 0, false, 2), (2, 0, false, 4)] + ); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(record.completed_at.is_some()); + assert_eq!(record.file_index, Some(2)); + assert_eq!(record.items_applied, Some(4)); + let desc = fake.tables.lock().unwrap()["MusicRestored"].clone(); + assert_eq!(desc.table_status, TableStatus::Active); + assert_eq!( + desc.item_count, 0, + "the restored table's ItemCount is 0 at ACTIVE, as the service reports it" + ); + assert!( + restore_writes(&fake).is_empty(), + "the flip is the backend's one write" + ); + assert_eq!( + fake.finish_sizes.lock().unwrap().as_slice(), + [0], + "the restored table's TableSizeBytes is 0 at ACTIVE, as the service reports it" + ); + assert_eq!(runtime.free_slots(), 4); + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads, + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + ], + "every file hashed before any is decoded, each read twice in place" + ); + let bytes: u64 = manifest.data_files.iter().map(|f| f.bytes).sum(); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("succeeded".to_owned(), 1.0)] + ); + // Summed across the drain's buckets (one per wall-clock minute). + let downloaded: f64 = metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreBytesDownloaded) + .map(|(_, _, sum)| *sum) + .sum(); + assert!( + (downloaded - bytes as f64).abs() < f64::EPSILON, + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob"), + "{metrics:?}" + ); + let started = lines.containing(": started, 3 data file(s), 9 items, key-ordered"); + assert_eq!(started.len(), 1, "{started:?}"); + let active = lines.containing(": ACTIVE, 9 items (9 applied by this run), "); + assert_eq!(active.len(), 1, "{active:?}"); +} + +/// The request's order on the manifest path, each refusal before the next +/// step and the registration: a table of the target's name is +/// `TableAlreadyExists` with no row read; a missing sidecar or manifest is +/// `BackupNotFound` with the operator's line naming `--prune-missing`; a +/// sidecar that does not match, a manifest naming another backup or another +/// table are `ValidationException`s; none of them registers anything or +/// takes a slot for longer than the request. +#[tokio::test] +async fn the_manifest_request_refuses_in_order_before_the_registration() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + + // A taken name: before the row. + fake.tables.lock().unwrap().insert( + "MusicRestored".to_owned(), + TableDescription { + table_name: "MusicRestored".to_owned(), + table_status: TableStatus::Active, + ..Default::default() + }, + ); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableAlreadyExistsException(_)), + "{err:?}" + ); + // The name is checked after the row is read (the service's order: a + // missing backup answers first) and before any store read. + assert_eq!(fake.row_reads.lock().unwrap().len(), 1); + assert!(mem.gets().is_empty()); + fake.row_reads.lock().unwrap().clear(); + fake.tables.lock().unwrap().remove("MusicRestored"); + + // A missing sidecar, then a missing manifest. + let sidecar_key = format!("{prefix}{MANIFEST_CHECKSUM_FILE}"); + let manifest_key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let sidecar = mem.objects.lock().unwrap().remove(&sidecar_key).unwrap(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupNotFoundException(ref m) if m.contains(&arn(ACCOUNT))), + "{err:?}" + ); + let prune = lines.containing("backup sync --prune-missing"); + assert_eq!(prune.len(), 1, "{prune:?}"); + assert!(prune[0].contains(MANIFEST_CHECKSUM_FILE), "{}", prune[0]); + mem.insert(&sidecar_key, &sidecar); + let manifest = mem.objects.lock().unwrap().remove(&manifest_key).unwrap(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupNotFoundException(_)), + "{err:?}" + ); + assert_eq!(lines.containing("backup sync --prune-missing").len(), 2); + + // A sidecar that does not match the manifest. + mem.insert(&manifest_key, &manifest); + mem.insert( + &sidecar_key, + b"0000000000000000000000000000000000000000000000000000000000000000\n", + ); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!(msg.contains("checksum sidecar does not match"), "{msg}"); + mem.insert(&sidecar_key, &sidecar); + + // A manifest naming another backup (another account's, the same id), + // with a sidecar that matches it. + let other = arn("999999999999"); + let mut parsed: Value = serde_json::from_slice(&manifest).unwrap(); + parsed["backup_arn"] = json!(other); + parsed["source_table"]["table_arn"] = json!(format!( + "arn:aws:dynamodb:{REGION}:999999999999:table/Music" + )); + let other_bytes = serde_json::to_vec(&parsed).unwrap(); + let other_sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&other_bytes) + ) + }; + mem.insert(&manifest_key, &other_bytes); + mem.insert(&sidecar_key, other_sidecar.as_bytes()); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!( + msg.contains("names another backup") && msg.contains(&other), + "{msg}" + ); + + // A manifest whose source table is not the ARN's: the reader's own + // identity rule refuses it at the parse. + let mut parsed: Value = serde_json::from_slice(&manifest).unwrap(); + parsed["source_table"]["table_name"] = json!("Other"); + let renamed = serde_json::to_vec(&parsed).unwrap(); + let renamed_sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&renamed) + ) + }; + mem.insert(&manifest_key, &renamed); + mem.insert(&sidecar_key, renamed_sidecar.as_bytes()); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!( + msg.contains("cannot restore from") + && msg.contains("names table \"Music\" but source_table.table_name is \"Other\""), + "{msg}" + ); + + assert!( + fake.restore_claims.lock().unwrap().is_empty(), + "nothing was registered" + ); + assert_eq!(runtime.free_slots(), 4, "every refusal returned its slot"); + assert_eq!(runtime.running_jobs(), 0); +} + +/// The maintenance claim is checked after the registration, not before: a +/// held claim leaves a record that was FAILED `maintenance` from RUNNING +/// under the job's claim, its target removed, and the request answered +/// `LimitExceededException`; the registration ran (the row is what the +/// command's wait sees). A claim that cannot be read fails the record with +/// the read's error and answers it. +#[tokio::test] +async fn a_manifest_restore_under_a_maintenance_claim_fails_its_record_after_registering() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + *fake.maintenance_held.lock().unwrap() = true; + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(ref m) if m.contains("maintenance claim")), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_claims.lock().unwrap().len(), + 1, + "the registration ran first" + ); + assert_eq!( + restore_writes(&fake), + [( + JobStatus::Running, + JobStatus::Failed, + Some(job::MAINTENANCE.to_owned()) + )] + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target and its record are gone" + ); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert_eq!(runtime.free_slots(), 4); + // A FAILED record is a job end: the outcome beside the worker error. + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)], + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::RestoreJobDuration && op == "failed"), + "{metrics:?}" + ); + + *fake.maintenance_held.lock().unwrap() = false; + *fake.maintenance_error.lock().unwrap() = Some("pool exhausted".to_owned()); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 2); + assert!( + writes[1].2.as_deref().unwrap().contains("pool exhausted"), + "{writes:?}" + ); +} + +/// The registration's own refusals reach the client as the legacy +/// backend's do: a name taken under the registration is +/// `TableAlreadyExistsException`, a CREATING restore target +/// `TableInUseException`, a backup deleted under the lock +/// `BackupNotFoundException`, a backup in use `BackupInUseException`; the +/// slot is returned each time and no job runs. With no slot free the +/// request is `LimitExceededException` before any store read. +#[tokio::test] +async fn the_registrations_refusals_keep_their_classes_and_return_the_slot() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + type Expected = fn(&DynamoDbError) -> bool; + let cases: [(StorageError, Expected); 4] = [ + ( + StorageError::TableAlreadyExists("MusicRestored".to_owned()), + |e| matches!(e, DynamoDbError::TableAlreadyExistsException(_)), + ), + ( + StorageError::TableBeingRestored { + table_name: "MusicRestored".to_owned(), + backup_arn: arn(ACCOUNT), + }, + |e| matches!(e, DynamoDbError::TableInUseException(_)), + ), + ( + extenddb_storage::backup_catalog::backup_not_found(&arn(ACCOUNT)), + |e| matches!(e, DynamoDbError::BackupNotFoundException(_)), + ), + ( + extenddb_storage::backup_catalog::backup_in_use("Earlier", &arn(ACCOUNT)), + |e| matches!(e, DynamoDbError::BackupInUseException(_)), + ), + ]; + for (error, expected) in cases { + let shown = format!("{error:?}"); + *fake.claim_restore_error.lock().unwrap() = Some(error); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!(expected(&err), "{shown}: {err:?}"); + wait_for_jobs(&runtime).await; + assert_eq!(runtime.free_slots(), 4, "{shown}"); + } + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!(restore_writes(&fake).is_empty()); + + // No wait for a slot: the refusal is what the cell reads. + let full = job_runtime(&mem).with_permit_wait(Duration::ZERO); + let mut held = Vec::new(); + for _ in 0..4 { + held.push(full.take_permit().await.unwrap()); + } + let before = mem.gets().len(); + let err = restore_on(&fake, &full, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(_)), + "{err:?}" + ); + assert_eq!(mem.gets().len(), before, "no store read before the permit"); +} + +/// A second restore while the first's target is CREATING is +/// `BackupInUseException` naming the backup, through the catalog's in-use +/// rule the registration wrote; a second restore into the same name, of the +/// same backup or of another, is `TableInUseException` (the target name is +/// checked once the backup is known to exist and before its in-use rule, the +/// service's measured order); once the first is ACTIVE the backup is free +/// again. +#[tokio::test] +async fn a_backup_being_restored_is_in_use_until_its_target_is_active() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + seed_manifest_backup( + &fake, + &mem, + &arn_n(2), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + assert_eq!( + runtime.free_slots(), + 3, + "the job holds its slot past the reply, until it ends" + ); + let err = restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "Second" })), + ) + .await + .unwrap_err(); + // The service's text names the backup, not the target (the log line + // names the target). + let in_use = extenddb_storage::backup_catalog::backup_in_use_message(&arn(ACCOUNT)); + assert!( + matches!(err, DynamoDbError::BackupInUseException(ref m) if *m == in_use), + "{err:?}" + ); + assert_eq!(runtime.free_slots(), 3, "a refusal returns its own slot"); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableInUseException(_)), + "the same backup into its own CREATING target (measured on DynamoDB): {err:?}" + ); + let err = restore_on( + &fake, + &runtime, + &restore_body(json!({ "BackupArn": arn_n(2) })), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableInUseException(_)), + "another backup into the name being restored into: {err:?}" + ); + gate.add_permits(1); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + runtime.free_slots(), + 4, + "the slot is free once the job ended" + ); + *fake.bulk_load_gate.lock().unwrap() = None; + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "Second" })), + ) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(fake.restored_items("Second").len(), 2); +} + +/// The tamper cell: file 2 of 3 does not hash to the manifest's digest. +/// Every file is verified before any is loaded, so no `bulk_load` is +/// called at all, the record is FAILED naming the file, the target is +/// removed (the FAILED write matched), and the failed outcome is recorded. +#[tokio::test] +async fn a_corrupt_second_file_fails_before_any_load() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 3), songs_from(6, 3)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000002.json.gz"); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + let last = bytes.len() - 1; + bytes[last] ^= 0xff; + mem.insert(&key, &bytes); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no file was loaded" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz does not match the manifest: "), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)).len(), + 4, + "sidecar, manifest, files 1 and 2" + ); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, sum)| *m == MetricName::WorkerErrorCount + && op == "RestoreJob" + && (*sum - 1.0).abs() < f64::EPSILON), + "a FAILED job is a worker error: {metrics:?}" + ); + let failed = lines.containing(": FAILED (data file data/000002.json.gz does not match"); + assert_eq!(failed.len(), 1, "{failed:?}"); + assert!( + failed[0].ends_with("the target was removed"), + "{}", + failed[0] + ); +} + +/// A transport failure at the `get` and one mid-body are each retried +/// after a backoff and the restore completes; the store's error is counted +/// once per failed attempt; a denial is not retried and fails the restore +/// naming the file; a file missing from the store fails it at once. +#[tokio::test] +async fn transport_failures_are_retried_and_other_failures_are_not() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, 1); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // The first data-file get failed; its retry, the next data-file get, + // stops mid-body. + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while gets_under(&mem, &arn(ACCOUNT)).len() < 3 { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + mem.fail_get_after_bytes(5); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 4); + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads, + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000001.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + ], + "file 1 took three attempts" + ); + // Summed across the drain's buckets: the two errors are 1 to 2 s apart + // and a run that crosses a wall-clock minute puts them in two. + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert!(errors.iter().all(|(_, op, _)| op == "get"), "{errors:?}"); + let total: f64 = errors.iter().map(|(_, _, sum)| *sum).sum(); + assert!((total - 2.0).abs() < f64::EPSILON, "{errors:?}"); + + // A denial: no retry. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::PermissionDenied, 1); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason + .starts_with("reading data file data/000001.json.gz from the store: permission denied"), + "{reason}" + ); + assert_eq!(gets_under(&mem, &arn(ACCOUNT)).len(), 3); + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert_eq!( + errors, + [(MetricName::BackupStoreErrors, "get".to_owned(), 1.0)], + "a denial is the store's error too" + ); + + // A file that is not there. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + mem.objects + .lock() + .unwrap() + .remove(&format!("{prefix}data/000002.json.gz")); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes[0].2.as_deref(), + Some("data file data/000002.json.gz is not in the store") + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert_eq!( + errors, + [(MetricName::BackupStoreErrors, "get".to_owned(), 1.0)], + "an absent file is the store's error too" + ); +} + +/// A store that keeps failing is retried for the bound and then fails the +/// restore naming the file and the bound, with the record heartbeated +/// meanwhile and no `bulk_load`; the backoff never exceeds a minute. +#[tokio::test(start_paused = true)] +async fn a_download_exhausting_the_retry_bound_fails_the_restore() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, usize::MAX); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let started = tokio::time::Instant::now(); + // The paused clock runs ahead through the backoffs, so the job's end is + // waited for without `wait_for_jobs`'s wall-clock bound. + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(50)).await; + } + let elapsed = started.elapsed(); + assert!( + elapsed >= Duration::from_secs(15 * 60) && elapsed < Duration::from_secs(16 * 60 + 2), + "{elapsed:?}" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("downloading data/000001.json.gz: transport error"), + "{reason}" + ); + assert!(reason.contains("the bound is 900 s"), "{reason}"); + let attempts = gets_under(&mem, &arn(ACCOUNT)).len() - 2; + // 1+2+4+8+16+32 s, then 60 s each: about 20 attempts in 15 minutes. + assert!((18..=24).contains(&attempts), "{attempts} attempts"); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) > 100, + "the record was heartbeated through the wait" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// The count after the last file: a backend that reports one item fewer +/// than the manifest (one of two rows with one key kept) fails the restore +/// with the count reason, after every file was loaded; and the duplicate +/// rule of the backend itself (an unordered backup holding one key in two +/// files) fails the second file's load with the backend's text. +#[tokio::test] +async fn a_duplicate_key_fails_the_restore_by_the_count_or_by_the_load() { + // The count runs on both orders. + for key_ordered in [false, true] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + key_ordered, + ); + let runtime = job_runtime(&mem); + *fake.count_override.lock().unwrap() = Some(3); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.bulk_loads.lock().unwrap().len(), + 2, + "every file was loaded first ({key_ordered})" + ); + let writes = restore_writes(&fake); + assert_eq!( + writes[0].2.as_deref(), + Some("restored 3 items, the manifest lists 4: the backup holds a key twice"), + "{key_ordered}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); + } + + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + let mut second = songs_from(5, 2); + second.push(song("artist-00001", "title-00001")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 3), second], + false, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_loads.lock().unwrap().len(), 2); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: ") + && reason.contains("holds a primary key the target already has"), + "{reason}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); + assert_eq!(fake.restored_items("MusicRestored").len(), 0); +} + +/// A key-ordered file whose first item is below its recorded range fails +/// the file through the reader's range check, naming the file; nothing of +/// the file is applied and the target is removed. +#[tokio::test] +async fn a_file_leaving_its_key_range_fails_the_restore() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 3)]; + let source_table = music_source_table(); + let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap(); + let prefix = parsed.store_prefix(); + let order = KeyOrder::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .unwrap(); + let mut tracker = KeyRangeTracker::new(&order); + let mut entries = Vec::new(); + for (i, items) in files.iter().enumerate() { + let (bytes, checksums) = encode_data_file(items.iter().cloned()).unwrap(); + for item in items { + tracker.push(item).unwrap(); + } + let mut range = tracker.next_file().unwrap(); + if i == 1 { + // The recorded range starts at the file's second item. + range.first = order.endpoint_key(&items[1]).unwrap(); + } + let key = data_file_key(u32::try_from(i + 1).unwrap()); + mem.insert(&format!("{prefix}{key}"), &bytes); + entries.push(( + DataFileEntry::from_checksums(key, &checksums, Some(range)), + None, + )); + } + seed_manifest_documents(&fake, &mem, &arn(ACCOUNT), source_table, entries, true); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert!(loads[1].stream_error.is_some(), "{loads:?}"); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: "), + "{reason}" + ); + assert!( + reason.contains("range") || reason.contains("key"), + "{reason}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); +} + +/// An item the validator refuses (a `Year` of `1E126`, #618's shape) fails +/// the file before the backend sees it, naming the key and the attribute; +/// the backend's stream ended with the error and nothing was applied. +#[tokio::test] +async fn a_validator_refusal_fails_the_restore_naming_the_item() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut second = songs_from(2, 1); + second.push(song_with_bad_number("artist-00003", "title-00003", "1E126")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), second], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert_eq!(loads[1].items.len(), 0, "the chunk was never sent"); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with( + "data file data/000002.json.gz: item cannot be restored: key \ + {\"Artist\":{\"S\":\"artist-00003\"},\"SongTitle\":{\"S\":\"title-00003\"}}, attribute Year: " + ), + "{reason}" + ); + assert_eq!( + loads[0].items.len(), + 2, + "file 1 was applied before file 2 failed" + ); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target was removed" + ); +} + +/// The FAILED-write gate: a record another owner took (its generation +/// changed) is not this job's to write, so a failing job makes no table +/// access: no `remove_restore_target`, the record untouched. An unknown +/// outcome is re-read: a write that landed is matched and the target is +/// removed with one write; one that was lost is sent once more; one that +/// was lost and whose record a successor then wrote FAILED under its own +/// generation is not this job's, so no removal. +#[tokio::test] +async fn a_failed_write_that_does_not_match_removes_no_target_and_an_unknown_one_is_re_read() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + // The job is held at its first `bulk_load`; the record changes hands + // meanwhile, so the load fails under the old generation and the FAILED + // write matches no record. + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + let other = { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.owner_generation = new_generation(); + record.owner_generation + }; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!(writes[0].1, JobStatus::Failed); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, other); + let line = lines.containing("no table access was made"); + assert_eq!(line.len(), 1, "{line:?}"); + + // A corrupt file, so the failure path runs with the claim intact. A + // write lost with its connection and followed by a takeover whose + // successor wrote FAILED itself re-reads FAILED under another + // generation: the record is not this job's, so no removal. + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000001.json.gz"); + { + let (lines, _guard) = LogLines::capture(); + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + bytes[0] ^= 0xff; + mem.insert(&key, &bytes); + *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::TakenAndWritten); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(restore_writes(&fake).len(), 1, "no second write"); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access: the FAILED record is the successor's" + ); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let line = + lines.containing("the record is not this job's to write (the row carries generation"); + assert_eq!(line.len(), 1, "{line:?}"); + } + for (unknown, expected_writes) in [(UnknownWrite::Applied, 1), (UnknownWrite::Lost, 2)] { + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + bytes[0] ^= 0xff; + mem.insert(&key, &bytes); + *fake.unknown_write.lock().unwrap() = Some(unknown); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(restore_writes(&fake).len(), expected_writes, "{unknown:?}"); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1, "{unknown:?}"); + assert!(fake.restore_job("MusicRestored").is_none(), "{unknown:?}"); + } +} + +/// A lost heartbeat ends the job with no catalog write and no table access: +/// the job is held inside the first `bulk_load`, the claim is reported lost +/// at the next beat, and the job stops; the record stays RUNNING under its +/// generation, nothing is removed, the outcome is `failed`. +#[tokio::test] +async fn a_lost_heartbeat_stops_the_restore_with_no_catalog_write_and_no_table_access() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // The job reaches the gate; then the claim is lost at the next beat. + wait_for_bulk_loads_started(&fake, 1).await; + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + wait_for_jobs(&runtime).await; + assert!(restore_writes(&fake).is_empty(), "no catalog write"); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!(fake.restored_items("MusicRestored").is_empty()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + // The stop line names the timer's reason, not the two-period clause. + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost: the row is gone \ + or carries another generation; the record is left to its owner or to the reconciler", + ); + assert_eq!(stopped.len(), 1, "{:?}", lines.containing("stopped with")); + let raised = lines.containing( + "the job's heartbeat raised the loss of its claim; the job stops at its next check: the \ + claim was lost: the row is gone or carries another generation", + ); + assert_eq!(raised.len(), 1, "{raised:?}"); + assert_eq!(lines.at_level(tracing::Level::WARN), raised); + assert_eq!(runtime.free_slots(), 4); +} + +/// A loss the timer raised while the job was inside a step (here the count) +/// is named by the next check between steps: the pre-flip check's stop line +/// carries the timer's reason, not the two-period clause. +#[tokio::test] +async fn a_loss_raised_inside_a_step_is_named_by_the_next_check() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let count_gate = closed_gate(); + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.count_calls.load(std::sync::atomic::Ordering::Acquire) == 0 { + assert!(tokio::time::Instant::now() < deadline, "no count began"); + tokio::time::sleep(Duration::from_millis(5)).await; + } + // The claim is lost at the next beat, while the job waits in the count. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + while lines.containing("raised the loss of its claim").is_empty() { + assert!( + tokio::time::Instant::now() < deadline, + "the loss was not raised" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + count_gate.add_permits(1); + wait_for_jobs(&runtime).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 0, "no DONE write"); + assert!(restore_writes(&fake).is_empty(), "no catalog write"); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running + ); + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost: the row is gone \ + or carries another generation; the record is left to its owner or to the reconciler", + ); + assert_eq!(stopped.len(), 1, "{:?}", lines.containing("stopped with")); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// Shutdown between files: the job stops at its next file with no status +/// write, the record stays RUNNING with its progress, the spool entry stays +/// with every file verified, the outcome is `cancelled` and no worker +/// error is recorded; `drain_jobs` returns with every job ended. +#[tokio::test] +async fn a_shutdown_leaves_the_record_running_with_its_spool() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let s3_arn = arn(ACCOUNT); + let files = [songs_from(0, 2), songs_from(2, 2), songs_from(4, 2)]; + seed_manifest_backup(&fake, &mem, &s3_arn, music_source_table(), &files, true); + fake.rows.lock().unwrap().get_mut(&s3_arn).unwrap().store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!( + fake.restore_job("MusicRestored") + .unwrap() + .spool_host + .as_deref(), + Some("host-a:/var/spool") + ); + // File 1 loads; file 2 waits at the gate. + wait_for_bulk_loads_started(&fake, 2).await; + let left = runtime.drain_jobs(Duration::from_millis(100)).await; + assert_eq!(left, 1, "the job waits at the gate past the bound"); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert!( + restore_writes(&fake).is_empty(), + "no status write at shutdown" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!( + record.file_index.is_some(), + "the progress of the loaded file(s) stays" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.is_dir(), "the spool entry stays"); + for i in 0..3 { + assert!(entry.join(format!("{i:06}.json.gz")).is_file()); + assert!(entry.join(format!("{i:06}.json.gz.verified")).is_file()); + } + assert!(entry.join(EXTENDDB_MANIFEST_FILE).is_file()); + assert!(entry.join(MANIFEST_CHECKSUM_FILE).is_file()); + // One drain, both assertions read from it (a second drain reads an + // empty collector and asserts nothing). + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("cancelled".to_owned(), 1.0)] + ); + assert!( + !metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "a clean shutdown is not a worker error: {metrics:?}" + ); +} + +/// On an S3 store the files are spooled: each is downloaded once into the +/// entry, marked `.verified`, and the load reads the spool, not the store; +/// the entry is removed when the job ends; and a completed job's second +/// read of the store is never made. +#[tokio::test] +async fn an_s3_restore_spools_every_file_and_removes_its_entry_at_the_end() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while !entry.join("000001.json.gz.verified").is_file() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&entry).unwrap().permissions().mode() & 0o777, + 0o700 + ); + assert_eq!( + std::fs::metadata(entry.join("000000.json.gz")) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o600 + ); + } + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 4); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000002.json.gz", + ], + "each file read from the store once" + ); + assert!(!entry.exists(), "the entry is removed at the end"); +} + +/// A resumed job skips the files and items the record and the progress rows +/// account for: the first open file is loaded from its `start_at` with +/// `reapplying`, the rest from 0 without it, and the count covers the whole +/// manifest. +#[tokio::test] +async fn a_resume_loads_from_the_recorded_progress() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 3), songs_from(6, 3)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + // A dead owner's record: file 0 complete, one item of file 1 applied, + // the target holding those four items. + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(1); + record.items_applied = Some(1); + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.items.truncate(4); + target.active = false; + } + fake.bulk_loads.lock().unwrap().clear(); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_load_files(), [(1, 1, true, 2), (2, 0, false, 3)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// The vector refusal on the manifest path, narrowed to a target without +/// vector support: a manifest with a non-empty `vector_indexes` into a +/// target whose engine has no vector search is the legacy text's +/// `ValidationException` with nothing registered; the same manifest into a +/// target with vector support registers the table with its vector indexes; +/// `null` and `[]` register on both. +#[tokio::test] +async fn vector_index_backups_are_refused_in_the_engine() { + let spec: extenddb_core::types::VectorIndexSpecification = serde_json::from_value(json!({ + "IndexName": "vidx", + "VectorAttribute": { "AttributeName": "emb" }, + "Dimensions": 3, + "DistanceFunction": "COSINE", + "Projection": { "ProjectionType": "ALL" } + })) + .unwrap(); + let cases: [( + Option>, + bool, + bool, + ); 6] = [ + (Some(vec![spec.clone()]), false, false), + (Some(vec![spec.clone()]), true, true), + (None, false, true), + (None, true, true), + (Some(Vec::new()), false, true), + (Some(Vec::new()), true, true), + ]; + for (vector_indexes, capable, registered) in cases { + let what = format!("{vector_indexes:?} into a target with vector support {capable}"); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + fake.vector_search + .store(capable, std::sync::atomic::Ordering::Release); + let mut source_table = on_demand_source_table(); + source_table.vector_indexes = vector_indexes.clone(); + seed_manifest_backup(&fake, &mem, &arn(ACCOUNT), source_table, &[songs(1)], true); + let runtime = job_runtime(&mem); + let result = restore_on(&fake, &runtime, &restore_body(json!({}))).await; + wait_for_jobs(&runtime).await; + if registered { + result.unwrap_or_else(|e| panic!("{what}: {e:?}")); + let claims = fake.restore_claims.lock().unwrap(); + assert_eq!(claims.len(), 1, "{what}"); + let expected = vector_indexes.filter(|v| !v.is_empty()); + assert_eq!(claims[0].1.vector_indexes, expected, "{what}"); + } else { + let msg = validation_message(result.unwrap_err()); + assert_eq!( + msg, + format!( + "backup {} has 1 vector index(es); restoring a table with vector indexes is \ + not supported by this storage backend", + arn(ACCOUNT) + ), + "{what}" + ); + assert!(fake.restore_claims.lock().unwrap().is_empty(), "{what}"); + } + assert_eq!(runtime.free_slots(), 4, "{what}"); + } +} + +/// The legacy reader's vector refusals stay as they were: a definition +/// naming a vector index and a pre-0.0.4 source counting one. +#[tokio::test] +async fn legacy_vector_backups_keep_the_reader_refusal() { + let mut def = definition("PAY_PER_REQUEST"); + def.vector_index_names.push("v".to_owned()); + let fake = Fake::with_source(source(Some(def))); + let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); + assert!(msg.contains("vector index"), "{msg}"); + assert!(fake.restores().is_empty()); + + let mut legacy = source(None); + legacy.legacy_vector_index_count = 1; + let fake = Fake::with_source(legacy); + let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); + assert!(msg.contains("vector index"), "{msg}"); + assert!(fake.restores().is_empty()); +} + +/// A multi-part-key table (two HASH attributes, backed up unordered) +/// registers on the manifest path when the target's limits allow the shape: +/// `ensure_single_part_base_key` is the legacy reader's rule alone. +#[tokio::test] +async fn a_multi_part_key_manifest_is_registered() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut source_table = music_source_table(); + source_table.key_schema = vec![ + key("Artist", KeyType::Hash), + key("Label", KeyType::Hash), + key("SongTitle", KeyType::Range), + ]; + source_table.attribute_definitions.push(attr("Label")); + source_table + .attribute_definitions + .retain(|a| a.attribute_name != "Genre"); + source_table.global_secondary_indexes.clear(); + let mut items = songs(2); + for item in &mut items { + item.insert( + "Label".to_owned(), + extenddb_core::types::AttributeValue::S("indie".to_owned()), + ); + } + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + source_table, + &[items.clone()], + false, + ); + let runtime = job_runtime(&mem); + let limits = LimitsConfig { + allow_multipart_table_keys: true, + ..LimitsConfig::default() + }; + let (_, response) = restore_table_from_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &limits, + &restore_body(json!({})), + ) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + wait_for_jobs(&runtime).await; + let claims = fake.restore_claims.lock().unwrap(); + assert_eq!(claims[0].1.key_schema.len(), 3); + assert_eq!(fake.restored_items("MusicRestored"), items); +} + +/// The override contract on the manifest path is the legacy path's: +/// `restored_input` runs every case through both and asserts they agree, +/// so one list of cases binds both (the override tests above). This cell +/// pins the comparison itself with a case of each kind: an applied +/// override and a refused one. +#[tokio::test] +async fn the_override_cases_run_through_both_paths() { + let input = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ + "BillingModeOverride": "PROVISIONED", + "ProvisionedThroughputOverride": { "ReadCapacityUnits": 11, "WriteCapacityUnits": 12 }, + "GlobalSecondaryIndexOverride": [gsi_entry(Some((3, 4)))], + }), + ) + .await + .unwrap(); + assert_eq!(input.billing_mode, Some(BillingMode::Provisioned)); + assert_eq!(input.provisioned_throughput, Some(throughput(11, 12))); + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 } }), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::ValidationException(_)), + "{err:?}" + ); +} + +// --- the reconciler's restore duties ------------------------------------------- + +/// A dead RUNNING manifest record of the configured store whose owner was +/// on another host: the reconciler takes it over (a fresh generation, this +/// host's `spool_host`, `heartbeat_at` refreshed so a second taker finds the +/// record live) and spawns the resume, which reads the manifest from the +/// store, removes the entry this host's earlier run of the job left (its +/// files were verified under a record that has since been another host's), +/// downloads every file again and finishes the restore from the recorded +/// progress; a live record is left alone. +#[tokio::test] +async fn the_reconciler_resumes_a_dead_manifest_restore_with_its_spool_host() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + // A record left by a dead owner on another host, file 0 applied. + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 2).await; + runtime.drain_jobs(Duration::from_millis(100)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let before = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(before.status, JobStatus::Running); + assert_eq!( + before.file_index, + Some(1), + "file 1 was in its load when the shutdown fired" + ); + *fake.bulk_load_gate.lock().unwrap() = None; + fake.bulk_loads.lock().unwrap().clear(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + + // A live owner on this host: nothing happens, and the sweep keeps the + // entry of a RUNNING record that names this host. + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert!(report.manifest_restores_resumed.is_empty()); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert!(report.spool_entries_swept.is_empty()); + assert!(entry.join("000000.json.gz.verified").is_file()); + assert!(entry.join("000001.json.gz.verified").is_file()); + + { + // As a backend whose file 1 write did not land would have left it: + // the record at file 0, the target holding file 0's items, the + // owner on another host (the record moved there and its job died), + // its heartbeat old. This host's entry from the stopped job still + // holds both files verified; it is not the record's and the resume + // removes it before use (the sweep would, a cycle later). + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.spool_host = Some("host-b:/var/spool".to_owned()); + record.heartbeat_at = 1234.5; + record.file_index = Some(0); + record.items_applied = Some(2); + fake.restore_tables + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .items + .truncate(2); + } + mem.gets.lock().unwrap().clear(); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + assert!(report.acted()); + let after = fake.restore_job("MusicRestored").unwrap(); + assert_ne!(after.owner_generation, before.owner_generation); + assert_eq!(after.spool_host.as_deref(), Some("host-a:/var/spool")); + assert!( + after.heartbeat_at > 1234.5, + "the takeover refreshed the heartbeat: {}", + after.heartbeat_at + ); + assert_eq!( + fake.restore_takeovers.lock().unwrap().as_slice(), + [( + "tid-MusicRestored".to_owned(), + Some("host-a:/var/spool".to_owned()) + )] + ); + wait_for_jobs(&runtime2).await; + assert_eq!(fake.bulk_load_files(), [(1, 0, true, 2)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + EXTENDDB_MANIFEST_FILE, + MANIFEST_CHECKSUM_FILE, + "data/000002.json.gz", + ], + "the manifest from the store; file 0 was recorded applied, so only file 1 is \ + downloaded (the resume downloads only the files its progress leaves to load)" + ); + assert_eq!( + lines + .containing( + "has no live owner; taken over by this instance and resumed with 2 items of data \ + file 0 recorded as applied" + ) + .len(), + 1, + "the reconciler's line states the record's progress" + ); + assert_eq!( + lines + .containing("started, 2 data file(s), 4 items, key-ordered; resumed from data file 1 after 0 of its items") + .len(), + 1, + "the job's line names the first open file" + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from the store") + .len(), + 1 + ); + assert_eq!( + lines + .containing("is left from a run whose record moved to another host") + .len(), + 1 + ); + assert!(!entry.exists()); +} + +/// The manifest arm's refusals: no free permit skips the record to the next +/// cycle with no takeover; a held maintenance claim fails it `OWNER_DEAD` +/// and removes its target like a legacy record; a record of another store +/// is FAILED naming `--move-store` and its target left to the FAILED-target +/// duty, which removes it next cycle; an unhealthy store skips it. +#[tokio::test] +async fn the_reconciler_skips_or_fails_a_dead_manifest_restore_it_cannot_resume() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + + // No permit. + let full = job_runtime(&mem); + let mut held = Vec::new(); + for _ in 0..4 { + held.push(full.take_permit().await.unwrap()); + } + let report = cycle_on(&fake, &full).await; + assert_eq!(report.restores_skipped_no_permit, ["MusicRestored"]); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert!(!report.acted()); + drop(held); + + // The store unhealthy: the cycle's own check, which runs first, finds + // the store down. + mem.set_down(true); + let unhealthy = runtime_unhealthy(&mem, StoreProblem::Transport("down".to_owned())) + .with_job_settings(job_settings()); + let report = cycle_on(&fake, &unhealthy).await; + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert_eq!(report.store_healthy, Some(false)); + assert_eq!( + lines.containing("dead store-backed restores are not resumed this cycle"), + [ + "the configured backup store is unhealthy; dead store-backed restores are not \ + resumed this cycle" + ] + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + mem.set_down(false); + + // The maintenance claim held: counted once, as failed and removed. + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &full).await; + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(report.manifest_restores_failed.is_empty()); + assert!(report.legacy_restores_removed.is_empty()); + assert!(report.acted()); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + *fake.maintenance_held.lock().unwrap() = false; + + // Another store. + let mem2 = Arc::new(MemStore::default()); + mem2.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem2, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem2); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let other = runtime_on( + &mem2, + StoreIdentity { + store_id: "store-2".to_owned(), + ..store() + }, + ) + .with_job_settings(job_settings()); + let report = cycle_on(&fake, &other).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), OTHER_STORE.to_owned())] + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(OTHER_STORE)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target waits for the FAILED-target duty" + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + let report = cycle_on(&fake, &other).await; + assert_eq!(report.failed_targets_removed, ["MusicRestored"]); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); +} + +/// A dead record with no progress at all whose target already holds items +/// is FAILED `target_not_empty` under the takeover and its target left in +/// place; the FAILED-target duty skips it on the next cycle and logs the +/// operator's command; a record with no progress into an empty target is +/// resumed from the first file. +#[tokio::test] +async fn a_resume_without_progress_needs_an_empty_target() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record as a dead owner that registered and recorded nothing left + // it, over a target that holds an item the record does not account for. + dead_record_without_progress(&fake, true); + fake.bulk_loads.lock().unwrap().clear(); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert_eq!(fake.restore_takeovers.lock().unwrap().len(), 1); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is left" + ); + wait_for_jobs(&runtime).await; + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + // The FAILED-target duty leaves it to the operator. + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.targets_left_for_operator, ["MusicRestored"]); + assert!(report.failed_targets_removed.is_empty()); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let command = lines.containing(&format!( + "extenddb backup remove-target --account {ACCOUNT} MusicRestored" + )); + assert_eq!( + command.len(), + 2, + "the takeover's line and the duty's: {command:?}" + ); + + // An empty target with no progress is resumed from the first file. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.bulk_loads.lock().unwrap().clear(); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2)]); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// Under the maintenance claim a dead record with no progress at all whose +/// target holds items is FAILED `target_not_empty` through the unclaimed +/// write and its target left for the operator, not `OWNER_DEAD` with the +/// target removed: the command's wait is satisfied (the record is no longer +/// RUNNING) and `remove-target`, under the same claim, removes it; one +/// whose target is empty is `OWNER_DEAD` and removed, as before. +#[tokio::test] +async fn a_dead_record_without_progress_under_the_maintenance_claim_keeps_its_target() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, true); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert!(report.manifest_restores_failed_and_removed.is_empty()); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is left" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!( + fake.restore_takeovers.lock().unwrap().is_empty(), + "no takeover under the claim" + ); + assert_eq!( + fake.failed_jobs.lock().unwrap().as_slice(), + [(record.target.clone(), TARGET_NOT_EMPTY.to_owned())], + "the unclaimed write" + ); + assert_eq!( + lines + .containing(&format!( + "extenddb backup remove-target --account {ACCOUNT} MusicRestored" + )) + .len(), + 1 + ); + // The command removes it under the same claim. + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + + // An empty target: `OWNER_DEAD` and removed. + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(report.manifest_restores_failed.is_empty()); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); +} + +/// Once the server's shutdown has begun (the jobs' token is cancelled) a +/// cycle takes no dead record over, writes no FAILED on the maintenance or +/// other-store arms, and spawns nothing: the record stays RUNNING under its +/// owner's generation with its heartbeat as it was, for the next instance's +/// reconciler; the cycle's other duties still run. +#[tokio::test] +async fn the_reconciler_takes_nothing_over_once_the_shutdown_began() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + let before = fake.restore_job("MusicRestored").unwrap(); + let checks_before = fake.maintenance_checks.lock().unwrap().len(); + let shutdown = extenddb_storage::CancellationToken::new(); + let draining = job_runtime(&mem).with_shutdown(shutdown.clone()); + shutdown.cancel(); + + // The takeover arm, the maintenance arm and the other-store arm, each + // with the token cancelled. + let report = cycle_on(&fake, &draining).await; + assert!(report.manifest_restores_resumed.is_empty(), "{report:?}"); + assert!(!report.acted(), "{report:?}"); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &draining).await; + assert!( + report.manifest_restores_failed_and_removed.is_empty(), + "{report:?}" + ); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + *fake.maintenance_held.lock().unwrap() = false; + let mem2 = Arc::new(MemStore::default()); + mem2.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); + let other = runtime_on( + &mem2, + StoreIdentity { + store_id: "store-2".to_owned(), + ..store() + }, + ) + .with_job_settings(job_settings()) + .with_shutdown(shutdown.clone()); + let report = cycle_on(&fake, &other).await; + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + + assert!( + fake.restore_takeovers.lock().unwrap().is_empty(), + "no takeover" + ); + assert!( + fake.failed_jobs.lock().unwrap().is_empty(), + "no FAILED write" + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!(draining.running_jobs(), 0, "nothing spawned"); + assert_eq!(fake.restore_job("MusicRestored").unwrap(), before); + assert_eq!( + lines + .containing("has no live owner and the server is shutting down; it is left for the next reconciler") + .len(), + 3 + ); + assert_eq!( + fake.maintenance_checks.lock().unwrap().len() - checks_before, + 3, + "each cycle ran its other duties" + ); + + // The same runtime with the token intact takes it over. + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// A reconciler whose own token has fired while the jobs' token has not +/// (the server stopping its workers while it holds no serving +/// registration, to start a new set once it holds one) takes no dead record +/// over and spawns no resume: a resume is a job, on the jobs' token, which +/// the workers' stop would not stop. The record stays as it was, the cycle's +/// other duties still run, and the next set's reconciler (a fresh state) +/// takes it over. +#[tokio::test] +async fn a_stopping_reconciler_takes_nothing_over() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + let before = fake.restore_job("MusicRestored").unwrap(); + let checks_before = fake.maintenance_checks.lock().unwrap().len(); + // The token fires once the cycle lists the dead records, after its + // store check, as a stop that lands mid-cycle does. + let stopped = extenddb_storage::CancellationToken::new(); + *fake.cancel_on_list_dead.lock().unwrap() = Some(stopped.clone()); + + let report = reconciler::cycle( + &fake, + &runtime, + Duration::from_secs(300), + "test", + &mut reconciler::ReconcilerState::with_shutdown(stopped.clone()), + ) + .await + .unwrap(); + assert!(stopped.is_cancelled(), "the cycle reached the dead records"); + assert!(report.manifest_restores_resumed.is_empty(), "{report:?}"); + assert!( + !runtime.shutdown().is_cancelled(), + "the jobs' token is intact" + ); + assert!( + fake.restore_takeovers.lock().unwrap().is_empty(), + "no takeover" + ); + assert_eq!(runtime.running_jobs(), 0, "nothing spawned"); + assert_eq!(fake.restore_job("MusicRestored").unwrap(), before); + assert_eq!( + lines + .containing("has no live owner and the reconciler is stopping; it is left for the next reconciler") + .len(), + 1 + ); + assert_eq!( + fake.maintenance_checks.lock().unwrap().len() - checks_before, + 1, + "the cycle ran its other duties" + ); + + // The next set's reconciler takes it over. + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// Rewrite the finished restore into `MusicRestored` as a dead owner's +/// RUNNING record with no progress recorded, its target CREATING and +/// holding three items, one more than the backup's two, when `with_item`, +/// else empty. +fn dead_record_without_progress(fake: &Fake, with_item: bool) { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = None; + record.items_applied = None; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.clear(); + if with_item { + // Three items against the backup's two: more than the backup's total, + // which the `target_not_empty` rule requires (a count at or below it + // is a first file applied in part and is resumed). + target.items.push(song("x", "y")); + target.items.push(song("x", "z")); + target.items.push(song("y", "z")); + } + if let Some(desc) = fake.tables.lock().unwrap().get_mut("MusicRestored") { + desc.table_status = TableStatus::Creating; + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); +} + +/// The two sweeps: progress rows of tables that are not RUNNING restore +/// targets go, a RUNNING target's stay; spool entries of restores not +/// RUNNING with this host's `spool_host` go (a dead one, one taken over by +/// another host), a live one of this host and every other name stay; both +/// run only without the maintenance claim; a backend that cannot list the +/// records skips both and the cycle goes on. +#[tokio::test] +async fn the_reconciler_sweeps_progress_rows_and_spool_entries() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let runtime = spooled_runtime(&mem, scratch.path()); + for (id, host) in [ + ("live", "host-a:/var/spool"), + ("moved", "host-b:/var/spool"), + ] { + fake.restore_jobs.lock().unwrap().insert( + id.to_owned(), + RestoreJobRow { + target: RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: id.to_owned(), + table_id: id.to_owned(), + }, + source_backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: 1, + heartbeat_at: seconds_since_epoch(), + store: Some(s3_identity()), + spool_host: Some(host.to_owned()), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + } + { + let mut rows = fake.progress_rows.lock().unwrap(); + rows.insert("live".to_owned(), vec![(0, 10)]); + rows.insert("gone".to_owned(), vec![(0, 10), (1, 5)]); + } + for name in ["restore-live", "restore-moved", "restore-dead", "other"] { + std::fs::create_dir(scratch.path().join(name)).unwrap(); + } + std::fs::write(scratch.path().join(".lock"), b"").unwrap(); + + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.progress_rows_swept, 0); + assert!(report.spool_entries_swept.is_empty()); + assert!(fake.progress_sweeps.lock().unwrap().is_empty()); + assert!(scratch.path().join("restore-dead").is_dir()); + *fake.maintenance_held.lock().unwrap() = false; + + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.progress_rows_swept, 1); + let mut swept = report.spool_entries_swept.clone(); + swept.sort(); + assert_eq!(swept, ["dead", "moved"]); + assert!(report.acted()); + assert_eq!( + fake.progress_sweeps.lock().unwrap().as_slice(), + [vec!["live".to_owned(), "moved".to_owned()]], + "the RUNNING targets' ids, in id order" + ); + assert_eq!( + fake.progress_rows + .lock() + .unwrap() + .keys() + .cloned() + .collect::>(), + ["live"] + ); + let mut left: Vec = std::fs::read_dir(scratch.path()) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + left.sort(); + assert_eq!(left, [".lock", "other", "restore-live"]); + assert_eq!( + fake.restore_lists.lock().unwrap().as_slice(), + [Some(JobStatus::Running)] + ); +} + +/// The sweeps on a backend whose records cannot be listed yet: the cycle +/// goes on, logs once, and sweeps nothing. +#[tokio::test] +async fn a_backend_that_cannot_list_restore_records_skips_the_sweeps() { + let (lines, _guard) = LogLines::capture(); + let fake = Arc::new(Fake::default()); + fake.progress_rows + .lock() + .unwrap() + .insert("gone".to_owned(), vec![(0, 1)]); + fake.restore_list_error + .store(true, std::sync::atomic::Ordering::Release); + let report = cycle_on(&fake, &runtime()).await; + assert_eq!(report.progress_rows_swept, 0); + assert_eq!( + lines + .containing("the restore job records could not be listed") + .len(), + 1 + ); + assert_eq!(fake.progress_rows.lock().unwrap().len(), 1, "nothing swept"); +} + +// --- the restore's liveness checks, spool rule and load bounds ----------------- + +/// The heartbeat's first confirmed beat is the instant `claim_restore` +/// returned, not the instant before it: a registration that takes four +/// periods answers CREATING to a job that runs to DONE, where a clock taken +/// before the registration would end it at its first check as a lost claim. +#[tokio::test] +async fn a_slow_registration_does_not_end_the_job_at_its_first_check() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let period = job_settings().heartbeat_interval; + *fake.claim_restore_delay.lock().unwrap() = Some(4 * period); + let runtime = job_runtime(&mem); + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 2); + assert_eq!(restore_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); +} + +/// The job's first beat, sent right after the registration and before the +/// reply, refreshes the record's `heartbeat_at`: a registration that +/// stamped it three periods in the past (a transaction-start `now()` behind +/// a lock, as on PostgreSQL) answers CREATING with the record's heartbeat of +/// this instant, one beat received before the reply, and the job runs to +/// DONE; a first beat that finds the claim lost fails the record before the +/// job starts, with the reason, and answers `InternalServerError`. On a +/// paused clock, so no timer beat runs between the reply and the read of +/// the record. +#[tokio::test(start_paused = true)] +async fn the_jobs_first_beat_refreshes_the_registrations_heartbeat_before_the_reply() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let period = job_settings().heartbeat_interval; + *fake.registration_heartbeat_age.lock().unwrap() = Some(3.0 * period.as_secs_f64() + 10.0); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + let age = seconds_since_epoch() - fake.restore_job("MusicRestored").unwrap().heartbeat_at; + assert!( + age < 1.0, + "the record's heartbeat is this instant's by the reply, not the registration's: {age}" + ); + assert!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) >= 1, + "one beat before the reply" + ); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(restore_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); + + // The first beat finds the claim lost. + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + let runtime = job_runtime(&mem); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert_eq!( + restore_writes(&fake), + [( + JobStatus::Running, + JobStatus::Failed, + Some("the claim was lost before the job started".to_owned()) + )] + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty(), "no load"); + assert_eq!( + lines + .containing("FAILED (the claim was lost before the job started) before it started;") + .len(), + 1 + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + assert_eq!(runtime.free_slots(), 4); +} + +/// The registration's bound: a `claim_restore` that takes longer than +/// `RESTORE_REGISTRATION_TIMEOUT` has the request answer +/// `InternalServerError` at the bound with the line naming it, while the +/// task goes on and the job runs to DONE once the registration returns. +#[tokio::test(start_paused = true)] +async fn the_request_stops_waiting_at_the_registration_bound_and_the_task_goes_on() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + // The design's figure, 60 s on every backend; the registration takes + // one second longer. + assert_eq!(RESTORE_REGISTRATION_TIMEOUT, Duration::from_secs(60)); + *fake.claim_restore_delay.lock().unwrap() = Some(Duration::from_secs(61)); + let runtime = job_runtime(&mem); + let started = tokio::time::Instant::now(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + assert!( + started.elapsed() >= Duration::from_secs(60) && started.elapsed() < Duration::from_secs(61), + "answered at the bound: {:?}", + started.elapsed() + ); + assert_eq!(runtime.running_jobs(), 1, "the task goes on"); + assert_eq!( + lines.containing("the registration did not answer within 60 s; the task goes on"), + [format!( + "RestoreTableFromBackup of {} into MusicRestored (account {ACCOUNT}): the \ + registration did not answer within 60 s; the task goes on and the target, if \ + registered, appears through DescribeTable", + arn(ACCOUNT) + )] + ); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 2); + assert_eq!(runtime.free_slots(), 4); +} + +/// The pre-write check's second clause on the restore: with the job held +/// inside a `bulk_load` and its heartbeat timer ended without raising the +/// loss, the next check (before the next file, before the count, before the +/// DONE write) finds the last confirmed beat more than two periods old and +/// ends the job as a lost claim naming what it was about to do, with no +/// further `bulk_load`, no count, no `finish_restore` and no status write. +#[tokio::test] +async fn a_stopped_heartbeat_timer_stops_the_restore_before_its_next_write() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + // (the target's name, the bulk_load the job is held in, the reason the + // next check names, the bulk_loads and counts made) + let cells: [(&str, usize, &str, usize, usize); 3] = [ + ( + "TimerStoppedBeforeFile", + 1, + "no heartbeat confirmed the claim recently enough to load data/000002.json.gz", + 1, + 0, + ), + ( + "TimerStoppedBeforeCount", + 2, + "no heartbeat confirmed the claim recently enough to count the target", + 2, + 0, + ), + ( + "TimerStoppedBeforeFlip", + 0, + "no heartbeat confirmed the claim recently enough for the DONE write", + 2, + 1, + ), + ]; + for (table, held_in, reason, loads, counts) in cells { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = Arc::new(tokio::sync::Semaphore::new(held_in.saturating_sub(1))); + let count_gate = closed_gate(); + if held_in > 0 { + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + } else { + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + } + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": table })), + ) + .await + .unwrap(); + let table_id = format!("tid-{table}"); + if held_in > 0 { + wait_for_bulk_loads_started(&fake, held_in).await; + } else { + // Both files loaded; the job waits in the count. + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.bulk_loads.lock().unwrap().len() < 2 { + assert!(tokio::time::Instant::now() < deadline, "{table}"); + tokio::time::sleep(Duration::from_millis(5)).await; + } + tokio::time::sleep(period / 2).await; + } + restore::test_support::stop_timer(&table_id); + tokio::time::sleep(3 * period).await; + gate.add_permits(10); + count_gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_loads.lock().unwrap().len(), loads, "{table}"); + assert_eq!( + fake.count_calls.load(std::sync::atomic::Ordering::Acquire), + counts, + "{table}" + ); + assert!(restore_writes(&fake).is_empty(), "{table}: no status write"); + assert!(fake.removed_targets.lock().unwrap().is_empty(), "{table}"); + let record = fake.restore_job(table).unwrap(); + assert_eq!(record.status, JobStatus::Running, "{table}"); + let stopped = lines.containing(&format!( + "into {table} (account {ACCOUNT}): stopped with no further catalog or table access: \ + {reason}" + )); + assert_eq!(stopped.len(), 1, "{table}: {stopped:?}"); + assert_eq!( + restore_outcomes(&runtime), + [("failed".to_owned(), 1.0)], + "{table}" + ); + } +} + +/// The DONE write's `NotMatched` outcome: the generation moved after the +/// last `bulk_load` returned (the job held in the count), so +/// `finish_restore` matches no record; the job removes no target, writes no +/// status, records `RestoreJobs{failed}` and the worker error, logs the +/// "matched no record" line, and leaves its spool entry to the sweep (the +/// record's owner may be reading it). +#[tokio::test] +async fn a_done_write_that_matches_no_record_removes_no_target_and_keeps_the_entry() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + let count_gate = closed_gate(); + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.bulk_loads.lock().unwrap().is_empty() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + let other = { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.owner_generation = new_generation(); + record.owner_generation + }; + count_gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.count_calls.load(std::sync::atomic::Ordering::Acquire), + 1 + ); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + assert!(restore_writes(&fake).is_empty(), "no status write"); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, other); + assert!(!fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active); + let line = lines.containing("the DONE write matched no record"); + assert_eq!(line.len(), 1, "{line:?}"); + assert!(line[0].contains("no table access was made"), "{}", line[0]); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + scratch.path().join("restore-tid-MusicRestored").is_dir(), + "the entry is left to the sweep" + ); + assert_eq!(runtime.free_slots(), 4); +} + +/// A job that learnt the record is another's leaves its spool entry in +/// place for the sweep: one whose claim is lost while it is held inside a +/// `bulk_load`, and one whose FAILED write matches no record (the +/// generation moved under it). Both end with no table access. +#[tokio::test] +async fn a_job_that_learnt_the_record_is_anothers_leaves_its_spool_entry() { + for lost_claim in [true, false] { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.join("000000.json.gz.verified").is_file()); + if lost_claim { + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + } else { + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .owner_generation = new_generation(); + gate.add_permits(10); + } + wait_for_jobs(&runtime).await; + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "{lost_claim}: no table access" + ); + assert_eq!( + restore_writes(&fake).len(), + usize::from(!lost_claim), + "{lost_claim}: a lost claim writes nothing, a failing job one FAILED write" + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "{lost_claim}" + ); + assert!( + entry.join("000000.json.gz.verified").is_file(), + "{lost_claim}: the entry is left to the sweep" + ); + assert_eq!( + restore_outcomes(&runtime), + [("failed".to_owned(), 1.0)], + "{lost_claim}" + ); + } +} + +/// `finish_restore` answering `Pending` (MongoDB: DONE with the index +/// backfill left to the backend's worker) is a success: the backfill line, +/// `RestoreJobs{succeeded}`, `WorkerLastSuccess{RestoreJob}`, the spool entry +/// removed and the record DONE, the table not flipped by the job. +#[tokio::test] +async fn a_pending_finish_is_a_success_whose_flip_is_the_backends() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + fake.finish_pending + .store(true, std::sync::atomic::Ordering::Release); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(record.completed_at.is_some()); + assert!( + !fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active, + "the flip is the backend's worker's" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!(!scratch.path().join("restore-tid-MusicRestored").exists()); + let pending = + lines.containing("index backfill pending, the backend's worker flips the table ACTIVE"); + assert_eq!(pending.len(), 1, "{pending:?}"); + assert!( + pending[0].contains(": DONE, 2 items (2 applied by this run)"), + "{}", + pending[0] + ); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("succeeded".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + !metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); +} + +/// The retry bound is the job's, from its first download attempt: the first +/// file succeeds after ten minutes of transport failures, the second fails +/// from then on, and the restore is FAILED naming the second file and the +/// bound once fifteen minutes have passed since the job's first attempt +/// (a clock per file would run it fifteen minutes longer). +#[tokio::test(start_paused = true)] +async fn the_download_retry_bound_is_the_jobs_not_each_files() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + *mem.fail_data_gets_until.lock().unwrap() = + Some(tokio::time::Instant::now() + Duration::from_secs(10 * 60)); + *mem.fail_gets_of.lock().unwrap() = Some("data/000002.json.gz".to_owned()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let started = tokio::time::Instant::now(); + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(50)).await; + } + let elapsed = started.elapsed(); + assert!( + elapsed >= Duration::from_secs(15 * 60) && elapsed < Duration::from_secs(16 * 60 + 2), + "{elapsed:?}" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("downloading data/000002.json.gz: transport error"), + "{reason}" + ); + assert!(reason.contains("the bound is 900 s"), "{reason}"); + let reads = gets_under(&mem, &arn(ACCOUNT)); + let first: Vec<&String> = reads + .iter() + .filter(|k| k.ends_with("000001.json.gz")) + .collect(); + let second: Vec<&String> = reads + .iter() + .filter(|k| k.ends_with("000002.json.gz")) + .collect(); + // The backoff restarts per file (1+2+4+8+16+32 s, then 60 s each): about + // ten minutes of attempts on the first file, five on the second. + assert!( + (13..=18).contains(&first.len()), + "{} attempts on file 1", + first.len() + ); + assert!( + (9..=13).contains(&second.len()), + "{} attempts on file 2", + second.len() + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// The download's own pre-attempt check: with the first attempt failed and +/// the job in its backoff, a heartbeat timer ended without raising the loss +/// stops the job before the next attempt, naming the file; no second +/// attempt is made and nothing is written. +#[tokio::test(start_paused = true)] +async fn a_stopped_heartbeat_timer_stops_the_download_before_its_next_attempt() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, 1); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "TimerStoppedInBackoff" })), + ) + .await + .unwrap(); + // The first attempt failed; the job sleeps its first backoff (at least + // one second), during which the timer is ended. + while gets_under(&mem, &arn(ACCOUNT)).len() < 3 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + restore::test_support::stop_timer("tid-TimerStoppedInBackoff"); + tokio::time::sleep(3 * period).await; + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)).len(), + 3, + "no second attempt" + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: no heartbeat confirmed the claim \ + recently enough to download data/000001.json.gz", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + fake.restore_job("TimerStoppedInBackoff").unwrap().status, + JobStatus::Running + ); +} + +/// A decoder that dies on the blocking pool (its reader panics mid-file) +/// drops the channel without an `Err` item, so the backend sees a clean end +/// of a short stream; the job fails the file rather than reporting it +/// applied, and the target is removed. +#[tokio::test] +async fn a_decoder_that_dies_fails_the_file_rather_than_completing_it() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + // The gets: the sidecar, the manifest, the file's hashing pass, then + // the file's load pass, whose body dies. + mem.panic_body_of_get + .store(4, std::sync::atomic::Ordering::Release); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + assert_eq!(loads[0].stream_error, None, "the backend saw a clean end"); + assert!(loads[0].items.len() < 2, "{} items", loads[0].items.len()); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!( + writes[0].2.as_deref(), + Some("data file data/000001.json.gz: the decoder stopped unexpectedly") + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.restore_job("MusicRestored").is_none()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// Stop a restore into `MusicRestored` on `runtime` at its first `bulk_load` +/// through the shutdown, so its spool entry holds every file verified; then +/// rewrite its record as a dead owner's with no progress recorded over an +/// empty target (the shutdown lets the held file finish, so the record +/// would otherwise name file 0). +async fn stop_a_spooled_restore_before_its_first_load( + fake: &Arc, + runtime: &BackupRuntime, + gate: &Arc, +) { + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(gate)); + restore_on(fake, runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(fake, 1).await; + runtime.drain_jobs(Duration::from_millis(100)).await; + gate.add_permits(10); + wait_for_jobs(runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + fake.bulk_loads.lock().unwrap().clear(); + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + record.file_index = None; + record.items_applied = None; + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.items.clear(); + target.active = false; + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); +} + +/// A same-host resume with every file verified in the spool reads the +/// manifest and sidecar from the spool and downloads nothing: no store +/// read at all, no `RestoreBytesDownloaded`. +#[tokio::test] +async fn a_same_host_resume_reads_the_spool_and_downloads_nothing() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + mem.gets.lock().unwrap().clear(); + let _ = drained(&runtime); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert!( + gets_under(&mem, &arn(ACCOUNT)).is_empty(), + "{:?}", + gets_under(&mem, &arn(ACCOUNT)) + ); + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2), (1, 0, false, 2)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from this host's spool") + .len(), + 1 + ); + assert!( + !drained(&runtime2) + .iter() + .any(|(m, _, _)| *m == MetricName::RestoreBytesDownloaded), + "nothing downloaded" + ); + let active = lines.containing(": ACTIVE, 4 items (4 applied by this run), 0 bytes downloaded"); + assert_eq!(active.len(), 1, "{active:?}"); +} + +/// A `.verified` marker binds the digest: a marker naming another digest +/// does not make its file verified, and a same-host resume downloads that +/// file again and only it. +#[tokio::test] +async fn a_marker_naming_another_digest_has_its_file_downloaded_again() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + let marker = entry.join("000000.json.gz.verified"); + let recorded = std::fs::read_to_string(&marker).unwrap(); + assert_eq!(recorded.trim_end(), files_sha256(&mem, &arn(ACCOUNT), 1)); + std::fs::write(&marker, format!("{}\n", "0".repeat(64))).unwrap(); + mem.gets.lock().unwrap().clear(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + ["data/000001.json.gz"], + "the file whose marker names another digest, and nothing else" + ); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(!entry.exists(), "the entry is removed with the job's end"); +} + +/// The lowercase hex sha256 of data file `n` (1-based key) of `arn` in `mem`. +fn files_sha256(mem: &MemStore, arn: &str, n: u32) -> String { + use sha2::Digest; + let prefix = BackupArn::parse(arn).unwrap().store_prefix(); + let key = format!("{prefix}{}", data_file_key(n)); + let bytes = mem.objects.lock().unwrap()[&key].clone(); + format!("{:x}", sha2::Sha256::digest(&bytes)) +} + +/// The spool states a crash can leave, and what a same-host resume +/// downloads: file 0 verified (file and marker), file 1 a `.partial` only, +/// file 2 a file without its marker, file 3 a marker without its file. The +/// verified file is kept; the other three are downloaded again, every file +/// is loaded and the restore completes. +#[tokio::test] +async fn a_resume_downloads_the_spool_files_a_crash_left_unverified() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [ + songs_from(0, 2), + songs_from(2, 2), + songs_from(4, 2), + songs_from(6, 2), + ]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + for i in 0..4 { + assert!(entry.join(format!("{i:06}.json.gz.verified")).is_file()); + } + std::fs::rename( + entry.join("000001.json.gz"), + entry.join("000001.json.gz.partial"), + ) + .unwrap(); + std::fs::remove_file(entry.join("000001.json.gz.verified")).unwrap(); + std::fs::remove_file(entry.join("000002.json.gz.verified")).unwrap(); + std::fs::remove_file(entry.join("000003.json.gz")).unwrap(); + mem.gets.lock().unwrap().clear(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + "data/000002.json.gz", + "data/000003.json.gz", + "data/000004.json.gz" + ], + "the verified file is kept; the .partial, the unmarked file and the marker without a \ + file are downloaded again" + ); + assert_eq!(fake.bulk_load_files().len(), 4); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(!entry.exists()); +} + +/// Songs `from..from + n` each carrying a `Blob` of `bytes` bytes. +fn padded_songs(from: usize, n: usize, bytes: usize, pattern: impl Fn(usize) -> u8) -> Vec { + songs_from(from, n) + .into_iter() + .enumerate() + .map(|(i, mut item)| { + let blob: Vec = (0..bytes) + .map(|j| pattern(i * 7919 + j * 104_729)) + .collect(); + item.insert( + "Blob".to_owned(), + extenddb_core::types::AttributeValue::B(blob), + ); + item + }) + .collect() +} + +/// `padded_songs` whose blob is `zeros` zero bytes followed by bytes from +/// `pattern`: the zeros compress to nothing, so one item's `zeros` tunes the +/// compressed length of a file by under a byte per step while the +/// uncompressed length and the item count stay the same. +fn padded_songs_with_zeros( + from: usize, + n: usize, + bytes: usize, + zeros_in_first: usize, + pattern: impl Fn(usize) -> u8, +) -> Vec { + let mut items = padded_songs(from, n, bytes, pattern); + if let Some(first) = items.first_mut() { + let blob: Vec = (0..bytes) + .map(|j| { + if j < zeros_in_first { + 0 + } else { + (j.wrapping_mul(2_654_435_761) >> 7) as u8 + } + }) + .collect(); + first.insert( + "Blob".to_owned(), + extenddb_core::types::AttributeValue::B(blob), + ); + } + items +} + +/// The blob byte pattern of `seed`: bytes with no structure a compressor +/// finds, so two seeds give files of nearly the same compressed length. +fn blob_pattern(seed: usize) -> impl Fn(usize) -> u8 + Copy { + move |x: usize| { + let mut z = (x as u64).wrapping_add((seed as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)); + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + (z ^ (z >> 31)) as u8 + } +} + +/// A data file of the same item count, uncompressed length and compressed +/// length as `original` (the file of `blob_pattern(0)`), holding other items +/// of the same keys and blob length (so only the digests differ): the blobs +/// are seeded differently per try, and within a try the zeros at the head of +/// the first item's blob are swept until the compressed length matches. +fn same_length_replacement(original: &[u8], from: usize, n: usize, bytes: usize) -> Vec { + for seed in 1usize..200 { + let pattern = blob_pattern(seed); + let (base, _) = + encode_data_file(padded_songs_with_zeros(from, n, bytes, 0, pattern)).unwrap(); + if base.len() < original.len() || base.len() > original.len() + bytes / 2 { + continue; + } + for zeros in 0..=bytes { + let (candidate, _) = + encode_data_file(padded_songs_with_zeros(from, n, bytes, zeros, pattern)).unwrap(); + if candidate.len() == original.len() { + assert_ne!(candidate, original); + return candidate; + } + if candidate.len() + 8 < original.len() { + break; + } + } + } + panic!( + "no replacement of {} compressed bytes found", + original.len() + ) +} + +/// On an in-place (filesystem) store a data file replaced between the +/// hashing pass and the load pass by another valid file of the same item +/// count, uncompressed length and compressed length (so only its digests +/// differ from the manifest's): the items of the replacement reach the +/// backend as they decode (two chunks of a thousand), the end-of-stream +/// check then fails the file on its sha256, since the load's second read is +/// hashed again, the restore is FAILED naming the file and the target is +/// removed. +/// +/// The job runs at a 1 s heartbeat, not the 50 ms of `job_settings`: the +/// hashing pass over both files runs before the first load's claim check, +/// on this test's one runtime thread with the heartbeat's timer, and under +/// a loaded host it can outlast two 50 ms periods, which ends the job as a +/// lost claim before any load. +#[tokio::test] +async fn a_file_replaced_between_the_two_passes_fails_in_the_load() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + // Payloads that compress little, so the compressed body is long and the + // decoder's first chunks are sent before the stream ends. + let blob = 256; + let files = [ + songs_from(0, 2), + padded_songs(2, 2500, blob, blob_pattern(0)), + ]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + false, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000002.json.gz"); + let original = mem.objects.lock().unwrap()[&key].clone(); + let replacement = same_length_replacement(&original, 2, 2500, blob); + assert_eq!(replacement.len(), original.len()); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + heartbeat_interval: Duration::from_secs(1), + ..job_settings() + }); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // Both files hashed (pass 1) and file 1's second get made; file 2 not + // yet re-read. + wait_for_bulk_loads_started(&fake, 1).await; + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads + .iter() + .filter(|k| k.ends_with("000002.json.gz")) + .count(), + 1, + "{reads:?}" + ); + mem.insert(&key, &replacement); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes.len(), + 1, + "the second pass hashes the file again and fails the restore: {writes:?}" + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: data file sha256 checksum mismatch"), + "the second pass hashes the file again: {reason}" + ); + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert_eq!( + loads[1].items.len(), + 2000, + "two chunks of the replacement reached the backend before the end-of-stream check" + ); + assert!(loads[1].stream_error.is_some(), "{loads:?}"); + assert!( + fake.restore_job("MusicRestored").is_none(), + "target removed" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); +} + +/// A file over one chunk by line bytes (the job's bound set small for the +/// test; the items are well under the count bound) is fed in chunks that +/// close at the byte bound, and the shutdown is read at each chunk boundary. +/// The backend is held after it pulled the first item of the second chunk, +/// so the decoder has sent two chunks and holds at most two more in the +/// channel and one past its check; the shutdown fired then is seen at the +/// close of a later chunk, so the backend receives a whole number of chunks +/// (two to five of six) and then the shutdown's `Err` item, never the whole +/// file. +#[tokio::test] +async fn a_file_over_one_chunk_by_bytes_is_fed_in_chunks_and_stops_at_a_boundary() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + // The job's chunk bound is 32 KiB (the production bound, 32 MiB, would + // make this a 192 MiB file); items of 1,500-byte blobs whose lines are + // the same length, so a chunk is a fixed number of items. + let chunk_bytes: u64 = 32 * 1024; + restore::test_support::set_chunk_bytes("tid-ChunkedByBytes", chunk_bytes); + let blob = 1500; + let line = line_bytes(&padded_songs(0, 1, blob, |_| b'a')[0]); + let per_chunk = usize::try_from(chunk_bytes.div_ceil(line)).unwrap(); + assert!( + (10..=20).contains(&per_chunk), + "{per_chunk} items of {line} bytes per chunk" + ); + let total = 6 * per_chunk; + let items = padded_songs(0, total, blob, |_| b'a'); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[items], + true, + ); + let runtime = job_runtime(&mem); + let item_gate = Arc::new(tokio::sync::Semaphore::new(per_chunk + 1)); + *fake.bulk_load_item_gate.lock().unwrap() = Some(Arc::clone(&item_gate)); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "ChunkedByBytes" })), + ) + .await + .unwrap(); + wait_for_items_seen(&fake, per_chunk + 1).await; + let left = runtime.drain_jobs(Duration::from_millis(100)).await; + assert_eq!(left, 1, "the job waits at the gate past the bound"); + item_gate.add_permits(total + 10); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + let received = loads[0].items.len(); + assert!( + received % per_chunk == 0 && received >= 2 * per_chunk && received < total, + "{received} items received of {total}, {per_chunk} per chunk" + ); + assert_eq!( + loads[0].stream_error.as_deref(), + Some("Internal error: the restore was stopped by the server's shutdown"), + "{:?}", + loads[0].stream_error + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!( + fake.restore_job("ChunkedByBytes").unwrap().status, + JobStatus::Running + ); + assert_eq!(restore_outcomes(&runtime), [("cancelled".to_owned(), 1.0)]); +} + +/// Wait until `bulk_load` has pulled `n` items from its stream. +async fn wait_for_items_seen(fake: &Fake, n: usize) { + // 120 s: the deadline covers the decoder's hashing pass over the file and + // the first chunk's decoding on the test's one runtime thread, which under + // a loaded machine (the suite runs beside the other crates' tests in CI) + // outlasted the 30 s this had, failing the two cells below while the code + // was right. + let deadline = tokio::time::Instant::now() + Duration::from_secs(120); + while fake + .bulk_load_items_seen + .load(std::sync::atomic::Ordering::Acquire) + < n + { + assert!( + tokio::time::Instant::now() < deadline, + "the backend pulled no item" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// The decoder's claim check at each chunk boundary: the backend is held +/// after it pulled the first item of the second chunk (the decoder has sent +/// two chunks of a thousand and holds at most two more and one past its +/// check), the heartbeat timer is ended without raising the loss, and the +/// decoder ends the stream at the close of a later chunk with the +/// claim-lost reason: a whole number of chunks (two to five of six) reach +/// the backend, then the `Err` item, and the job stops with no catalog +/// write. +#[tokio::test] +async fn a_feeder_whose_timer_stopped_ends_the_stream_with_the_claim_lost_reason() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + let per_chunk = restore::load::LOAD_CHUNK_ITEMS; + let total = 6 * per_chunk; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, total)], + true, + ); + let runtime = job_runtime(&mem); + let item_gate = Arc::new(tokio::sync::Semaphore::new(per_chunk + 1)); + *fake.bulk_load_item_gate.lock().unwrap() = Some(Arc::clone(&item_gate)); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "FeederTimerStopped" })), + ) + .await + .unwrap(); + wait_for_items_seen(&fake, per_chunk + 1).await; + restore::test_support::stop_timer("tid-FeederTimerStopped"); + tokio::time::sleep(3 * period).await; + item_gate.add_permits(total + 10); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + let received = loads[0].items.len(); + assert!( + received % per_chunk == 0 && received >= 2 * per_chunk && received < total, + "{received} items received of {total}" + ); + assert_eq!( + loads[0].stream_error.as_deref(), + Some("Internal error: the claim was lost while the file was being decoded"), + "{:?}", + loads[0].stream_error + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost while the file \ + was being decoded", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + fake.restore_job("FeederTimerStopped").unwrap().status, + JobStatus::Running + ); +} + +/// The manifest arm reads the health this cycle's check left: a store that +/// passed a check once (its identity known) and is down when the cycle +/// runs is unhealthy to the arm, which skips the record with the line +/// naming the state; a store whose check cannot read the catalog and so has +/// not yet been checked is skipped with "not yet checked"; once the store +/// answers again the next cycle resumes the record. +#[tokio::test] +async fn an_unhealthy_or_unchecked_store_with_a_known_identity_skips_the_resume() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.bulk_loads.lock().unwrap().clear(); + + // Healthy with its identity known, then down when the cycle runs. + let runtime = job_runtime(&mem); + assert_eq!( + runtime.configured_store().unwrap().identity(), + Some(store()) + ); + mem.set_down(true); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(false)); + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + lines.containing("dead store-backed restores are not resumed this cycle"), + [ + "the configured backup store is unhealthy; dead store-backed restores are not \ + resumed this cycle" + ] + ); + mem.set_down(false); + + // Not yet checked: the store's first check cannot read the catalog. + let checking = self::runtime() + .with_store(ConfiguredStore::checking( + StoreKind::Filesystem, + store().store_root, + Arc::clone(&mem) as Arc, + None, + )) + .with_job_settings(job_settings()) + .with_metrics(Arc::new(extenddb_core::metrics::MetricsCollector::new())); + *fake.catalog_id_error.lock().unwrap() = true; + let report = cycle_on(&fake, &checking).await; + assert_eq!(report.store_healthy, Some(false)); + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + lines + .containing("the configured backup store is not yet checked; dead store-backed restores are not resumed this cycle") + .len(), + 1 + ); + *fake.catalog_id_error.lock().unwrap() = false; + + // The store answers again: resumed. + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +// --- the sweep's read order, the DONE write's unknown outcome, the resume's +// instant and documents, the request's bounds ----------------------------------- + +/// A restore that registers (RUNNING, this host's `spool_host`) and opens +/// its spool entry after the reconciler read the RUNNING records: the spool +/// was listed before the records, so the entry is not a candidate and the +/// live job keeps it (the design: the sweep never removes the spool of the +/// job that owns the record). +#[tokio::test] +async fn a_restore_registered_between_the_two_reads_keeps_its_spool_entry() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let runtime = spooled_runtime(&mem, scratch.path()); + // An entry of a dead restore, so the sweep has work to do in the same + // cycle and the live entry's survival is not the sweep doing nothing. + std::fs::create_dir(scratch.path().join("restore-dead")).unwrap(); + let spool_dir = scratch.path().to_path_buf(); + *fake.sweep_hook.lock().unwrap() = Some(Box::new(move |fake: &Fake| { + fake.restore_jobs.lock().unwrap().insert( + "late".to_owned(), + RestoreJobRow { + target: RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "late".to_owned(), + table_id: "late".to_owned(), + }, + source_backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: 1, + heartbeat_at: seconds_since_epoch(), + store: Some(s3_identity()), + spool_host: Some("host-a:/var/spool".to_owned()), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + // The job's `Entry::open` and its first download. + let entry = restore::spool::Entry::open( + &spool_dir, + "late", + Some("host-a:/var/spool"), + "host-a:/var/spool", + ) + .unwrap(); + std::fs::write(entry.dir().join("000000.json.gz.partial"), b"bytes").unwrap(); + })); + let report = cycle_on(&fake, &runtime).await; + assert!( + fake.sweep_hook.lock().unwrap().is_none(), + "the registration landed between the two reads" + ); + assert_eq!( + fake.restore_job("late").map(|r| r.status), + Some(JobStatus::Running), + "the record is RUNNING with this host's spool_host" + ); + assert_eq!( + report.spool_entries_swept, + ["dead"], + "the dead entry went; the live restore's entry did not" + ); + assert!( + scratch.path().join("restore-late").is_dir(), + "the live job's spool entry was removed from under it" + ); + assert!( + scratch + .path() + .join("restore-late/000000.json.gz.partial") + .is_file() + ); + // The next cycle, with the record RUNNING and this host's, keeps it too. + let report = cycle_on(&fake, &runtime).await; + assert!(report.spool_entries_swept.is_empty()); + assert!(scratch.path().join("restore-late").is_dir()); +} + +/// A `finish_restore` that returns an error is a write whose outcome is +/// unknown, and the record decides: (a) the flip landed and the call then +/// failed (PostgreSQL's progress delete after the commit): the re-read finds +/// DONE under this generation, the restore is complete and `succeeded`, with +/// a WARN naming the error, no FAILED write, no table access and the entry +/// removed; (b) the call failed before anything was written: the re-read +/// finds RUNNING, the write is sent once more and lands; (c) it fails twice: +/// the record stays RUNNING for the reconciler, no FAILED write, no +/// removal, the job `failed` with the worker error and the entry left; (d) +/// it fails and a successor wrote DONE under its own generation before the +/// re-read: the write matched no record, the job `failed`, no table access. +#[tokio::test] +async fn a_failed_done_write_is_re_read_and_the_record_decides() { + async fn run_with( + errors: &[FinishError], + ) -> (Arc, Vec<(MetricName, String, f64)>, LogLines) { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + fake.finish_errors + .lock() + .unwrap() + .extend(errors.iter().copied()); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let entry_left = scratch.path().join("restore-tid-MusicRestored").exists(); + let mut metrics = drained(&runtime); + metrics.push(( + MetricName::RestoreJobs, + format!("entry_left={entry_left}"), + 0.0, + )); + (fake, metrics, lines) + } + fn restore_jobs(metrics: &[(MetricName, String, f64)]) -> Vec<(String, f64)> { + metrics + .iter() + .filter(|(m, op, _)| *m == MetricName::RestoreJobs && !op.starts_with("entry_left")) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect() + } + fn entry_left(metrics: &[(MetricName, String, f64)]) -> bool { + metrics.iter().any(|(_, op, _)| op == "entry_left=true") + } + + // (a) The flip landed, then the error. + let (fake, metrics, lines) = run_with(&[FinishError::AfterFlip]).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 1, "one DONE write"); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active); + assert!( + restore_writes(&fake).is_empty(), + "no FAILED write over a DONE record: {:?}", + restore_writes(&fake) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(restore_jobs(&metrics), [("succeeded".to_owned(), 1.0)]); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob") + ); + assert!(!entry_left(&metrics), "the entry is removed on a success"); + let warned = lines.containing( + "the DONE write returned an error (Internal error: the data pool failed after the flip's \ + commit) and the record is DONE under this job's generation: the write landed and the \ + restore is complete", + ); + assert_eq!(warned.len(), 1, "{warned:?}"); + assert_eq!( + lines + .containing(": complete (DONE), 2 items (2 applied by this run)") + .len(), + 1, + "the line says DONE, not ACTIVE: the table's flip is the backend's" + ); + assert!(lines.containing(": ACTIVE, ").is_empty()); + + // (b) The error before anything was written: once more, and it lands. + let (fake, metrics, lines) = run_with(&[FinishError::BeforeFlip]).await; + assert_eq!( + fake.finish_sizes.lock().unwrap().len(), + 2, + "the write sent once more" + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!(restore_jobs(&metrics), [("succeeded".to_owned(), 1.0)]); + assert_eq!( + lines + .containing("the record is still RUNNING under this claim; writing once more") + .len(), + 1 + ); + + // (d) The error, and a takeover whose successor wrote DONE before the + // re-read: the record is DONE under another generation, so the write + // matched no record; no second write, no table access, the entry left. + let (fake, metrics, lines) = run_with(&[FinishError::TakenAndDone]).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 1, "one DONE write"); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(restore_jobs(&metrics), [("failed".to_owned(), 1.0)]); + assert!(entry_left(&metrics), "the entry is left to the sweep"); + let ended = lines.containing( + "the DONE write matched no record (the DONE write failed (Internal error: the claim \ + connection failed before the DONE write) and the record carries generation", + ); + assert_eq!(ended.len(), 1, "{ended:?}"); + assert!( + lines.containing("complete (DONE)").is_empty(), + "a successor's DONE is not this job's" + ); + + // (c) Twice: no third write, no FAILED write, no removal; the record is + // the reconciler's. + let (fake, metrics, lines) = + run_with(&[FinishError::BeforeFlip, FinishError::BeforeFlip]).await; + assert_eq!( + fake.finish_sizes.lock().unwrap().len(), + 2, + "two writes at most" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!( + restore_writes(&fake).is_empty(), + "no FAILED write: {:?}", + restore_writes(&fake) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target stands" + ); + assert_eq!(restore_jobs(&metrics), [("failed".to_owned(), 1.0)]); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob") + ); + assert!(entry_left(&metrics), "the entry is left to the sweep"); + let ended = lines.containing( + "the DONE write matched no record (the DONE write failed twice (Internal error: the claim \ + connection failed before the DONE write) and the record is still RUNNING under this \ + claim; the reconciler resumes it once the record is stale); the record is not this \ + job's to write and no table access was made", + ); + assert_eq!(ended.len(), 1, "{ended:?}"); +} + +/// A resumed job's first confirmed beat is the instant the takeover +/// returned, not the instant the job started: with the record read held +/// three periods after the takeover, the job's first check finds the beat +/// older than two periods and stops as a lost claim with no load and no +/// status write (the record is one a second taker may hold by then); with +/// the read held one period the job runs to DONE. +#[tokio::test(start_paused = true)] +async fn a_resumes_first_beat_is_the_takeovers_return() { + async fn resume_held(periods: u32) -> (Arc, Arc, LogLines) { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record as a dead owner left it after file 0. + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(0); + record.items_applied = Some(2); + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.truncate(2); + if let Some(desc) = fake.tables.lock().unwrap().get_mut("MusicRestored") { + desc.table_status = TableStatus::Creating; + } + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + fake.bulk_loads.lock().unwrap().clear(); + mem.gets.lock().unwrap().clear(); + *fake.read_restore_job_delay.lock().unwrap() = + Some(job_settings().heartbeat_interval * periods); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + (fake, mem, lines) + } + + let (fake, mem, lines) = resume_held(3).await; + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no load under a beat three periods old: {:?}", + fake.bulk_load_files() + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "no status write" + ); + assert!(restore_writes(&fake).is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: no heartbeat confirmed the claim \ + recently enough to download data/000002.json.gz", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [EXTENDDB_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE], + "the two documents of the resume's preparation and no data file" + ); + + let (fake, _mem, _lines) = resume_held(1).await; + assert_eq!(fake.bulk_load_files(), [(1, 0, true, 2)]); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// A same-host resume whose spooled manifest a crash left short reads the +/// store's manifest and sidecar instead (one WARN), writes them to the entry +/// again, and resumes with the spooled files; the restore completes. +#[tokio::test] +async fn a_truncated_spooled_manifest_has_the_resume_read_the_store() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + mem.gets.lock().unwrap().clear(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + let spooled_manifest = std::fs::read(entry.join(EXTENDDB_MANIFEST_FILE)).unwrap(); + // A crash during the manifest's write, as an in-place write leaves it. + std::fs::write( + entry.join(EXTENDDB_MANIFEST_FILE), + &spooled_manifest[..spooled_manifest.len() / 2], + ) + .unwrap(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + fake.restore_job("MusicRestored").map(|r| r.status), + Some(JobStatus::Done), + "the resume read the store's documents and completed: {:?}", + restore_writes(&fake) + ); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [EXTENDDB_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE], + "the two documents from the store, no data file (the spooled files are verified)" + ); + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2), (1, 0, false, 2)]); + assert_eq!( + lines + .containing( + "the spooled manifest and sidecar of the restore into MusicRestored (account \ + 123456789012) do not verify (the manifest's checksum sidecar does not match" + ) + .len(), + 1 + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from the store") + .len(), + 1 + ); +} + +/// A resume whose store documents fail verification is FAILED with the +/// reason, its target removed, and its duration recorded from the resume's +/// start: a sidecar that does not match the manifest; a manifest naming +/// another backup under a sidecar that matches it. +#[tokio::test] +async fn a_resume_whose_store_documents_fail_verification_is_failed() { + async fn resume_over(tamper: impl Fn(&MemStore, &str)) -> (Arc, BackupRuntime, LogLines) { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(0); + record.items_applied = Some(2); + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.truncate(2); + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + fake.bulk_loads.lock().unwrap().clear(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + tamper(&mem, &prefix); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + drop(_guard); + (fake, runtime, lines) + } + fn failed_with(fake: &Fake, runtime: &BackupRuntime, reason_prefix: &str) { + let writes = restore_writes(fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!(reason.starts_with(reason_prefix), "{reason}"); + assert!(fake.bulk_loads.lock().unwrap().is_empty(), "no load"); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target and its record are removed" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + let metrics = drained(runtime); + let durations: Vec = metrics + .iter() + .filter(|(m, op, _)| *m == MetricName::RestoreJobDuration && op == "failed") + .map(|(_, _, sum)| *sum) + .collect(); + assert_eq!(durations.len(), 1, "{metrics:?}"); + assert!( + durations[0] > 0.0, + "the duration runs from the resume's start: {metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob") + ); + } + + // The sidecar names another digest. + let (fake, runtime, lines) = resume_over(|mem, prefix| { + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + format!("{} {EXTENDDB_MANIFEST_FILE}\n", "0".repeat(64)).as_bytes(), + ); + }) + .await; + failed_with( + &fake, + &runtime, + "the manifest's checksum sidecar does not match", + ); + assert_eq!( + lines + .containing("FAILED on resume (the manifest's checksum sidecar does not match") + .len(), + 1 + ); + + // The manifest names another backup of the same table and instant + // (another id; the parser's identity rule admits it), under a sidecar + // that matches it. + let (fake, runtime, lines) = resume_over(|mem, prefix| { + let other = BackupArn::parse(&arn(ACCOUNT).replace("73d8d5bc", "0badf00d")).unwrap(); + let bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")].clone(); + let text = String::from_utf8(bytes).unwrap(); + let own_id = BackupArn::parse(&arn(ACCOUNT)).unwrap(); + let renamed = text.replace( + own_id.to_arn_string().as_str(), + other.to_arn_string().as_str(), + ); + assert_ne!(renamed, text); + let manifest = renamed.into_bytes(); + let sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&manifest) + ) + }; + mem.insert(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}"), &manifest); + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + sidecar.as_bytes(), + ); + }) + .await; + failed_with(&fake, &runtime, "the manifest names another backup, "); + assert_eq!( + lines + .containing("FAILED on resume (the manifest names another backup, ") + .len(), + 1 + ); +} + +/// A same-host resume that fails before it runs (the spooled pair a crash +/// left short, and the store's sidecar naming another digest) removes this +/// host's spool entry with the failure path, as the job removes its entry +/// at every failed end: the record is FAILED naming the mismatch, the +/// target removed, and the entry gone with its verified files. +#[tokio::test] +async fn a_resume_that_fails_before_it_runs_removes_this_hosts_spool_entry() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.join("000000.json.gz.verified").is_file()); + let spooled_manifest = std::fs::read(entry.join(EXTENDDB_MANIFEST_FILE)).unwrap(); + std::fs::write( + entry.join(EXTENDDB_MANIFEST_FILE), + &spooled_manifest[..spooled_manifest.len() / 2], + ) + .unwrap(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + format!("{} {EXTENDDB_MANIFEST_FILE}\n", "0".repeat(64)).as_bytes(), + ); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("the manifest's checksum sidecar does not match"), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + !entry.exists(), + "this host's entry goes with the failed resume" + ); + let ended = + lines.containing("FAILED on resume (the manifest's checksum sidecar does not match"); + assert_eq!(ended.len(), 1, "{ended:?}"); + assert!(ended[0].ends_with("; the target was removed"), "{ended:?}"); +} + +/// A spool document write that fails (the `.partial` path is taken by a +/// directory, as a disk full fails the same arm) fails the restore naming +/// the document, removes the target, and removes the entry the job had +/// opened, as every failed end does. +#[tokio::test] +async fn a_spool_document_write_failure_fails_the_restore_and_removes_the_entry() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + std::fs::create_dir_all(entry.join(format!("{EXTENDDB_MANIFEST_FILE}.partial"))).unwrap(); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with(&format!("writing {EXTENDDB_MANIFEST_FILE} to the spool: ")), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + !entry.exists(), + "the entry is removed after a FAILED write that matched" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// `Progress::from_records` on each backend's record shape: PostgreSQL's +/// (the record carries no `file_index`; the progress rows name each +/// recorded file's count), MongoDB's (`file_index` with no `items_applied`: +/// the recorded file is complete), and SQLite's and Cassandra's (both, +/// mid-file); a `file_index` past the manifest records nothing more. +#[test] +fn progress_from_records_reads_each_backends_shape() { + let mem = MemStore::default(); + let fake = Fake::default(); + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 3), songs_from(3, 2), songs_from(5, 4)], + true, + ); + let mut row = RestoreJobRow::reconciler_row( + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "T".to_owned(), + table_id: "tid".to_owned(), + }, + arn(ACCOUNT), + RestoreKind::Manifest, + JobStatus::Running, + 1, + 0.0, + ); + + // PostgreSQL: files 0 and 2 recorded whole, file 1 not; the resume + // opens file 1 from 0. + let progress = restore::Progress::from_records(&row, &[(0, 3), (2, 4)], &manifest); + assert_eq!(progress.recorded(), [Some(3), None, Some(4)]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 0))); + // A PostgreSQL row of a file a decoder panic left short (n1): the file + // opens from its row. + let progress = restore::Progress::from_records(&row, &[(0, 3), (1, 1)], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(1), None]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 1))); + + // MongoDB: `file_index` 1 with no `items_applied`: files 0 and 1 are + // complete, file 2 opens from 0. + row.file_index = Some(1); + row.items_applied = None; + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), None]); + assert_eq!(progress.first_open_in(&manifest), Some((2, 0))); + + // SQLite and Cassandra: file 1 after 1 of its items. + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(1), None]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 1))); + + // Every file recorded: nothing opens. + row.file_index = Some(2); + row.items_applied = Some(4); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.first_open_in(&manifest), None); + + // A `file_index` past the manifest: every file complete, the count + // decides. + row.file_index = Some(7); + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), Some(4)]); + assert_eq!(progress.first_open_in(&manifest), None); + + // Nothing recorded. + row.file_index = None; + row.items_applied = None; + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [None, None, None]); + assert_eq!(progress.first_open_in(&manifest), Some((0, 0))); + + // The record and a progress row for one file: the larger count is the + // file's, whichever source carries it (no backend writes both today). + row.file_index = Some(1); + row.items_applied = Some(2); + let progress = restore::Progress::from_records(&row, &[(1, 1)], &manifest); + assert_eq!( + progress.recorded(), + [Some(3), Some(2), None], + "a smaller progress row lowered the record's count" + ); + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[(1, 2)], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), None]); +} + +/// A record gone between the takeover and the reads that follow (whoever +/// moved it owns it) is left to its owner with the info line, not reported +/// as a cycle error. +#[tokio::test] +async fn a_record_gone_after_the_takeover_is_left_to_its_owner() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.takeover_removes_record + .store(true, std::sync::atomic::Ordering::Release); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(fake.restore_takeovers.lock().unwrap().len(), 1); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(report.manifest_restores_failed.is_empty()); + assert_eq!( + lines + .containing("was gone when its progress was read; left to whoever moved it") + .len(), + 1 + ); + assert!( + lines + .containing("was taken over but its progress could not be read") + .is_empty() + ); + assert_eq!(lines.at_level(tracing::Level::ERROR), Vec::::new()); +} + +/// A registration that fails after the request stopped waiting for it +/// (the 60 s bound answered `InternalServerError`) is logged at ERROR with +/// the ARN, the target, the account and the error, since the request's own +/// line told the operator the target appears if registered. +#[tokio::test] +async fn a_registration_that_fails_after_the_request_stopped_waiting_is_logged() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let store = runtime.configured_store_arc().unwrap(); + let env = job::JobEnv::new(&runtime, store).unwrap(); + let permit = runtime.take_permit().await.unwrap(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let manifest_bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")].clone(); + let sidecar_bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{MANIFEST_CHECKSUM_FILE}")].clone(); + let input = extenddb_storage::backup_definition::restore_input_from_manifest( + &manifest.source_table, + "MusicRestored", + &RestoreTableOverrides::default(), + ); + let key_schema = extenddb_core::validation::RestorableKeySchema::resolve( + &manifest.source_table.key_schema, + &manifest.source_table.attribute_definitions, + ) + .unwrap(); + let parts = restore::RestoreRequestParts { + account_id: ACCOUNT.to_owned(), + arn: BackupArn::parse(&arn(ACCOUNT)).unwrap(), + backup: BackupKey::from_arn(&BackupArn::parse(&arn(ACCOUNT)).unwrap()), + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits: (&LimitsConfig::default()).into(), + registration: None, + }; + *fake.claim_restore_error.lock().unwrap() = + Some(StorageError::TableAlreadyExists("MusicRestored".to_owned())); + let (reply, answer) = tokio::sync::oneshot::channel(); + // The request gave up at its bound. + drop(answer); + restore::start(Arc::clone(&fake), env, permit, parts, reply).await; + let logged = lines.containing( + "the registration failed after the request had stopped waiting for it; no target was \ + created", + ); + assert_eq!(logged.len(), 1, "{logged:?}"); + assert!( + logged[0].starts_with(&format!( + "restore of {} into MusicRestored (account {ACCOUNT}): ", + arn(ACCOUNT) + )), + "{}", + logged[0] + ); + assert!(logged[0].contains("MusicRestored"), "{}", logged[0]); + assert_eq!(runtime.free_slots(), 4, "the permit is returned"); +} + +/// The request's two document reads collect at most their bound: a sidecar +/// over 4 KiB or a manifest over 16 MiB behind an AVAILABLE row is refused +/// as a corrupt prefix (`ValidationException`), not read whole; a resume +/// meeting one is FAILED with the same reason. +#[tokio::test] +async fn the_document_reads_are_bounded() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let sidecar_key = format!("{prefix}{MANIFEST_CHECKSUM_FILE}"); + let good_sidecar = mem.objects.lock().unwrap()[&sidecar_key].clone(); + let mut oversize = good_sidecar.clone(); + oversize.extend(std::iter::repeat_n(b' ', 4 * 1024)); + mem.insert(&sidecar_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert_eq!( + validation_message(err), + format!( + "Backup {} has a {MANIFEST_CHECKSUM_FILE} over 4096 bytes, which no backup this \ + server wrote produces; the backup's prefix is corrupt", + arn(ACCOUNT) + ) + ); + assert!(fake.restore_claims.lock().unwrap().is_empty()); + mem.insert(&sidecar_key, &good_sidecar); + + // The manifest's bound, one byte over. + let manifest_key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let good_manifest = mem.objects.lock().unwrap()[&manifest_key].clone(); + let mut oversize = good_manifest.clone(); + oversize.resize(16 * 1024 * 1024 + 1, b' '); + mem.insert(&manifest_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert_eq!( + validation_message(err), + format!( + "Backup {} has a {EXTENDDB_MANIFEST_FILE} over 16777216 bytes, which no backup this \ + server wrote produces; the backup's prefix is corrupt", + arn(ACCOUNT) + ) + ); + // At the bound, the read goes through and the checksum decides. + oversize.truncate(16 * 1024 * 1024); + mem.insert(&manifest_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + validation_message(err).contains("whose checksum sidecar does not match"), + "a manifest at the bound is read and verified" + ); + mem.insert(&manifest_key, &good_manifest); + + // A resume over an oversize sidecar is FAILED with the reason. + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + let mut oversize = good_sidecar.clone(); + oversize.extend(std::iter::repeat_n(b' ', 4 * 1024)); + mem.insert(&sidecar_key, &oversize); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes.last().and_then(|w| w.2.clone()).as_deref(), + Some( + "the store's extenddb-manifest.sha256 for the backup is over 4096 bytes, which no \ + backup this server wrote produces" + ), + "{writes:?}" + ); +} + +// --- `backup remove-target` (`maintenance::remove_target`) ------------------------ + +/// The request `remove_target` runs with, against the `Fake`. +fn remove_target_request(table: &str, owner_window: Duration) -> maintenance::RemoveTarget { + maintenance::RemoveTarget { + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + holder: "extenddb backup remove-target on host-a (pid 1)".to_owned(), + stale_after: Duration::from_secs(300), + owner_window, + heartbeat_interval: Duration::from_millis(20), + } +} + +/// `backup remove-target` under the maintenance claim: a FAILED record's +/// CREATING target (`target_not_empty`, the reason the reconciler leaves +/// for it) is removed with its record, and the claim was taken for the +/// removal and released after it; a RUNNING record the backend's dead-job +/// listing does not name is refused and nothing is removed, whatever its +/// `heartbeat_at` reads on this host's clock (one second old; 1234.5, an +/// age a host whose clock runs ahead of the catalog's would compute for a +/// live owner); a RUNNING record the listing names is removed, with a +/// heartbeat this host's clock reads as fresh; a DONE record (the table +/// ACTIVE) and a table no record names are refused as "not a restore +/// target"; a claim a live holder has refuses the command before any read. +#[tokio::test] +async fn remove_target_removes_a_failed_or_dead_target_under_the_claim_and_refuses_the_rest() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + + // DONE: the table is ACTIVE. + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "MusicRestored (account {ACCOUNT}) is not a restore target: its restore from {} is \ + DONE (the table is ACTIVE, or awaits its index backfill)", + arn(ACCOUNT) + ) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(fake.maintenance_holders.lock().unwrap().len(), 1); + assert!( + *fake.maintenance_holders.lock().unwrap()[0] + .released + .lock() + .unwrap(), + "the claim is released after a refusal too" + ); + + // No record names the table. + let err = maintenance::remove_target( + &*fake, + &remove_target_request("Nothing", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!("Nothing (account {ACCOUNT}) is not a restore target: no restore record names it") + ); + + // RUNNING with a live owner: the dead-job listing does not name the + // record (the backend's age test on the catalog's clock), whatever its + // heartbeat reads against this host's clock. + dead_record_without_progress(&fake, true); + fake.restore_dead.lock().unwrap().clear(); + for heartbeat_at in [seconds_since_epoch() - 1.0, 1234.5] { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = heartbeat_at; + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "the restore into MusicRestored (account {ACCOUNT}) from {} is RUNNING with a \ + live owner (a heartbeat within stale_job_after_secs); stop it or wait for it", + arn(ACCOUNT) + ), + "heartbeat_at {heartbeat_at}" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + + // The reconciler finds the dead owner, no progress and a non-empty + // target: FAILED `target_not_empty`, the target left; the command + // removes it. + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!( + removed, + maintenance::RemovedTarget { + table_id: "tid-MusicRestored".to_owned(), + backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Failed, + failure_reason: Some(TARGET_NOT_EMPTY.to_owned()), + } + ); + assert!( + !fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is gone" + ); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the record is gone" + ); + assert_eq!( + fake.removed_targets.lock().unwrap().as_slice(), + [RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "MusicRestored".to_owned(), + table_id: "tid-MusicRestored".to_owned(), + }] + ); + { + let holders = fake.maintenance_holders.lock().unwrap(); + assert_eq!(holders.len(), 5, "a claim per command"); + assert!( + holders.iter().all(|h| *h.released.lock().unwrap()), + "every claim released" + ); + } + assert!( + fake.maintenance_claim_held_by.lock().unwrap().is_none(), + "no claim left held" + ); + + // RUNNING with a dead owner: the listing names the record (under the + // 300 s window with a heartbeat this host's clock reads as fresh; under + // the zero window SQLite's serve lock gives, with a heartbeat an hour + // ahead): removed with its record, no FAILED write. + for window in [Duration::from_secs(300), Duration::ZERO] { + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, true); + if window.is_zero() { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = seconds_since_epoch() + 3600.0; + } else { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = seconds_since_epoch(); + } + let removed = + maintenance::remove_target(&*fake, &remove_target_request("MusicRestored", window)) + .await + .unwrap(); + assert_eq!(removed.status, JobStatus::Running); + assert_eq!(removed.failure_reason, None); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!( + fake.failed_jobs.lock().unwrap().is_empty(), + "no FAILED write" + ); + } + + // A claim a live holder has: refused before any read. + let fake = Arc::new(Fake::default()); + *fake.maintenance_claim_held_by.lock().unwrap() = Some("extenddb backup purge".to_owned()); + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + "the maintenance claim is held by \"extenddb backup purge\"; one maintenance command \ + runs at a time, and a claim whose holder died expires after stale_job_after_secs (300 s)" + ); + assert!(fake.restore_lists.lock().unwrap().is_empty(), "no read"); +} + +/// A legacy record's target is removed the same way: the reconciler would +/// fail and remove it once stale; the command lets the operator do it now. +#[tokio::test] +async fn remove_target_removes_a_legacy_records_target() { + let fake = Arc::new(Fake::default()); + let target = RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "Legacy".to_owned(), + table_id: "tid-legacy".to_owned(), + }; + fake.restore_jobs.lock().unwrap().insert( + "tid-legacy".to_owned(), + RestoreJobRow::reconciler_row( + target.clone(), + arn(ACCOUNT), + RestoreKind::Legacy, + JobStatus::Running, + 1, + 1234.5, + ), + ); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-legacy".to_owned()); + *fake.target_still_creating.lock().unwrap() = true; + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("Legacy", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.kind, RestoreKind::Legacy); + assert_eq!(removed.table_id, "tid-legacy"); + assert_eq!(fake.removed_targets.lock().unwrap().as_slice(), [target]); + + // A target the backend finds no longer a CREATING restore target: the + // record went with it, so the re-read finds none and the refusal is not + // the lock's. + *fake.target_still_creating.lock().unwrap() = false; + let err = maintenance::remove_target( + &*fake, + &remove_target_request("Legacy", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "Legacy (account {ACCOUNT}) is not a restore target: it is no longer a CREATING \ + restore target with table id tid-legacy" + ) + ); +} + +/// `remove-target` of a CREATING restore target whose owner's lock is still +/// held (a frozen owner whose session outlives its heartbeat, on +/// PostgreSQL) reports the held lock as its own case, not "not a restore +/// target": the backend refused the removal and the record is unchanged, so +/// the table is still the target and the operator is told to wait for the +/// owner or stop it. Nothing is removed. With the lock released the same +/// command removes the target. +#[tokio::test] +async fn remove_target_reports_a_held_owner_lock_as_its_own_case() { + let fake = Arc::new(Fake::default()); + seed_failed_target(&fake, ACCOUNT, "T", "tid-a"); + *fake.owner_lock_held.lock().unwrap() = true; + let err = maintenance::remove_target( + &*fake, + &remove_target_request("T", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "the restore into T (account {ACCOUNT}) from {} is a CREATING restore target (table \ + id tid-a) whose owner's lock is still held: its owner's session is alive though \ + its heartbeat is stale; wait for the owner to stop, or stop it, and run the \ + command again", + arn(ACCOUNT) + ) + ); + assert!( + fake.restore_jobs.lock().unwrap().contains_key("tid-a"), + "the record stays" + ); + assert!( + fake.restore_tables.lock().unwrap().contains_key("tid-a"), + "the target stays" + ); + + *fake.owner_lock_held.lock().unwrap() = false; + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("T", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.table_id, "tid-a"); +} + +// --- the manifest total, remove-target's scope and lost claim, a target without +// --- tables, the registration retry --------------------------------------------- + +/// The final count above the manifest's total fails the restore and removes +/// its target, as a count below it does: the count check is `!=`, not `<` +/// (five restored against four listed). A fresh restore starts on an empty +/// target, so a count over the total means the target holds items the +/// backup does not account for. +#[tokio::test] +async fn a_count_above_the_manifest_total_fails_the_restore() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + // The manifest lists four items; the backend reports five. + *fake.count_override.lock().unwrap() = Some(5); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.bulk_loads.lock().unwrap().len(), + 2, + "every file was loaded before the count" + ); + let writes = restore_writes(&fake); + assert_eq!( + writes[0].2.as_deref(), + Some( + "restored 5 items, the manifest lists 4: the target holds items the backup does not \ + account for" + ) + ); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the record and its target are removed" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// Seed a FAILED `target_not_empty` restore record over a non-empty CREATING +/// target of `(account, name, id)`, the state `remove-target` acts on. +fn seed_failed_target(fake: &Fake, account: &str, name: &str, id: &str) { + fake.restore_jobs.lock().unwrap().insert( + id.to_owned(), + RestoreJobRow { + target: RestoreTargetKey { + account_id: account.to_owned(), + table_name: name.to_owned(), + table_id: id.to_owned(), + }, + source_backup_arn: arn(account), + kind: RestoreKind::Manifest, + status: JobStatus::Failed, + owner_generation: 1, + heartbeat_at: seconds_since_epoch(), + store: Some(s3_identity()), + spool_host: None, + file_index: None, + items_applied: None, + failure_reason: Some(TARGET_NOT_EMPTY.to_owned()), + completed_at: Some(seconds_since_epoch()), + }, + ); + fake.restore_tables.lock().unwrap().insert( + id.to_owned(), + RestoreTable { + name: name.to_owned(), + key_attributes: vec!["pk".to_owned()], + items: vec![song("x", "y")], + active: false, + }, + ); +} + +/// `remove-target --account A T` removes only account A's target: the record +/// `find` compares the account, so account B's target of the same name is +/// untouched, and a command for A when only B's target exists is refused: +/// the record's table name alone would select either account's target. +#[tokio::test] +async fn remove_target_is_scoped_to_the_named_account() { + const OTHER: &str = "210987654321"; + let fake = Arc::new(Fake::default()); + seed_failed_target(&fake, ACCOUNT, "T", "tid-a"); + seed_failed_target(&fake, OTHER, "T", "tid-b"); + + // The command for account A removes A's target alone. + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("T", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.table_id, "tid-a"); + assert_eq!( + fake.removed_targets.lock().unwrap().as_slice(), + [RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "T".to_owned(), + table_id: "tid-a".to_owned(), + }], + "only the named account's target was removed" + ); + assert!( + fake.restore_jobs.lock().unwrap().contains_key("tid-b"), + "the other account's record is untouched" + ); + + // With only account B's target left, the command for account A finds no + // record of its own. + let err = maintenance::remove_target( + &*fake, + &remove_target_request("T", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!("T (account {ACCOUNT}) is not a restore target: no restore record names it") + ); + assert!(fake.restore_jobs.lock().unwrap().contains_key("tid-b")); +} + +/// `remove-target` whose maintenance claim is lost between the record checks +/// and the removal stops before the removal, leaving the target and its +/// record: the pre-removal `claim.lost()` check is what stops it. +#[tokio::test] +async fn remove_target_stops_before_the_removal_when_the_claim_is_lost() { + let fake = Arc::new(Fake::default()); + seed_failed_target(&fake, ACCOUNT, "T", "tid-a"); + // The claim this command takes reports lost at once; the read before the + // removal runs long enough for the HeldClaim's timer to observe it. + *fake.next_claim_lost.lock().unwrap() = true; + *fake.list_jobs_delay.lock().unwrap() = Some(Duration::from_millis(80)); + let request = maintenance::RemoveTarget { + account_id: ACCOUNT.to_owned(), + table_name: "T".to_owned(), + holder: "extenddb backup remove-target on host-a (pid 1)".to_owned(), + stale_after: Duration::from_secs(300), + owner_window: Duration::from_secs(300), + heartbeat_interval: Duration::from_millis(5), + }; + let err = maintenance::remove_target(&*fake, &request) + .await + .unwrap_err(); + assert!(err.starts_with("stopped before the removal: "), "{err}"); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "nothing was removed" + ); + assert!( + fake.restore_jobs.lock().unwrap().contains_key("tid-a"), + "the record stays" + ); + assert!( + fake.restore_tables.lock().unwrap().contains_key("tid-a"), + "the target stays" + ); +} + +/// A dead manifest restore whose target has no data tables (its owner died +/// between the registration's commit and the data DDL) is failed and its +/// target removed, not retaken every stale window. Without the readiness +/// check the count on the missing table errors and the refreshed heartbeat +/// leaves the record to be retaken, never failing. +#[tokio::test] +async fn the_reconciler_fails_a_dead_restore_whose_target_has_no_tables() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record is RUNNING with a dead owner, its target's data tables gone. + dead_record_without_progress(&fake, false); + fake.targets_without_tables + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + fake.bulk_loads.lock().unwrap().clear(); + + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(report.manifest_restores_resumed.is_empty()); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the record is gone" + ); + assert!( + !fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is removed" + ); + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no resume was spawned" + ); +} + +/// A retry that meets its own registration in flight waits for it and is +/// answered the CREATING description, rather than registering a second +/// target or being refused; a restore of another backup into the same name +/// while that registration is in flight is `TableInUseException` (the 60 s +/// registration bound equals botocore's default read timeout, so a client +/// retries while the first registration still holds the lock). +#[tokio::test] +async fn a_retry_waits_on_its_registration_and_another_backup_is_refused() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let arn_b = + format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/01791976431439-aaaaaaaa"); + seed_manifest_backup(&fake, &mem, &arn_b, music_source_table(), &[songs(2)], true); + let runtime = Arc::new(job_runtime(&mem)); + // Hold the first registration mid-flight so the target is not yet + // created when the retry's name check runs. + *fake.claim_restore_delay.lock().unwrap() = Some(Duration::from_millis(400)); + + let first = { + let fake = Arc::clone(&fake); + let runtime = Arc::clone(&runtime); + tokio::spawn(async move { restore_on(&fake, &runtime, &restore_body(json!({}))).await }) + }; + // Wait for the first request's registration to be in flight. + let deadline = tokio::time::Instant::now() + Duration::from_secs(5); + while fake.restore_claims.lock().unwrap().is_empty() { + assert!( + tokio::time::Instant::now() < deadline, + "registration never started" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + + // The retry of the same request waits and is answered CREATING. + let retry = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!(retry["TableDescription"]["TableStatus"], "CREATING"); + assert_eq!( + retry["TableDescription"]["RestoreSummary"]["SourceBackupArn"], + arn(ACCOUNT) ); - mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - // Under the maintenance claim, the store healthy again: the - // dead-CREATING duty runs, the deleting duties and the registrations - // sweep (which the backend has something for) do not. - fake.swept_registrations.lock().unwrap().push(arn_n(9)); - *fake.maintenance_held.lock().unwrap() = true; - let report = cycle_on(&fake, &runtime).await; - assert_eq!(report.store_healthy, Some(true)); - assert!(report.maintenance_held); - assert_eq!(report.creating_resolved, [(dead, BackupStatus::Available)]); - assert!(report.failed_prefixes_deleted.is_empty()); - assert!(report.deleted_prefixes_swept.is_empty()); - assert!(report.tombstones_removed.is_empty()); + // A different backup into the same name is refused while the first is in + // flight. + let other = restore_on( + &fake, + &runtime, + &restore_body(json!({ "BackupArn": arn_b })), + ) + .await + .unwrap_err(); assert!( - report.registrations_swept.is_empty(), - "the sweep does not run under the claim" + matches!(other, DynamoDbError::TableInUseException(_)), + "{other:?}" ); - assert!(mem.deletes.lock().unwrap().is_empty()); - assert_eq!(fake.rows.lock().unwrap().len(), 9); - // An AVAILABLE row past the retention with an empty prefix: a tombstone - // is a FAILED or DELETED row and nothing else. It enters the catalog - // here, older than the row the cycle above resolved, because the - // store's check heads the newest AVAILABLE row's manifest and this row - // has none. - let old_available_with_an_empty_prefix = - format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music/backup/01700000000000-00000000"); - let mut r = row(&old_available_with_an_empty_prefix, BackupStatus::Available); - r.heartbeat_at = now - 1000.0; - r.completed_at = Some(now - 1000.0); - fake.rows - .lock() - .unwrap() - .insert(old_available_with_an_empty_prefix.clone(), r); + let first = first.await.unwrap().unwrap(); + assert_eq!(first["TableDescription"]["TableStatus"], "CREATING"); + wait_for_jobs(&runtime).await; + // One target, one record: the retry registered nothing of its own. + assert_eq!(fake.restore_claims.lock().unwrap().len(), 1); +} - // The claim released: each duty on its clock. - *fake.maintenance_held.lock().unwrap() = false; - let report = cycle_on(&fake, &runtime).await; - assert_eq!( - report.failed_prefixes_deleted, - std::slice::from_ref(&stale_failed) +// --- the count retry, the maintenance claim taken mid-cycle, the takeover +// --- bound, the first file resumed with upserts ---------------------------------- + +/// The final count that fails three times stops the job with the record +/// RUNNING and no FAILED write: every item is loaded and recorded, so the +/// reconciler's resume counts again rather than the restore failing and its +/// loaded target being removed. The count is attempted `COUNT_ATTEMPTS` +/// times (three), with a wait between them. +#[tokio::test] +async fn a_count_that_fails_three_times_leaves_the_record_running() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, ); + let runtime = job_runtime(&mem); + fake.count_errors + .store(3, std::sync::atomic::Ordering::Release); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs_within(&runtime, Duration::from_secs(20)).await; assert_eq!( - report.deleted_prefixes_swept, - [ - deleted_with_objects.clone(), - old_tombstone_with_objects.clone() - ] + fake.count_calls.load(std::sync::atomic::Ordering::Acquire), + 3, + "the count was attempted three times" ); - // The tombstone duty runs after the two deleting duties, so a prefix - // they emptied this cycle counts as empty here. - let mut removed = report.tombstones_removed.clone(); - removed.sort(); assert_eq!( - removed, - [ - stale_failed.clone(), - deleted_with_objects.clone(), - old_failed_tombstone.clone(), - old_deleted_tombstone.clone(), - old_tombstone_with_objects.clone(), - ] + fake.bulk_loads.lock().unwrap().len(), + 2, + "every file was loaded before the count" ); - assert_eq!(report.registrations_swept, [arn_n(9)]); - assert!(report.acted()); + let record = fake.restore_job("MusicRestored").expect("the record stays"); + assert_eq!(record.status, JobStatus::Running, "no FAILED was written"); assert_eq!( - objects_under(&mem, &fresh_failed).len(), - 1, - "a FAILED row with a live heartbeat keeps its prefix" + (record.file_index, record.items_applied), + (Some(1), Some(2)), + "the progress of the last file stands for the resume" ); - assert!(objects_under(&mem, &stale_failed).is_empty()); - assert!(objects_under(&mem, &deleted_with_objects).is_empty()); - assert_eq!( - objects_under(&mem, &old_failed_with_objects_and_a_live_heartbeat).len(), - 1, - "no duty empties the prefix of a FAILED row with a live heartbeat" + assert!( + restore_writes(&fake) + .iter() + .all(|(_, to, _)| *to != JobStatus::Failed && *to != JobStatus::Done), + "no status write ended the job: {:?}", + restore_writes(&fake) + ); + assert!( + fake.restore_tables + .lock() + .unwrap() + .contains_key("tid-MusicRestored"), + "the loaded target is kept" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// A command that takes the maintenance claim during the cycle, after the +/// cycle's first read of it and before the takeover, is seen by the second +/// read after `claim_restore_dead`: the record is taken over (its generation +/// moved, its heartbeat fresh) but not resumed, and is left for the +/// maintenance arm of a later cycle. +#[tokio::test] +async fn a_claim_taken_during_the_cycle_leaves_the_taken_over_record_unresumed() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.bulk_loads.lock().unwrap().clear(); + let before = fake.restore_job("MusicRestored").unwrap().owner_generation; + // The command takes the claim inside the takeover, so the cycle's first + // read (not held) and its second read (held) disagree. + *fake.takeover_hook.lock().unwrap() = Some(Box::new(|fake: &Fake| { + *fake.maintenance_held.lock().unwrap() = true; + })); + + let report = cycle_on(&fake, &runtime).await; + assert!(!report.maintenance_held, "the first read found no claim"); + assert!( + fake.takeover_hook.lock().unwrap().is_none(), + "the takeover ran" + ); + assert!( + report.manifest_restores_resumed.is_empty(), + "the record was not resumed: {report:?}" + ); + assert!(report.manifest_restores_failed.is_empty()); + assert!(report.manifest_restores_failed_and_removed.is_empty()); + wait_for_jobs(&runtime).await; + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no resume was spawned" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert_ne!(record.owner_generation, before, "the takeover itself stood"); +} + +/// A record whose every resume ends with it RUNNING again (here the final +/// count fails on every attempt, so each resume stops `LeftRunning` and the +/// record is retaken every stale window) is failed by the takeover after +/// this instance's fifth: FAILED `TAKEOVERS_EXHAUSTED`, the target removed. +/// Five cycles take it over and leave it RUNNING; the sixth fails it. +#[tokio::test] +async fn the_takeover_after_the_fifth_fails_a_record_that_never_ends() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record is RUNNING with its progress and a dead owner; the target's + // count fails on every attempt, so every resume ends `LeftRunning`. { - let rows = fake.rows.lock().unwrap(); - assert!(rows.contains_key(&fresh_failed), "its heartbeat is live"); - assert!( - rows.contains_key(&young_tombstone), - "its retention has not passed" - ); - assert!( - rows.contains_key(&old_failed_with_objects_and_a_live_heartbeat), - "its prefix still lists an object" - ); - assert!( - rows.contains_key(&old_available_with_an_empty_prefix), - "AVAILABLE is not a tombstone status" + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + fake.restore_tables + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .active = false; + if let Some(desc) = fake.tables.lock().unwrap().get_mut("MusicRestored") { + desc.table_status = TableStatus::Creating; + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + } + fake.count_errors + .store(u32::MAX, std::sync::atomic::Ordering::Release); + let mut state = reconciler::ReconcilerState::default(); + for cycle in 1..=reconciler::RESTORE_TAKEOVERS { + let report = reconciler::cycle( + &fake, + &runtime, + Duration::from_secs(300), + "test", + &mut state, + ) + .await + .unwrap(); + assert_eq!( + report.manifest_restores_resumed, + ["MusicRestored"], + "cycle {cycle} took the record over and resumed it" ); + wait_for_jobs_within(&runtime, Duration::from_secs(20)).await; + let record = fake.restore_job("MusicRestored").expect("the record stays"); assert_eq!( - rows[&old_available_with_an_empty_prefix].status, - BackupStatus::Available + record.status, + JobStatus::Running, + "cycle {cycle}: the resume ended with the record RUNNING" ); - assert_eq!(rows.len(), 5, "the four kept and the resolved CREATING row"); } + assert_eq!(fake.restore_takeovers.lock().unwrap().len(), 5); + + let report = reconciler::cycle( + &fake, + &runtime, + Duration::from_secs(300), + "test", + &mut state, + ) + .await + .unwrap(); + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"], + "the sixth takeover fails the record: {report:?}" + ); + assert!(report.manifest_restores_resumed.is_empty()); assert!( - !fake - .remove_attempts - .lock() - .unwrap() - .contains(&old_available_with_an_empty_prefix), - "the duty asks the catalog to remove FAILED and DELETED rows only; the backends would \ - refuse an AVAILABLE one, and the duty does not lean on that" + fake.restore_job("MusicRestored").is_none(), + "the record and its target are removed" + ); + assert!( + !fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is removed" + ); + let writes = restore_writes(&fake); + assert!( + writes.iter().any(|(from, to, reason)| { + *from == JobStatus::Running + && *to == JobStatus::Failed + && reason.as_deref() == Some(reconciler::TAKEOVERS_EXHAUSTED) + }), + "the FAILED write names the exhausted takeovers: {writes:?}" ); +} + +/// A dead record with no progress recorded over a target that holds items, +/// no more of them than the backup's total, is the state a crash inside the +/// first batch leaves on Cassandra (progress is recorded after the batch's +/// writes): it is resumed, not failed `target_not_empty`, and the resume +/// re-applies the first data file with `reapplying` set (idempotent +/// upserts), so the restore ends ACTIVE with the exact count. Only a count +/// above the backup's total is `target_not_empty` +/// (`a_resume_without_progress_needs_an_empty_target`). +#[tokio::test] +async fn a_half_applied_first_file_with_no_progress_is_resumed_with_upserts() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record with no progress, the target holding one of the backup's + // two items (the first batch's writes landed, its progress did not). + dead_record_without_progress(&fake, false); + let first = songs(2).remove(0); + fake.restore_tables + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .items + .push(first); + fake.bulk_loads.lock().unwrap().clear(); - // The next cycle has nothing left to do. let report = cycle_on(&fake, &runtime).await; - assert!(report.tombstones_removed.is_empty()); - assert!(report.deleted_prefixes_swept.is_empty()); - assert!(report.failed_prefixes_deleted.is_empty()); - assert!(report.creating_resolved.is_empty()); + assert_eq!( + report.manifest_restores_resumed, + ["MusicRestored"], + "one item of two is resumed, not failed: {report:?}" + ); + assert!(report.manifest_restores_failed.is_empty()); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap(); + assert_eq!(loads.len(), 1, "the one file was loaded again"); + assert!( + loads[0].file.reapplying, + "the first file is re-applied with upserts: {:?}", + loads[0].file + ); + assert_eq!((loads[0].file.file_index, loads[0].file.start_at), (0, 0)); + drop(loads); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done, "{record:?}"); + assert_eq!( + fake.restore_tables.lock().unwrap()["tid-MusicRestored"] + .items + .len(), + 2, + "the duplicate of the half-applied item was suppressed and the count is exact" + ); } // --- the review's cells (PR 8, Lee's items) ------------------------------- @@ -11916,7 +19120,7 @@ async fn the_jobs_failed_deletion_and_the_probes_failed_listing_are_counted() { // The probe's listing. let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); let dead_prefix = BackupArn::parse(&dead).unwrap().store_prefix(); *mem.fail_lists_of.lock().unwrap() = Some(dead_prefix); @@ -12030,8 +19234,7 @@ mod review_round_3 { let handle = job::heartbeat::HeartbeatHandle::start( Arc::clone(&claim), period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); // The first tick's beat parks at one period; at two and a half the @@ -12072,8 +19275,7 @@ mod review_round_3 { let handle = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); tokio::time::sleep(5 * period).await; @@ -12119,8 +19321,7 @@ mod review_round_3 { let handle = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now() - 2 * period, ); // The first tick's beat starts at one period; half a period in it @@ -12159,22 +19360,24 @@ mod review_round_3 { /// `TableInUseException`): an existing table of the name answers /// `TableAlreadyExistsException` though the backup is in use, a CREATING /// restore target `TableInUseException` naming its source backup, and a free -/// name reaches the in-use rule and then this release's refusal of a -/// store-backed restore. A row that is not AVAILABLE answers for itself -/// before the name is read. +/// name reaches the in-use rule and then the manifest read (here +/// `BackupNotFoundException`: the store holds nothing under the row's +/// prefix). A row that is not AVAILABLE answers for itself before the name +/// is read. #[tokio::test] async fn a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_its_in_use_rule() { // The row's store is the configured, healthy one, so a free name reaches - // the store's refusal after the in-use rule. - let runtime = runtime_on(&Arc::new(MemStore::default()), store()); - let fake = fake_with_row(BackupStatus::Available); + // the store after the in-use rule. + let mem = Arc::new(MemStore::default()); + let runtime = runtime_on(&mem, store()); + let fake = Arc::new(fake_with_row(BackupStatus::Available)); fake.in_use .lock() .unwrap() .insert(arn(ACCOUNT), "Elsewhere".to_owned()); let limits = LimitsConfig::default(); let body = restore_body(json!({})); - let restore_on = async |fake: &Fake| { + let restore_on = async |fake: &Arc| { restore_table_from_backup(fake, &runtime, ACCOUNT, REGION, &limits, &body).await }; @@ -12220,12 +19423,16 @@ async fn a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_i "a free name reaches the in-use rule: {err:?}" ); fake.in_use.lock().unwrap().clear(); + assert!( + mem.gets().is_empty(), + "no store read before the in-use rule" + ); let err = restore_on(&fake).await.unwrap_err(); assert!( - matches!(err, DynamoDbError::ValidationException(ref m) - if m.contains("is in a backup store; this release restores in-catalog backups only")), + matches!(err, DynamoDbError::BackupNotFoundException(_)), "{err:?}" ); + assert!(!mem.gets().is_empty(), "the free name reached the store"); // A row that is not AVAILABLE: its own answer, the name unread. A // CREATING row is the backup being created (not measured on the service: @@ -12234,7 +19441,7 @@ async fn a_catalog_row_restore_checks_the_target_name_after_the_row_and_before_i (BackupStatus::Deleted, "not found"), (BackupStatus::Creating, "being created"), ] { - let fake = fake_with_row(status); + let fake = Arc::new(fake_with_row(status)); fake.tables.lock().unwrap().insert( "MusicRestored".to_owned(), TableDescription { diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index b6c0e25b9..aa0e9eebf 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -10,9 +10,9 @@ mod backup; pub use backup::{ BackupRuntime, ConfiguredStore, HeldClaim, JobSettings, KillPoint, LOCAL_CHECK_TIMEOUT, - S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, - StoreProblem, check_configured_store, maintenance, read_identity_bounded, reconciler, - store_identity, + OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, + StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, TARGET_NOT_EMPTY, + check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity, }; mod batch_get_item; mod batch_write_item; diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml index ed72f54fb..653bd6c80 100755 --- a/crates/server/Cargo.toml +++ b/crates/server/Cargo.toml @@ -9,9 +9,15 @@ license.workspace = true [features] mongodb-test-hooks = [] -# The backup job's kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), for the -# binary-level crash tests. -backup-test-hooks = ["extenddb-engine/backup-test-hooks"] +# The backup and restore jobs' kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), +# for the binary-level crash tests. The engine's points are its own feature; +# the points inside the backends' writes (`restore_sqlite_mid_batch`, +# `restore_postgres_before_progress_delete`, +# `restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`) +# are calls of `extenddb_storage::backup_catalog::restore_pause_point`, so +# `extenddb-storage/test-hooks` arms them in whichever backend the build +# selects without naming a backend crate here (the server depends on none). +backup-test-hooks = ["extenddb-engine/backup-test-hooks", "extenddb-storage/test-hooks"] [dependencies] extenddb-cache = { workspace = true } diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs index 8fc5f3518..3f81615ef 100644 --- a/crates/server/src/backup_store.rs +++ b/crates/server/src/backup_store.rs @@ -974,6 +974,37 @@ mod tests { ) -> BoxedFuture<'a, Result>, StorageError>> { unused() } + fn claim_restore<'a>( + &'a self, + _account_id: &'a str, + _input: extenddb_core::types::CreateTableInput, + _registration: &'a extenddb_storage::backup_catalog::RestoreRegistrationRow, + ) -> BoxedFuture< + 'a, + Result<(extenddb_core::types::TableDescription, Box), StorageError>, + > { + unused() + } + fn claim_restore_dead<'a>( + &'a self, + _key: &'a RestoreTargetKey, + _stale_after: std::time::Duration, + _spool_host: Option<&'a str>, + ) -> BoxedFuture<'a, Result>, StorageError>> { + unused() + } + fn read_restore_job<'a>( + &'a self, + _key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + unused() + } + fn list_restore_jobs<'a>( + &'a self, + _status: Option, + ) -> BoxedFuture<'a, Result, StorageError>> { + unused() + } fn list_backup_rows<'a>( &'a self, _status: Option, diff --git a/crates/server/src/metrics_endpoint.rs b/crates/server/src/metrics_endpoint.rs index 9803f3b23..0c24473bf 100755 --- a/crates/server/src/metrics_endpoint.rs +++ b/crates/server/src/metrics_endpoint.rs @@ -378,6 +378,9 @@ mod tests { MetricName::BackupBytesUploaded, MetricName::BackupSnapshotAge, MetricName::BackupStoreErrors, + MetricName::RestoreJobs, + MetricName::RestoreJobDuration, + MetricName::RestoreBytesDownloaded, ]; let rows: Vec = names .iter() diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs index b4d9c1b67..8505c0b75 100644 --- a/crates/server/src/serve.rs +++ b/crates/server/src/serve.rs @@ -683,7 +683,8 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { backup_runtime .with_metrics(metrics.clone()) .with_shutdown(shutdown.child_token()) - .with_source(&backend_name, build.version), + .with_source(&backend_name, build.version) + .with_limits(Arc::clone(&limits)), ); let backup_runtime_for_reconciler = Arc::clone(&backup_runtime); let state = AppState { diff --git a/crates/storage-cassandra/Cargo.toml b/crates/storage-cassandra/Cargo.toml index 4d104122b..ad9bb4b7e 100644 --- a/crates/storage-cassandra/Cargo.toml +++ b/crates/storage-cassandra/Cargo.toml @@ -33,8 +33,10 @@ chrono = { workspace = true } # Entry points the integration tests build catalogs with. test-util = ["extenddb-storage/test-util"] # Coordination hooks used only by this crate's integration test binary -# (`src/test_hooks.rs`). Production builds do not compile them. -test-hooks = [] +# (`src/test_hooks.rs`). Production builds do not compile them. Enabling it +# also arms `extenddb_storage::backup_catalog::restore_pause_point`, the +# `bulk_load` kill point the restore kill-point harness drives. +test-hooks = ["extenddb-storage/test-hooks"] [dev-dependencies] extenddb-storage = { workspace = true, features = ["test-util"] } diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs index f24e1f0a7..36f767dd7 100644 --- a/crates/storage-cassandra/src/backup_catalog.rs +++ b/crates/storage-cassandra/src/backup_catalog.rs @@ -73,11 +73,12 @@ use cdrs_tokio::types::IntoRustByName; use cdrs_tokio::types::rows::Row; use cdrs_tokio::types::value::Value; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::CreateTableInput; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, billing_mode_name, - new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, billing_mode_name, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -173,7 +174,7 @@ fn outcome_unknown(error: &DriverError) -> bool { } /// Run a conditional write and report what it did. -async fn conditional_write( +pub(crate) async fn conditional_write( session: &CassandraSession, query: &str, values: QueryValues, @@ -572,6 +573,80 @@ impl CassandraEngine { Ok(arns) } + /// Re-apply the target's `active_restores[table_id] = table_name` entry + /// on the backup's row when it is absent, under the takeover of a dead + /// manifest record (the trait doc of `claim_restore_dead`): one serial + /// read of the map, then a lightweight transaction on the entry + /// conditioned on the backup still AVAILABLE. A backup no longer + /// AVAILABLE (a delete won while nothing held it) keeps no entry: the + /// resumed load fails on the missing prefix and the target is removed. + async fn reapply_restore_registration( + &self, + key: &RestoreTargetKey, + backup_arn: &str, + ) -> Result<(), StorageError> { + use cdrs_tokio::types::AsRust; + use cdrs_tokio::types::map::Map; + let catalog = self.catalog_keyspace(); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &format!( + "SELECT active_restores FROM {catalog}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(key.account_id.as_str(), backup_arn), + "restore registration", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(()); + }; + let map: Option = row.get_by_name("active_restores").ok().flatten(); + let entries: std::collections::HashMap = match map { + Some(map) => map + .as_rust() + .map_err(|e| StorageError::Internal(format!("Parse active_restores: {e}")))? + .unwrap_or_default(), + None => std::collections::HashMap::new(), + }; + if entries.contains_key(&key.table_id) { + return Ok(()); + } + let applied = conditional_write( + &self.session, + &format!( + "UPDATE {catalog}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ? IF backup_status = 'AVAILABLE'" + ), + QueryValues::SimpleValues(vec![ + Value::from(key.table_id.as_str()), + Value::from(key.table_name.as_str()), + Value::from(key.account_id.as_str()), + Value::from(backup_arn), + ]), + ) + .await?; + match applied { + WriteOutcome::Matched => tracing::warn!( + "restore of table {}: the backup {backup_arn} did not list the target as in \ + use (its owner died between the target's creation and the registration); the \ + entry was re-applied by the takeover", + key.table_name + ), + WriteOutcome::NotMatched => tracing::warn!( + "restore of table {}: the backup {backup_arn} is no longer AVAILABLE and does \ + not list the target; the resumed load fails on the missing prefix", + key.table_name + ), + WriteOutcome::Unknown => tracing::warn!( + "restore of table {}: re-applying the backup's in-use entry had no known \ + outcome; the registration sweep removes a stale one", + key.table_name + ), + } + Ok(()) + } + /// The `tables` row of a restore target, read at quorum. async fn restore_target_row( &self, @@ -631,6 +706,187 @@ impl RestoreTargetRow { } } +/// The `tables` columns one restore job record fills, in the order +/// [`restore_job_row_from`] reads them. +const RESTORE_JOB_COLUMNS: &str = "account_id, table_name, table_id, table_status, \ + restore_source_backup_arn, restore_kind, restore_status, restore_owner_generation, \ + restore_heartbeat_at, restore_store_kind, restore_store_root, restore_store_id, \ + restore_spool_host, restore_file_index, restore_items_applied, restore_failure_reason, \ + restore_completed_at"; + +/// One [`RestoreJobRow`] from a `tables` row that holds a restore record +/// (every member [`RESTORE_JOB_COLUMNS`] selects): the store the restore +/// reads, the claimant's spool, the recorded progress, the failure reason +/// and the completion time, beyond the six the reconciler's legacy and dead +/// arms read. +fn restore_job_row_from(row: &Row) -> Result { + let column = |name: &str| -> Result { + crate::cassandra_util::get_column(row, name, "restore job") + }; + let optional = |name: &str| -> Option { row.get_by_name(name).ok().flatten() }; + let store = optional("restore_store_kind") + .as_deref() + .and_then(StoreKind::parse) + .map(|store_kind| StoreIdentity { + store_kind, + store_root: optional("restore_store_root").unwrap_or_default(), + store_id: optional("restore_store_id").unwrap_or_default(), + }); + let generation: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + let completed_at: Option = row.get_by_name("restore_completed_at").ok().flatten(); + let file_index: Option = row.get_by_name("restore_file_index").ok().flatten(); + let items_applied: Option = row.get_by_name("restore_items_applied").ok().flatten(); + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id: column("account_id")?, + table_name: column("table_name")?, + table_id: column("table_id")?, + }, + source_backup_arn: column("restore_source_backup_arn")?, + kind: RestoreKind::parse(&column("restore_kind")?)?, + status: JobStatus::parse(&column("restore_status")?)?, + owner_generation: generation_from_column(generation), + heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), + store, + spool_host: optional("restore_spool_host"), + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason: optional("restore_failure_reason"), + completed_at: completed_at.map(millis_to_seconds), + }) +} + +/// A store-backed restore job's handle on its target's `tables` row. Unlike +/// the legacy copy's self-managed [`crate::backup_engine::JobHeartbeat`], +/// the engine drives this claim's heartbeat and status writes; every write +/// is a lightweight transaction conditioned on the restore generation and on +/// the record being RUNNING, and leaves `table_status` to +/// [`CassandraEngine::activate_restore_target_row`] (the ACTIVE flip) and to +/// `remove_restore_target` (the CREATING to DELETING step of a removal). +struct CassandraRestoreClaim { + engine: CassandraEngine, + key: RestoreTargetKey, + generation: u64, +} + +impl JobClaim for CassandraRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let query = format!( + "UPDATE {}.tables SET restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = 'RUNNING'", + self.engine.catalog_keyspace() + ); + let result = crate::cassandra_util::query_lwt( + &self.engine.session, + &query, + cdrs_tokio::query_values!( + self.key.account_id.as_str(), + self.key.table_name.as_str(), + self.key.table_id.as_str(), + generation_column(self.generation) + ), + ) + .await?; + Ok(if crate::cassandra_util::lwt_applied(&result)? { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The record only; `table_status` is left to the ACTIVE flip and + // to the removal's CREATING to DELETING step (4(e)'s two-write + // rule). A restore record has no size/item/source-table columns, + // so those `StatusFields` are ignored. + let mut assignments = vec!["restore_status = ?".to_owned()]; + let mut values = vec![Value::from(to.as_str())]; + if let Some(reason) = fields.failure_reason.as_deref() { + assignments.push("restore_failure_reason = ?".to_owned()); + values.push(Value::from(reason)); + } + if fields.completed { + assignments.push("restore_completed_at = toTimestamp(now())".to_owned()); + } + values.extend([ + Value::from(self.key.account_id.as_str()), + Value::from(self.key.table_name.as_str()), + Value::from(self.key.table_id.as_str()), + Value::from(generation_column(self.generation)), + Value::from(from.as_str()), + ]); + let query = format!( + "UPDATE {}.tables SET {} WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = ?", + self.engine.catalog_keyspace(), + assignments.join(", ") + ); + conditional_write( + &self.engine.session, + &query, + QueryValues::SimpleValues(values), + ) + .await + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let query = format!( + "SELECT restore_status, restore_owner_generation, restore_heartbeat_at, \ + restore_failure_reason FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.engine.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.engine.session, + &query, + cdrs_tokio::query_values!( + self.key.account_id.as_str(), + self.key.table_name.as_str() + ), + "restore job row", + ) + .await?; + let row = rows.first().ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.key.table_name + )) + })?; + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let status = status.ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.key.table_name + )) + })?; + let generation: Option = + row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + Ok(JobRow { + status: JobStatus::parse(&status)?, + owner_generation: generation_from_column(generation), + heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), + failure_reason: row.get_by_name("restore_failure_reason").ok().flatten(), + }) + }) + } +} + impl BackupCatalog for CassandraEngine { fn catalog_id(&self) -> BoxFuture<'_, Result> { Box::pin(async move { @@ -1361,7 +1617,22 @@ impl BackupCatalog for CassandraEngine { else { return Ok(None); }; - self.creating_restore_target(&stored).await + // The backup's `active_restores` map names the targets whose + // registration committed its second write. A store-backed + // registration creates the target row (`tables` with + // `restore_source_backup_arn`, CREATING) before it writes that + // map entry, so a second restore's in-use check in that window + // would miss the first and fall through to its own override + // validation (`ValidationException` naming the override's index + // where the other backends answer `BackupInUseException`). Read + // the target rows too: a + // CREATING restore target of this backup is in use whether or not + // its `active_restores` entry has landed yet. + if let Some(target) = self.creating_restore_target(&stored).await? { + return Ok(Some(target)); + } + self.creating_restore_target_of_backup(&key.account_id, &key.backup_arn) + .await }) } @@ -1436,13 +1707,11 @@ impl BackupCatalog for CassandraEngine { // window, every RUNNING record the scans saw is listed). A range // scan has no Paxos read (a serial level resolves to a local // quorum there), so it stays at quorum. - let columns = "account_id, table_name, table_id, table_status, \ - restore_source_backup_arn, restore_kind, restore_status, \ - restore_owner_generation, restore_heartbeat_at, restore_completed_at"; let mut scanned = Vec::new(); for status in ["RUNNING", "FAILED"] { let query = format!( - "SELECT {columns} FROM {}.tables WHERE restore_status = ? ALLOW FILTERING", + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables WHERE restore_status = ? \ + ALLOW FILTERING", self.catalog_keyspace() ); let rows = crate::cassandra_util::query_rows_quorum::( @@ -1464,15 +1733,16 @@ impl BackupCatalog for CassandraEngine { row.get_by_name("restore_heartbeat_at").ok().flatten(); let completed_at: Option = row.get_by_name("restore_completed_at").ok().flatten(); - let column = |name: &str| -> Result { - crate::cassandra_util::get_column(&row, name, "dead restore jobs") - }; // A RUNNING record always carries a heartbeat (the insert // or the backfill wrote it); one without is as dead as the // epoch. let age_of = if status == "FAILED" { let target = RestoreTargetRow { - table_id: column("table_id")?, + table_id: crate::cassandra_util::get_column( + &row, + "table_id", + "dead restore jobs", + )?, table_status: row.get_by_name("table_status").ok().flatten(), source_backup_arn: row .get_by_name("restore_source_backup_arn") @@ -1490,34 +1760,27 @@ impl BackupCatalog for CassandraEngine { if age_of.is_some_and(|at| at >= cutoff) { continue; } - let generation: Option = - row.get_by_name("restore_owner_generation").ok().flatten(); - // A row this build cannot read (a kind no release writes) - // is logged and skipped, as ListBackups skips a row it - // cannot read, so one such row does not stop the cycle. - let kind = match RestoreKind::parse(&column("restore_kind")?) { - Ok(kind) => kind, + // A row this build cannot read (a kind or status no + // release writes) is logged and skipped, as ListBackups + // skips a row it cannot read, so one such row does not + // stop the cycle. + match restore_job_row_from(&row) { + Ok(job) => dead.push(job), Err(e) => { tracing::warn!( "reconciler: restore job record of {} (account {}) skipped: {e}", - column("table_name").unwrap_or_default(), - column("account_id").unwrap_or_default() + row.get_by_name("table_name") + .ok() + .flatten() + .unwrap_or_else(String::new), + row.get_by_name("account_id") + .ok() + .flatten() + .unwrap_or_else(String::new) ); continue; } - }; - dead.push(RestoreJobRow { - target: RestoreTargetKey { - account_id: column("account_id")?, - table_name: column("table_name")?, - table_id: column("table_id")?, - }, - source_backup_arn: column("restore_source_backup_arn")?, - kind, - status: JobStatus::parse(status)?, - owner_generation: generation_from_column(generation), - heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), - }); + } } } Ok(dead) @@ -1531,9 +1794,8 @@ impl BackupCatalog for CassandraEngine { // ahead of the coordinator's clock (skew between coordinators), // is listed too. let query = format!( - "SELECT account_id, table_name, table_id, restore_source_backup_arn, \ - restore_kind, restore_owner_generation, restore_heartbeat_at \ - FROM {}.tables WHERE restore_status = 'RUNNING' ALLOW FILTERING", + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables WHERE restore_status = 'RUNNING' \ + ALLOW FILTERING", self.catalog_keyspace() ); let rows = crate::cassandra_util::query_rows_quorum::( @@ -1543,30 +1805,220 @@ impl BackupCatalog for CassandraEngine { "running restore jobs", ) .await?; - let mut running = Vec::with_capacity(rows.len()); - for row in rows { - let column = |name: &str| -> Result { - crate::cassandra_util::get_column(&row, name, "running restore jobs") - }; - let kind = column("restore_kind")?; - let generation: Option = - row.get_by_name("restore_owner_generation").ok().flatten(); - let heartbeat_at: Option = - row.get_by_name("restore_heartbeat_at").ok().flatten(); - running.push(RestoreJobRow { - target: RestoreTargetKey { - account_id: column("account_id")?, - table_name: column("table_name")?, - table_id: column("table_id")?, + rows.iter().map(restore_job_row_from).collect() + }) + } + + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + Self::validate_account_id(account_id)?; + let (description, generation) = self + .register_manifest_restore_target(account_id, input, registration) + .await?; + let key = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: description.table_name.clone(), + table_id: description.table_id.clone(), + }; + Ok(( + description, + Box::new(CassandraRestoreClaim { + engine: self.clone(), + key, + generation, + }) as Box, + )) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The record is read at LOCAL_SERIAL, so a heartbeat an LWT just + // wrote is seen; dead is `restore_heartbeat_at` older than the + // window on the coordinator's clock. The takeover is conditioned + // on the generation, RUNNING, the manifest kind and the heartbeat + // as read: a beat landing between the read and the write (the + // owner alive after all), another taker's generation, or a record + // that left RUNNING leaves the row and answers `None`. The write + // sets the generation, the claimant's `spool_host` (`None` clears + // it) and `restore_heartbeat_at` to the coordinator's clock, so a + // second taker's read finds the record live at once (the trait + // doc says why a restore record is refreshed where a backup row + // is not). A legacy record is not a manifest restore's to take. + let catalog = self.catalog_keyspace(); + let read = format!( + "SELECT table_id, restore_status, restore_kind, restore_owner_generation, \ + restore_heartbeat_at, restore_source_backup_arn FROM {catalog}.tables \ + WHERE account_id = ? AND table_name = ?" + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &read, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "dead restore job", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(None); + }; + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + if table_id.as_deref() != Some(key.table_id.as_str()) { + return Ok(None); + } + let status: Option = row.get_by_name("restore_status").ok().flatten(); + if status.as_deref() != Some(JobStatus::Running.as_str()) { + return Ok(None); + } + let owner_generation: Option = + row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + let source_backup_arn: Option = + row.get_by_name("restore_source_backup_arn").ok().flatten(); + let now = coordinator_now_millis(self).await?; + let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX); + if heartbeat_at.is_some_and(|at| at >= now.saturating_sub(stale_ms)) { + return Ok(None); + } + // A test may interpose a beat here, between the read and the + // write, to prove the heartbeat condition refuses the takeover. + crate::test_hooks::before_restore_takeover(&key.table_id).await; + let generation = new_generation(); + let take_over = format!( + "UPDATE {catalog}.tables SET restore_owner_generation = ?, \ + restore_spool_host = ?, restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF restore_owner_generation = ? AND restore_status = 'RUNNING' \ + AND restore_kind = 'manifest' AND restore_heartbeat_at = ?" + ); + let taken = conditional_write( + &self.session, + &take_over, + QueryValues::SimpleValues(vec![ + Value::from(generation_column(generation)), + match spool_host { + Some(host) => Value::from(host), + None => Value::from(Option::::None), }, - source_backup_arn: column("restore_source_backup_arn")?, - kind: RestoreKind::parse(&kind)?, - status: JobStatus::Running, - owner_generation: generation_from_column(generation), - heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), - }); + Value::from(key.account_id.as_str()), + Value::from(key.table_name.as_str()), + Value::from(owner_generation), + Value::from(heartbeat_at), + ]), + ) + .await?; + let claimed = match taken { + WriteOutcome::Matched => true, + WriteOutcome::NotMatched => false, + // A takeover whose answer was lost may have committed: a + // record carrying this call's generation is this call's claim. + WriteOutcome::Unknown => { + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &format!( + "SELECT restore_owner_generation FROM {catalog}.tables \ + WHERE account_id = ? AND table_name = ?" + ), + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "dead restore job", + ) + .await?; + let current: Option = rows + .first() + .and_then(|r| r.get_by_name("restore_owner_generation").ok().flatten()); + current == Some(generation_column(generation)) + } + }; + if !claimed { + return Ok(None); + } + // The backup's `active_restores` entry for the target, re-applied + // when it is absent: the registration writes the target's row and + // its DDL first and the entry second, so an owner that died + // between the two left a resumable record whose backup nothing + // held in use. Without the entry `start_backup_delete` would + // succeed under the resumed load. + if let Some(backup_arn) = source_backup_arn { + self.reapply_restore_registration(key, &backup_arn).await?; + } + Ok(Some(Box::new(CassandraRestoreClaim { + engine: self.clone(), + key: key.clone(), + generation, + }) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let query = format!( + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables \ + WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore job", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(None); + }; + // No record, or a table re-created under the name with another id, + // is not this target's job. + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + if table_id.as_deref() != Some(key.table_id.as_str()) || status.is_none() { + return Ok(None); + } + Ok(Some(restore_job_row_from(row)?)) + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + // CQL filters one equality per scan, so a status filter is one + // scan and "every record" is a scan per restore status. + let statuses = match status { + Some(status) => vec![status], + None => vec![JobStatus::Running, JobStatus::Done, JobStatus::Failed], + }; + let mut jobs = Vec::new(); + for status in statuses { + let query = format!( + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables WHERE restore_status = ? \ + ALLOW FILTERING", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(status.as_str()), + "restore jobs", + ) + .await?; + for row in rows { + jobs.push(restore_job_row_from(&row)?); + } } - Ok(running) + Ok(jobs) }) } diff --git a/crates/storage-cassandra/src/backup_engine.rs b/crates/storage-cassandra/src/backup_engine.rs index 1284c3511..ecb73e318 100755 --- a/crates/storage-cassandra/src/backup_engine.rs +++ b/crates/storage-cassandra/src/backup_engine.rs @@ -80,11 +80,13 @@ use cdrs_tokio::query::BatchQueryBuilder; use cdrs_tokio::types::map::Map; use cdrs_tokio::types::{AsRust, IntoRustByName, rows::Row}; use extenddb_core::types::{ - AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, DeleteTableInput, - GsiInput, KeySchemaElement, LsiInput, ProvisionedThroughput, SourceTableDetails, - TableDescription, TableKeyInfo, + AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, CreateTableInput, + DeleteTableInput, GsiInput, KeySchemaElement, LsiInput, ProvisionedThroughput, + SourceTableDetails, TableDescription, TableKeyInfo, +}; +use extenddb_storage::backup_catalog::{ + AbandonedLegacyBackup, RestoreRegistrationRow, new_generation, }; -use extenddb_storage::backup_catalog::{AbandonedLegacyBackup, new_generation}; use extenddb_storage::backup_definition::{ BACKUP_DEFINITION_VERSION, BackupTableDefinition, throughput_from_catalog, }; @@ -106,8 +108,9 @@ use crate::delete_table::DeleteTableCaller; const BACKUP_ITEMS_PER_BUCKET: i64 = 64; const BACKUP_SCAN_PAGE_SIZE: i64 = 1_000; -/// Items of one bucket a restore copy writes concurrently. -const RESTORE_WRITE_CONCURRENCY: usize = 8; +/// Items of one bucket a restore copy writes concurrently. Shared by the +/// legacy copy and the store-backed `bulk_load`. +pub(crate) const RESTORE_WRITE_CONCURRENCY: usize = 8; /// Heartbeats that error in a row after which a job stops: three beats could /// not confirm the claim, so the row may already be another owner's. One @@ -533,6 +536,26 @@ impl CassandraEngine { Ok(backup) } + /// The AVAILABLE catalog-index backup (`store_kind` set) `backup_arn` + /// names; an in-catalog row is "not found" here, the mirror of + /// [`Self::load_available_backup`]. A store-backed restore reads this + /// row, not the legacy one. + async fn load_available_index_backup( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result { + let backup = self + .load_stored_backup(account_id, backup_arn) + .await? + .filter(|backup| !backup.is_in_catalog()) + .ok_or_else(|| backup_not_found(backup_arn))?; + if backup.backup_status != "AVAILABLE" { + return Err(backup_not_found(backup_arn)); + } + Ok(backup) + } + /// Delete a backup's `backup_items` buckets: the `bucket_count(item_count)` /// the row accounts for, then every further bucket until an empty one. /// The create writes buckets in order and publishes `item_count` last, so @@ -745,6 +768,24 @@ impl CassandraEngine { self.ensure_backup_not_restoring(&backup).await } + /// [`Self::registration_refusal`] for a store-backed (index) backup: it + /// re-reads the catalog-index row rather than the in-catalog one. + async fn registration_refusal_index( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result<(), StorageError> { + let backup = self + .load_stored_backup(account_id, backup_arn) + .await? + .filter(|backup| !backup.is_in_catalog()) + .ok_or_else(|| backup_not_found(backup_arn))?; + if backup.backup_status != "AVAILABLE" { + return Err(backup_not_found(backup_arn)); + } + self.ensure_backup_not_restoring(&backup).await + } + async fn unregister_restore( &self, account_id: &str, @@ -832,6 +873,39 @@ impl CassandraEngine { }) } + /// The name of a CREATING restore target whose `restore_source_backup_arn` + /// is `backup_arn`, read from the `tables` rows directly rather than from + /// the backup's `active_restores` map: a store-backed registration writes + /// the target row before the map entry, so this catches an in-flight + /// registration the map does not yet name (the in-use check would + /// otherwise miss it and a second restore's own override validation + /// would answer first). A table + /// scan filtered on the two columns; the in-use check is a control-plane + /// path (DeleteBackup, a restore's registration), not a hot path. + pub(crate) async fn creating_restore_target_of_backup( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result, StorageError> { + let query = format!( + "SELECT table_name FROM {}.tables \ + WHERE account_id = ? AND restore_source_backup_arn = ? AND table_status = 'CREATING' \ + ALLOW FILTERING", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(account_id, backup_arn), + "restore target of backup", + ) + .await?; + Ok(rows.first().and_then(|row| { + let name: Option = row.get_by_name("table_name").ok().flatten(); + name + })) + } + /// The name of a CREATING restore target registered on `backup` and still /// being restored (the in-use rule: it reads the target's status, not the /// registration alone); `None` when there is none. Every registration @@ -944,6 +1018,30 @@ impl CassandraEngine { job: &RestoreJob, item_count: i64, table_size_bytes: i64, + ) -> Result { + self.activate_restore_target_row( + &job.account_id, + &job.target_name, + &job.table_id, + job.generation, + item_count, + table_size_bytes, + ) + .await + } + + /// The ACTIVE flip's lightweight transaction, by the target's address and + /// the claim's generation. Shared by the legacy copy + /// ([`Self::activate_restore_target`]) and the store-backed + /// [`extenddb_storage::restore_engine::RestoreEngine::finish_restore`]. + pub(crate) async fn activate_restore_target_row( + &self, + account_id: &str, + target_name: &str, + table_id: &str, + generation: u64, + item_count: i64, + table_size_bytes: i64, ) -> Result { let query = format!( "UPDATE {}.tables SET table_status = 'ACTIVE', status_transition_at = NULL, \ @@ -960,10 +1058,10 @@ impl CassandraEngine { cdrs_tokio::query_values!( item_count, table_size_bytes, - job.account_id.as_str(), - job.target_name.as_str(), - job.table_id.as_str(), - crate::backup_catalog::generation_column(job.generation) + account_id, + target_name, + table_id, + crate::backup_catalog::generation_column(generation) ), ) .await?; @@ -1323,6 +1421,9 @@ impl CassandraEngine { source_table_arn, restore_date_time_ms, owner_generation: generation, + kind: extenddb_storage::backup_catalog::RestoreKind::Legacy, + store: None, + spool_host: None, }; let table_id = columns.table_id.clone(); @@ -1401,6 +1502,218 @@ impl CassandraEngine { }; Ok((description, job, heartbeat)) } + + /// Claim the target a registration created for its own removal: CREATING + /// to DELETING and the record FAILED with `reason`, one lightweight + /// transaction conditioned on the table id, on CREATING, on the + /// registrant's generation and on RUNNING (the write + /// [`Self::abandon_restore_target`] makes). `false` when the record is no + /// longer this registration's: the registration ran no heartbeat during + /// the target's DDL, a reconciler whose stale window the DDL outlasted + /// took the record over, and the target is the successor's to load, so + /// the registrant removes nothing. The state a crash between the claim + /// and the drop leaves (DELETING with its registration and a FAILED + /// record) is the one `remove_restore_target` and the FAILED-target duty + /// resume from. + async fn claim_registered_target_for_removal( + &self, + account_id: &str, + target_name: &str, + table_id: &str, + generation: u64, + reason: &str, + ) -> Result { + let query = format!( + "UPDATE {}.tables SET table_status = 'DELETING', restore_status = 'FAILED', \ + restore_failure_reason = ?, restore_completed_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND table_status = 'CREATING' AND restore_owner_generation = ? \ + AND restore_status = 'RUNNING'", + self.catalog_keyspace() + ); + let result = crate::cassandra_util::query_lwt( + &self.session, + &query, + cdrs_tokio::query_values!( + reason, + account_id, + target_name, + table_id, + crate::backup_catalog::generation_column(generation) + ), + ) + .await?; + crate::cassandra_util::lwt_applied(&result) + } + + /// The error of a registration whose record another generation took over + /// while its target's DDL ran: the target is left to the record's owner. + fn registration_claim_lost(target_name: &str, backup_arn: &str) -> StorageError { + StorageError::Internal(format!( + "restore of table {target_name} from {backup_arn}: the registration lost its \ + record to a takeover during the target's creation (the record is not RUNNING \ + under the registration's generation); the target is left to the record's owner, \ + which finishes the restore" + )) + } + + /// Register a store-backed (manifest) restore target and return the + /// CREATING description and the claim's generation. The manifest analogue + /// of [`Self::register_restore_target`]: it reads the catalog-index backup + /// row (`store_kind` set) rather than the in-catalog one, writes the job + /// record with `restore_kind = 'manifest'` and the store columns, and + /// starts no self-managed heartbeat (the engine heartbeats through the + /// claim). The ordering against DeleteBackup is the same `active_restores` + /// lightweight transaction on the backup's partition, which + /// [`crate::backup_catalog`]'s `start_backup_delete` conflicts with. Two + /// concurrent registrations of one backup are ordered by that transaction + /// too: the in-use pre-read (`ensure_backup_not_restoring`) is not atomic + /// with it, so both may pass the pre-read and create their targets, but + /// its `active_restores = null` condition admits one; the other finds the + /// registered target in its re-read, is `BackupInUse` naming it, and + /// removes its own target, so one target is left. With no heartbeat + /// during the DDL, a registration whose DDL outlasts the stale window can + /// lose its record to a reconciler's takeover; its removal of the target + /// on a refusal is therefore conditioned on its own generation + /// ([`Self::claim_registered_target_for_removal`]) and removes nothing + /// then, answering the lost claim instead of the refusal. + pub(crate) async fn register_manifest_restore_target( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result<(TableDescription, u64), StorageError> { + let backup_arn = registration.backup.backup_arn.clone(); + // The name is checked first (unfenced), so a restore into a CREATING + // restore target's name is refused as the name (TableBeingRestored), + // not as the backup, as on the other backends and the legacy path. + if let Some(source_backup_arn) = self + .creating_restore_target_under_name(account_id, &input.table_name) + .await? + { + return Err(StorageError::TableBeingRestored { + table_name: input.table_name.clone(), + backup_arn: source_backup_arn, + }); + } + let backup = self + .load_available_index_backup(account_id, &backup_arn) + .await?; + self.ensure_backup_not_restoring(&backup).await?; + + let target_name = input.table_name.clone(); + // The claim is the row itself: a fresh generation, written by the + // IF NOT EXISTS insert, fences every later write of this restore. + let generation = new_generation(); + #[allow(clippy::cast_possible_truncation)] + let restore_date_time_ms = (registration.restore_date_time * 1_000.0).round() as i64; + let columns = RestoreTargetColumns { + table_id: uuid::Uuid::new_v4().to_string(), + source_backup_arn: backup_arn.clone(), + source_table_arn: registration.source_table_arn.clone(), + restore_date_time_ms, + owner_generation: generation, + kind: extenddb_storage::backup_catalog::RestoreKind::Manifest, + store: Some(registration.store.clone()), + spool_host: registration.spool_host.clone(), + }; + let table_id = columns.table_id.clone(); + + // Step 1: the target row (IF NOT EXISTS reserves the name), its + // indexes and its data tables, exactly as `input` describes. + let description = match self + .create_table_for_restore_manifest(account_id, input, &columns) + .await + { + Ok(description) => description, + Err(StorageError::TableAlreadyExists(name)) => { + return Err(StorageError::TableAlreadyExists(name)); + } + Err(error) => { + if let Err(cleanup) = self + .drop_restore_target(account_id, &target_name, &table_id, &backup_arn) + .await + { + tracing::warn!( + "Restore of table {target_name}: removing the partly created \ + target failed ({cleanup}); the reconciler removes it" + ); + } + return Err(error); + } + }; + + // Step 2: register on the backup, fenced on AVAILABLE and on no other + // registration. A refusal caused only by stale registrations is + // retried once they are cleared, a bounded number of times. A test + // may hold here, between the target's creation and the fence, to + // move the backup to DELETING. + crate::test_hooks::before_restore_registration(&backup_arn).await; + let mut registered = Ok(false); + for _ in 0..REGISTRATION_ATTEMPTS { + registered = self + .register_restore(account_id, &backup_arn, &table_id, &target_name) + .await; + match registered { + Ok(true) => break, + Ok(false) => { + if let Err(refusal) = self + .registration_refusal_index(account_id, &backup_arn) + .await + { + registered = Err(refusal); + break; + } + } + Err(_) => break, + } + } + if !matches!(registered, Ok(true)) { + // The removal is fenced on this registration's generation: a + // record a reconciler took over during the DDL is the + // successor's, and so is the target. + let refusal = registration_exhausted(registered, &backup_arn); + let owned = match self + .claim_registered_target_for_removal( + account_id, + &target_name, + &table_id, + generation, + &format!("the registration was refused: {refusal}"), + ) + .await + { + Ok(owned) => owned, + Err(error) => { + tracing::warn!( + "Restore of table {target_name}: claiming the unregistered target \ + for removal failed ({error}); the reconciler removes it" + ); + return Err(refusal); + } + }; + if !owned { + tracing::warn!( + "Restore of table {target_name} from {backup_arn}: the registration's \ + record was taken over during the target's creation; the target is left \ + to its new owner" + ); + return Err(Self::registration_claim_lost(&target_name, &backup_arn)); + } + if let Err(cleanup) = self + .drop_restore_target(account_id, &target_name, &table_id, &backup_arn) + .await + { + tracing::warn!( + "Restore of table {target_name}: removing the unregistered target \ + failed ({cleanup}); the reconciler removes it" + ); + } + return Err(refusal); + } + + Ok((description, generation)) + } } impl LegacyBackupReader for CassandraEngine { diff --git a/crates/storage-cassandra/src/create_table.rs b/crates/storage-cassandra/src/create_table.rs index 72d436181..6758e71e8 100644 --- a/crates/storage-cassandra/src/create_table.rs +++ b/crates/storage-cassandra/src/create_table.rs @@ -9,6 +9,7 @@ use extenddb_core::types::{ ProvisionedThroughputDescription, RestoreSummary, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::{RestoreKind, StoreIdentity}; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; @@ -123,6 +124,25 @@ impl CassandraEngine { Ok((description, heartbeat)) } + /// Create a store-backed (manifest) restore target: CREATING, carrying + /// the `restore_kind = 'manifest'` job record and the store columns in + /// the one `INSERT ... IF NOT EXISTS`. No self-managed [`JobHeartbeat`] + /// is started: the engine owns the manifest restore's heartbeat through + /// the [`extenddb_storage::backup_catalog::JobClaim`] this registration + /// returns. The target's DDL that follows the insert is bounded by a + /// few schema statements, inside the engine's registration timeout. + pub(crate) async fn create_table_for_restore_manifest( + &self, + account_id: &str, + input: CreateTableInput, + restore: &RestoreTargetColumns, + ) -> Result { + let (description, _heartbeat) = self + .create_table_impl_inner(account_id, input, Some(restore)) + .await?; + Ok(description) + } + async fn create_table_impl_inner( &self, account_id: &str, @@ -267,8 +287,8 @@ impl CassandraEngine { ]; let mut placeholders = vec!["?"; values.len()]; if let Some(r) = restore { - // The job record: RUNNING under the claim's generation, kind - // legacy, the heartbeat stamped on the coordinator's clock. + // The job record: RUNNING under the claim's generation, the + // restore kind, the heartbeat stamped on the coordinator's clock. columns.push_str( ", restore_source_backup_arn, restore_source_table_arn, restore_date_time, \ restore_status, restore_kind, restore_owner_generation, restore_heartbeat_at", @@ -278,11 +298,29 @@ impl CassandraEngine { Value::from(r.source_table_arn.as_str()), Value::from(r.restore_date_time_ms), Value::from("RUNNING"), - Value::from("legacy"), + Value::from(r.kind.as_str()), Value::from(crate::backup_catalog::generation_column(r.owner_generation)), ]); placeholders.extend(["?"; 6]); placeholders.push("toTimestamp(now())"); + // A store-backed restore records the store it reads and the + // claimant's spool, so the reconciler can resume it. + if let Some(store) = &r.store { + columns.push_str( + ", restore_store_kind, restore_store_root, restore_store_id, \ + restore_spool_host", + ); + values.extend([ + Value::from(store.store_kind.as_str()), + Value::from(store.store_root.as_str()), + Value::from(store.store_id.as_str()), + match r.spool_host.as_deref() { + Some(host) => Value::from(host), + None => Value::NotSet, + }, + ]); + placeholders.extend(["?"; 4]); + } } let placeholders = placeholders.join(", "); let insert_table_cql = format!( @@ -336,9 +374,11 @@ impl CassandraEngine { } } - // A restore target's heartbeat runs from here, before the DDL below. - // An error return from this function drops it, which stops it. - let heartbeat = restore.map(|r| { + // A legacy restore target's heartbeat runs from here, before the DDL + // below, and is dropped (stopped) on an error return. A store-backed + // (manifest) restore target has no self-managed heartbeat: the engine + // heartbeats it through the claim the registration returns. + let heartbeat = restore.filter(|r| r.kind == RestoreKind::Legacy).map(|r| { self.start_restore_heartbeat( account_id, &input.table_name, @@ -674,4 +714,16 @@ pub(crate) struct RestoreTargetColumns { /// The claim's fencing token (`tables.restore_owner_generation`): every /// write the copy makes to the target row is conditioned on it. pub(crate) owner_generation: u64, + /// `legacy` for an in-catalog restore (whose copy reads `backup_items`), + /// `manifest` for a store-backed one (whose copy the engine streams from + /// the store). A legacy target carries a self-managed heartbeat; a + /// manifest target is heartbeated by the engine's claim. + pub(crate) kind: RestoreKind, + /// The store a manifest restore reads its backup from; `None` on a legacy + /// restore, which reads the catalog. + pub(crate) store: Option, + /// `:` of the instance downloading the backup into + /// its spool; `None` for a filesystem store (read in place) and on a + /// legacy restore. + pub(crate) spool_host: Option, } diff --git a/crates/storage-cassandra/src/data/put_get_item.rs b/crates/storage-cassandra/src/data/put_get_item.rs index b74db89dc..c626383ef 100644 --- a/crates/storage-cassandra/src/data/put_get_item.rs +++ b/crates/storage-cassandra/src/data/put_get_item.rs @@ -80,6 +80,103 @@ impl CassandraEngine { .map(|_| ()) } + /// The write a store-backed restore makes for one backup item, at the + /// write timestamp `stamp` (microseconds since the epoch, the domain of + /// `version`): the base row and every GSI and LSI row of the item in one + /// logged batch carrying `USING TIMESTAMP stamp`, with no lightweight + /// transaction and no read before it. A CREATING restore target has no + /// writer but its restore job, whose writes are idempotent upserts of the + /// backup's items, so the put path's version fence has nothing to + /// serialize against; what the timestamp buys is the fence the restore + /// needs instead (`restore_engine`'s module doc): a job reads `stamp` + /// before it checks its claim, so a batch it lands late, after a takeover, + /// carries a timestamp below every write and delete the successor made, + /// and Cassandra keeps the later cell. `version` is set to `stamp`, so a + /// later ordinary write's `next_row_version` is above it and its index + /// rows resolve in order (`data::version`). `indexes` are the target's, + /// read once per data file by the caller, each with its propagation delay + /// zeroed so every GSI is written in the batch as LSIs always are. + pub(crate) async fn put_restored_item_at( + &self, + key_info: &TableKeyInfo, + indexes: &[super::index::IndexMeta], + item: &Item, + stamp: i64, + ) -> Result<(), StorageError> { + let data_keyspace = self.account_keyspace(&key_info.account_id); + let ddb_table = data_table_name(&key_info.table_id); + let pk_text = composite_pk_to_text(item, &key_info.key_schema)?; + let item_text = serde_json::to_value(item) + .map_err(|e| StorageError::Internal(e.to_string()))? + .to_string(); + if !indexes.is_empty() { + extenddb_core::validation::validate_index_keys( + item, + &super::index::index_key_refs(indexes), + &key_info.attribute_definitions, + ) + .map_err(|e| StorageError::Validation(e.to_string()))?; + } + let (cql, values) = if let Some((sk_name, sk_type)) = + sk_info(&key_info.key_schema, &key_info.attribute_definitions) + { + let sk_value = item + .get(sk_name) + .ok_or_else(|| StorageError::Internal("missing sort key".to_owned()))?; + let sk = parse_sk(sk_value, sk_type)?; + let sk_col = sk_column(sk_type); + ( + format!( + "UPDATE {data_keyspace}.{ddb_table} SET item_data = ?, version = ? \ + WHERE pk = ? AND {sk_col} = ?" + ), + vec![ + item_text.into(), + stamp.into(), + cdrs_tokio::types::value::Value::from(pk_text.as_str()), + super::index::sk_to_value(&sk), + ], + ) + } else { + ( + format!( + "UPDATE {data_keyspace}.{ddb_table} SET item_data = ?, version = ? WHERE pk = ?" + ), + vec![ + item_text.into(), + stamp.into(), + cdrs_tokio::types::value::Value::from(pk_text.as_str()), + ], + ) + }; + // The batch carries the timestamp, so the index statements pass none + // of their own (Cassandra refuses a batch with both). + let mut batch = BatchQueryBuilder::new() + .with_consistency(Consistency::LocalQuorum) + .with_timestamp(stamp) + .add_query(cql, cdrs_tokio::query::QueryValues::SimpleValues(values)); + if !indexes.is_empty() { + super::index::sync_indexes_at( + &mut batch, + &data_keyspace, + &key_info.key_schema, + &key_info.attribute_definitions, + indexes, + None, + Some(item), + 0, + None, + )?; + } + let built = batch + .build() + .map_err(|error| StorageError::Internal(error.to_string()))?; + self.session.batch(built).await.map_err(|error| { + StorageError::Internal(format!("put_restored_item_at batch: {error}")) + })?; + Ok(()) + } + #[allow(clippy::too_many_arguments)] async fn put_item_with_retry( &self, diff --git a/crates/storage-cassandra/src/lib.rs b/crates/storage-cassandra/src/lib.rs index bff16c1d4..15c04a742 100644 --- a/crates/storage-cassandra/src/lib.rs +++ b/crates/storage-cassandra/src/lib.rs @@ -32,6 +32,7 @@ pub mod migrations; pub mod operations; mod password_hash_store; pub(crate) mod propagation_hold; +mod restore_engine; mod snapshot_engine; mod stream_engine; pub mod stream_util; diff --git a/crates/storage-cassandra/src/migrations/mod.rs b/crates/storage-cassandra/src/migrations/mod.rs index 7015b82f6..8818dcd3c 100644 --- a/crates/storage-cassandra/src/migrations/mod.rs +++ b/crates/storage-cassandra/src/migrations/mod.rs @@ -150,6 +150,11 @@ pub fn migration(id: MigrationId) -> Migration { m0006_backup_catalog_index::SOURCE, )], }, + MigrationId::RestoreProgress => Migration::NotApplicable( + "a store-backed restore records its progress on the target's catalog tables row \ + (restore_file_index, restore_items_applied, which migration 6 adds); the \ + restore_progress table is PostgreSQL's data database's", + ), } } diff --git a/crates/storage-cassandra/src/migrations/tests.rs b/crates/storage-cassandra/src/migrations/tests.rs index 037323f88..fd75a11b4 100644 --- a/crates/storage-cassandra/src/migrations/tests.rs +++ b/crates/storage-cassandra/src/migrations/tests.rs @@ -69,6 +69,7 @@ fn every_migration_records_a_row_in_the_catalog_and_names_its_stores() { for id in [ MigrationId::SortKeyDefinitions, MigrationId::IndexTableBaseKey, + MigrationId::RestoreProgress, ] { assert!(matches!(migration(id), Migration::NotApplicable(_)), "{id}"); } diff --git a/crates/storage-cassandra/src/restore_engine.rs b/crates/storage-cassandra/src/restore_engine.rs new file mode 100644 index 000000000..0cc25ad5f --- /dev/null +++ b/crates/storage-cassandra/src/restore_engine.rs @@ -0,0 +1,599 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on Cassandra: the +//! [`RestoreEngine`] methods the engine's restore job calls. +//! +//! `bulk_load` writes one verified data file's items into the CREATING +//! target, each item's base row and every GSI and LSI row in one logged +//! batch ([`CassandraEngine::put_restored_item_at`], no lightweight +//! transaction: a CREATING restore target has no writer but its job), +//! `RESTORE_WRITE_CONCURRENCY` in flight, in batches of +//! at most [`RESTORE_BATCH_ITEMS`] items or [`RESTORE_BATCH_BYTES`] of +//! serialized item text (a batch closes at the first item that takes its +//! `item_data` text past the bound, so the batch term of a restore's memory +//! is bounded in bytes, not only in items). Before each batch's writes it +//! reads the coordinator's clock (the batch's write timestamp) and then +//! runs a lightweight transaction on the record (`confirm_restore_claim`: +//! the heartbeat refreshed `IF` the record is RUNNING under the claim's +//! generation) and ends the file as a lost claim when it is not applied, so +//! the check is linearized with a takeover's own transaction rather than a +//! quorum read beside it. The batch's writes into the account keyspace +//! carry no condition of their own (a Cassandra write cannot be conditioned +//! on another partition); the fence is the timestamp: every write of the +//! batch lands `USING TIMESTAMP` the reading taken before the check, so a +//! writer that stalls between the check and its writes and lands them after +//! a takeover lands them below every write and delete the successor made +//! (the successor's batches carry readings taken after the takeover, its +//! `finish_restore` and the deletes through the ACTIVE table are later +//! still), and Cassandra keeps the later cell or tombstone; the stalled +//! batch changes nothing the successor wrote or removed. The stamp is read +//! from the ExtendDB host's clock (`data::version::now_micros`, the clock +//! the item versions already use), while the writes it must lose to are +//! lightweight transactions and the deletes through the ACTIVE table, +//! whose write times the Cassandra coordinator sets from its own clock, so +//! the rule needs the ExtendDB hosts' clocks and the Cassandra nodes' clocks +//! to agree, as the version scheme already needs the hosts' to. After +//! each batch it records the file's progress +//! with a lightweight transaction conditioned on the restore generation and +//! on the record being RUNNING; `[applied] = false` there is the same lost +//! claim, and the load stops without touching the target again; an unknown +//! outcome of that transaction is decided by a LOCAL_SERIAL re-read of the +//! record (applied when it carries the values written, one retry when it +//! carries the earlier ones, the lost claim when it is not RUNNING under the +//! generation; `write_restore_progress`). The writes are idempotent upserts, +//! so a re-applied batch after a resume rewrites the same base and index rows +//! at a later timestamp (the base row's `version` is the timestamp, so a +//! later ordinary write's version is above it and its index rows resolve in +//! order, ADR-0027), and a key the backup +//! holds twice collapses into one row: a shortfall `count_restore_target` +//! finds against the manifest's total is the duplicate (the engine's count +//! check, 5.3 step 6). Progress is recorded after the batch's writes, so a +//! crash inside the first batch leaves items with no progress; the engine +//! resumes such a record with the first file re-applied (`reapplying`) when +//! the target holds no more than the backup's total, and fails it +//! `target_not_empty` above the total. An empty file records its progress +//! too (`items_applied = start_at`), so a resume does not re-read it. +//! +//! `finish_restore` is [`CassandraEngine::activate_restore_target_row`]'s +//! lightweight transaction (DONE, the table and its indexes ACTIVE, the +//! counts, conditioned on CREATING, RUNNING and the generation). The index +//! rows are created ACTIVE with the table, so Cassandra never returns +//! [`Finished::Pending`]. + +use cdrs_tokio::types::IntoRustByName; +use extenddb_core::types::{Item, TableKeyInfo}; +use extenddb_storage::backup_catalog::{ + JobClaim, JobStatus, RESTORE_CASSANDRA_BEFORE_PROGRESS, RestoreTargetKey, WriteOutcome, + restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +pub(crate) use extenddb_storage::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; +use futures::{StreamExt, TryStreamExt}; + +use crate::CassandraEngine; +use crate::backup_engine::RESTORE_WRITE_CONCURRENCY; +use crate::data::ddl::{data_table_name, index_table_name}; + +/// The error of a `bulk_load` whose progress write found the record no longer +/// this claim's: the reconciler took it over while the file was loading, and +/// whoever holds the record now owns the target. Nothing further is touched. +fn claim_lost(table_name: &str) -> StorageError { + StorageError::Internal(format!( + "restore of table {table_name}: the claim on the target was lost while a data file \ + loaded (the record is not RUNNING under this generation); the record's owner \ + finishes it" + )) +} + +impl CassandraEngine { + /// The key shape of a CREATING restore target, read from its `tables` + /// row by hand (the put path refuses a CREATING table through + /// `fetch_table_key_info`, so the restore builds the info itself, as the + /// legacy copy does). `TableNotFound` when `key` is not a CREATING + /// restore target with this table id, as the other backends answer. The + /// put path reads the target's indexes from the catalog by `table_id`, + /// so the routing fields are left empty. + async fn restore_target_key_info( + &self, + key: &RestoreTargetKey, + ) -> Result { + let query = format!( + "SELECT table_id, table_status, restore_source_backup_arn, key_schema, \ + attribute_definitions FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore target key info", + ) + .await?; + let row = rows.first().filter(|row| { + let id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("table_status").ok().flatten(); + let source: Option = + row.get_by_name("restore_source_backup_arn").ok().flatten(); + id.as_deref() == Some(&key.table_id) + && status.as_deref() == Some("CREATING") + && source.is_some() + }); + let row = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + let ks_text: String = + crate::cassandra_util::get_column(row, "key_schema", "restore target key info")?; + let ad_text: String = crate::cassandra_util::get_column( + row, + "attribute_definitions", + "restore target key info", + )?; + let key_schema: Vec = + serde_json::from_str(&ks_text) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?; + let attribute_definitions = serde_json::from_str(&ad_text) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?; + Ok(TableKeyInfo { + table_name: key.table_name.clone(), + account_id: key.account_id.clone(), + table_id: key.table_id.clone(), + base_key_schema: key_schema.clone(), + key_schema, + attribute_definitions, + has_lsi: false, + global_secondary_indexes: Vec::new(), + local_secondary_indexes: Vec::new(), + stream_specification: None, + vector_indexes: Vec::new(), + }) + } + + /// The claim's liveness before each batch's writes: a lightweight + /// transaction on the record (`restore_heartbeat_at` refreshed, `IF` the + /// table id, the generation and RUNNING), linearized on the target's + /// partition with the takeover's own transaction, so the check and a + /// `claim_restore_dead` order wholly one before the other where a + /// quorum read could run beside one. `Err` (a lost claim) when the + /// transaction is not applied: the reconciler took the record over, and + /// whoever holds it owns the target, so nothing further is written. An + /// unknown outcome is decided by a LOCAL_SERIAL re-read of the record. + /// The writes themselves carry no condition on the record: a batch landed + /// after a takeover is fenced by its write timestamp, read before this + /// check (the module doc). + async fn confirm_restore_claim( + &self, + key: &RestoreTargetKey, + generation: u64, + ) -> Result<(), StorageError> { + let query = format!( + "UPDATE {}.tables SET restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = 'RUNNING'", + self.catalog_keyspace() + ); + let outcome = crate::backup_catalog::conditional_write( + &self.session, + &query, + cdrs_tokio::query::QueryValues::SimpleValues(vec![ + cdrs_tokio::types::value::Value::from(key.account_id.as_str()), + cdrs_tokio::types::value::Value::from(key.table_name.as_str()), + cdrs_tokio::types::value::Value::from(key.table_id.as_str()), + cdrs_tokio::types::value::Value::from(crate::backup_catalog::generation_column( + generation, + )), + ]), + ) + .await?; + match outcome { + WriteOutcome::Matched => return Ok(()), + WriteOutcome::NotMatched => return Err(claim_lost(&key.table_name)), + WriteOutcome::Unknown => {} + } + let query = format!( + "SELECT table_id, restore_status, restore_owner_generation FROM {}.tables \ + WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore claim", + ) + .await?; + let live = rows.first().is_some_and(|row| { + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let owner: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + table_id.as_deref() == Some(key.table_id.as_str()) + && status.as_deref() == Some(JobStatus::Running.as_str()) + && owner == Some(crate::backup_catalog::generation_column(generation)) + }); + if live { + Ok(()) + } else { + Err(claim_lost(&key.table_name)) + } + } + + /// Write one batch of items into the target, each item's base and index + /// rows in its own logged batch at one write timestamp, + /// `RESTORE_WRITE_CONCURRENCY` in flight, after confirming the claim. + /// The timestamp is read before the claim check (the module doc's + /// fence). Drains `batch`. + async fn flush_restore_batch( + &self, + key: &RestoreTargetKey, + generation: u64, + key_info: &TableKeyInfo, + indexes: &[crate::data::index::IndexMeta], + batch: &mut Vec, + ) -> Result { + let stamp = crate::data::version::now_micros(); + self.confirm_restore_claim(key, generation).await?; + crate::test_hooks::after_restore_claim_check(&key.table_id).await; + let count = u64::try_from(batch.len()).unwrap_or(u64::MAX); + let mut writes = futures::stream::iter(batch.drain(..)) + .map(|item| async move { + self.put_restored_item_at(key_info, indexes, &item, stamp) + .await + }) + .buffer_unordered(RESTORE_WRITE_CONCURRENCY); + while let Some(result) = writes.next().await { + result?; + } + Ok(count) + } + + /// The progress lightweight transaction alone: `restore_file_index` and + /// `restore_items_applied`, conditioned on the table id, the restore + /// generation and the record being RUNNING. A test may make it answer + /// `Unknown`, with or without the transaction run. + async fn progress_write( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result { + let fault = crate::test_hooks::progress_write_fault(&key.account_id); + if fault == Some(crate::test_hooks::ProgressWriteFault::Lost) { + return Ok(WriteOutcome::Unknown); + } + let query = format!( + "UPDATE {}.tables SET restore_file_index = ?, restore_items_applied = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = 'RUNNING'", + self.catalog_keyspace() + ); + let outcome = crate::backup_catalog::conditional_write( + &self.session, + &query, + cdrs_tokio::query::QueryValues::SimpleValues(vec![ + cdrs_tokio::types::value::Value::from( + i32::try_from(file_index).unwrap_or(i32::MAX), + ), + cdrs_tokio::types::value::Value::from( + i64::try_from(items_applied).unwrap_or(i64::MAX), + ), + cdrs_tokio::types::value::Value::from(key.account_id.as_str()), + cdrs_tokio::types::value::Value::from(key.table_name.as_str()), + cdrs_tokio::types::value::Value::from(key.table_id.as_str()), + cdrs_tokio::types::value::Value::from(crate::backup_catalog::generation_column( + generation, + )), + ]), + ) + .await?; + if fault == Some(crate::test_hooks::ProgressWriteFault::Landed) { + return Ok(WriteOutcome::Unknown); + } + Ok(outcome) + } + + /// What the record says after a progress write whose outcome is unknown, + /// read at LOCAL_SERIAL so a transaction that did commit is seen: RUNNING + /// under `generation` with this table id and carrying the values just + /// written is `Applied`; RUNNING under `generation` with other values + /// (the earlier progress: nothing but this job writes the record's + /// progress while it owns it) is `Pending`; anything else is `Lost`. + async fn progress_write_state( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result { + let query = format!( + "SELECT table_id, restore_status, restore_owner_generation, restore_file_index, \ + restore_items_applied FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore progress", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(ProgressWriteState::Lost); + }; + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let owner: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + let owned = table_id.as_deref() == Some(key.table_id.as_str()) + && status.as_deref() == Some(JobStatus::Running.as_str()) + && owner == Some(crate::backup_catalog::generation_column(generation)); + if !owned { + return Ok(ProgressWriteState::Lost); + } + let recorded_file: Option = row.get_by_name("restore_file_index").ok().flatten(); + let recorded_items: Option = row.get_by_name("restore_items_applied").ok().flatten(); + let written = ( + Some(i32::try_from(file_index).unwrap_or(i32::MAX)), + Some(i64::try_from(items_applied).unwrap_or(i64::MAX)), + ); + if (recorded_file, recorded_items) == written { + Ok(ProgressWriteState::Applied) + } else { + Ok(ProgressWriteState::Pending) + } + } + + /// Record the file's progress (`restore_file_index`, the running total + /// `restore_items_applied`) with a lightweight transaction conditioned on + /// the restore generation and on the record being RUNNING. `[applied] = + /// false` is a lost claim. An unknown outcome (a write timeout or failure, + /// a lost connection: the Paxos round may still have committed) is + /// decided by re-reading the record at LOCAL_SERIAL: RUNNING under this + /// generation with the values just written is applied and the file goes + /// on; RUNNING under this generation with the earlier values is one + /// retry of the transaction, whose own unknown outcome is re-read the + /// same way, and a second unknown outcome that the record does not carry + /// fails the file (the error names it; the engine then fails the restore + /// and removes the target); anything else is the lost claim. + async fn write_restore_progress( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result<(), StorageError> { + restore_pause_point(RESTORE_CASSANDRA_BEFORE_PROGRESS).await; + let mut retried = false; + loop { + match self + .progress_write(key, generation, file_index, items_applied) + .await? + { + WriteOutcome::Matched => return Ok(()), + WriteOutcome::NotMatched => return Err(claim_lost(&key.table_name)), + WriteOutcome::Unknown => {} + } + match self + .progress_write_state(key, generation, file_index, items_applied) + .await? + { + ProgressWriteState::Applied => return Ok(()), + ProgressWriteState::Lost => return Err(claim_lost(&key.table_name)), + ProgressWriteState::Pending if !retried => { + tracing::warn!( + "restore of table {}: the progress write for data file {file_index} \ + had no known outcome and the record does not carry it; written once \ + more", + key.table_name + ); + retried = true; + } + ProgressWriteState::Pending => { + return Err(StorageError::Internal(format!( + "restore of table {}: the progress write for data file {file_index} \ + had no known outcome twice and the record still carries the earlier \ + progress; the restore fails and its target is removed", + key.table_name + ))); + } + } + } + } +} + +/// What the record says about a progress write whose outcome was unknown. +#[derive(Debug, PartialEq, Eq)] +enum ProgressWriteState { + /// RUNNING under the claim's generation, the written values on it. + Applied, + /// RUNNING under the claim's generation, the earlier values on it. + Pending, + /// Not RUNNING under the claim's generation, or gone. + Lost, +} + +impl RestoreEngine for CassandraEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let generation = claim.generation(); + let key_info = self.restore_target_key_info(target).await?; + // The target's indexes, once per file (a CREATING target's do not + // change), every GSI synchronous as the put path's + // `IndexWrites::AllSynchronous` makes them. + let mut indexes = crate::data::index::fetch_indexes_for_table( + &target.table_id, + &self.session, + &self.catalog_keyspace(), + ) + .await?; + for index in &mut indexes { + index.propagation_delay_ms = Some(0); + } + let batch_size = crate::test_hooks::restore_batch_size(&target.account_id).max(1); + let mut items = items; + let mut batch: Vec = Vec::with_capacity(batch_size.min(1024)); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + let mut wrote_progress = false; + while let Some(item) = items.try_next().await? { + // The item's `item_data` text, the unit the byte bound is + // measured in (the put path serializes it again for the row). + let text_len = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))? + .len(); + batch_bytes = batch_bytes.saturating_add(text_len); + batch.push(item); + if batch.len() >= batch_size || batch_bytes >= RESTORE_BATCH_BYTES { + applied += self + .flush_restore_batch(target, generation, &key_info, &indexes, &mut batch) + .await?; + batch_bytes = 0; + self.write_restore_progress( + target, + generation, + file.file_index, + file.start_at.saturating_add(applied), + ) + .await?; + wrote_progress = true; + } + } + if !batch.is_empty() { + applied += self + .flush_restore_batch(target, generation, &key_info, &indexes, &mut batch) + .await?; + self.write_restore_progress( + target, + generation, + file.file_index, + file.start_at.saturating_add(applied), + ) + .await?; + wrote_progress = true; + } + // An empty file still records that it is complete, so a resume + // does not re-read it. + if !wrote_progress { + self.write_restore_progress(target, generation, file.file_index, file.start_at) + .await?; + } + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let activated = self + .activate_restore_target_row( + &target.account_id, + &target.table_name, + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + ) + .await?; + if activated { + Ok(Finished::Active) + } else { + Ok(Finished::NotMatched(format!( + "the restore record of {} is not RUNNING under generation {}, or the table \ + is not a CREATING restore target", + target.table_name, + claim.generation() + ))) + } + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // A full COUNT(*) over the target's data table: a single scan of + // a CREATING table, run once per restore (the cost is stated in + // the limits doc). The id names the data table; the account + // keyspace scopes it to the target's account, and the catalog is + // read first so a key that is not a CREATING restore target with + // this id is `TableNotFound`, as on the other backends. + self.restore_target_key_info(key).await?; + let account_keyspace = self.account_keyspace(&key.account_id); + let data_table = data_table_name(&key.table_id); + let query = format!("SELECT COUNT(*) AS c FROM {account_keyspace}.{data_table}"); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(), + "count restore target", + ) + .await?; + let count: i64 = rows + .first() + .map(|row| crate::cassandra_util::get_column(row, "c", "count restore target")) + .transpose()? + .unwrap_or(0); + Ok(u64::try_from(count).unwrap_or(0)) + }) + } + + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The catalog first, so a bare id reads no other account's table; + // then the data table's row in `system_schema.tables` and one per + // GSI and LSI the catalog names (the registration LWT lands + // before the data DDL, which creates the data table and then + // each index table in its own statement, so a crash between them + // leaves a target missing an index table). Unquoted CQL + // identifiers are stored lowercased. + self.restore_target_key_info(key).await?; + let account_keyspace = self.account_keyspace(&key.account_id).to_lowercase(); + let indexes = crate::data::index::fetch_indexes_for_table( + &key.table_id, + &self.session, + &self.catalog_keyspace(), + ) + .await?; + let mut tables = vec![data_table_name(&key.table_id).to_lowercase()]; + tables.extend( + indexes + .iter() + .map(|index| index_table_name(&index.index_id).to_lowercase()), + ); + for table in tables { + let rows = crate::cassandra_util::query_rows::( + &self.session, + "SELECT table_name FROM system_schema.tables WHERE keyspace_name = ? AND \ + table_name = ?", + cdrs_tokio::query_values!(account_keyspace.clone(), table), + "restore target data table", + ) + .await?; + if rows.is_empty() { + return Ok(false); + } + } + Ok(true) + }) + } +} diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs index 0e70abcb8..42c48cf57 100644 --- a/crates/storage-cassandra/src/test_hooks.rs +++ b/crates/storage-cassandra/src/test_hooks.rs @@ -30,20 +30,45 @@ //! a test can hold two takers of one stale row there and see one win; //! * a pause in `claim_dead_backup` between its `LOCAL_SERIAL` read and its //! takeover LWT, so a test can land the owner's beat in that gap and see -//! the LWT's `heartbeat_at = ` condition refuse the takeover. +//! the LWT's `heartbeat_at = ` condition refuse the takeover; +//! * a pause in a store-backed registration between the target's creation +//! and its `active_restores` lightweight transaction, so a test can move +//! the backup to DELETING in that gap and see the transaction's own +//! `backup_status = 'AVAILABLE'` condition refuse the registration; +//! * a pause in `claim_restore_dead` between its read and its takeover +//! lightweight transaction, and the batch size of a store-backed +//! `bulk_load`; +//! * a fault on the progress lightweight transaction of a store-backed +//! `bulk_load`, so a test can see the re-read that decides an unknown +//! outcome; +//! * a pause in a store-backed `bulk_load` between a batch's claim check and +//! its writes, so a test can let a successor write and delete in that gap +//! and see the stalled batch's timestamp lose to the successor's state. use std::time::Duration; use extenddb_storage::error::StorageError; +/// How a test makes a progress lightweight transaction answer +/// `WriteOutcome::Unknown`: with the transaction run first (`Landed`: the +/// answer was lost on its way back) or not run at all (`Lost`: the proposal +/// never reached a replica). Each fault is taken by one write, in order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ProgressWriteFault { + Landed, + Lost, +} + #[cfg(feature = "test-hooks")] mod armed { - use std::collections::HashMap; + use std::collections::{HashMap, VecDeque}; use std::sync::{Arc, LazyLock, Mutex}; use std::time::Duration; use tokio::sync::Notify; + use super::ProgressWriteFault; + /// A pause a test arms at one point of one operation: the operation /// signals `reached` when it gets there and waits for `released`. pub struct Pause { @@ -84,6 +109,11 @@ mod armed { pub(super) lose_maintenance_insert_answer: bool, pub(super) before_maintenance_takeover: Option>, pub(super) before_takeover_write: HashMap>, + pub(super) before_restore_takeover: HashMap>, + pub(super) before_restore_registration: HashMap>, + pub(super) restore_batch_size: HashMap, + pub(super) progress_write_faults: HashMap>, + pub(super) after_restore_claim_check: HashMap>, } pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default); @@ -154,6 +184,39 @@ pub fn pause_before_abandoned_fence(backup_arn: &str) -> std::sync::Arc { pause } +/// Arm a pause in the next `claim_restore_dead` of the restore target +/// `table_id`, between its serial read (and staleness check) and its +/// takeover lightweight transaction. A test releases it after writing a +/// fresh heartbeat, to prove the takeover's `restore_heartbeat_at = ` +/// condition refuses a beat that lands in between. Taken once, then disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_before_restore_takeover(table_id: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .before_restore_takeover + .insert(table_id.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + +/// Arm a pause in the next store-backed registration of a restore of +/// `backup_arn`, between the target's creation (its row and DDL) and the +/// `active_restores` lightweight transaction on the backup. A test moves the +/// backup to DELETING in the gap to prove the transaction's own +/// `backup_status = 'AVAILABLE'` condition, not the unfenced pre-read, +/// refuses the registration. Taken once, then disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_before_restore_registration(backup_arn: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .before_restore_registration + .insert(backup_arn.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + /// Arm a pause in the next `claim_dead_backup` of `backup_arn`, between its /// read of the row and its takeover write. The pause is taken once and then /// disarmed. @@ -168,6 +231,59 @@ pub fn pause_before_takeover_write(backup_arn: &str) -> std::sync::Arc { pause } +/// Make the store-backed `bulk_load` of jobs of `account_id` flush and record +/// progress every `size` items instead of the production batch size, so a +/// test can cross a batch boundary without writing ten thousand items. +#[cfg(feature = "test-hooks")] +pub fn set_restore_batch_size(account_id: &str, size: usize) { + armed::with_hooks(|hooks| { + hooks.restore_batch_size.insert(account_id.to_owned(), size); + }); +} + +/// Arm a pause in the next batch of a store-backed `bulk_load` into the +/// restore target `table_id`, between the batch's claim check (its +/// lightweight transaction on the record, run after the batch's write +/// timestamp was read) and the batch's writes. A test takes the record over +/// in the gap, lets the successor write and delete, then releases the pause +/// to prove the stalled batch's writes, at the earlier timestamp, change +/// nothing the successor wrote or removed. Taken once, then disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_after_restore_claim_check(table_id: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .after_restore_claim_check + .insert(table_id.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + +/// Hold here if a test armed a pause after the claim check of a batch into +/// the restore target `table_id`. +pub(crate) async fn after_restore_claim_check(table_id: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = armed::with_hooks(|hooks| hooks.after_restore_claim_check.remove(table_id)) + { + pause.hold().await; + } + let _ = table_id; +} + +/// Make the next progress lightweight transactions of the store-backed +/// `bulk_load` of jobs of `account_id` answer `WriteOutcome::Unknown`, one +/// fault per write in the order given: [`ProgressWriteFault::Landed`] runs +/// the transaction and drops its answer, [`ProgressWriteFault::Lost`] drops +/// the transaction. Writes after the last fault run as usual. +#[cfg(feature = "test-hooks")] +pub fn fail_progress_writes(account_id: &str, faults: &[ProgressWriteFault]) { + armed::with_hooks(|hooks| { + hooks + .progress_write_faults + .insert(account_id.to_owned(), faults.iter().copied().collect()); + }); +} + /// The period the jobs of `account_id` heartbeat at. pub(crate) fn heartbeat_interval(account_id: &str) -> Duration { #[cfg(feature = "test-hooks")] @@ -223,6 +339,28 @@ pub(crate) async fn before_abandoned_fence(backup_arn: &str) { let _ = backup_arn; } +/// Hold here if a test armed a pause for the takeover of restore target +/// `table_id`. +pub(crate) async fn before_restore_takeover(table_id: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = armed::with_hooks(|hooks| hooks.before_restore_takeover.remove(table_id)) { + pause.hold().await; + } + let _ = table_id; +} + +/// Hold here if a test armed a pause for the store-backed registration of a +/// restore of `backup_arn`. +pub(crate) async fn before_restore_registration(backup_arn: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = + armed::with_hooks(|hooks| hooks.before_restore_registration.remove(backup_arn)) + { + pause.hold().await; + } + let _ = backup_arn; +} + /// Hold here if a test armed a pause for the takeover of `backup_arn`. pub(crate) async fn before_takeover_write(backup_arn: &str) { #[cfg(feature = "test-hooks")] @@ -278,3 +416,36 @@ pub(crate) fn maintenance_insert_answer_lost() -> bool { #[cfg(not(feature = "test-hooks"))] false } + +/// The number of items the store-backed `bulk_load` of jobs of `account_id` +/// applies before each progress write (the byte bound closes a batch +/// earlier). The production default is `RESTORE_BATCH_ITEMS`; a test may set +/// a small size with [`set_restore_batch_size`]. +pub(crate) fn restore_batch_size(account_id: &str) -> usize { + #[cfg(feature = "test-hooks")] + if let Some(size) = armed::with_hooks(|hooks| hooks.restore_batch_size.get(account_id).copied()) + { + return size; + } + let _ = account_id; + crate::restore_engine::RESTORE_BATCH_ITEMS +} + +/// The fault the next progress lightweight transaction of a job of +/// `account_id` takes, if a test armed one. +pub(crate) fn progress_write_fault(account_id: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + armed::with_hooks(|hooks| { + hooks + .progress_write_faults + .get_mut(account_id) + .and_then(std::collections::VecDeque::pop_front) + }) + } + #[cfg(not(feature = "test-hooks"))] + { + let _ = account_id; + None + } +} diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index 067d49b61..28d1dd3a4 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -20,14 +20,15 @@ use extenddb_core::types::{ TimeToLiveStatus, }; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, backup_in_use_message, }; use extenddb_storage::error::StorageError; use extenddb_storage::migrations::{CatalogStatus, MigrationError, MigrationId}; use extenddb_storage::{ - BackupCatalog, DataEngine, JobPermit, LegacyBackupReader, RestoreRequest, - RestoreTableOverrides, TableEngine, + BackupCatalog, BulkLoadFile, DataEngine, Finished, JobPermit, LegacyBackupReader, + RestoreEngine, RestoreRequest, RestoreTableOverrides, TableEngine, }; use extenddb_storage_cassandra::CassandraEngine; use futures::{StreamExt, TryStreamExt}; @@ -4164,12 +4165,18 @@ async fn test_a_migration_6_that_fails_in_its_backfill_stays_pending_and_resumes let (engine, account_id, _drop_keyspaces) = isolated_engine("mf", true).await; let catalog = engine.catalog_keyspace(); let prefix = prefix_of(&engine); + // Behind at migration 5 with migration 6 pending (and 7, which does not + // apply on Cassandra and is recorded after it). let behind_at_5 = |status: CatalogStatus| match status { CatalogStatus::Behind { version, pending } => { version == Some(MigrationId::DropContinuousBackups.number()) && pending .iter() - .all(|p| p.id == MigrationId::BackupCatalogIndex.number()) + .any(|p| p.id == MigrationId::BackupCatalogIndex.number()) + && pending.iter().all(|p| { + p.id == MigrationId::BackupCatalogIndex.number() + || p.id == MigrationId::RestoreProgress.number() + }) } _ => false, }; @@ -5810,6 +5817,1967 @@ async fn test_the_registrations_sweep_removes_stale_entries_and_keeps_a_live_one ); } +// ── Store-backed (manifest) restore ────────────────────────────────────────── + +/// An AVAILABLE catalog-index backup row (`store_kind` set) for `table`, the +/// row a store-backed restore registers against. +async fn available_index_backup( + engine: &CassandraEngine, + account_id: &str, + table: &str, + name: &str, + store: &StoreIdentity, +) -> BackupRow { + let row = index_row(account_id, table, name, store); + let claim = engine.claim_backup(&row).await.expect("claim backup"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("make the backup AVAILABLE"), + WriteOutcome::Matched + ); + row +} + +/// A hash-only `CreateTableInput` for a restore target. +fn manifest_target_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + } +} + +/// The registration a store-backed restore of `row` writes, with `spool_host`. +fn manifest_registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:us-east-1:{}:table/{}", + row.account_id, row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: UNOWNED_GENERATION, + } +} + +/// A hash-keyed item with key `k{i:05}`. +fn id_item(i: usize) -> Item { + let mut it = BTreeMap::new(); + it.insert("id".to_owned(), AttributeValue::S(format!("k{i:05}"))); + it.insert("v".to_owned(), AttributeValue::S(format!("val{i}"))); + it +} + +/// The items `lo..hi` as a `bulk_load` stream. +fn item_stream( + lo: usize, + hi: usize, +) -> futures::stream::BoxStream<'static, Result> { + let items: Vec = (lo..hi).map(id_item).collect(); + futures::stream::iter(items.into_iter().map(Ok)).boxed() +} + +/// `claim_restore` registers a CREATING target with a RUNNING `manifest` +/// record carrying the store, the spool host and the claim's generation, the +/// `active_restores` entry on the backup, and a `RestoreSummary` on the +/// description; the claim heartbeats and reads the record. +#[tokio::test] +async fn test_claim_restore_registers_a_manifest_target_with_its_record() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let registration = manifest_registration(&row, Some("host-a:/var/spool")); + + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("RManifest"), + ®istration, + ) + .await + .expect("the registration creates the target and the record"); + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert!(summary.restore_in_progress); + assert_ne!(claim.generation(), UNOWNED_GENERATION); + + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let record = engine + .read_restore_job(&key) + .await + .expect("read") + .expect("the record exists"); + assert_eq!(record.kind, RestoreKind::Manifest); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, claim.generation()); + assert_eq!(record.store.as_ref(), Some(&store)); + assert_eq!(record.spool_host.as_deref(), Some("host-a:/var/spool")); + assert_eq!(record.source_backup_arn, row.backup_arn); + + // The backup is in use by the target, and the heartbeat confirms and the + // read returns RUNNING. + assert_eq!( + engine.backup_in_use(&row.key()).await.unwrap().as_deref(), + Some("RManifest") + ); + assert_eq!( + claim.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + assert_eq!(claim.read().await.expect("read").status, JobStatus::Running); +} + +/// The two orderings against `start_backup_delete` on the backup's +/// `backup_index` row: a registration that lands first leaves the backup in +/// use; a DeleteBackup that lands first leaves the registration with no +/// backup to read. +#[tokio::test] +async fn test_claim_restore_and_start_backup_delete_order_on_the_backup_index_row() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + + // Registration wins: DeleteBackup finds the backup in use by the target. + let store = unique_store(); + let won = available_index_backup(&engine, &account_id, "Music", "a", &store).await; + let (desc, _claim) = engine + .claim_restore( + &account_id, + manifest_target_input("WonTarget"), + &manifest_registration(&won, None), + ) + .await + .expect("register"); + assert_eq!(desc.table_status, TableStatus::Creating); + assert_eq!( + engine.start_backup_delete(&won.key()).await.unwrap(), + DeleteStart::InUse("WonTarget".to_owned()) + ); + + // DeleteBackup wins: the registration finds the backup gone. + let lost = available_index_backup(&engine, &account_id, "Music", "b", &store).await; + assert_eq!( + engine.start_backup_delete(&lost.key()).await.unwrap(), + DeleteStart::Deleting + ); + let refused = engine + .claim_restore( + &account_id, + manifest_target_input("LostTarget"), + &manifest_registration(&lost, None), + ) + .await + .err() + .expect("a registration against a DELETING backup is Backup not found"); + assert!( + matches!(refused, StorageError::Validation(ref m) if m.contains("Backup not found")), + "{refused}" + ); + // The lost registration left no target behind. + assert!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "LostTarget".to_owned() + } + ) + .await + .is_err() + ); +} + +/// A second restore of a backup whose target is CREATING is `BackupInUse`, +/// and a restore into a taken name is `TableAlreadyExists`. +#[tokio::test] +async fn test_a_second_restore_is_in_use_and_a_taken_name_is_refused() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + + let (_desc, _claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FirstTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("the first restore registers"); + + // A second restore of the same backup, into a different name, is in use. + let in_use = engine + .claim_restore( + &account_id, + manifest_target_input("SecondTarget"), + &manifest_registration(&row, None), + ) + .await + .err() + .expect("the backup is in use by the first restore"); + // The service's text names the backup, not the target. + assert!( + matches!(in_use, StorageError::BackupInUse(ref m) if *m == backup_in_use_message(&row.backup_arn)), + "{in_use}" + ); + + // A restore into an existing table's name is TableAlreadyExists. + create_active_table( + &engine, + &account_id, + CreateTableInput { + table_name: "Taken".to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + }, + ) + .await; + let other = available_index_backup(&engine, &account_id, "Music", "other", &store).await; + let taken = engine + .claim_restore( + &account_id, + manifest_target_input("Taken"), + &manifest_registration(&other, None), + ) + .await + .err() + .expect("the name is taken"); + assert!( + matches!(taken, StorageError::TableAlreadyExists(ref name) if name == "Taken"), + "{taken}" + ); +} + +/// `claim_restore_dead`: a live owner, and a legacy record, are not taken +/// over; a stale manifest record is, under a fresh generation, with the +/// claimant's `restore_spool_host` written and `restore_heartbeat_at` +/// refreshed to the coordinator's clock in the same lightweight transaction, +/// so a second taker right after the first (before the successor's first +/// beat) finds the record live and is refused while the first taker's beat +/// confirms; the old claim's writes are then fenced. +#[tokio::test] +async fn test_claim_restore_dead_takes_over_a_stale_manifest_record() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("Takeover"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let window = Duration::from_secs(1); + let before = engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .heartbeat_at; + assert!( + engine + .claim_restore_dead(&key, Duration::from_secs(300), Some("other:/s")) + .await + .expect("claim") + .is_none(), + "a live owner within a long window is not taken over" + ); + + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, window, Some("successor:/spool")) + .await + .expect("claim") + .expect("the window passed: taken over"); + assert_ne!(successor.generation(), owner.generation()); + let after = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(after.owner_generation, successor.generation()); + assert_eq!(after.status, JobStatus::Running); + assert_eq!( + after.spool_host.as_deref(), + Some("successor:/spool"), + "the takeover wrote the claimant's spool host" + ); + // A second taker right after the first, before the successor's first + // beat: its serial read finds the fresh heartbeat and it is refused; the + // first taker's beat confirms. + let second = engine + .claim_restore_dead(&key, window, Some("second:/s")) + .await + .expect("a second takeover attempt"); + assert!( + second.is_none(), + "a second taker right after the first is refused: {:?}", + second.as_ref().map(|c| c.generation()) + ); + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed, + "the first taker keeps the record" + ); + assert!( + after.heartbeat_at > before + 1.0, + "the takeover refreshed the heartbeat: {} was {before}", + after.heartbeat_at + ); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("epoch") + .as_secs_f64(); + assert!( + (now - after.heartbeat_at).abs() < 5.0, + "the refreshed heartbeat is the coordinator's now: {} vs {now}", + after.heartbeat_at + ); + assert_eq!(owner.heartbeat().await.expect("heartbeat"), Heartbeat::Lost); + assert_eq!( + owner + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched, + "the old generation's write is fenced" + ); + + // The successor heartbeats again: a third taker finds it live. + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + assert!( + engine + .claim_restore_dead(&key, window, Some("third:/s")) + .await + .expect("claim") + .is_none(), + "the successor's heartbeat is fresh" + ); + + // A legacy record is not a manifest restore's to take, whatever its age. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_kind = 'legacy' \ + WHERE account_id = ? AND table_name = ? IF table_id = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str() + ), + ) + .await; + tokio::time::sleep(Duration::from_millis(1_300)).await; + assert!( + engine + .claim_restore_dead(&key, window, Some("x:/s")) + .await + .expect("claim") + .is_none(), + "a legacy record is not taken over by the manifest takeover" + ); +} + +/// `claim_restore_dead` refuses the takeover when a heartbeat lands between +/// its serial read and its lightweight transaction: the LWT conditions on the +/// heartbeat it read, which the beat has moved. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_claim_restore_dead_refuses_a_beat_between_the_read_and_the_write() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("BeatRace"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + tokio::time::sleep(Duration::from_millis(1_300)).await; + + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_takeover(&key.table_id); + let takeover = { + let engine = engine.clone(); + let key = key.clone(); + tokio::spawn(async move { + engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + }) + }; + pause.reached().await; + // A beat lands after the takeover read its stale heartbeat. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? IF table_id = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str() + ), + ) + .await; + pause.release(); + assert!( + takeover + .await + .expect("takeover task") + .expect("claim") + .is_none(), + "the beat between the read and the write refuses the takeover" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + record.owner_generation, + owner.generation(), + "the record is still the original owner's" + ); + assert_eq!( + record.spool_host.as_deref(), + Some("owner:/s"), + "the refused takeover did not rewrite the spool host" + ); +} + +/// A record whose backup does not list its target (the owner died between +/// the target's creation and the `active_restores` registration): the +/// takeover re-applies the entry, so the backup is in use for the resumed +/// load and `start_backup_delete` is refused. +#[tokio::test] +async fn test_claim_restore_dead_reapplies_a_missing_active_restores_entry() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("Unlisted"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The crash's shape: the target and its RUNNING record exist, the + // backup's entry does not. + apply( + &engine, + &format!( + "DELETE active_restores[?] FROM {}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + key.table_id.as_str(), + account_id.as_str(), + row.backup_arn.as_str() + ), + ) + .await; + assert!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .is_empty(), + "the entry is gone" + ); + drop(owner); + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + .expect("claim") + .expect("the stale record is taken over"); + let entries = registrations(&engine, &account_id, &row.backup_arn).await; + assert_eq!( + entries.get(&key.table_id).map(String::as_str), + Some("Unlisted"), + "the takeover re-applied the backup's entry for the target: {entries:?}" + ); + assert_eq!( + engine.start_backup_delete(&row.key()).await.unwrap(), + DeleteStart::InUse("Unlisted".to_owned()), + "the backup is in use for the resumed load" + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.unwrap().as_deref(), + Some("Unlisted") + ); + drop(successor); +} + +/// A DeleteBackup that lands between a registration's unfenced pre-read and +/// its `active_restores` lightweight transaction: the transaction's own +/// `backup_status = 'AVAILABLE'` condition refuses the registration, which +/// answers `Backup not found` and leaves no target and no entry. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_delete_between_the_registrations_pre_read_and_its_fence_leaves_no_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "raced", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let registration = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, None); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + manifest_target_input("RacedTarget"), + ®istration, + ) + .await + }) + }; + // The registration has read the backup AVAILABLE and created the target; + // the delete lands before its fence. + pause.reached().await; + assert_eq!( + engine.start_backup_delete(&row.key()).await.unwrap(), + DeleteStart::Deleting, + "nothing holds the backup before the registration's fence" + ); + pause.release(); + let refused = registration + .await + .expect("registration task") + .err() + .expect("the fence refuses the registration"); + assert!( + matches!(refused, StorageError::Validation(ref m) if m.contains("Backup not found")), + "{refused}" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "RacedTarget".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the refused registration left no target" + ); + assert!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .is_empty(), + "no entry on the DELETING backup" + ); +} + +/// Two concurrent registrations of one backup leave one target: the in-use +/// pre-read is not atomic with the `active_restores` lightweight +/// transaction, so both can pass it (the first held before its fence while +/// the second runs whole), but the fence's `active_restores = null` admits +/// one; the other finds the registered target in its re-read, is +/// `BackupInUse` naming it, and removes its own target. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_two_concurrent_registrations_of_one_backup_leave_one_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "twice", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let first = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, None); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + manifest_target_input("FirstOfTwo"), + ®istration, + ) + .await + }) + }; + // The first registration has passed the pre-read (no restore registered) + // and created its target; the second runs whole before the first's fence. + pause.reached().await; + let (second_desc, second_claim) = engine + .claim_restore( + &account_id, + manifest_target_input("SecondOfTwo"), + &manifest_registration(&row, None), + ) + .await + .expect("the second registration, whole before the first's fence"); + pause.release(); + let refused = first + .await + .expect("registration task") + .err() + .expect("the first's fence finds the second registered"); + assert!( + matches!(refused, StorageError::BackupInUse(ref m) if *m == backup_in_use_message(&row.backup_arn)), + "{refused}" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "FirstOfTwo".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the refused registration left no target" + ); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(second_desc.table_id.clone(), "SecondOfTwo".to_owned())]), + "one entry on the backup, the second's" + ); + assert_eq!( + engine + .backup_in_use(&row.key()) + .await + .expect("in use") + .as_deref(), + Some("SecondOfTwo") + ); + assert_eq!( + second_claim.heartbeat().await.expect("beat"), + Heartbeat::Confirmed, + "the second registration's claim is live" + ); +} + +/// A registration whose record a reconciler took over during its target's +/// DDL (no heartbeat runs there, so a DDL that outlasts the stale window +/// leaves the record dead) removes nothing: its `active_restores` transaction +/// finds the successor's entry, and its removal of the target is conditioned +/// on its own generation, which the takeover moved. The target stays, the +/// record is the successor's, and the registration answers an error naming +/// the lost claim rather than the refusal. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "slowddl", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let registration = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, Some("registrant:/s")); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + composite_target_input("SlowDdl"), + ®istration, + ) + .await + }) + }; + // The target's row and DDL exist; the registration is held before its + // `active_restores` transaction, as a DDL past the stale window would + // hold it. The record's heartbeat is the row insert's. + pause.reached().await; + let desc = describe(&engine, &account_id, "SlowDdl").await; + assert_eq!(desc.table_status, TableStatus::Creating); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let registrant_generation = engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .owner_generation; + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + .expect("claim") + .expect("the registration's record is stale and taken over"); + assert_ne!(successor.generation(), registrant_generation); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(key.table_id.clone(), "SlowDdl".to_owned())]), + "the takeover re-applied the target's entry" + ); + + pause.release(); + let outcome = registration.await.expect("registration task"); + + // The successor's target, record and registration stand. + let target = engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "SlowDdl".to_owned(), + }, + ) + .await; + assert!( + matches!(&target, Ok(desc) if desc.table_status == TableStatus::Creating), + "the registrant's refusal path left the successor's target in place: {target:?}" + ); + let lost = outcome + .err() + .expect("the registration does not return a claim on a record another owns"); + assert!( + matches!(&lost, StorageError::Internal(m) if m.contains("lost its record to a takeover")), + "the registration names the lost claim, not the refusal: {lost}" + ); + let record = engine + .read_restore_job(&key) + .await + .expect("read") + .expect("the record stands"); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, successor.generation()); + assert_eq!(record.spool_host.as_deref(), Some("successor:/s")); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(key.table_id.clone(), "SlowDdl".to_owned())]), + "the successor's entry on the backup stands" + ); + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + // The successor loads into the target the registrant left: its data and + // index tables are there. + assert_eq!( + engine + .bulk_load( + &key, + &*successor, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + futures::stream::iter((0..6).map(composite_item).map(Ok)).boxed(), + ) + .await + .expect("the successor's load"), + 6 + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 6); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 3 + ); +} + +/// A composite-key restore target with a GSI and an LSI. +fn composite_target_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![ks("id", KeyType::Hash), ks("sk", KeyType::Range)], + attribute_definitions: vec![ + attr("id", ScalarAttributeType::S), + attr("sk", ScalarAttributeType::N), + attr("g", ScalarAttributeType::S), + attr("l", ScalarAttributeType::S), + ], + billing_mode: Some(BillingMode::PayPerRequest), + global_secondary_indexes: Some(vec![GsiInput { + index_name: "gix".to_owned(), + key_schema: vec![ks("g", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::KeysOnly, + non_key_attributes: None, + }, + provisioned_throughput: None, + }]), + local_secondary_indexes: Some(vec![LsiInput { + index_name: "lix".to_owned(), + key_schema: vec![ks("id", KeyType::Hash), ks("l", KeyType::Range)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + }]), + ..Default::default() + } +} + +/// Item `i` of the composite fixture: `id` one of five, `sk` the index, +/// every second item carrying the GSI's key `g`, every third the LSI's `l`. +fn composite_item(i: usize) -> Item { + let mut it = BTreeMap::new(); + it.insert("id".to_owned(), AttributeValue::S(format!("p{}", i % 5))); + it.insert("sk".to_owned(), AttributeValue::N(i.to_string())); + if i.is_multiple_of(2) { + it.insert("g".to_owned(), AttributeValue::S(format!("g{}", i % 4))); + } + if i.is_multiple_of(3) { + it.insert("l".to_owned(), AttributeValue::S(format!("l{i}"))); + } + it +} + +/// The rows of the target's secondary index `index_name`. +async fn index_rows( + engine: &CassandraEngine, + account_id: &str, + table_id: &str, + index_name: &str, +) -> i64 { + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT index_id FROM {}.indexes WHERE table_id = ? AND index_name = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!(table_id, index_name), + "index id", + ) + .await + .expect("read the index"); + let index_id: String = rows + .first() + .expect("the index exists") + .get_by_name("index_id") + .ok() + .flatten() + .expect("index_id"); + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT COUNT(*) AS c FROM {}.{}", + engine.account_keyspace(account_id), + extenddb_storage_cassandra::data::ddl::index_table_name(&index_id) + ), + cdrs_tokio::query_values!(), + "index rows", + ) + .await + .expect("count the index rows"); + rows.first() + .expect("a count") + .get_by_name("c") + .ok() + .flatten() + .expect("c") +} + +/// Per item of the composite fixture (keyed by its `sk`), the base row's +/// `version` when `index_name` is `None`, otherwise the write timestamp of +/// the item's row in that secondary index (`WRITETIME(item_data)`). +#[cfg(feature = "test-hooks")] +async fn stamps_by_sk( + engine: &CassandraEngine, + account_id: &str, + table_id: &str, + index_name: Option<&str>, +) -> BTreeMap { + use cdrs_tokio::types::IntoRustByName; + let keyspace = engine.account_keyspace(account_id); + let cql = match index_name { + None => format!( + "SELECT item_data, version AS stamp FROM {keyspace}.items_{}", + table_id.replace('-', "_") + ), + Some(index_name) => { + let rows = + extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT index_id FROM {}.indexes WHERE table_id = ? AND index_name = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!(table_id, index_name), + "index id", + ) + .await + .expect("read the index"); + let index_id: String = rows + .first() + .expect("the index exists") + .get_by_name("index_id") + .ok() + .flatten() + .expect("index_id"); + format!( + "SELECT item_data, WRITETIME(item_data) AS stamp FROM {keyspace}.{}", + extenddb_storage_cassandra::data::ddl::index_table_name(&index_id) + ) + } + }; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &cql, + cdrs_tokio::query_values!(), + "stamps", + ) + .await + .expect("read the stamps"); + rows.into_iter() + .map(|row| { + let json: String = row.get_r_by_name("item_data").expect("item_data"); + let image: Item = serde_json::from_str(&json).expect("an item image"); + let Some(AttributeValue::N(sk)) = image.get("sk") else { + panic!("no sk in {image:?}"); + }; + let stamp: i64 = row.get_r_by_name("stamp").expect("a stamp"); + (sk.clone(), stamp) + }) + .collect() +} + +/// Every GSI and LSI row of the composite target carries its base row's +/// `version` as its write timestamp, the stamp the write path gives the +/// index rows of each version it stores (ADR-0027); returns the base +/// versions. +#[cfg(feature = "test-hooks")] +async fn assert_index_rows_carry_the_base_versions( + engine: &CassandraEngine, + account_id: &str, + table_id: &str, +) -> BTreeMap { + let versions = stamps_by_sk(engine, account_id, table_id, None).await; + for index_name in ["gix", "lix"] { + let stamps = stamps_by_sk(engine, account_id, table_id, Some(index_name)).await; + assert!(!stamps.is_empty(), "{index_name} has rows"); + for (sk, stamp) in &stamps { + assert_eq!( + Some(stamp), + versions.get(sk), + "the {index_name} row of item {sk} carries its base row's version" + ); + } + } + versions +} + +/// `bulk_load` into a composite-key target with a GSI and an LSI: a file +/// resumed after its first three items (`start_at: 3`) applies exactly the +/// rest and records `start_at` plus the items applied as the file's total; +/// the GSI and LSI rows are written with the base rows; an empty file, and a +/// file resumed at its end, record their progress. +#[tokio::test] +async fn test_bulk_load_resumes_mid_file_with_its_index_rows_and_records_an_empty_file() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + composite_target_input("Composite"), + &manifest_registration(&row, None), + ) + .await + .expect("register a composite target"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // File 0 resumed after three of its ten items. + let rest: Vec = (3..10).map(composite_item).collect(); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 3, + reapplying: false, + }, + futures::stream::iter(rest.into_iter().map(Ok)).boxed(), + ) + .await + .expect("the rest of file 0"); + assert_eq!(applied, 7, "exactly the rest"); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(0)); + assert_eq!( + record.items_applied, + Some(10), + "the running total is start_at plus the items applied" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 3, + "the GSI rows of items 4, 6 and 8" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + 3, + "the LSI rows of items 3, 6 and 9" + ); + + // An empty file records its progress as complete. + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(Vec::>::new()).boxed(), + ) + .await + .expect("an empty file"); + assert_eq!(applied, 0); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(1), Some(0)), + "an empty file's progress is recorded" + ); + // A file resumed at its end: `start_at` is the recorded total. + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 2, + start_at: 4, + reapplying: false, + }, + futures::stream::iter(Vec::>::new()).boxed(), + ) + .await + .expect("a file resumed at its end"); + assert_eq!(applied, 0); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(2), Some(4)) + ); + + // Another account's key, and the target once ACTIVE, name no CREATING + // restore target: `count_restore_target` is `TableNotFound`. + let foreign = RestoreTargetKey { + account_id: "111111111111".to_owned(), + ..key.clone() + }; + assert!(matches!( + engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert_eq!( + engine + .finish_restore(&key, &*claim, 7, 4_096) + .await + .expect("finish"), + Finished::Active + ); + assert!( + matches!( + engine.count_restore_target(&key).await, + Err(StorageError::TableNotFound(_)) + ), + "an ACTIVE table is not a CREATING restore target" + ); +} + +/// A file resumed mid-way (`start_at: 3`) with more than one batch left: +/// the progress written after each batch inside the loop is `start_at` plus +/// the items applied so far, as the write after the tail is, so a second +/// crash between two of the resumed file's batches resumes from a running +/// total that counts the items the first run applied. Three items per batch, +/// seven items: the record reads 6 after the first batch, 9 after the +/// second, 10 at the end. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_mid_file_resume_records_start_at_in_every_batch_progress() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("ResumedBatches"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream reads the record when asked for the first item of the + // second and of the third batch: by then the previous batch's progress + // is written. + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let seen_by_stream = std::sync::Arc::clone(&seen); + let reader = engine.clone(); + let read_key = key.clone(); + let rest: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(3usize, move |i| { + let reader = reader.clone(); + let key = read_key.clone(); + let seen = std::sync::Arc::clone(&seen_by_stream); + async move { + if i >= 10 { + return None; + } + if i == 6 || i == 9 { + let record = reader.read_restore_job(&key).await.unwrap().unwrap(); + seen.lock() + .expect("seen") + .push((record.file_index, record.items_applied)); + } + Some((Ok(id_item(i)), i + 1)) + } + })); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 3, + reapplying: false, + }, + rest, + ) + .await + .expect("the rest of file 0"); + assert_eq!(applied, 7, "exactly the rest"); + assert_eq!( + *seen.lock().expect("seen"), + vec![(Some(0), Some(6)), (Some(0), Some(9))], + "each batch's progress is start_at plus the items applied so far" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(10)), + "the tail's progress is the file's total" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); +} + +/// A progress lightweight transaction whose outcome is unknown (a write +/// timeout, a lost connection) does not fail the file: the record is re-read +/// at LOCAL_SERIAL. One whose transaction committed (the answer alone was +/// lost) is found applied and the file goes on; one whose transaction never +/// ran is found carrying the earlier progress and is written once more; a +/// second unknown outcome the record does not carry fails the file with an +/// error naming it, and the batches before it stand with their progress. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_an_unknown_progress_write_outcome_is_decided_by_a_re_read() { + use extenddb_storage_cassandra::test_hooks::ProgressWriteFault; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("UnknownProgress"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + + // File 0, three batches: the first batch's write lands but its answer is + // lost (the re-read finds the values written), the second's never runs + // (the re-read finds the first batch's values; written once more), the + // tail's runs as usual. The file completes with every item and its + // progress. + extenddb_storage_cassandra::test_hooks::fail_progress_writes( + &account_id, + &[ProgressWriteFault::Landed, ProgressWriteFault::Lost], + ); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 7), + ) + .await + .expect("an unknown outcome the record decides does not fail the file"); + assert_eq!(applied, 7); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(7)), + "the file's progress is recorded" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); + + // File 1: the first batch's write is lost twice over; the second unknown + // outcome fails the file. The batch's items landed (the writes are + // idempotent upserts a resume rewrites) and the record carries file 0's + // progress, so a resume re-applies file 1 from its start. + extenddb_storage_cassandra::test_hooks::fail_progress_writes( + &account_id, + &[ProgressWriteFault::Lost, ProgressWriteFault::Lost], + ); + let failed = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + item_stream(7, 14), + ) + .await + .expect_err("two unknown outcomes the record does not carry fail the file"); + assert!( + matches!(&failed, StorageError::Internal(m) if m.contains("had no known outcome twice")), + "{failed}" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(7)), + "file 1 recorded no progress" + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 10, + "the first batch of file 1 landed before its progress write" + ); + assert_eq!( + claim.heartbeat().await.expect("beat"), + Heartbeat::Confirmed, + "the claim is this job's throughout" + ); +} + +/// A batch closes at the first item whose text takes it past +/// `RESTORE_BATCH_BYTES`, long before the count cap: a file of 100 items of +/// about 350 KB each (35 MB of text against the 32 MiB bound) is written in +/// two batches, the first's progress recorded before the stream hands over +/// its last item. +#[tokio::test] +async fn test_a_batch_closes_at_the_byte_bound_before_the_count_cap() { + if crate::helpers::skip_without_cassandra() { + return; + } + const N: usize = 100; + const TEXT: usize = 350_000; + const { + assert!(N < extenddb_storage::restore_engine::RESTORE_BATCH_ITEMS); + // 100 items cross the byte bound once. + assert!( + N * TEXT > extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + && (N - 2) * TEXT < 2 * extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + ); + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("BigItems"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream reads the record when asked for the last item: by then the + // byte bound has closed the first batch and recorded its progress. + let seen = std::sync::Arc::new(std::sync::Mutex::new(None)); + let seen_by_stream = std::sync::Arc::clone(&seen); + let reader = engine.clone(); + let read_key = key.clone(); + let big: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let reader = reader.clone(); + let key = read_key.clone(); + let seen = std::sync::Arc::clone(&seen_by_stream); + async move { + if i >= N { + return None; + } + if i == N - 1 { + let record = reader.read_restore_job(&key).await.unwrap().unwrap(); + *seen.lock().expect("seen") = Some((record.file_index, record.items_applied)); + } + let mut item = id_item(i); + item.insert("v".to_owned(), AttributeValue::S("x".repeat(TEXT))); + Some((Ok(item), i + 1)) + } + })); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + big, + ) + .await + .expect("bulk_load"); + assert_eq!(applied as usize, N); + let (file_index, items_applied) = seen + .lock() + .expect("seen") + .expect("the stream read the record at its last item"); + assert_eq!(file_index, Some(0), "the first batch recorded its progress"); + let first_batch = items_applied.expect("the first batch's total") as usize; + assert!( + 0 < first_batch && first_batch < N, + "the first batch closed at the byte bound with {first_batch} of {N} items" + ); + assert!( + first_batch * TEXT + < extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + TEXT + 1_000 * N, + "the batch closed at the first item past the bound: {first_batch} items" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(N as u64)) + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count") as usize, + N + ); +} + +/// A claim lost between two batches of one file (the record's generation +/// moved by a takeover while the load ran): the read before the next batch +/// finds the record no longer this claim's, the file ends as a lost claim, +/// and the second batch's items never land. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_bulk_load_stops_at_a_lost_claim_between_two_batches() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("LostMidFile"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream hands the first batch's three items, then moves the + // record's generation when asked for the first item of the second. + let mover = engine.clone(); + let moved_key = key.clone(); + let generation = i64::try_from(claim.generation()).unwrap(); + let stream: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let mover = mover.clone(); + let key = moved_key.clone(); + async move { + if i >= 7 { + return None; + } + if i == 3 { + apply( + &mover, + &format!( + "UPDATE {}.tables SET restore_owner_generation = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ?", + mover.catalog_keyspace() + ), + cdrs_tokio::query_values!( + generation + 1, + key.account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str(), + generation + ), + ) + .await; + } + Some((Ok(id_item(i)), i + 1)) + } + })); + let out = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream, + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the read before the second batch finds another owner: {out:?}" + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 3, + "the first batch's items alone landed" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(3)), + "the first batch's progress stands" + ); +} + +/// `bulk_load` crosses a batch boundary, recording the file's running total +/// after each batch, and a re-applied batch changes nothing observable: the +/// writes are idempotent upserts of the base rows and of each GSI and LSI row +/// through the ordinary put path (an index row whose key did not change is +/// rewritten by its `INSERT` alone, and each index row carries its base row's +/// `version` as its write timestamp, ADR-0027), so the base count and both +/// index row counts equal those of the first load, after the last batch alone +/// is re-applied (`start_at: 8`, a resume's shape) and after the whole file +/// is, and every index row's stamp follows its base row's version. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_bulk_load_crosses_a_batch_boundary_and_a_reapply_changes_nothing() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + // Four items per batch, so twelve items cross two boundaries. + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 4); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + composite_target_input("BulkTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let composite_items = |range: std::ops::Range| { + futures::stream::iter(range.map(composite_item).map(Ok)).boxed() + }; + // Items 0, 2, 4, 6, 8 and 10 carry the GSI's key; 0, 3, 6 and 9 the LSI's. + let (gsi_rows, lsi_rows) = (6, 4); + + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + composite_items(0..12), + ) + .await + .expect("bulk_load"); + assert_eq!(applied, 12); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(0)); + assert_eq!(record.items_applied, Some(12)); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 12); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + gsi_rows + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + lsi_rows + ); + let loaded = + assert_index_rows_carry_the_base_versions(&engine, &account_id, &key.table_id).await; + assert_eq!(loaded.len(), 12); + + // The last batch re-applied (a resume whose progress write was lost): + // the same base and index rows. + let reapplied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 8, + reapplying: true, + }, + composite_items(8..12), + ) + .await + .expect("bulk_load re-apply of the last batch"); + assert_eq!(reapplied, 4); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 12); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + gsi_rows, + "a re-applied batch keeps every GSI row" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + lsi_rows, + "a re-applied batch keeps every LSI row" + ); + // The re-applied items' base rows moved to a later version and their + // index rows with them; the others kept both. + let reapplied_versions = + assert_index_rows_carry_the_base_versions(&engine, &account_id, &key.table_id).await; + for (sk, version) in &reapplied_versions { + if sk.parse::().expect("an integer sk") >= 8 { + assert!(version > &loaded[sk], "item {sk} was written again"); + } else { + assert_eq!(version, &loaded[sk], "item {sk} was not written again"); + } + } + + // The whole file re-applied: the count stays 12 and the index rows stand. + let reapplied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: true, + }, + composite_items(0..12), + ) + .await + .expect("bulk_load re-apply"); + assert_eq!(reapplied, 12); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 12, + "a re-applied file changes nothing" + ); + assert_eq!( + ( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + index_rows(&engine, &account_id, &key.table_id, "lix").await + ), + (gsi_rows, lsi_rows), + "a re-applied file keeps every index row" + ); + assert_index_rows_carry_the_base_versions(&engine, &account_id, &key.table_id).await; +} + +/// An unordered backup carrying one key in two data files applies both items +/// (the second overwrites the first), so `count_restore_target` is one short +/// of the manifest's total: the shortfall is the duplicate, which the +/// engine's count check turns into a FAILED restore. +#[tokio::test] +async fn test_an_unordered_duplicate_key_is_caught_by_the_count_alone() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("DupTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // File 0: items 0..6. File 1: items 6..12 and item 0 again (the backup + // holds k00000 twice). The manifest's total is 13 items. + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 6), + ) + .await + .expect("file 0"); + let file_one: Vec = (6..12) + .map(id_item) + .chain(std::iter::once(id_item(0))) + .collect(); + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(file_one.into_iter().map(Ok)).boxed(), + ) + .await + .expect("file 1"); + let count = engine.count_restore_target(&key).await.expect("count"); + assert_eq!(count, 12, "the duplicated key collapsed into one row"); + assert!(count < 13, "the count is one short of the manifest's 13"); +} + +/// `bulk_load` stops with a lost-claim error when the record's generation has +/// moved (the reconciler took it over): the progress lightweight transaction +/// matches no RUNNING record under the claim's generation. +#[tokio::test] +async fn test_bulk_load_is_refused_when_the_claim_was_lost() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("LostTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // Another owner bumps the generation. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_owner_generation = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + i64::try_from(claim.generation()).unwrap() + 1, + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str(), + i64::try_from(claim.generation()).unwrap() + ), + ) + .await; + let error = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 3), + ) + .await + .expect_err("the progress write finds another owner"); + assert!( + error.to_string().contains("claim on the target was lost"), + "{error}" + ); +} + +/// `finish_restore` flips the target ACTIVE and marks the record DONE, and a +/// stale generation's `finish_restore` is `NotMatched`. +#[tokio::test] +async fn test_finish_restore_flips_the_target_and_refuses_a_stale_generation() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FinishTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 5), + ) + .await + .expect("bulk_load"); + let count = engine.count_restore_target(&key).await.expect("count"); + assert_eq!(count, 5); + + assert_eq!( + engine + .finish_restore(&key, &*claim, count, 4_096) + .await + .expect("finish"), + Finished::Active + ); + let described = describe(&engine, &account_id, "FinishTarget").await; + assert_eq!(described.table_status, TableStatus::Active); + assert_eq!(described.item_count, 5); + + // A second flip, the record no longer RUNNING, matches nothing. + assert!(matches!( + engine + .finish_restore(&key, &*claim, count, 4_096) + .await + .expect("finish again"), + Finished::NotMatched(_) + )); +} + +/// After a claim writes the record FAILED (leaving `table_status` alone), +/// `remove_restore_target` performs the two-write removal (CREATING to +/// DELETING, then the drop) and the target is gone. +#[tokio::test] +async fn test_remove_restore_target_after_a_failed_write() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FailTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + assert_eq!( + claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("a data file failed".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("fail the record"), + WriteOutcome::Matched + ); + // The FAILED write left the table CREATING; the record carries the reason. + assert_eq!( + describe(&engine, &account_id, "FailTarget") + .await + .table_status, + TableStatus::Creating, + "the FAILED write leaves table_status alone for the removal's two-write step" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some("a data file failed")); + + assert!( + engine.remove_restore_target(&key).await.expect("remove"), + "the two-write removal of a FAILED target succeeds" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "FailTarget".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the target is gone" + ); + assert!( + engine.read_restore_job(&key).await.expect("read").is_none(), + "the record went with the target" + ); +} + /// The liveness test inside the takeover write: with the owner's beat /// landing between `claim_dead_backup`'s `LOCAL_SERIAL` read (which found /// the heartbeat stale) and its LWT, the LWT's `heartbeat_at = ` @@ -5868,3 +7836,523 @@ async fn test_a_beat_between_the_takeovers_read_and_its_write_refuses_the_takeov .expect("taken over once no beat lands in the gap"); assert_ne!(successor.generation(), owner.generation()); } + +// ── The batch's timestamp fence, the first batch resumed without progress, the +// ── readiness check's index tables, the in-use rule of a running restore ─────── + +/// The items of a target's data table (rows with an `item_data`: a delete +/// leaves the partition's static delete stamp behind, so `COUNT(*)` would +/// count a deleted item's partition), counted whatever the table's status +/// (`count_restore_target` refuses a target no longer CREATING). +#[cfg(feature = "test-hooks")] +async fn base_rows(engine: &CassandraEngine, account_id: &str, table_id: &str) -> i64 { + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT COUNT(item_data) AS c FROM {}.items_{}", + engine.account_keyspace(account_id), + table_id.replace('-', "_") + ), + cdrs_tokio::query_values!(), + "base rows", + ) + .await + .expect("count the base rows"); + rows.first() + .expect("a count") + .get_by_name("c") + .ok() + .flatten() + .expect("c") +} + +/// Microseconds since the epoch on this host's clock, the domain of the +/// restore's write timestamps. +#[cfg(feature = "test-hooks")] +fn micros_now() -> i64 { + i64::try_from( + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("after the epoch") + .as_micros(), + ) + .expect("fits") +} + +/// The batch fence (the module doc of `restore_engine`): a writer paused +/// between a batch's claim check and the batch's writes, taken over, whose +/// successor re-applies the batch, flips the target ACTIVE and deletes two +/// of its items, lands its batch when released at the timestamp it read +/// before the check, below every write and delete the successor made: the +/// deleted items stay deleted, their GSI and LSI rows stay gone, and the +/// surviving rows carry the successor's versions, not the stalled writer's. +/// Without the timestamp (the writes stamped when they land) the stalled +/// batch would resurrect the deleted items and their index rows. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 4); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + composite_target_input("Fenced"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let composite_items = |range: std::ops::Range| { + futures::stream::iter(range.map(composite_item).map(Ok)).boxed() + }; + + // The owner's one batch of four, held after its claim check. + let pause = + extenddb_storage_cassandra::test_hooks::pause_after_restore_claim_check(&key.table_id); + let stalled = { + let engine = engine.clone(); + let key = key.clone(); + tokio::spawn(async move { + engine + .bulk_load( + &key, + &*owner, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + composite_items(0..4), + ) + .await + }) + }; + pause.reached().await; + assert_eq!( + base_rows(&engine, &account_id, &key.table_id).await, + 0, + "nothing is written before the check" + ); + + // The owner's heartbeat (the check refreshed it) goes stale; the + // successor takes the record over, writes the batch itself at later + // timestamps, finishes, and two items are deleted through the ACTIVE + // table. + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor_from = micros_now(); + let successor = engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + .expect("claim") + .expect("the stale record is taken over"); + let applied = engine + .bulk_load( + &key, + &*successor, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: true, + }, + composite_items(0..4), + ) + .await + .expect("the successor's load"); + assert_eq!(applied, 4); + assert_eq!( + engine + .finish_restore(&key, &*successor, 4, 1_024) + .await + .expect("finish"), + Finished::Active + ); + let key_info = engine + .table_key_info(&account_id, "Fenced") + .await + .expect("the ACTIVE target's key info"); + // The deletes' GSI tombstones are written in the delete's own batch (no + // propagation worker runs in these tests). + make_gsis_synchronous(&engine, &key.table_id, &["gix"]).await; + // Items 0 and 2 carry the GSI's key `g`; item 0 the LSI's `l` as well. + for i in [0usize, 2] { + let mut k = BTreeMap::new(); + k.insert("id".to_owned(), AttributeValue::S(format!("p{}", i % 5))); + k.insert("sk".to_owned(), AttributeValue::N(i.to_string())); + engine + .delete_item(&key_info, &k, false, None, &ExpressionMaps::default(), None) + .await + .expect("delete through the ACTIVE table"); + } + assert_eq!(base_rows(&engine, &account_id, &key.table_id).await, 2); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 0 + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + 1 + ); + + // The stalled writer lands its batch now. Its progress write then finds + // the record no longer its own (the lost claim); its writes carry the + // timestamp read before its check, so they change nothing. + pause.release(); + let out = stalled.await.expect("the owner's task"); + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the stalled writer's progress write reports the lost claim: {out:?}" + ); + assert_eq!( + base_rows(&engine, &account_id, &key.table_id).await, + 2, + "the deleted items stay deleted" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 0, + "the deleted items' GSI rows stay gone" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + 1, + "the deleted item's LSI row stays gone" + ); + for i in [0usize, 2] { + let mut k = BTreeMap::new(); + k.insert("id".to_owned(), AttributeValue::S(format!("p{}", i % 5))); + k.insert("sk".to_owned(), AttributeValue::N(i.to_string())); + assert!( + engine.get_item(&key_info, &k).await.expect("get").is_none(), + "item {i} was not resurrected" + ); + } + let versions = live_versions_by_sk(&engine, &account_id, &key.table_id).await; + assert_eq!(versions.keys().collect::>(), ["1", "3"]); + for (sk, version) in &versions { + assert!( + *version >= successor_from, + "item {sk} carries the successor's version ({version}), not the stalled writer's \ + (before {successor_from})" + ); + } + // The surviving LSI row (item 3's) carries its base row's version. + assert_eq!( + stamps_by_sk(&engine, &account_id, &key.table_id, Some("lix")).await, + BTreeMap::from([("3".to_owned(), versions["3"])]) + ); +} + +/// Per live item of the composite fixture (keyed by its `sk`), the base +/// row's `version`; a deleted item's partition, left with its static delete +/// stamp alone, is skipped. +#[cfg(feature = "test-hooks")] +async fn live_versions_by_sk( + engine: &CassandraEngine, + account_id: &str, + table_id: &str, +) -> BTreeMap { + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT item_data, version FROM {}.items_{}", + engine.account_keyspace(account_id), + table_id.replace('-', "_") + ), + cdrs_tokio::query_values!(), + "versions", + ) + .await + .expect("read the versions"); + rows.into_iter() + .filter_map(|row| { + let json: Option = row.get_by_name("item_data").ok().flatten(); + let image: Item = serde_json::from_str(&json?).expect("an item image"); + let Some(AttributeValue::N(sk)) = image.get("sk") else { + panic!("no sk in {image:?}"); + }; + let version: i64 = row.get_r_by_name("version").expect("a version"); + Some((sk.clone(), version)) + }) + .collect() +} + +/// A first batch whose writes landed and whose progress was never recorded +/// (here the progress lightweight transaction's outcome is unknown twice and +/// the load fails, the state a crash before the first progress record +/// leaves) is resumed with the first file re-applied (`reapplying`, from +/// `start_at: 0`): the upserts rewrite the landed rows and write the rest, +/// and the base count and both index-row counts are exact. The engine +/// takes this path for a target holding no more than the backup's total +/// (`a_half_applied_first_file_with_no_progress_is_resumed_with_upserts`). +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_first_batch_applied_without_progress_resumes_to_the_exact_count() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + composite_target_input("HalfFirst"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let composite_items = |range: std::ops::Range| { + futures::stream::iter(range.map(composite_item).map(Ok)).boxed() + }; + // The first batch's writes land; its progress write is lost twice and the + // load fails with no progress recorded. + extenddb_storage_cassandra::test_hooks::fail_progress_writes( + &account_id, + &[ + extenddb_storage_cassandra::test_hooks::ProgressWriteFault::Lost, + extenddb_storage_cassandra::test_hooks::ProgressWriteFault::Lost, + ], + ); + let out = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + composite_items(0..7), + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("no known outcome twice")), + "{out:?}" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (None, None), + "no progress is recorded" + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 3, + "the first batch's items landed" + ); + + // The resume: the whole first file re-applied with upserts. + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: true, + }, + composite_items(0..7), + ) + .await + .expect("the resume's load"); + assert_eq!(applied, 7); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); + // Items 0, 2, 4 and 6 carry `g`; 0, 3 and 6 carry `l`. + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 4 + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + 3 + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(7)) + ); + assert_index_rows_carry_the_base_versions(&engine, &account_id, &key.table_id).await; +} + +/// `restore_target_ready` reads the index tables as well as the data table: +/// a target whose GSI table is missing (the registration creates the data +/// table and then each index table in its own statement, so a crash between +/// them leaves this) is not ready, where the data table alone would pass. +#[tokio::test] +async fn test_restore_target_ready_requires_every_index_table() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, _claim) = engine + .claim_restore( + &account_id, + composite_target_input("NoGsiTable"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + assert!( + engine.restore_target_ready(&key).await.expect("ready"), + "the registration created every table" + ); + // The GSI's table is dropped, as a crash between the data table's DDL + // and the index table's would leave it. + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT index_id FROM {}.indexes WHERE table_id = ? AND index_name = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!(key.table_id.as_str(), "gix"), + "index id", + ) + .await + .expect("read the index"); + let index_id: String = rows + .first() + .expect("the GSI exists") + .get_by_name("index_id") + .ok() + .flatten() + .expect("index_id"); + engine + .session() + .query(format!( + "DROP TABLE {}.{}", + engine.account_keyspace(&account_id), + extenddb_storage_cassandra::data::ddl::index_table_name(&index_id) + )) + .await + .expect("drop the GSI's table"); + assert!( + !engine.restore_target_ready(&key).await.expect("ready"), + "a target missing its GSI table is not ready" + ); +} + +/// The in-use rule's input on Cassandra while a store-backed restore of the +/// backup runs: `backup_in_use` names the CREATING target, twice (the +/// engine reads it before it validates a second restore's overrides, so an +/// unknown GSI override, a throughput override and a plain restore are each +/// `BackupInUseException`, as on the other backends); it names the target +/// still when the backup's `active_restores` entry has not landed (the +/// registration writes the target row first), through the target rows; and +/// it names nothing once the target is ACTIVE. +#[tokio::test] +async fn test_a_running_store_backed_restore_keeps_its_backup_in_use_until_active() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + None + ); + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("InUseTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + for _ in 0..2 { + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + Some("InUseTarget".to_owned()), + "a running store-backed restore keeps the backup in use" + ); + let second = engine + .claim_restore( + &account_id, + manifest_target_input("SecondTarget"), + &manifest_registration(&row, None), + ) + .await + .err() + .expect("a second restore is refused"); + assert!( + matches!(second, StorageError::BackupInUse(ref m) if *m == backup_in_use_message(&row.backup_arn)), + "{second}" + ); + } + // The registration's `active_restores` entry removed: the target row + // alone keeps the backup in use. + apply( + &engine, + &format!( + "DELETE active_restores[?] FROM {}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + key.table_id.as_str(), + account_id.as_str(), + row.backup_arn.as_str() + ), + ) + .await; + assert!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .is_empty() + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + Some("InUseTarget".to_owned()), + "the CREATING target row keeps the backup in use before the map entry lands" + ); + // ACTIVE: the backup is free. + assert_eq!( + engine + .finish_restore(&key, &*claim, 0, 0) + .await + .expect("finish"), + Finished::Active + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + None + ); +} diff --git a/crates/storage-mongodb/Cargo.toml b/crates/storage-mongodb/Cargo.toml index 383f7e6ca..510805d92 100644 --- a/crates/storage-mongodb/Cargo.toml +++ b/crates/storage-mongodb/Cargo.toml @@ -10,9 +10,10 @@ license.workspace = true [features] # Enables deterministic coordination hooks used only by the MongoDB integration -# test binary (the mongodb-test-hooks server build). Production builds do not -# compile these hooks. -test-hooks = [] +# test binary (the mongodb-test-hooks server build), and the restore pause +# point of `extenddb-storage` the kill-point harness drives. Production builds +# do not compile these hooks. +test-hooks = ["extenddb-storage/test-hooks"] # Compiles the snapshot page-size setter, the open pause and the shared # snapshot conformance check the crate's snapshot tests use. Separate from # `test-hooks` so the crate's own tests build the production commit and diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs index ee901e850..cfbfb83af 100644 --- a/crates/storage-mongodb/src/backup_catalog.rs +++ b/crates/storage-mongodb/src/backup_catalog.rs @@ -40,10 +40,12 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary, TableDescription}; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::TryStreamExt; @@ -52,7 +54,9 @@ use futures::stream::BoxStream; use mongodb::bson::{self, Bson, Document, doc}; use crate::MongoEngine; -use crate::backup_engine::{age_is_stale, restoring_target_filter}; +use crate::backup_engine::{ + RestoreRecord, RestoreRegistration, age_is_stale, restoring_target_filter, +}; use crate::data_engine::{ CommitOutcome, TRANSIENT_RETRY_ATTEMPTS, TxErr, backoff_sleep, is_duplicate_key, }; @@ -70,10 +74,20 @@ pub(crate) const LOCK_SEQ: &str = "lock_seq"; /// document, with the field names the job record has on every backend. pub(crate) const RESTORE_JOB: &str = "restore_job"; pub(crate) const RESTORE_JOB_STATUS: &str = "restore_job.status"; +pub(crate) const RESTORE_JOB_KIND: &str = "restore_job.kind"; pub(crate) const RESTORE_JOB_GENERATION: &str = "restore_job.owner_generation"; pub(crate) const RESTORE_JOB_HEARTBEAT: &str = "restore_job.heartbeat_at"; pub(crate) const RESTORE_JOB_FAILURE_REASON: &str = "restore_job.failure_reason"; pub(crate) const RESTORE_JOB_COMPLETED_AT: &str = "restore_job.completed_at"; +pub(crate) const RESTORE_JOB_SPOOL_HOST: &str = "restore_job.spool_host"; +pub(crate) const RESTORE_JOB_FILE_INDEX: &str = "restore_job.file_index"; +/// Items of `file_index` applied so far, written with each batch's inserts +/// in one transaction (`restore_engine.rs`). +pub(crate) const RESTORE_JOB_ITEMS_APPLIED: &str = "restore_job.items_applied"; +/// Whether the registration's DDL finished: `false` from the record's +/// insert, `true` once the target's collections and index metadata exist; +/// the reconciler's resume refuses a record left `false`. +pub(crate) const RESTORE_JOB_DDL_DONE: &str = "restore_job.ddl_done"; fn internal(e: impl std::fmt::Display) -> StorageError { StorageError::Internal(e.to_string()) @@ -211,7 +225,7 @@ fn key_filter(key: &BackupKey) -> Document { /// `set` as the `$set` stage of a pipeline update: `$$NOW` stays the server's /// clock and every other value is wrapped in `$literal`, so a string, or a /// key inside the source table object, is never read as a field path. -fn literal_set(set: Document) -> Document { +pub(crate) fn literal_set(set: Document) -> Document { set.into_iter() .map(|(k, v)| { let v = match v { @@ -296,28 +310,6 @@ pub(crate) fn dead_restore_job_filter(stale_after: Duration) -> Document { } } -/// A restore job record as the two listings read a table document. -fn restore_job_row(doc: &Document) -> Result { - let id = doc - .get_document("_id") - .map_err(|_| StorageError::Internal("table is missing _id".to_owned()))?; - let job = doc.get_document(RESTORE_JOB).map_err(|_| { - StorageError::Internal("restore target is missing its job record".to_owned()) - })?; - Ok(RestoreJobRow { - target: RestoreTargetKey { - account_id: required_str(id, "account_id")?, - table_name: required_str(id, "table_name")?, - table_id: required_str(doc, "table_id")?, - }, - source_backup_arn: required_str(doc, "restore_source_backup_arn")?, - kind: RestoreKind::parse(&required_str(job, "kind")?)?, - status: JobStatus::parse(&required_str(job, "status")?)?, - owner_generation: generation_from_field(job, "owner_generation"), - heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0), - }) -} - /// A backup job's handle on its `backup_index` document. struct MongoBackupClaim { index: mongodb::Collection, @@ -420,6 +412,165 @@ impl JobClaim for MongoBackupClaim { } } +/// The filter of the `tables` document `key` names: the `_id` and the table +/// id, so a table re-created under the name is never the one addressed. +fn target_filter(key: &RestoreTargetKey) -> Document { + doc! { + "_id": { "account_id": &key.account_id, "table_name": &key.table_name }, + "table_id": &key.table_id, + } +} + +/// One restore target's `tables` document as a [`RestoreJobRow`]: the key, +/// the provenance and every member of the `restore_job` subdocument; the +/// store is read when all three of its fields are there. +pub(crate) fn restore_job_from_doc(doc: &Document) -> Result { + let id = doc + .get_document("_id") + .map_err(|_| StorageError::Internal("table is missing _id".to_owned()))?; + let job = doc.get_document(RESTORE_JOB).map_err(|_| { + StorageError::Internal("restore target is missing its job record".to_owned()) + })?; + let source_backup_arn = required_str(doc, "restore_source_backup_arn")?; + let store = match ( + job.get_str("store_kind"), + job.get_str("store_root"), + job.get_str("store_id"), + ) { + (Ok(kind), Ok(store_root), Ok(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root: store_root.to_owned(), + store_id: store_id.to_owned(), + }), + _ => None, + }; + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id: required_str(id, "account_id")?, + table_name: required_str(id, "table_name")?, + table_id: required_str(doc, "table_id")?, + }, + source_backup_arn, + kind: RestoreKind::parse(&required_str(job, "kind")?)?, + status: JobStatus::parse(&required_str(job, "status")?)?, + owner_generation: generation_from_field(job, "owner_generation"), + heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0), + store, + spool_host: job.get_str("spool_host").ok().map(str::to_owned), + file_index: match job.get("file_index") { + Some(Bson::Int32(i)) => u32::try_from(*i).ok(), + Some(Bson::Int64(i)) => u32::try_from(*i).ok(), + _ => None, + }, + items_applied: match job.get("items_applied") { + Some(Bson::Int32(i)) => u64::try_from(*i).ok(), + Some(Bson::Int64(i)) => u64::try_from(*i).ok(), + _ => None, + }, + failure_reason: job.get_str("failure_reason").ok().map(str::to_owned), + completed_at: optional_datetime(job, "completed_at"), + }) +} + +/// A store-backed restore job's handle on the job record of its target's +/// `tables` document: every write carries the target's key, its table id +/// and the generation, and the heartbeat and the status writes also require +/// the record RUNNING (or the `from` status). +struct MongoRestoreClaim { + tables: mongodb::Collection, + key: RestoreTargetKey, + generation: u64, +} + +impl MongoRestoreClaim { + /// The filter of the record while it is in `status` under this claim. + fn owned_filter(&self, status: JobStatus) -> Document { + let mut filter = target_filter(&self.key); + filter.insert(RESTORE_JOB_STATUS, status.as_str()); + filter.insert(RESTORE_JOB_GENERATION, generation_field(self.generation)); + filter + } +} + +impl JobClaim for MongoRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let result = self + .tables + .update_one( + self.owned_filter(JobStatus::Running), + vec![doc! { "$set": { RESTORE_JOB_HEARTBEAT: "$$NOW" } }], + ) + .await + .map_err(|e| StorageError::Internal(format!("restore heartbeat: {e}")))?; + Ok(if result.matched_count == 1 { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let mut set = doc! { RESTORE_JOB_STATUS: to.as_str() }; + if let Some(reason) = &fields.failure_reason { + set.insert(RESTORE_JOB_FAILURE_REASON, reason.as_str()); + } + if fields.completed { + set.insert(RESTORE_JOB_COMPLETED_AT, "$$NOW"); + } + let set = literal_set(set); + match self + .tables + .update_one(self.owned_filter(from), vec![doc! { "$set": set }]) + .await + { + Ok(result) if result.matched_count == 1 => Ok(WriteOutcome::Matched), + Ok(_) => Ok(WriteOutcome::NotMatched), + Err(e) if outcome_unknown(&e) => Ok(WriteOutcome::Unknown), + Err(e) => Err(internal(e)), + } + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let doc = self + .tables + .find_one(target_filter(&self.key)) + .await + .map_err(internal)? + .ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} (table id {}) has no catalog document", + self.key.table_name, self.key.table_id + )) + })?; + let row = restore_job_from_doc(&doc)?; + Ok(JobRow { + status: row.status, + owner_generation: row.owner_generation, + heartbeat_at: row.heartbeat_at, + failure_reason: row.failure_reason, + }) + }) + } +} + impl MongoEngine { pub(crate) fn backup_index(&self) -> mongodb::Collection { self.catalog_db.collection::(BACKUP_INDEX) @@ -521,6 +672,143 @@ impl MongoEngine { key.backup_arn ))) } + + /// A store-backed restore's registration (the design's 5.3 step 4) as + /// one session transaction: `$inc` the `backup_index` row's `lock_seq` + /// while it is AVAILABLE (the lock `start_backup_delete` takes, so the + /// two order wholly one before the other), run the in-use rule, insert + /// the CREATING target with its provenance and its RUNNING `kind = + /// manifest` record under a fresh generation, the store and the + /// claimant's spool, with the server's clock on its heartbeat. The + /// transaction commits right after the insert, before the target's + /// collections and indexes are created; one that lost a write conflict + /// (a DeleteBackup or another registration of the backup) is re-run and + /// sees the other's result. + /// + /// Refusals: a missing or non-AVAILABLE row is `Backup not found`; a + /// CREATING target of the backup is `BackupInUse` naming it; a taken + /// name is what `create_table_for_restore` reports (`TableAlreadyExists`, + /// or `TableBeingRestored` when the name is a CREATING restore target). + async fn claim_restore_tx( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result { + let target_table_name = input.table_name.clone(); + let backup_arn = registration.backup.backup_arn.as_str(); + let index = self.backup_index(); + let mut session = self.client.start_session().await.map_err(internal)?; + let tx_options = self.write_transaction_options(); + let generation = new_generation(); + for attempt in 0..TRANSIENT_RETRY_ATTEMPTS { + session + .start_transaction() + .with_options(tx_options.clone()) + .await + .map_err(internal)?; + let body: Result, TxErr> = async { + let mut lock = key_filter(®istration.backup); + lock.insert("status", BackupStatus::Available.as_str()); + let locked = index + .find_one_and_update(lock, doc! { "$inc": { LOCK_SEQ: 1_i64 } }) + .session(&mut session) + .await + .map_err(TxErr::from)?; + if locked.is_none() { + return Ok(Err(not_found(backup_arn))); + } + if let Some(restoring) = self + .creating_restore_target(account_id, backup_arn, Some(&mut session)) + .await + .map_err(TxErr::from)? + { + return Ok(Err(backup_in_use(&restoring, backup_arn))); + } + let mut target = RestoreRegistration { + backup_arn, + source_table_arn: ®istration.source_table_arn, + restore_date_time: epoch_secs_datetime(registration.restore_date_time), + owner_generation: generation, + // This process's clock, replaced by the server's inside + // the transaction (`stamp_server_clock`); test builds set + // it behind to prove the replacement. + first_heartbeat: self.local_now(), + record: RestoreRecord::Manifest { + store: ®istration.store, + spool_host: registration.spool_host.as_deref(), + }, + session: Some(&mut session), + }; + match self + .create_table_for_restore(account_id, input.clone(), &mut target) + .await + { + Ok(desc) => Ok(Ok(desc)), + Err(StorageError::Transient(_)) => Err(TxErr::Transient), + Err(e) => { + // A creation that failed after its insert committed + // (a collection or index could not be created) left a + // registered target under this generation; a name + // refusal left nothing, and the removal matches + // nothing. + if let Err(cleanup) = self + .remove_target_of_generation(account_id, &target_table_name, generation) + .await + { + tracing::error!( + "restore of {backup_arn} into {target_table_name} failed during \ + target creation ({e}), and cleanup failed: {cleanup}" + ); + } + Ok(Err(e)) + } + } + } + .await; + match body { + // The insert committed its transaction before the target's + // collections were created; nothing is left to commit. + Ok(Ok(mut desc)) => { + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(backup_arn.to_owned()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let key = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: target_table_name, + table_id: desc.table_id.clone(), + }; + return Ok(( + desc, + Box::new(MongoRestoreClaim { + tables: self.tables(), + key, + generation, + }) as Box, + )); + } + Ok(Err(refusal)) => { + let _ = session.abort_transaction().await; + return Err(refusal); + } + Err(TxErr::Transient) => { + let _ = session.abort_transaction().await; + backoff_sleep(attempt).await; + } + Err(TxErr::Fatal(e)) => { + let _ = session.abort_transaction().await; + return Err(e); + } + } + } + Err(StorageError::Internal(format!( + "the registration of the restore of {backup_arn} into {target_table_name} did not \ + commit after {TRANSIENT_RETRY_ATTEMPTS} conflicting attempts" + ))) + } } impl BackupCatalog for MongoEngine { @@ -884,7 +1172,7 @@ impl BackupCatalog for MongoEngine { // cannot read, so one such document does not stop the cycle. Ok(docs .iter() - .filter_map(|doc| match restore_job_row(doc) { + .filter_map(|doc| match restore_job_from_doc(doc) { Ok(row) => Some(row), Err(e) => { tracing::warn!( @@ -898,6 +1186,97 @@ impl BackupCatalog for MongoEngine { }) } + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(self.claim_restore_tx(account_id, input, registration)) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The heartbeat is the liveness test, as for a backup row: a + // RUNNING manifest record whose heartbeat is older than the window + // on the server's clock has no owner, and the one write below + // tests that and takes the record under a fresh generation with + // the claimant's spool and `heartbeat_at` at the server's clock, + // so a second taker (another instance's reconciler in the period + // before the successor's first beat) finds it live; the trait doc + // says why a restore record is refreshed where a backup row is + // not. A legacy record is never taken over (the reconciler fails + // it), and a record that is not RUNNING is its owner's or + // finished. + let generation = new_generation(); + let mut filter = target_filter(key); + filter.insert(RESTORE_JOB_STATUS, JobStatus::Running.as_str()); + filter.insert(RESTORE_JOB_KIND, RestoreKind::Manifest.as_str()); + filter.insert("$expr", age_is_stale(&[RESTORE_JOB_HEARTBEAT], stale_after)); + let mut update = vec![doc! { "$set": literal_set(doc! { + RESTORE_JOB_GENERATION: generation_field(generation), + RESTORE_JOB_HEARTBEAT: "$$NOW", + }) }]; + match spool_host { + Some(host) => update.push(doc! { "$set": literal_set(doc! { + RESTORE_JOB_SPOOL_HOST: host, + }) }), + None => update.push(doc! { "$unset": RESTORE_JOB_SPOOL_HOST }), + } + let taken = self + .tables() + .find_one_and_update(filter, update) + .await + .map_err(internal)?; + Ok(taken.map(|_| { + Box::new(MongoRestoreClaim { + tables: self.tables(), + key: key.clone(), + generation, + }) as Box + })) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let mut filter = target_filter(key); + filter.insert(RESTORE_JOB, doc! { "$exists": true }); + let doc = self.tables().find_one(filter).await.map_err(internal)?; + doc.as_ref().map(restore_job_from_doc).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let mut filter = doc! { RESTORE_JOB: { "$exists": true } }; + if let Some(status) = status { + filter.insert(RESTORE_JOB_STATUS, status.as_str()); + } + let docs: Vec = self + .tables() + .find(filter) + .sort(doc! { "_id": 1 }) + .await + .map_err(internal)? + .try_collect() + .await + .map_err(internal)?; + docs.iter().map(restore_job_from_doc).collect() + }) + } + fn list_running_restore_jobs(&self) -> BoxFuture<'_, Result, StorageError>> { Box::pin(async move { // By status alone: a record stamped in this read's instant, or @@ -910,7 +1289,7 @@ impl BackupCatalog for MongoEngine { .try_collect() .await .map_err(internal)?; - docs.iter().map(restore_job_row).collect() + docs.iter().map(restore_job_from_doc).collect() }) } diff --git a/crates/storage-mongodb/src/backup_engine.rs b/crates/storage-mongodb/src/backup_engine.rs index 91e76856b..ebb160046 100644 --- a/crates/storage-mongodb/src/backup_engine.rs +++ b/crates/storage-mongodb/src/backup_engine.rs @@ -96,7 +96,7 @@ use extenddb_core::types::{ }; use extenddb_storage::backup_catalog::{ AbandonedLegacyBackup, BackupCatalog, JOB_HEARTBEAT_INTERVAL, JobStatus, RestoreKind, - RestoreTargetKey, new_generation, + RestoreTargetKey, StoreIdentity, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; @@ -353,6 +353,19 @@ pub(crate) fn age_is_stale(fields: &[&str], stale_after: Duration) -> Document { } } +/// How the job of a registration reads its backup, and what the record +/// carries for it beyond the members every record has. +pub(crate) enum RestoreRecord<'a> { + /// The in-catalog backup: `kind = legacy`, nothing more. + Legacy, + /// A backup in the store: `kind = manifest`, the store the backup row + /// names and the claimant's spool. + Manifest { + store: &'a StoreIdentity, + spool_host: Option<&'a str>, + }, +} + /// What a restore's registration writes into the target's first catalog /// document: the provenance `DescribeTable` reports and the job record. pub(crate) struct RestoreRegistration<'a> { @@ -363,8 +376,11 @@ pub(crate) struct RestoreRegistration<'a> { pub(crate) owner_generation: u64, /// The record's first heartbeat. An insert cannot use `$$NOW`, so this is /// the inserting process's clock; the copy task's first refresh replaces - /// it with the server's as soon as the task starts. + /// it with the server's as soon as the task starts. A manifest record + /// has it rewritten with the server's clock inside the registration's + /// transaction ([`Self::stamp_server_clock`]). pub(crate) first_heartbeat: bson::DateTime, + pub(crate) record: RestoreRecord<'a>, /// The session whose transaction the target's insert runs in and which /// the insert commits; `None` inserts and commits nothing beyond the /// document itself. @@ -376,15 +392,60 @@ impl RestoreRegistration<'_> { table_doc.insert("restore_source_backup_arn", self.backup_arn); table_doc.insert("restore_source_table_arn", self.source_table_arn); table_doc.insert("restore_date_time", self.restore_date_time); - table_doc.insert( - RESTORE_JOB, - doc! { - "status": JobStatus::Running.as_str(), - "kind": RestoreKind::Legacy.as_str(), - "owner_generation": generation_field(self.owner_generation), - "heartbeat_at": self.first_heartbeat, - }, - ); + let kind = match self.record { + RestoreRecord::Legacy => RestoreKind::Legacy, + RestoreRecord::Manifest { .. } => RestoreKind::Manifest, + }; + let mut job = doc! { + "status": JobStatus::Running.as_str(), + "kind": kind.as_str(), + "owner_generation": generation_field(self.owner_generation), + "heartbeat_at": self.first_heartbeat, + // Set once the target's collections and index metadata exist + // (`create_table_impl_inner`); a resume refuses an unmarked + // record. + "ddl_done": false, + }; + if let RestoreRecord::Manifest { store, spool_host } = &self.record { + job.insert("store_kind", store.store_kind.as_str()); + job.insert("store_root", &store.store_root); + job.insert("store_id", &store.store_id); + if let Some(spool_host) = spool_host { + job.insert("spool_host", *spool_host); + } + } + table_doc.insert(RESTORE_JOB, job); + } + + /// Rewrite the record's first heartbeat with the server's clock, inside + /// the registration's transaction: a manifest record is heartbeated by + /// the engine's timer, whose first beat comes one period after the + /// registration, so the record must not start on the inserting + /// process's clock. The driver's error is returned as it is, so a write + /// conflict keeps its transient label for the caller's retry. Nothing + /// for a legacy record, whose copy task beats at once. + pub(crate) async fn stamp_server_clock( + &mut self, + tables_coll: &mongodb::Collection, + table_id: &str, + ) -> Result<(), mongodb::error::Error> { + if !matches!(self.record, RestoreRecord::Manifest { .. }) { + return Ok(()); + } + let Some(session) = self.session.as_deref_mut() else { + return Ok(()); + }; + tables_coll + .update_one( + doc! { + "table_id": table_id, + RESTORE_JOB_GENERATION: generation_field(self.owner_generation), + }, + vec![doc! { "$set": { RESTORE_JOB_HEARTBEAT: "$$NOW" } }], + ) + .session(session) + .await + .map(|_| ()) } } @@ -447,7 +508,7 @@ impl MongoEngine { /// This process's clock, for the one heartbeat that cannot use the /// server's (the target insert). Test builds can set it behind. - fn local_now(&self) -> bson::DateTime { + pub(crate) fn local_now(&self) -> bson::DateTime { let now = bson::DateTime::now(); #[cfg(test)] let now = bson::DateTime::from_millis( @@ -580,6 +641,7 @@ impl MongoEngine { restore_date_time: epoch_secs_datetime(*restore_date_time), owner_generation, first_heartbeat: self.local_now(), + record: RestoreRecord::Legacy, session: Some(&mut session), }; match self @@ -650,7 +712,7 @@ impl MongoEngine { /// Remove the CREATING restore target `target_table_name` registered /// under `owner_generation`, with its collections; `Ok(false)` when there /// is none. - async fn remove_target_of_generation( + pub(crate) async fn remove_target_of_generation( &self, account_id: &str, target_table_name: &str, @@ -766,6 +828,52 @@ impl MongoEngine { Ok(result.matched_count == 1) } + /// The end of a restore's load (the design's 5.3 step 6 on this + /// backend): the job record DONE and the target handed to the GSI + /// backfill worker (`restore_backfill_pending`), with `item_count` and, + /// when given, `table_size_bytes`, in one write conditioned on the + /// generation, on the record RUNNING and on the table CREATING. From + /// then on the worker owns the target: it promotes each index and flips + /// the table ACTIVE, conditioned on the flag and on CREATING, and the + /// reconciler leaves a DONE record alone. `Ok(false)` when the record is + /// not RUNNING under `owner_generation` or the table is not CREATING. + pub(crate) async fn hand_over_to_backfill( + &self, + target: &RestoreTargetKey, + owner_generation: u64, + item_count: u64, + size_bytes: Option, + ) -> Result { + let mut set = doc! { + "item_count": { "$literal": i64::try_from(item_count).unwrap_or(i64::MAX) }, + "restore_backfill_pending": { "$literal": true }, + RESTORE_JOB_STATUS: { "$literal": JobStatus::Done.as_str() }, + RESTORE_JOB_COMPLETED_AT: "$$NOW", + }; + if let Some(size_bytes) = size_bytes { + set.insert( + "table_size_bytes", + doc! { "$literal": i64::try_from(size_bytes).unwrap_or(i64::MAX) }, + ); + } + let handed_over = self + .catalog_db + .collection::("tables") + .update_one( + doc! { + "_id": { "account_id": &target.account_id, "table_name": &target.table_name }, + "table_id": &target.table_id, + "table_status": "CREATING", + RESTORE_JOB_STATUS: JobStatus::Running.as_str(), + RESTORE_JOB_GENERATION: generation_field(owner_generation), + }, + vec![doc! { "$set": set }], + ) + .await + .map_err(internal)?; + Ok(handed_over.matched_count == 1) + } + /// The detached copy of a restore: copy the backup's items into the /// target with `$out` while the job record is heartbeated, then hand the /// target to the backfill worker, marking the record DONE in the same @@ -824,26 +932,18 @@ impl MongoEngine { // the flag and on CREATING, and the reconciler leaves a DONE // record alone. let handed_over = self - .catalog_db - .collection::("tables") - .update_one( - doc! { - "_id": { "account_id": account_id, "table_name": target_table_name }, - "table_id": table_id, - "table_status": "CREATING", - RESTORE_JOB_STATUS: JobStatus::Running.as_str(), - RESTORE_JOB_GENERATION: generation_field(owner_generation), + .hand_over_to_backfill( + &RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: target_table_name.to_owned(), + table_id: table_id.to_owned(), }, - vec![doc! { "$set": { - "item_count": { "$literal": i64::try_from(item_count).unwrap_or(i64::MAX) }, - "restore_backfill_pending": { "$literal": true }, - RESTORE_JOB_STATUS: { "$literal": JobStatus::Done.as_str() }, - RESTORE_JOB_COMPLETED_AT: "$$NOW", - } }], + owner_generation, + item_count, + None, ) - .await - .map_err(internal)?; - Ok::<_, StorageError>((item_count, handed_over.matched_count == 1)) + .await?; + Ok::<_, StorageError>((item_count, handed_over)) }; let heartbeat = self.restore_heartbeat_loop(account_id, target_table_name, table_id, owner_generation); @@ -1824,8 +1924,9 @@ mod tests { #[cfg(test)] mod definition_tests { use super::{ - LockMiss, RestoreRegistration, backup_definition_from_doc, description_from_doc, - gone_statuses, lock_miss, restore_source_from_doc, restoring_target_filter, + LockMiss, RestoreRecord, RestoreRegistration, backup_definition_from_doc, + description_from_doc, gone_statuses, lock_miss, restore_source_from_doc, + restoring_target_filter, }; use extenddb_core::types::{ OnDemandThroughput, ProvisionedThroughput, StreamViewType, TimeToLiveStatus, @@ -2069,6 +2170,7 @@ mod definition_tests { restore_date_time: bson::DateTime::from_millis(1_791_976_431_438), owner_generation: 7, first_heartbeat: bson::DateTime::from_millis(1_791_976_500_000), + record: RestoreRecord::Legacy, session: None, }; let mut table_doc = doc! { "table_status": "CREATING" }; @@ -2096,6 +2198,71 @@ mod definition_tests { assert!(!table_doc.contains_key("restore_operation_id")); assert!(!table_doc.contains_key("restore_heartbeat_at")); } + + /// The manifest arm writes `kind = manifest`, the store the backup row + /// names and the claimant's spool; a filesystem store leaves + /// `spool_host` out. + #[test] + fn a_manifest_registration_writes_the_kind_store_and_spool() { + use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind}; + let store = StoreIdentity { + store_kind: StoreKind::S3, + store_root: "s3://bucket/prefix".to_owned(), + store_id: "store-7".to_owned(), + }; + let registration = RestoreRegistration { + backup_arn: ARN, + source_table_arn: "arn:aws:dynamodb:us-east-1:123456789012:table/src", + restore_date_time: bson::DateTime::from_millis(1_791_976_431_438), + owner_generation: 9, + first_heartbeat: bson::DateTime::from_millis(1_791_976_500_000), + record: RestoreRecord::Manifest { + store: &store, + spool_host: Some("host-b:/var/spool"), + }, + session: None, + }; + let mut table_doc = doc! { "table_status": "CREATING" }; + registration.insert_into(&mut table_doc); + let job = table_doc.get_document("restore_job").expect("job record"); + assert_eq!(job.get_str("status"), Ok("RUNNING")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!(job.get_str("store_kind"), Ok(StoreKind::S3.as_str())); + assert_eq!(job.get_str("store_root"), Ok("s3://bucket/prefix")); + assert_eq!(job.get_str("store_id"), Ok("store-7")); + assert_eq!(job.get_str("spool_host"), Ok("host-b:/var/spool")); + + // A filesystem store spools nothing. + let fs = StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/var/lib/extenddb/backups".to_owned(), + store_id: "store-1".to_owned(), + }; + let registration = RestoreRegistration { + backup_arn: ARN, + source_table_arn: "x", + restore_date_time: bson::DateTime::from_millis(0), + owner_generation: 9, + first_heartbeat: bson::DateTime::from_millis(0), + record: RestoreRecord::Manifest { + store: &fs, + spool_host: None, + }, + session: None, + }; + let mut table_doc = doc! { "table_status": "CREATING" }; + registration.insert_into(&mut table_doc); + let job = table_doc.get_document("restore_job").expect("job record"); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!( + job.get_str("store_kind"), + Ok(StoreKind::Filesystem.as_str()) + ); + assert!( + job.get_str("spool_host").is_err(), + "no spool for a filesystem store" + ); + } } /// The protocol model; see the module documentation. diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index bdb8b6c2f..4c55ae396 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -30,23 +30,28 @@ use extenddb_core::types::{ ProjectionType, ScalarAttributeType, TableStatus, }; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, RestoreTargetKey, - StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, - WriteOutcome, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, }; use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; use extenddb_storage::{ BackupCatalog, JobPermit, LegacyBackupReader, RestoreRequest, RestoreTableOverrides, TableEngine, }; -use futures::{FutureExt, TryStreamExt}; +use futures::stream::BoxStream; +use futures::{FutureExt, StreamExt, TryStreamExt}; use mongodb::bson::{self, Document, doc}; use super::{AfterBackupDrop, BackupLiveness, HeartbeatFault}; use crate::MongoEngine; -use crate::backup_catalog::{BACKUP_INDEX, RESTORE_JOB, RESTORE_JOB_HEARTBEAT, RESTORE_JOB_STATUS}; -use crate::data::{data_collection_name, item_to_document}; +use crate::backup_catalog::{ + BACKUP_INDEX, RESTORE_JOB, RESTORE_JOB_GENERATION, RESTORE_JOB_HEARTBEAT, RESTORE_JOB_STATUS, +}; +use crate::data::{data_collection_name, document_to_item, item_to_document}; use crate::migrations::{MongoTarget, run_backup_catalog_index}; +use crate::restore_engine::INSERT_MANY_DOCUMENTS; const ACCOUNT: &str = "123456789012"; const OTHER_ACCOUNT: &str = "210987654321"; @@ -420,6 +425,25 @@ async fn registered_targets(engine: &MongoEngine, targets: &[String]) -> Vec bson::DateTime { + let mut cursor = engine + .catalog_db + .collection::("tables") + .aggregate(vec![ + doc! { "$limit": 1 }, + doc! { "$project": { "_id": 0, "now": "$$NOW" } }, + ]) + .await + .expect("aggregate"); + let doc = cursor + .next() + .await + .expect("one document") + .expect("the server's clock"); + *doc.get_datetime("now").expect("now") +} + async fn job_record(engine: &MongoEngine, name: &str) -> Document { table_doc(engine, name) .await @@ -3354,6 +3378,1054 @@ async fn live_stream_record_created_at_matches_the_record_timestamp() { fx.finish().await; } +// ───────────────────────────────────────────────────────────────────────── +// Store-backed restore: claim_restore, the claim, claim_restore_dead, +// read/list_restore_jobs, bulk_load, finish_restore, count_restore_target. +// The registration takes the `backup_index` row's lock (as the legacy path +// takes the `backups` row's), writes the record `kind = manifest` with the +// store and the claimant's spool, and the record's heartbeat is the server's +// clock from the registration on; bulk_load writes base documents only, the +// GSI backfill worker fills the indexes and flips the table ACTIVE. +// ───────────────────────────────────────────────────────────────────────── + +/// A composite-key input with a GSI (ALL) and an LSI (KEYS_ONLY), the shape +/// `source_table_for` creates, for a restore target. +fn manifest_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![key("pk", KeyType::Hash), key("sk", KeyType::Range)], + attribute_definitions: vec![ + attr("pk", ScalarAttributeType::S), + attr("sk", ScalarAttributeType::N), + attr("gpk", ScalarAttributeType::S), + attr("lsk", ScalarAttributeType::N), + ], + billing_mode: Some(BillingMode::PayPerRequest), + global_secondary_indexes: Some(vec![GsiInput { + index_name: "g1".to_owned(), + key_schema: vec![key("gpk", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + provisioned_throughput: None, + }]), + local_secondary_indexes: Some(vec![LsiInput { + index_name: "l1".to_owned(), + key_schema: vec![key("pk", KeyType::Hash), key("lsk", KeyType::Range)], + projection: Projection { + projection_type: ProjectionType::KeysOnly, + non_key_attributes: None, + }, + }]), + ..Default::default() + } +} + +/// A `backup_index` row of a backup of `table`, left AVAILABLE: what a +/// store-backed restore registers against. Returns the row (its `key()` is +/// the registration's backup key). +async fn available_backup(engine: &MongoEngine, table: &str, name: &str) -> BackupRow { + let row = row(ACCOUNT, table, name); + let claim = engine.claim_backup(&row).await.expect("claim backup row"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields::default() + ) + .await + .expect("to AVAILABLE"), + WriteOutcome::Matched + ); + drop(claim); + row +} + +/// The registration row for `row`, with the claimant's `spool_host` +/// (`None` for a filesystem store). `owner_generation` is unowned: the +/// backend mints its own. +fn registration(row: &BackupRow, spool: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: store(), + spool_host: spool.map(str::to_owned), + owner_generation: UNOWNED_GENERATION, + } +} + +/// Item `i` of a restore: every attribute the source shape carries, both +/// index keys, and a 38-digit non-key number that survives as a string. +fn manifest_item(i: usize) -> Item { + [ + ("pk".to_owned(), AttributeValue::S(format!("p{}", i % 50))), + ("sk".to_owned(), AttributeValue::N(i.to_string())), + ("gpk".to_owned(), AttributeValue::S(format!("g{}", i % 7))), + ("lsk".to_owned(), AttributeValue::N((i * 3).to_string())), + ( + "big".to_owned(), + AttributeValue::N("12345678901234567890123456789012345678".to_owned()), + ), + ] + .into_iter() + .collect() +} + +/// The items `lo..hi` as the stream `bulk_load` reads. +fn item_stream(lo: usize, hi: usize) -> BoxStream<'static, Result> { + let items: Vec<_> = (lo..hi).map(manifest_item).collect(); + futures::stream::iter(items.into_iter().map(Ok)).boxed() +} + +/// The registration writes the target CREATING with its `RestoreSummary` +/// and a RUNNING `kind = manifest` record carrying the store, the +/// claimant's `spool_host`, a fresh generation and the server's clock on +/// its heartbeat (the registering process's clock is an hour behind, and +/// the stored heartbeat is within seconds of the server's `$$NOW`); the +/// claim heartbeats and reads the record; `read_restore_job` and +/// `list_restore_jobs` fill the store and spool; `backup_in_use` names the +/// target. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_registers_the_target_and_its_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let reg = registration(&backup, Some("host-a:/var/spool/x")); + let behind = fx + .second_instance(BackupLiveness { + local_clock_behind: Duration::from_secs(3_600), + ..test_liveness() + }) + .await; + let (desc, claim) = behind + .claim_restore(ACCOUNT, manifest_input("dst"), ®) + .await + .expect("register"); + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(backup.backup_arn.as_str()) + ); + assert!(summary.restore_in_progress); + let generation = claim.generation(); + assert_ne!(generation, UNOWNED_GENERATION); + + let doc = table_doc(&fx.engine, "dst").await.expect("target"); + let job = doc.get_document(RESTORE_JOB).expect("job record"); + assert_eq!(job.get_str("status"), Ok("RUNNING")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!( + u64::try_from(job.get_i64("owner_generation").unwrap()).unwrap(), + generation + ); + assert_eq!( + job.get_str("store_kind"), + Ok(StoreKind::Filesystem.as_str()) + ); + assert_eq!(job.get_str("store_root"), Ok(store().store_root.as_str())); + assert_eq!(job.get_str("store_id"), Ok(store().store_id.as_str())); + assert_eq!(job.get_str("spool_host"), Ok("host-a:/var/spool/x")); + // The heartbeat is the server's clock stamped inside the registration, + // not the inserting process's (an hour behind): within seconds of the + // server's `$$NOW`, read back through an aggregation. + let heartbeat = *job.get_datetime("heartbeat_at").expect("heartbeat"); + let now = server_now(&fx.engine).await; + let age_ms = now.timestamp_millis() - heartbeat.timestamp_millis(); + assert!( + (-2_000..=5_000).contains(&age_ms), + "the record's heartbeat is the server's clock, not the slow process's: {heartbeat:?} \ + is {age_ms} ms behind {now:?}" + ); + + let key = target_key(&doc); + let read = fx + .engine + .read_restore_job(&key) + .await + .expect("read") + .expect("a record"); + assert_eq!(read.kind, RestoreKind::Manifest); + assert_eq!(read.status, JobStatus::Running); + assert_eq!(read.owner_generation, generation); + assert_eq!( + read.store.as_ref().map(|s| s.store_kind), + Some(StoreKind::Filesystem) + ); + assert_eq!(read.spool_host.as_deref(), Some("host-a:/var/spool/x")); + assert_eq!(read.file_index, None); + + let listed = fx + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].target, key); + assert!(listed[0].store.is_some()); + assert_eq!(listed[0].spool_host.as_deref(), Some("host-a:/var/spool/x")); + + assert_eq!( + fx.engine + .backup_in_use(&backup.key()) + .await + .unwrap() + .as_deref(), + Some("dst") + ); + assert_eq!(claim.heartbeat().await.unwrap(), Heartbeat::Confirmed); + drop(claim); + fx.finish().await; +} + +/// A DeleteBackup and a store-backed registration of one backup order wholly +/// one before the other, on the `backup_index` row's lock: a delete that +/// wins leaves the registration `Backup not found` and no target; a +/// registration that wins makes the delete `BackupInUse`. A loop of +/// concurrent races holds the invariant that no target is ever registered +/// against a row that is not AVAILABLE. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_the_two_orderings_against_delete_backup_on_the_backup_index_row() { + let fx = Fixture::new().await; + fx.worker.abort(); + + // Order A: the delete wins. + let a = available_backup(&fx.engine, "srcA", "ba").await; + assert_eq!( + fx.engine.start_backup_delete(&a.key()).await.unwrap(), + DeleteStart::Deleting + ); + let refused = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dstA"), ®istration(&a, None)) + .await + .err(); + assert!( + matches!(&refused, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "the registration finds the backup gone: {refused:?}" + ); + assert!(table_doc(&fx.engine, "dstA").await.is_none()); + + // Order B: the registration wins. + let b = available_backup(&fx.engine, "srcB", "bb").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dstB"), ®istration(&b, None)) + .await + .expect("register"); + assert_eq!( + fx.engine.start_backup_delete(&b.key()).await.unwrap(), + DeleteStart::InUse("dstB".to_owned()) + ); + assert_eq!( + fx.engine + .read_backup_row(&arn_of(&b)) + .await + .unwrap() + .unwrap() + .status, + BackupStatus::Available, + "a refused delete leaves the row AVAILABLE" + ); + drop(claim); + + // Concurrent races: the lock keeps the two consistent every time, and + // both outcomes occur across the loop (no frozen answer). + let mut delete_won = 0; + let mut restore_won = 0; + for i in 0..16 { + let backup = available_backup(&fx.engine, &format!("srcC{i}"), "bc").await; + let name = format!("dstC{i}"); + let e1 = fx.engine.clone(); + let e2 = fx.engine.clone(); + let key = backup.key(); + let reg = registration(&backup, None); + let del = tokio::spawn(async move { e1.start_backup_delete(&key).await }); + let res = + tokio::spawn( + async move { e2.claim_restore(ACCOUNT, manifest_input(&name), ®).await }, + ); + let (del, res) = (del.await.unwrap().unwrap(), res.await.unwrap()); + let res_ok = res.is_ok(); + let res_err = res.err(); + let target = table_doc(&fx.engine, &format!("dstC{i}")).await; + let status = fx + .engine + .read_backup_row(&arn_of(&backup)) + .await + .unwrap() + .expect("the backup_index row") + .status; + if target.is_some() { + assert_eq!( + status, + BackupStatus::Available, + "a target was registered, so the delete must have lost" + ); + assert!( + matches!(del, DeleteStart::InUse(_)), + "iteration {i}: {del:?}" + ); + assert!(res_ok, "iteration {i}"); + restore_won += 1; + } else { + assert!( + matches!(&res_err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "iteration {i}: no target but the registration did not refuse: {res_err:?}" + ); + assert_eq!(status, BackupStatus::Deleting, "iteration {i}"); + delete_won += 1; + } + } + assert!( + delete_won > 0 && restore_won > 0, + "both orders occur: {delete_won} / {restore_won}" + ); + fx.finish().await; +} + +/// Two concurrent registrations of one backup leave one CREATING target and +/// answer `BackupInUse` to the other (the in-use rule under the lock). +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_two_concurrent_registrations_of_one_backup_leave_one_target() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let e1 = fx.engine.clone(); + let e2 = fx.engine.clone(); + let r1 = registration(&backup, None); + let r2 = registration(&backup, None); + let t1 = + tokio::spawn(async move { e1.claim_restore(ACCOUNT, manifest_input("r1"), &r1).await }); + let t2 = + tokio::spawn(async move { e2.claim_restore(ACCOUNT, manifest_input("r2"), &r2).await }); + let o1 = t1.await.unwrap(); + let o2 = t2.await.unwrap(); + let oks = [&o1, &o2].iter().filter(|o| o.is_ok()).count(); + let in_use = [&o1, &o2] + .iter() + .filter(|o| matches!(o, Err(StorageError::BackupInUse(_)))) + .count(); + let errs: Vec<_> = [&o1, &o2] + .iter() + .filter_map(|o| o.as_ref().err().map(ToString::to_string)) + .collect(); + assert_eq!( + (oks, in_use), + (1, 1), + "one target, one BackupInUse: oks={oks}, errs={errs:?}" + ); + let targets = registered_targets(&fx.engine, &["r1".to_owned(), "r2".to_owned()]).await; + assert_eq!( + targets.len(), + 1, + "exactly one target registered: {targets:?}" + ); + for (_, claim) in [o1, o2].into_iter().flatten() { + drop(claim); + } + fx.finish().await; +} + +/// A name a table or a CREATING restore target already holds is refused as +/// `TableAlreadyExists` or `TableBeingRestored`, with nothing registered. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_refuses_a_taken_name() { + let fx = Fixture::new().await; + fx.worker.abort(); + // A plain ACTIVE table. + fx.engine + .create_table_impl(ACCOUNT, manifest_input("taken"), false) + .await + .expect("create a table"); + let backup = available_backup(&fx.engine, "src", "b").await; + let exists = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("taken"), + ®istration(&backup, None), + ) + .await + .err(); + assert!( + matches!(&exists, Some(StorageError::TableAlreadyExists(n)) if n == "taken"), + "{exists:?}" + ); + + // A CREATING restore target. + let (_, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("being"), + ®istration(&backup, None), + ) + .await + .expect("first registration"); + let b2 = available_backup(&fx.engine, "src2", "b2").await; + let restoring = fx + .engine + .claim_restore(ACCOUNT, manifest_input("being"), ®istration(&b2, None)) + .await + .err(); + assert!( + matches!( + &restoring, + Some(StorageError::TableBeingRestored { table_name, backup_arn }) + if table_name == "being" && backup_arn == &backup.backup_arn + ), + "{restoring:?}" + ); + drop(claim); + fx.finish().await; +} + +/// `claim_restore_dead` takes over a RUNNING manifest record whose heartbeat +/// is stale: a fresh generation, the claimant's `spool_host` and +/// `heartbeat_at` refreshed to the server's clock in one write, so a second +/// taker issued right after the first (before the successor's first beat) +/// finds the record live and is refused, and the first taker's beat +/// confirms; the old generation's writes are fenced out. A live record, and +/// a legacy record whatever its age, are left alone. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_dead_takes_over_a_stale_manifest_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("t"), + ®istration(&backup, Some("old:/s")), + ) + .await + .expect("register"); + let desc_doc = table_doc(&fx.engine, "t").await.expect("target"); + let key = target_key(&desc_doc); + let g1 = claim.generation(); + let _ = doc; + + // Live: fresh heartbeat, so not dead. + assert!( + fx.engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s2")) + .await + .unwrap() + .is_none(), + "a live record is not taken over" + ); + + // Age it past the window and take it over. + age_running_record(&fx.engine, "t", Duration::from_secs(600)).await; + let aged = *job_record(&fx.engine, "t") + .await + .get_datetime("heartbeat_at") + .expect("heartbeat"); + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s2")) + .await + .expect("takeover") + .expect("a stale record is taken over"); + let g2 = taken.generation(); + assert_ne!(g2, g1, "a fresh generation"); + let after = job_record(&fx.engine, "t").await; + assert_eq!( + u64::try_from(after.get_i64("owner_generation").unwrap()).unwrap(), + g2 + ); + assert_eq!( + after.get_str("spool_host"), + Ok("new:/s2"), + "the claimant's spool is written" + ); + // A second taker right after the first, before the successor's first + // beat: the record is live to it, and the first taker's beat confirms. + let second = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, Some("third:/s3")) + .await + .expect("a second takeover attempt"); + assert!( + second.is_none(), + "a second taker right after the first is refused: {:?}", + second.as_ref().map(|c| c.generation()) + ); + assert_eq!( + taken.heartbeat().await.unwrap(), + Heartbeat::Confirmed, + "the first taker keeps the record" + ); + let refreshed = *after.get_datetime("heartbeat_at").expect("heartbeat"); + assert!( + refreshed > aged, + "the takeover refreshed heartbeat_at: {refreshed:?} was {aged:?}" + ); + let now = server_now(&fx.engine).await; + let age_ms = now.timestamp_millis() - refreshed.timestamp_millis(); + assert!( + (-2_000..=5_000).contains(&age_ms), + "the refreshed heartbeat is the server's now: {refreshed:?} is {age_ms} ms behind {now:?}" + ); + assert!( + fx.engine + .list_dead_restore_jobs(STALE_AFTER) + .await + .unwrap() + .is_empty(), + "the taken-over record is live to the listing at once" + ); + + // The old generation is fenced out; the new one owns the record. + assert_eq!( + claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .unwrap(), + WriteOutcome::NotMatched + ); + assert!( + matches!( + fx.engine.finish_restore(&key, &*claim, 0, 0).await.unwrap(), + Finished::NotMatched(_) + ), + "the dead generation cannot finish the restore" + ); + drop(claim); + drop(taken); + fx.finish().await; +} + +/// A legacy record, and a record that is not RUNNING, are never taken over. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_dead_leaves_a_legacy_or_finished_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("leg"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "leg").await.expect("target")); + let _ = doc; + // Make the record legacy and stale. + fx.engine + .catalog_db + .collection::("tables") + .update_one( + doc! { "_id": { "account_id": ACCOUNT, "table_name": "leg" } }, + doc! { "$set": { "restore_job.kind": "legacy" } }, + ) + .await + .expect("to legacy"); + age_running_record(&fx.engine, "leg", Duration::from_secs(600)).await; + // age_running_record rewrites the record RUNNING; set kind legacy again. + fx.engine + .catalog_db + .collection::("tables") + .update_one( + doc! { "_id": { "account_id": ACCOUNT, "table_name": "leg" } }, + doc! { "$set": { "restore_job.kind": "legacy" } }, + ) + .await + .expect("to legacy"); + assert!( + fx.engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s")) + .await + .unwrap() + .is_none(), + "a legacy record is not taken over" + ); + drop(claim); + fx.finish().await; +} + +/// `bulk_load` writes a file's base documents and records `(file_index, +/// items_applied)` in the same transaction as each batch, so a resume +/// continues from the exact item `items_applied` names and re-applies +/// nothing; a duplicate key on a file being loaded for the first time (a +/// backup that holds a key twice) fails the file. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_records_each_batch_and_a_resume_continues_from_items_applied() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dst"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "dst").await.expect("target")); + let _ = doc; + + // File 0: 2500 items across three insert_many batches, each committed + // with its progress in one transaction. + let applied = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 2500), + ) + .await + .expect("file 0"); + assert_eq!(applied, 2500); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 2500); + let record = fx.engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(0)); + assert_eq!(record.items_applied, Some(2500)); + + // File 1's first 1000 items commit (one batch), then the process dies: + // the record names file 1 with 1000 applied. + let applied = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + item_stream(2500, 3500), + ) + .await + .expect("file 1, first batch"); + assert_eq!(applied, 1000); + let record = fx.engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(1)); + assert_eq!(record.items_applied, Some(1000)); + + // The resume continues file 1 from item 1000: the engine skipped the + // 1000 already applied and streams only the rest, so nothing is + // re-applied and no duplicate arises (the `reapplying` flag the engine + // sets for the first resumed file is vestigial on this backend, since + // the progress is exact). + let resumed = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 1000, + reapplying: true, + }, + item_stream(3500, 4000), + ) + .await + .expect("resume file 1"); + assert_eq!(resumed, 500); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 4000); + let record = fx.engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(1)); + assert_eq!(record.items_applied, Some(1500)); + + // A file whose items the target already holds (a backup that holds a + // key twice) fails the file; the unique `_id` raises E11000. + let dup = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 2, + start_at: 0, + reapplying: false, + }, + item_stream(2500, 3500), + ) + .await; + assert!( + matches!(&dup, Err(StorageError::Validation(m)) if m.contains("holds a key twice")), + "{dup:?}" + ); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 4000); + drop(claim); + fx.finish().await; +} + +/// A claim lost between two `insert_many` calls of one file (the record's +/// generation moved by a takeover while the load ran): the read before the +/// next call finds the record no longer this claim's, the file ends as a +/// lost claim, and the second call's documents never land. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_calls() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_doc, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("lost"), + ®istration(&backup, None), + ) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "lost").await.expect("target")); + // The stream hands the first call's documents, then moves the record's + // generation when asked for the first item of the second call. + let tables = fx.engine.catalog_db.collection::("tables"); + let moved_key = key.clone(); + let stream: BoxStream<'static, Result> = Box::pin(futures::stream::unfold( + 0usize, + move |i| { + let tables = tables.clone(); + let key = moved_key.clone(); + async move { + if i >= 2 * INSERT_MANY_DOCUMENTS + 500 { + return None; + } + if i == INSERT_MANY_DOCUMENTS { + tables + .update_one( + doc! { "_id": { "account_id": &key.account_id, "table_name": &key.table_name } }, + doc! { "$inc": { RESTORE_JOB_GENERATION: 1_i64 } }, + ) + .await + .expect("another owner takes the record"); + } + Some((Ok(manifest_item(i)), i + 1)) + } + }, + )); + let out = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream, + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the second batch's fence finds another owner: {out:?}" + ); + assert_eq!( + count(&fx.engine, &data_collection_name(&key.table_id)).await, + INSERT_MANY_DOCUMENTS as u64, + "the first batch's documents alone landed; the second batch is fenced out" + ); + // The first batch committed its documents and its progress in one + // transaction, so the record carries that batch's progress; the second + // batch, fenced out, recorded nothing. + let record = fx.engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + record.file_index, + Some(0), + "the first batch's progress is recorded" + ); + assert_eq!(record.items_applied, Some(INSERT_MANY_DOCUMENTS as u64)); + drop(claim); + fx.finish().await; +} + +/// `bulk_load` refuses an `N` sort key beyond Decimal128's precision with +/// the write path's `ValidationException`, and a 38-digit non-key number +/// survives a restore as a string. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_refuses_a_long_sort_key_and_keeps_a_long_number() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dec"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "dec").await.expect("target")); + + let long_sk: Item = [ + ("pk".to_owned(), AttributeValue::S("p".to_owned())), + // 35 significant digits: beyond Decimal128. + ( + "sk".to_owned(), + AttributeValue::N("12345678901234567890123456789012345".to_owned()), + ), + ("gpk".to_owned(), AttributeValue::S("g".to_owned())), + ("lsk".to_owned(), AttributeValue::N("1".to_owned())), + ] + .into_iter() + .collect(); + let refused = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(vec![Ok(long_sk)]).boxed(), + ) + .await; + assert!( + matches!(&refused, Err(StorageError::Validation(m)) if m.contains("Decimal128")), + "{refused:?}" + ); + + // A good item with a 38-digit non-key number restores intact. + fx.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 1), + ) + .await + .expect("a good item"); + let stored = fx + .engine + .data_db + .collection::(&data_collection_name(&key.table_id)) + .find_one(doc! {}) + .await + .unwrap() + .expect("the item"); + let item = document_to_item(&stored).expect("item"); + assert_eq!( + item.get("big"), + Some(&AttributeValue::N( + "12345678901234567890123456789012345678".to_owned() + )), + "the 38-digit non-key number survives" + ); + drop(claim); + fx.finish().await; +} + +/// `finish_restore` hands the target to the backfill worker (`Pending`, +/// DONE + `restore_backfill_pending`), the backup stays in use until the +/// worker flips the table ACTIVE, and `list_dead_restore_jobs` leaves the +/// DONE record alone. A stale generation's `finish_restore` is `NotMatched`. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_finish_restore_hands_over_and_the_worker_flips_active() { + const FEW: usize = 300; + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("fin"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "fin").await.expect("target")); + fx.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, FEW), + ) + .await + .expect("load"); + + // A stale generation cannot finish it. + age_running_record(&fx.engine, "fin", Duration::from_secs(600)).await; + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, None) + .await + .unwrap() + .expect("takeover"); + assert_eq!( + fx.engine + .finish_restore(&key, &*claim, FEW as u64, 1) + .await + .unwrap(), + Finished::NotMatched(format!( + "the job record of fin is not RUNNING under generation {}, or the table is not \ + CREATING with table id {}", + claim.generation(), + key.table_id + )) + ); + + assert_eq!( + fx.engine + .finish_restore(&key, &*taken, FEW as u64, 1) + .await + .unwrap(), + Finished::Pending + ); + let doc = table_doc(&fx.engine, "fin").await.expect("target"); + assert_eq!(doc.get_str("table_status"), Ok("CREATING")); + assert!(doc.get_bool("restore_backfill_pending").unwrap_or(false)); + assert_eq!( + doc.get_document(RESTORE_JOB).unwrap().get_str("status"), + Ok("DONE") + ); + assert_eq!(doc.get_i64("item_count"), Ok(FEW as i64)); + + // Between the DONE write and the worker's flip the backup is in use and + // the DONE record is not dead. + assert_eq!( + fx.engine + .backup_in_use(&backup.key()) + .await + .unwrap() + .as_deref(), + Some("fin") + ); + assert!( + fx.engine + .list_dead_restore_jobs(STALE_AFTER) + .await + .unwrap() + .iter() + .all(|j| j.target.table_name != "fin"), + "the DONE+pending record is not dead" + ); + + // The worker flips it ACTIVE with every index populated. + let worker = spawn_worker(&fx.engine); + let active = wait_for_status(&fx.engine, "fin", "ACTIVE").await; + let table_id = active.get_str("table_id").expect("table_id"); + assert_eq!(active.get_i64("item_count"), Ok(FEW as i64)); + assert_eq!( + count(&fx.engine, &data_collection_name(table_id)).await, + FEW as u64 + ); + for (index_name, index_id, status) in index_ids(&fx.engine, table_id).await { + assert_eq!(status, "ACTIVE", "{index_name}"); + assert_eq!( + count(&fx.engine, &data_collection_name(&index_id)).await, + FEW as u64, + "index {index_name}" + ); + } + let job = active.get_document(RESTORE_JOB).expect("job record"); + assert_eq!(job.get_str("status"), Ok("DONE")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert!(!active.contains_key("restore_backfill_pending")); + assert_eq!(fx.engine.backup_in_use(&backup.key()).await.unwrap(), None); + worker.abort(); + drop(claim); + drop(taken); + fx.finish().await; +} + +/// The `restore_mongodb_after_insert_many` kill point and the file-index +/// write's fence: a claim taken over while `bulk_load` is paused between its +/// `insert_many` and the record write finds the write matched nothing (the +/// generation moved), so the lost claim's progress never lands. Runs only +/// under `test-hooks` (the pause point is compiled then) and must run alone +/// (`--test-threads=1`): the pause is driven by a process-wide environment +/// variable. +#[cfg(feature = "test-hooks")] +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set and --features test-hooks, run with --test-threads=1"] +async fn a_takeover_during_bulk_load_fences_the_file_index_write() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("fence"), + ®istration(&backup, None), + ) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "fence").await.expect("target")); + + // Take the record over before the owner's batch runs: its heartbeat is + // aged and a second owner claims it under a fresh generation. + age_running_record(&fx.engine, "fence", Duration::from_secs(600)).await; + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, None) + .await + .expect("takeover") + .expect("a stale record is taken over"); + + // The old claim's load is fenced inside the batch transaction: the + // `find_one_and_update` on the record (RUNNING under the old generation) + // matches nothing, so neither the documents nor the progress of the + // batch are written. + let out = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 10), + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the fenced-out load must not write the batch: {out:?}" + ); + assert_eq!( + count(&fx.engine, &data_collection_name(&key.table_id)).await, + 0, + "no document lands after the takeover" + ); + // The successor's record never advanced its file_index. + assert_eq!( + fx.engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .file_index, + None, + "the lost claim's batch wrote no progress" + ); + + drop(taken); + fx.finish().await; +} + /// The registration itself writes the record's first heartbeat on the /// server's clock (`$$NOW`, in the registration's transaction): a target /// registered with a first heartbeat ten minutes behind, as a host whose @@ -3380,6 +4452,7 @@ async fn live_a_registration_writes_its_first_heartbeat_on_the_server_clock() { restore_date_time: bson::DateTime::now(), owner_generation: 7, first_heartbeat: behind, + record: crate::backup_engine::RestoreRecord::Legacy, session: None, }; fx.engine diff --git a/crates/storage-mongodb/src/lib.rs b/crates/storage-mongodb/src/lib.rs index 14c735a72..cd061d0da 100644 --- a/crates/storage-mongodb/src/lib.rs +++ b/crates/storage-mongodb/src/lib.rs @@ -23,6 +23,7 @@ pub mod migrations; mod operations; mod password_hash_store; pub mod pushdown; +mod restore_engine; mod snapshot_engine; mod stream_engine; mod table_engine; diff --git a/crates/storage-mongodb/src/migrations/mod.rs b/crates/storage-mongodb/src/migrations/mod.rs index 89f8f18f7..1f2cc9aa7 100644 --- a/crates/storage-mongodb/src/migrations/mod.rs +++ b/crates/storage-mongodb/src/migrations/mod.rs @@ -160,6 +160,11 @@ pub fn migration(id: MigrationId) -> Migration { m0006_backup_catalog_index::SOURCE, )], }, + MigrationId::RestoreProgress => Migration::NotApplicable( + "a store-backed restore records its progress on the target's restore job record \ + (restore_job.file_index); the restore_progress table is PostgreSQL's data \ + database's", + ), } } diff --git a/crates/storage-mongodb/src/migrations/tests.rs b/crates/storage-mongodb/src/migrations/tests.rs index 651ad34fe..7edae3bfb 100644 --- a/crates/storage-mongodb/src/migrations/tests.rs +++ b/crates/storage-mongodb/src/migrations/tests.rs @@ -39,7 +39,7 @@ fn every_migration_records_a_row_in_the_catalog_and_names_its_stores() { } #[test] -fn every_body_is_code_and_three_and_four_do_not_apply() { +fn every_body_is_code_and_three_four_and_seven_do_not_apply() { for id in MigrationId::ALL { for body in migration(id).bodies() { assert!(matches!(body.body, Body::Code { .. }), "{id}"); @@ -48,6 +48,7 @@ fn every_body_is_code_and_three_and_four_do_not_apply() { for id in [ MigrationId::SortKeyDefinitions, MigrationId::IndexTableBaseKey, + MigrationId::RestoreProgress, ] { let Migration::NotApplicable(reason) = migration(id) else { panic!("{id} applies on MongoDB"); diff --git a/crates/storage-mongodb/src/restore_engine.rs b/crates/storage-mongodb/src/restore_engine.rs new file mode 100644 index 000000000..8f5b06d7c --- /dev/null +++ b/crates/storage-mongodb/src/restore_engine.rs @@ -0,0 +1,372 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on `MongoDB` +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target's data collection, the per-file record, the hand-over to +//! the index backfill worker and the count. +//! +//! The target's documents are built by [`item_to_document`], the write +//! path's function, so a restored table is laid out as a natively created +//! one and refuses what the write path refuses (an `N` sort key beyond +//! Decimal128's 34 significant digits is the write path's `Validation` +//! error, and fails the file). Base documents only: the target's indexes +//! are created CREATING with it and the GSI backfill worker fills them from +//! the base collection and flips the table ACTIVE once every index has +//! left CREATING, so `finish_restore` is the hand-over +//! ([`crate::MongoEngine::hand_over_to_backfill`]) and answers +//! [`Finished::Pending`]. +//! +//! One file is `insert_many` calls of [`INSERT_MANY_DOCUMENTS`] documents, +//! each one session transaction with the record's progress write: the +//! transaction first writes the record (`find_one_and_update` on the +//! target's `_id`, `table_id`, `restore_job.status = RUNNING` and +//! `restore_job.owner_generation` equal to the claim's, setting +//! `restore_job.file_index`, `restore_job.items_applied` to the file's +//! running total and `heartbeat_at` to the server's clock), and ends the +//! file as a lost claim when that write matches nothing; then inserts the +//! batch in the same transaction and commits. The record write is the +//! fence: a takeover is a write on the same document, so a writer whose +//! record another instance took over while it stalled finds its transaction +//! aborted for the write conflict (re-run, it meets the other generation and +//! stops) or its record write matching nothing, and no batch of it lands +//! after the takeover, as PostgreSQL's and SQLite's loads condition their +//! writes. The progress is per batch and in the batch's transaction, so a +//! job that dies leaves no documents its record does not account for: a +//! resume starts after `items_applied`, and `BulkLoadFile::reapplying` is +//! not read (nothing of a batch lands without its record). A duplicate key +//! (E11000, the data collection's unique `_id`) is the backup's and fails +//! the file, as any other write error does. The record carries the +//! progress, so `read_restore_progress` and `sweep_restore_progress` keep +//! their defaults. + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement}; +use extenddb_storage::backup_catalog::{ + JobClaim, JobStatus, RESTORE_MONGODB_AFTER_INSERT_MANY, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; +use mongodb::bson::{self, Document, doc}; + +use crate::MongoEngine; +use crate::backup_catalog::{ + RESTORE_JOB_FILE_INDEX, RESTORE_JOB_GENERATION, RESTORE_JOB_HEARTBEAT, + RESTORE_JOB_ITEMS_APPLIED, RESTORE_JOB_STATUS, generation_field, literal_set, +}; +use crate::data::{data_collection_name, item_to_document}; +use crate::data_engine::{CommitOutcome, TRANSIENT_RETRY_ATTEMPTS, TxErr, backoff_sleep}; + +/// Documents per `insert_many` call. Not a message bound: 1,000 documents at +/// the item size limit are about 390 MiB, and the driver splits one call by +/// the server's message and write-batch bounds on its own. It is the unit a +/// half-applied file is re-applied in, and the unit between two reads of the +/// record's claim. +pub(crate) const INSERT_MANY_DOCUMENTS: usize = 1000; + +/// The server's duplicate-key error code. +const DUPLICATE_KEY: i32 = 11000; + +fn internal(e: impl std::fmt::Display) -> StorageError { + StorageError::Internal(e.to_string()) +} + +/// The key shape of a CREATING restore target, read from its catalog +/// document. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +impl MongoEngine { + /// The target's catalog document, when `key` names a CREATING restore + /// target with this table id. + async fn restore_target_doc(&self, key: &RestoreTargetKey) -> Result { + self.catalog_db + .collection::("tables") + .find_one(doc! { + "_id": { "account_id": &key.account_id, "table_name": &key.table_name }, + "table_id": &key.table_id, + "table_status": "CREATING", + "restore_source_backup_arn": { "$exists": true }, + }) + .await + .map_err(internal)? + .ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + }) + } + + /// Write `batch` into the target's data collection in one transaction + /// with the record's progress (module doc) and clear it; the documents + /// the server took. `applied_total` is the file's running total once + /// the batch lands (`start_at` plus every batch of this call). `Err` on + /// a lost claim (the record is not RUNNING under `generation`, or a + /// takeover's write conflicted with this transaction), on a duplicate + /// key (the backup's), and on any other error. + async fn insert_restore_batch( + &self, + collection: &mongodb::Collection, + target: &RestoreTargetKey, + generation: u64, + file_index: u32, + applied_total: u64, + batch: &mut Vec, + ) -> Result { + let tables = self.catalog_db.collection::("tables"); + let fence = doc! { + "_id": { "account_id": &target.account_id, "table_name": &target.table_name }, + "table_id": &target.table_id, + RESTORE_JOB_STATUS: JobStatus::Running.as_str(), + RESTORE_JOB_GENERATION: generation_field(generation), + }; + let progress = vec![doc! { "$set": literal_set(doc! { + RESTORE_JOB_FILE_INDEX: i64::from(file_index), + RESTORE_JOB_ITEMS_APPLIED: i64::try_from(applied_total).unwrap_or(i64::MAX), + RESTORE_JOB_HEARTBEAT: "$$NOW", + }) }]; + let lost = || { + StorageError::Internal(format!( + "restore of table {}: the claim on the target was lost while a data file \ + loaded (the record is not RUNNING under generation {generation}); the \ + record's owner finishes it", + target.table_name + )) + }; + let mut session = self.client.start_session().await.map_err(internal)?; + let tx_options = self.write_transaction_options(); + for attempt in 0..TRANSIENT_RETRY_ATTEMPTS { + session + .start_transaction() + .with_options(tx_options.clone()) + .await + .map_err(internal)?; + let body: Result<(), TxErr> = async { + let fenced = tables + .find_one_and_update(fence.clone(), progress.clone()) + .session(&mut session) + .await + .map_err(TxErr::from)?; + if fenced.is_none() { + return Err(TxErr::Fatal(lost())); + } + if batch.is_empty() { + return Ok(()); + } + collection + .insert_many(batch.iter().cloned()) + .session(&mut session) + .await + .map_err(|e| { + if is_duplicate_key_anywhere(&e) { + // The engine prefixes the file's key, so the + // file is not named here. + TxErr::Fatal(StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds \ + a key twice): {e}" + ))) + } else { + TxErr::from(e) + } + })?; + Ok(()) + } + .await; + match body { + Ok(()) => match self.commit_write_transaction(&mut session, None).await? { + CommitOutcome::Committed => { + let sent = batch.len(); + batch.clear(); + return Ok(u64::try_from(sent).unwrap_or(u64::MAX)); + } + CommitOutcome::RerunBody => backoff_sleep(attempt).await, + }, + Err(TxErr::Transient) => { + let _ = session.abort_transaction().await; + backoff_sleep(attempt).await; + } + Err(TxErr::Fatal(e)) => { + let _ = session.abort_transaction().await; + return Err(e); + } + } + } + Err(StorageError::Internal(format!( + "restore of table {}: a batch of data file {file_index} did not commit after \ + {TRANSIENT_RETRY_ATTEMPTS} conflicting attempts", + target.table_name + ))) + } +} + +/// Whether `e` carries the server's duplicate-key code, as an insert's +/// write error, a bulk write's, or a command error inside a transaction. +fn is_duplicate_key_anywhere(e: &mongodb::error::Error) -> bool { + match &*e.kind { + mongodb::error::ErrorKind::InsertMany(failure) => failure + .write_errors + .as_ref() + .is_some_and(|errors| errors.iter().any(|w| w.code == DUPLICATE_KEY)), + mongodb::error::ErrorKind::Command(c) => c.code == DUPLICATE_KEY, + _ => crate::data_engine::is_duplicate_key(e), + } +} + +impl RestoreEngine for MongoEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + // The shape from the target's catalog document; each batch's + // transaction writes the record under the claim before its + // inserts (`insert_restore_batch`), which is the fence. + let generation = claim.generation(); + let table_doc = self.restore_target_doc(target).await?; + let shape = TargetShape { + key_schema: bson::from_bson( + table_doc + .get("key_schema") + .cloned() + .ok_or_else(|| internal("restore target is missing key_schema"))?, + ) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: bson::from_bson( + table_doc + .get("attribute_definitions") + .cloned() + .ok_or_else(|| { + internal("restore target is missing attribute_definitions") + })?, + ) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }; + let collection = self + .data_db + .collection::(&data_collection_name(&target.table_id)); + // The test-hook gate the in-catalog copy also honours: it holds + // the restore after the target's CREATING index metadata exists + // and before the first insert of its first file, so a test can + // run a backfill worker tick through that window and see that + // the worker completes no index of a restore whose base load has + // not started. Inert without the `test-hooks` feature and unless + // a test armed it. + if file.file_index == 0 && file.start_at == 0 { + self.wait_for_gsi_backfill_test_gate(&target.table_name) + .await?; + } + + let mut batch: Vec = Vec::with_capacity(INSERT_MANY_DOCUMENTS); + let mut applied: u64 = 0; + while let Some(item) = items.next().await { + let item = item?; + batch.push(item_to_document( + &item, + &shape.key_schema, + &shape.attribute_definitions, + )?); + if batch.len() == INSERT_MANY_DOCUMENTS { + let total = file.start_at + applied + batch.len() as u64; + applied += self + .insert_restore_batch( + &collection, + target, + generation, + file.file_index, + total, + &mut batch, + ) + .await?; + restore_pause_point(RESTORE_MONGODB_AFTER_INSERT_MANY).await; + } + } + // The file's last batch, or, for a file with no item left to + // apply, the file's record alone (the fence write with the + // running total and no insert). + let total = file.start_at + applied + batch.len() as u64; + applied += self + .insert_restore_batch( + &collection, + target, + generation, + file.file_index, + total, + &mut batch, + ) + .await?; + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let handed_over = self + .hand_over_to_backfill(target, claim.generation(), item_count, Some(size_bytes)) + .await?; + if handed_over { + Ok(Finished::Pending) + } else { + Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING with table id {}", + target.table_name, + claim.generation(), + target.table_id + ))) + } + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data collection; the key's account and name + // are checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_doc(key).await?; + self.data_db + .collection::(&data_collection_name(&key.table_id)) + .count_documents(doc! {}) + .await + .map_err(internal) + }) + } + + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The record's `ddl_done` marker, set after the last index + // metadata document and data collection were created + // (`create_table_impl_inner`); a record without it is one whose + // owner died between the registration's commit and the end of + // its DDL, and its collections and indexes are not the + // registration's. The catalog document is read first, so a bare + // id reads no other account's table. + let table_doc = self.restore_target_doc(key).await?; + Ok(table_doc + .get_document(crate::backup_catalog::RESTORE_JOB) + .ok() + .and_then(|job| job.get_bool("ddl_done").ok()) + .unwrap_or(false)) + }) + } +} diff --git a/crates/storage-mongodb/src/snapshot_engine.rs b/crates/storage-mongodb/src/snapshot_engine.rs index e3292a84e..b1201580a 100644 --- a/crates/storage-mongodb/src/snapshot_engine.rs +++ b/crates/storage-mongodb/src/snapshot_engine.rs @@ -74,6 +74,20 @@ //! `paging::SNAPSHOT_FETCH_BYTES` (64 MiB): the reply limit is already under //! that budget, and the byte cut the SQL backends make exists because their //! row-bounded fetches have no such limit. +//! +//! A snapshot read serves the majority-committed snapshot, which can trail +//! what a `readConcern: local` read of the primary already returns (the +//! catalog handle's level, so `DescribeTable`'s): a table flipped ACTIVE a +//! few milliseconds earlier can be missing, or still CREATING, at the +//! snapshot's time, and a CreateBackup sent as soon as DescribeTable said +//! ACTIVE would be refused as a table that does not exist. So an open whose +//! snapshot finds the table missing or not ACTIVE reads the `tables` +//! document at `local`; when that read shows it ACTIVE, the open starts over +//! with a fresh session after a short sleep (5 ms, doubling to 100 ms), +//! for at most [`MAJORITY_CATCH_UP`], and answers the snapshot's refusal +//! once a local read no longer shows the table ACTIVE or the bound has +//! passed. The bound is far above the lag a w:majority write leaves (one +//! journal commit, 100 ms at most by default). use extenddb_core::types::Item; use extenddb_storage::error::StorageError; @@ -105,6 +119,10 @@ const STANDALONE_SERVER: &str = "MongoDB refused the first read at readConcern s standalone server cannot serve snapshot reads, and table snapshots need a replica set \ member"; +/// How long an open waits for the majority-committed snapshot to show a +/// table that a `local` read shows ACTIVE (module doc). +const MAJORITY_CATCH_UP: std::time::Duration = std::time::Duration::from_secs(2); + /// Why an open cannot proceed when the first read succeeded but the session /// reports no snapshot time. const NO_SNAPSHOT_TIME: &str = "MongoDB served the first read without a snapshot time: a read \ @@ -119,13 +137,51 @@ impl SnapshotEngine for MongoEngine { let account_id = account_id.to_owned(); let table_name = table_name.to_owned(); Box::pin(async move { - self.open_table_snapshot_impl(&account_id, &table_name) - .await + let deadline = tokio::time::Instant::now() + MAJORITY_CATCH_UP; + let mut delay = std::time::Duration::from_millis(5); + loop { + match self + .open_table_snapshot_impl(&account_id, &table_name) + .await + { + Err(StorageError::TableNotFound(_) | StorageError::TableNotActive(_)) + if tokio::time::Instant::now() < deadline + && self + .table_is_active_locally(&account_id, &table_name) + .await? => + { + tokio::time::sleep(delay).await; + delay = (delay * 2).min(std::time::Duration::from_millis(100)); + } + outcome => return outcome, + } + } }) } } impl MongoEngine { + /// Whether a `local` read of the `tables` document shows the table + /// ACTIVE: what `DescribeTable` reads, and the snapshot's time may trail. + async fn table_is_active_locally( + &self, + account_id: &str, + table_name: &str, + ) -> Result { + Ok(self + .catalog_db + .collection::("tables") + .find_one(doc! { + "_id": { "account_id": account_id, "table_name": table_name }, + "table_status": "ACTIVE", + }) + .read_concern(ReadConcern::local()) + .projection(doc! { "_id": 1 }) + .await + .map_err(|e| map_mongo_error(&e))? + .is_some()) + } + async fn open_table_snapshot_impl( &self, account_id: &str, @@ -647,6 +703,78 @@ mod tests { .await; } + /// A table a `local` read shows ACTIVE opens although the majority- + /// committed snapshot does not show it yet. The flip to ACTIVE is + /// written at `w: 1` without the journal, so the majority snapshot + /// trails it by up to one journal commit; the test observes that + /// precondition with a snapshot read of its own before each open, and + /// asserts that it saw it at least once. + #[tokio::test] + async fn an_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active() { + let Some(uri) = uri() else { + eprintln!( + "SKIP an_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active: \ + no MongoDB" + ); + return; + }; + let _hooks = hooks().await; + let engine = fixture(&uri).await; + engine + .create_table(ACCOUNT, snapshot_conformance::table_input("lagging")) + .await + .expect("create"); + snapshot_conformance::wait_active(&*engine, ACCOUNT, "lagging", || async { + let _ = + extenddb_storage::WorkerStore::process_control_plane_transitions(&*engine).await; + }) + .await; + let tables = engine.catalog_db.collection::("tables"); + let id = doc! { "_id": { "account_id": ACCOUNT, "table_name": "lagging" } }; + let unjournaled = mongodb::options::WriteConcern::builder() + .w(mongodb::options::Acknowledgment::Nodes(1)) + .journal(false) + .build(); + let mut trailing = 0; + for _ in 0..20 { + set_updating(&engine, "lagging").await; + tables + .update_one(id.clone(), doc! { "$set": { "table_status": "ACTIVE" } }) + .write_concern(unjournaled.clone()) + .await + .expect("flip to ACTIVE"); + let mut session = engine + .client + .start_session() + .snapshot(true) + .await + .expect("session"); + let at_snapshot = tables + .find_one(id.clone()) + .read_concern(ReadConcern::snapshot()) + .session(&mut session) + .await + .expect("snapshot read") + .and_then(|d| d.get_str("table_status").ok().map(str::to_owned)); + if at_snapshot.as_deref() != Some("ACTIVE") { + trailing += 1; + } + let snapshot = engine + .open_table_snapshot(ACCOUNT, "lagging") + .await + .expect("a table a local read shows ACTIVE opens"); + assert_eq!( + snapshot.description.table_status, + extenddb_core::types::TableStatus::Active + ); + } + eprintln!("the snapshot trailed the local read in {trailing} of 20 flips"); + assert!( + trailing > 0, + "precondition: no flip left the majority snapshot behind the local read" + ); + } + /// A document the decoder cannot read ends the stream at its error, as on /// the other backends: nothing after it, so a consumer cannot mistake the /// rest for the whole table. diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index 32b61ef7d..0ad4bd12b 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -267,11 +267,28 @@ impl MongoEngine { let tables_coll = self.catalog_db.collection::("tables"); let for_restore = restore.is_some(); - let mut session = restore.and_then(|registration| registration.session.as_deref_mut()); - let inserted = match session.as_deref_mut() { + let mut restore = restore; + let inserted = match restore + .as_deref_mut() + .and_then(|registration| registration.session.as_deref_mut()) + { Some(session) => tables_coll.insert_one(table_doc).session(session).await, None => tables_coll.insert_one(table_doc).await, }; + // A manifest record's first heartbeat is rewritten with the server's + // clock in the same transaction (nothing for a legacy one). + let inserted = match inserted { + Ok(_) => match restore.as_deref_mut() { + Some(registration) => { + registration + .stamp_server_clock(&tables_coll, &table_id) + .await + } + None => Ok(()), + }, + Err(e) => Err(e), + }; + let mut session = restore.and_then(|registration| registration.session.as_deref_mut()); if let Err(e) = inserted { if let Some(session) = session { let _ = session.abort_transaction().await; @@ -565,6 +582,24 @@ impl MongoEngine { } } + // A restore target's DDL is complete here: the data collection, its + // unique key index and every GSI and LSI metadata document and data + // collection exist. The record was committed before any of them (the + // registration's transaction ends at the catalog insert), so the + // record carries `restore_job.ddl_done = false` until this write, + // and the reconciler's resume refuses a record left unmarked by an + // owner that died in between (`restore_target_ready`), rather than + // resuming into a target that is missing its indexes. + if for_restore { + tables_coll + .update_one( + doc! { "table_id": &table_id }, + doc! { "$set": { crate::backup_catalog::RESTORE_JOB_DDL_DONE: true } }, + ) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + // Derive the SSEDescription from the SSESpecification, mirroring the // postgres backend (table_helpers.rs:329-346). The specification's // `Enabled: true` becomes a KMS-status ENABLED description with a diff --git a/crates/storage-mongodb/src/ttl_worker.rs b/crates/storage-mongodb/src/ttl_worker.rs index 8864471d0..81de24c02 100644 --- a/crates/storage-mongodb/src/ttl_worker.rs +++ b/crates/storage-mongodb/src/ttl_worker.rs @@ -87,10 +87,12 @@ pub(crate) async fn stream_record_cleanup_worker( /// says the table has no indexes; an invalidated in-flight observation /// cannot publish that stale no-index result. /// -/// A restore target whose copy stopped before its hand-over is not this +/// A restore target whose load stopped before its hand-over is not this /// worker's: its job record keeps a stale heartbeat, and the backup -/// reconciler fails the record and removes the target. Returns once -/// `shutdown` fires, at its next pause between cycles. +/// reconciler acts on it, by its kind: a legacy (in-catalog) record is +/// failed and its target removed; a store-backed (`manifest`) record is +/// taken over and its load resumed, and the hand-over comes from the resume. +/// Returns once `shutdown` fires, at its next pause between cycles. pub(crate) async fn gsi_backfill_worker(storage: Arc, shutdown: CancellationToken) { while sleep_or_shutdown(&shutdown, GSI_BACKFILL_INTERVAL).await { let indexes_coll = storage.catalog_db.collection::("indexes"); @@ -287,9 +289,11 @@ async fn restore_backfill_is_ready( } /// Complete restored tables after every restored GSI/LSI has been populated. -/// The restore request records `restore_backfill_pending` only after the base -/// `$out` copy succeeds, so this remains safe to retry after a process restart -/// or a client disconnect during index backfill. +/// `restore_backfill_pending` is set only once every base document is in +/// place: by the legacy copy after its `$out` succeeds, and by a store-backed +/// restore's `finish_restore` after its count check (`hand_over_to_backfill`), +/// so this remains safe to retry after a process restart or a client +/// disconnect during index backfill. async fn finish_ready_restores(storage: &MongoEngine) { let tables_coll = storage.catalog_db.collection::("tables"); let mut cursor = match tables_coll diff --git a/crates/storage-mongodb/tests/migrations_fresh.rs b/crates/storage-mongodb/tests/migrations_fresh.rs index 10a8a2f33..9b5270035 100644 --- a/crates/storage-mongodb/tests/migrations_fresh.rs +++ b/crates/storage-mongodb/tests/migrations_fresh.rs @@ -322,6 +322,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { (4, d.target.catalog.clone()), (5, d.target.catalog.clone()), (6, d.target.catalog.clone()), + (7, d.target.catalog.clone()), ] ); let not_applicable: Vec = report @@ -330,7 +331,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { .filter(|s| s.outcome == Outcome::NotApplicable) .map(|s| s.id) .collect(); - assert_eq!(not_applicable, [3, 4], "{report:?}"); + assert_eq!(not_applicable, [3, 4, 7], "{report:?}"); assert_eq!( d.status().await, CatalogStatus::Current { @@ -348,6 +349,7 @@ async fn a_fresh_store_gets_every_migration_and_the_expected_definitions() { (4, Outcome::NotApplicable), (5, Outcome::Applied), (6, Outcome::Applied), + (7, Outcome::NotApplicable), ] ); assert!( @@ -442,8 +444,8 @@ async fn a_second_run_applies_nothing_and_reconverge_keeps_the_rows() { let reconverged = d.reconverge().await.expect("reconverge"); // Migration 1's steps in the catalog and the data database, and - // migrations 5's and 6's in the catalog: 2 is not convergent, and 3 and 4 - // do not apply on MongoDB. + // migrations 5's and 6's in the catalog: 2 is not convergent, and 3, 4 + // and 7 do not apply on MongoDB. let ran: Vec<(u32, String)> = reconverged .applied .iter() @@ -533,19 +535,19 @@ async fn a_changed_checksum_a_newer_catalog_and_an_older_release_are_reported() doc! { "_id": id, "name": "later", "kind": kind, "checksum": "c", "outcome": "applied", "notes": "", "applied_at": bson::DateTime::now(), "applied_by": "9.9.9" } }; - ledger.insert_one(row(7, "breaking")).await.expect("insert"); + ledger.insert_one(row(8, "breaking")).await.expect("insert"); let status = d.status().await; assert_eq!(status.label(), "AheadIncompatible"); assert_eq!(status.exit_code(), exit::AHEAD_INCOMPATIBLE); ledger - .delete_one(doc! { "_id": 7_i64 }) + .delete_one(doc! { "_id": 8_i64 }) .await .expect("delete"); - ledger.insert_one(row(7, "additive")).await.expect("insert"); + ledger.insert_one(row(8, "additive")).await.expect("insert"); assert_eq!(d.status().await.label(), "AheadCompatible"); assert!(d.migrate().await.expect("migrate").applied.is_empty()); ledger - .delete_one(doc! { "_id": 7_i64 }) + .delete_one(doc! { "_id": 8_i64 }) .await .expect("delete"); @@ -626,6 +628,6 @@ async fn a_ledger_document_this_binary_cannot_read_stops_the_status() { .expect_err("an error"); migrations.close().await; assert!(error.to_string().contains("\"sideways\""), "{error}"); - assert_eq!(MigrationId::LATEST.number(), 6); + assert_eq!(MigrationId::LATEST.number(), 7); d.drop_databases().await; } diff --git a/crates/storage-mongodb/tests/migrations_locks.rs b/crates/storage-mongodb/tests/migrations_locks.rs index 558128782..64be558be 100644 --- a/crates/storage-mongodb/tests/migrations_locks.rs +++ b/crates/storage-mongodb/tests/migrations_locks.rs @@ -53,7 +53,7 @@ async fn two_migrators_started_together_apply_once_and_the_other_finds_nothing() applied.sort_unstable(); assert_eq!( applied, - [0, 7], + [0, 8], "one applies every step and the other finds nothing" ); assert_eq!( diff --git a/crates/storage-mongodb/tests/migrations_released.rs b/crates/storage-mongodb/tests/migrations_released.rs index 5108b49f4..d85f0342e 100644 --- a/crates/storage-mongodb/tests/migrations_released.rs +++ b/crates/storage-mongodb/tests/migrations_released.rs @@ -161,7 +161,7 @@ async fn every_released_catalog_upgrades_to_the_definitions_of_a_fresh_install() panic!("{state} is not behind"); }; assert_eq!(version, None, "{state} has no ledger"); - assert_eq!(pending.len(), 7, "{state}"); + assert_eq!(pending.len(), 8, "{state}"); let report = d.migrate().await.expect("migrate"); assert_eq!( (report.from, report.to), diff --git a/crates/storage-postgres/data_migrations/0007_restore_progress.sql b/crates/storage-postgres/data_migrations/0007_restore_progress.sql new file mode 100644 index 000000000..417530889 --- /dev/null +++ b/crates/storage-postgres/data_migrations/0007_restore_progress.sql @@ -0,0 +1,18 @@ +-- Copyright 2026 ExtendDB contributors +-- SPDX-License-Identifier: Apache-2.0 +-- +-- Migration 7, restore_progress, PostgreSQL data database. +-- +-- The progress record of a store-backed restore: one row per data file the +-- restore applied into its CREATING target, written in the data transaction +-- that wrote the file's rows, so a file is recorded exactly when its rows +-- are durable. The catalog's `table_restores` row names the target; this +-- table lives beside the target's rows because the two databases cannot +-- share a transaction. The restore's end deletes the target's rows, and the +-- reconciler removes rows of targets no RUNNING restore owns. +CREATE TABLE IF NOT EXISTS restore_progress ( + table_id TEXT NOT NULL, + file_index BIGINT NOT NULL, + items_applied BIGINT NOT NULL, + PRIMARY KEY (table_id, file_index) +); diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs index d7109a12d..d973eaf00 100644 --- a/crates/storage-postgres/src/backup_catalog.rs +++ b/crates/storage-postgres/src/backup_catalog.rs @@ -2,7 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 //! The catalog index on `PostgreSQL`: `backup_index`, the restore job fields -//! of `table_restores`, and `backup_maintenance`. +//! of `table_restores`, and `backup_maintenance`. The data-plane side of a +//! store-backed restore is `restore_engine.rs`. //! //! The claim of a backup job is `pg_try_advisory_lock(hashtextextended( //! 'backup:' || backup_id, RESTORE_LOCK_SEED))` on a dedicated, non-pooled @@ -42,10 +43,12 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -55,8 +58,10 @@ use tokio::sync::Mutex; use crate::PostgresEngine; use crate::backup_engine::{ - ClaimSession, backup_lock_key, generation_column, try_claim, try_claim_with, + ClaimSession, RESTORE_HEARTBEAT, backup_lock_key, generation_column, restore_lock_key, + try_claim, try_claim_with, }; +use crate::create_table::RestoreSource; /// The name of a CREATING restore target registered from `backup_arn`, if /// any: the one in-use rule DeleteBackup, a restore's registration and @@ -300,6 +305,174 @@ impl JobClaim for PostgresBackupClaim { } } +/// A store-backed restore job's handle on its `table_restores` record: the +/// advisory lock on the target name (`restore_lock_key`) on the claim +/// session, the generation, and the pool for reads. The heartbeat and the +/// status writes go through the session, so a session the server dropped +/// surfaces as a lost heartbeat or an unknown write outcome. +struct PostgresRestoreClaim { + session: Mutex, + generation: u64, + table_id: String, + pool: sqlx::PgPool, +} + +impl JobClaim for PostgresRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let mut session = self.session.lock().await; + let confirmed = session + .confirm(RESTORE_HEARTBEAT, &[&self.table_id]) + .await + .map_err(|e| StorageError::Internal(format!("claim session: {e}")))?; + Ok(if confirmed { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let mut session = self.session.lock().await; + let written = sqlx::query( + "UPDATE table_restores SET status = $3, \ + failure_reason = COALESCE($4, failure_reason), \ + completed_at = CASE WHEN $5 THEN now() ELSE completed_at END \ + WHERE table_id = $1 AND owner_generation = $2 AND status = $6", + ) + .bind(&self.table_id) + .bind(generation_column(self.generation)) + .bind(to.as_str()) + .bind(fields.failure_reason.as_deref()) + .bind(fields.completed) + .bind(from.as_str()) + .execute(session.connection()) + .await; + match written { + Ok(result) if result.rows_affected() == 1 => Ok(WriteOutcome::Matched), + Ok(_) => Ok(WriteOutcome::NotMatched), + Err(e) if outcome_unknown(&e) => Ok(WriteOutcome::Unknown), + Err(e) => Err(db_err(e)), + } + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let row: Option<(String, i64, f64, Option)> = sqlx::query_as( + "SELECT status, owner_generation, EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8, \ + failure_reason FROM table_restores WHERE table_id = $1", + ) + .bind(&self.table_id) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (status, generation, heartbeat_at, failure_reason) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.table_id + )) + })?; + Ok(JobRow { + status: JobStatus::parse(&status)?, + owner_generation: u64::try_from(generation).unwrap_or_default(), + heartbeat_at, + failure_reason, + }) + }) + } +} + +/// The columns of one restore job record, joined with its table, in the +/// order [`RestoreJobTuple`] reads them. +const RESTORE_ROW_COLUMNS: &str = "r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ + r.kind, r.status, r.owner_generation, EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8, \ + r.store_kind, r.store_root, r.store_id, r.spool_host, r.file_index, r.items_applied, \ + r.failure_reason, EXTRACT(EPOCH FROM r.completed_at)::FLOAT8"; + +/// One restore job record as the driver reads it. +#[allow(clippy::type_complexity)] +type RestoreJobTuple = ( + String, + String, + String, + String, + String, + String, + i64, + f64, + Option, + Option, + Option, + Option, + Option, + Option, + Option, + Option, +); + +fn restore_job_from_tuple(t: RestoreJobTuple) -> Result { + let ( + table_id, + account_id, + table_name, + source_backup_arn, + kind, + status, + owner_generation, + heartbeat_at, + store_kind, + store_root, + store_id, + spool_host, + file_index, + items_applied, + failure_reason, + completed_at, + ) = t; + let store = match (store_kind, store_root, store_id) { + (Some(kind), Some(store_root), Some(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root, + store_id, + }), + _ => None, + }; + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id, + table_name, + table_id, + }, + source_backup_arn, + kind: RestoreKind::parse(&kind)?, + status: JobStatus::parse(&status)?, + owner_generation: u64::try_from(owner_generation).unwrap_or_default(), + heartbeat_at, + store, + spool_host, + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason, + completed_at, + }) +} + impl BackupCatalog for PostgresEngine { fn catalog_id(&self) -> BoxFuture<'_, Result> { Box::pin(async move { @@ -464,10 +637,9 @@ impl BackupCatalog for PostgresEngine { // row must meet: it locks the same row before its own in-use // check, so the two order wholly one before the other (a // registration that commits first leaves this `InUse`; one that - // waits behind this finds the row DELETING). On this head the - // legacy registration locks the legacy `backups` row and no - // restore of a `backup_index` row exists; PR 9a's registration - // is the first to take this lock. + // waits behind this finds the row DELETING). `claim_restore` + // (`create_table.rs`, the manifest arm) takes this lock; the + // legacy registration locks the legacy `backups` row instead. let status: Option = sqlx::query_scalar( "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ FOR UPDATE", @@ -638,41 +810,36 @@ impl BackupCatalog for PostgresEngine { fn remove_restore_target<'a>( &'a self, key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { self.remove_restore_target_locked(key).await }) + } + + fn remove_restore_target_exclusive<'a>( + &'a self, + key: &'a RestoreTargetKey, ) -> BoxFuture<'a, Result> { Box::pin(async move { - // The key must name a CREATING restore target with this table id; - // a table re-created under the name has another id and is not - // touched. `abort_restore` then claims it from CREATING in one - // statement, so it and the ACTIVE flip exclude each other. - let is_target: bool = sqlx::query_scalar( - "SELECT EXISTS(SELECT 1 FROM tables t JOIN table_restores r \ - ON r.table_id = t.table_id WHERE t.table_id = $1 AND t.account_id = $2 \ - AND t.table_name = $3 AND t.table_status = 'CREATING')", + // The command path (`remove-target`) takes the target name's + // advisory lock first, the one the owner and `claim_restore_dead` + // take, so a removal and a takeover of the same target exclude + // each other, and a frozen owner whose claim session still holds + // the lock (its heartbeat stale but its session alive) keeps its + // target. The job's own failure path and the + // reconciler's resume call the plain `remove_restore_target` + // while they already hold this lock through their claim, so the + // lock lives only on this command-only variant. + let Some(session) = try_claim( + &self.pool, + &restore_lock_key(&key.account_id, &key.table_name), + None, ) - .bind(&key.table_id) - .bind(&key.account_id) - .bind(&key.table_name) - .fetch_one(&self.pool) - .await - .map_err(db_err)?; - if !is_target { + .await? + else { return Ok(false); - } - // Progress rows belong to a later release's restore path; a - // catalog without that table has none to delete. - let has_progress: bool = - sqlx::query_scalar("SELECT to_regclass('restore_progress') IS NOT NULL") - .fetch_one(&self.pool) - .await - .map_err(db_err)?; - if has_progress { - sqlx::query("DELETE FROM restore_progress WHERE table_id = $1") - .bind(&key.table_id) - .execute(&self.pool) - .await - .map_err(db_err)?; - } - self.abort_restore(&key.table_id).await + }; + let outcome = self.remove_restore_target_locked(key).await; + session.close().await; + outcome }) } @@ -691,32 +858,25 @@ impl BackupCatalog for PostgresEngine { // this build cannot read (a kind or status no release writes) is // logged and skipped, as ListBackups skips a row it cannot read, // so one such row does not stop the cycle. - #[allow(clippy::type_complexity)] - let rows: Vec<(String, String, String, String, String, String, i64, f64)> = - sqlx::query_as( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, \ - EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8 \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE (r.status = 'RUNNING' \ - AND r.heartbeat_at < now() - make_interval(secs => $1)) \ - OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ - AND COALESCE(r.completed_at, r.heartbeat_at) \ - < now() - make_interval(secs => $1)) \ - ORDER BY t.account_id, t.table_name", - ) - .bind(stale_after.as_secs_f64()) - .fetch_all(&self.pool) - .await - .map_err(db_err)?; - // A row this build cannot read (a kind or status no release - // writes) is logged and skipped, as ListBackups skips a row it - // cannot read, so one such row does not stop the cycle. + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (r.status = 'RUNNING' \ + AND r.heartbeat_at < now() - make_interval(secs => $1)) \ + OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ + AND COALESCE(r.completed_at, r.heartbeat_at) \ + < now() - make_interval(secs => $1)) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(stale_after.as_secs_f64()) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; Ok(rows .into_iter() .filter_map(|row| { let (account_id, table_name) = (row.1.clone(), row.2.clone()); - match restore_job_row(row) { + match restore_job_from_tuple(row) { Ok(job) => Some(job), Err(e) => { tracing::warn!( @@ -735,19 +895,15 @@ impl BackupCatalog for PostgresEngine { Box::pin(async move { // By status alone: a record stamped in this statement's instant, // or ahead of the server's clock, is RUNNING too. - #[allow(clippy::type_complexity)] - let rows: Vec<(String, String, String, String, String, String, i64, f64)> = - sqlx::query_as( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, \ - EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8 \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name", - ) - .fetch_all(&self.pool) - .await - .map_err(db_err)?; - rows.into_iter().map(restore_job_row).collect() + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name" + )) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() }) } @@ -785,6 +941,158 @@ impl BackupCatalog for PostgresEngine { }) } + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let table_name = input.table_name.clone(); + // Ownership first, before the target exists, on the lock the + // legacy restore takes for a name (`restore_lock_key`), so a + // legacy and a store-backed restore into one name exclude each + // other too. A lock already held is another restore into this + // name in flight: TableBeingRestored naming its source once its + // CREATING row is visible, else the name is taken as CreateTable + // would report it. The record's `heartbeat_at` is `now()` of the + // registration transaction, its start (`create_table.rs`), so a + // registration that waited on the backup's row lock or whose data + // DDL took long returns a record already that old; the engine's + // first beat refreshes it before the reply. + let Some(session) = + try_claim(&self.pool, &restore_lock_key(account_id, &table_name), None).await? + else { + return Err(self.name_taken_error(account_id, &table_name, true).await); + }; + let generation = session.generation(); + let registration = RestoreRegistrationRow { + owner_generation: generation, + ..registration.clone() + }; + let mut desc = match self + .create_table_for_restore(account_id, input, RestoreSource::Manifest(®istration)) + .await + { + Ok(desc) => desc, + Err(e) => { + session.close().await; + return Err(e); + } + }; + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(registration.backup.backup_arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let table_id = desc.table_id.clone(); + Ok(( + desc, + Box::new(PostgresRestoreClaim { + session: Mutex::new(session), + generation, + table_id, + pool: self.pool.clone(), + }) as Box, + )) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The same advisory lock the owner took on the target name: a + // session that still holds it is alive whatever its heartbeat + // says, and a second taker is refused here. + let Some(mut session) = try_claim( + &self.pool, + &restore_lock_key(&key.account_id, &key.table_name), + None, + ) + .await? + else { + return Ok(None); + }; + let generation = session.generation(); + // The generation, the claimant's spool and a fresh heartbeat in + // one write: a second taker, on this or another instance, finds + // the record live at once (the trait doc says why a restore + // record is refreshed where a backup row is not). + let taken = sqlx::query( + "UPDATE table_restores r SET owner_generation = $4, spool_host = $5, \ + heartbeat_at = now() \ + FROM tables t WHERE t.table_id = r.table_id AND r.table_id = $1 \ + AND t.account_id = $2 AND t.table_name = $3 \ + AND r.status = 'RUNNING' AND r.kind = 'manifest' \ + AND r.heartbeat_at < now() - make_interval(secs => $6)", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .bind(generation_column(generation)) + .bind(spool_host) + .bind(stale_after.as_secs_f64()) + .execute(session.connection()) + .await + .map_err(db_err)? + .rows_affected(); + if taken != 1 { + session.close().await; + return Ok(None); + } + Ok(Some(Box::new(PostgresRestoreClaim { + session: Mutex::new(session), + generation, + table_id: key.table_id.clone(), + pool: self.pool.clone(), + }) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let row: Option = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.table_id = $1 AND t.account_id = $2 AND t.table_name = $3" + )) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + row.map(restore_job_from_tuple).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE ($1::TEXT IS NULL OR r.status = $1) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(status.map(JobStatus::as_str)) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() + }) + } + fn read_store_ids<'a>( &'a self, store_kind: StoreKind, @@ -899,34 +1207,6 @@ impl BackupCatalog for PostgresEngine { /// (`RESTORE_LOCK_SEED`) the job claims use. const MAINTENANCE_LOCK_KEY: &str = "backup_maintenance"; -/// A restore job record as the two listings select it. -#[allow(clippy::type_complexity)] -fn restore_job_row( - ( - table_id, - account_id, - table_name, - source_backup_arn, - kind, - status, - owner_generation, - heartbeat_at, - ): (String, String, String, String, String, String, i64, f64), -) -> Result { - Ok(RestoreJobRow { - target: RestoreTargetKey { - account_id, - table_name, - table_id, - }, - source_backup_arn, - kind: RestoreKind::parse(&kind)?, - status: JobStatus::parse(&status)?, - owner_generation: u64::try_from(owner_generation).unwrap_or_default(), - heartbeat_at, - }) -} - /// How many times `maintenance_claim` tries the exclusive lock (on its one /// claim connection), and the pause between tries. `check_maintenance_claim` holds the key's shared lock for /// one statement, and every CreateBackup and RestoreTableFromBackup runs one, diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs index 179302914..a37f1183e 100755 --- a/crates/storage-postgres/src/backup_engine.rs +++ b/crates/storage-postgres/src/backup_engine.rs @@ -31,10 +31,10 @@ use extenddb_storage::{BackupRestoreSource, JobPermit, LegacyBackupReader, Resto use futures::future::BoxFuture; use crate::PostgresEngine; -use crate::create_table::RestoreRegistration; +use crate::create_table::{RestoreRegistration, RestoreSource}; use crate::data::{ all_sort_key_info, data_table_name, index_table_name, insert_index_rows_multi, - item_has_index_keys, project_item_for_index, + item_has_index_keys, project_item_for_index, table_storage_relations, }; /// Convert a `PostgreSQL` `TIMESTAMPTZ` to epoch seconds as `f64`, keeping the @@ -65,10 +65,198 @@ fn pg_timestamp_to_epoch(ts: time::OffsetDateTime) -> f64 { pub const RESTORE_LOCK_SEED: i64 = 0x0045_4452; // 'E', 'D', 'R' /// A secondary index of a restore target, as the copy needs it. -struct RestoreIndex { - table: String, - key_schema: Vec, - projection: Projection, +pub(crate) struct RestoreIndex { + pub(crate) table: String, + pub(crate) key_schema: Vec, + pub(crate) projection: Projection, +} + +/// The secondary indexes of `table_id` as the catalog records them, each +/// with the data table its rows go into. +pub(crate) async fn restore_indexes( + pool: &sqlx::PgPool, + table_id: &str, +) -> Result, StorageError> { + let index_rows: Vec<(String, serde_json::Value, serde_json::Value)> = + sqlx::query_as("SELECT index_id, key_schema, projection FROM indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(pool) + .await + .map_err(db_err)?; + let mut indexes = Vec::with_capacity(index_rows.len()); + for (index_id, ks, proj) in index_rows { + indexes.push(RestoreIndex { + table: index_table_name(&index_id), + key_schema: serde_json::from_value(ks) + .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, + projection: serde_json::from_value(proj) + .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, + }); + } + Ok(indexes) +} + +/// Why [`insert_restore_rows`] stopped: the base table refused a key it +/// already holds (the backup carries a key twice), or anything else. The +/// legacy copy reports both as the database error ([`From`]); the +/// store-backed load names the data file for the first. +pub(crate) enum RowInsertError { + DuplicateKey(sqlx::Error), + Storage(StorageError), +} + +impl From for RowInsertError { + fn from(e: StorageError) -> Self { + Self::Storage(e) + } +} + +impl From for StorageError { + fn from(e: RowInsertError) -> Self { + match e { + RowInsertError::DuplicateKey(e) => db_err(e), + RowInsertError::Storage(e) => e, + } + } +} + +/// SQLSTATE `unique_violation`. +const SQLSTATE_UNIQUE_VIOLATION: &str = "23505"; + +/// A restore target as the row writes need it, built once per restore: the +/// base data table, the key shape, and the secondary indexes with their +/// data tables. `source` names the backup or data file in errors. +pub(crate) struct RestoreTarget<'a> { + pub(crate) ddb_table: String, + pub(crate) key_schema: &'a [KeySchemaElement], + pub(crate) attribute_definitions: &'a [extenddb_core::types::AttributeDefinition], + pub(crate) sort_keys: Vec<(&'a str, ScalarAttributeType)>, + pub(crate) indexes: Vec, + pub(crate) source: String, +} + +impl<'a> RestoreTarget<'a> { + /// The target `table_id` with `key_schema` and `attribute_definitions`, + /// its indexes read from the catalog. + pub(crate) async fn load( + pool: &sqlx::PgPool, + table_id: &str, + key_schema: &'a [KeySchemaElement], + attribute_definitions: &'a [extenddb_core::types::AttributeDefinition], + source: String, + ) -> Result { + Ok(Self { + ddb_table: data_table_name(table_id), + key_schema, + attribute_definitions, + sort_keys: all_sort_key_info(key_schema, attribute_definitions), + indexes: restore_indexes(pool, table_id).await?, + source, + }) + } +} + +/// Write one batch of restored items inside `tx`: one multi-row INSERT into +/// the base table, then one multi-row INSERT into each secondary index for +/// the batch's items that carry that index's keys. Plain INSERTs: two rows +/// for one key fail the statement rather than collapsing into one item. +pub(crate) async fn insert_restore_rows( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + target: &RestoreTarget<'_>, + batch: &[(Item, String)], +) -> Result<(), RowInsertError> { + if batch.is_empty() { + return Ok(()); + } + let RestoreTarget { + ddb_table, + key_schema, + attribute_definitions, + sort_keys, + indexes, + source, + } = target; + let mut cols = vec!["pk".to_owned()]; + cols.extend( + sort_keys + .iter() + .enumerate() + .map(|(i, &(_, t))| sk_column_n(i, t)), + ); + cols.push("item_data".to_owned()); + let width = cols.len(); + // The last column is the item, bound as text and cast in SQL. + let tuples: Vec = (0..batch.len()) + .map(|r| { + let ps: Vec = (1..=width) + .map(|c| { + if c == width { + format!("${}::jsonb", r * width + c) + } else { + format!("${}", r * width + c) + } + }) + .collect(); + format!("({})", ps.join(", ")) + }) + .collect(); + let sql = format!( + "INSERT INTO {ddb_table} ({}) VALUES {}", + cols.join(", "), + tuples.join(", ") + ); + let mut query = sqlx::query(&sql); + for (item, item_json) in batch { + query = query.bind(composite_pk_to_text(item, key_schema)?); + for &(name, sk_type) in sort_keys { + let value = item.get(name).ok_or_else(|| { + StorageError::Internal(format!("{source} has an item without sort key {name}")) + })?; + query = match parse_sk(value, sk_type)? { + SortKeyValue::S(v) => query.bind(v), + SortKeyValue::N(v) => query.bind(v), + SortKeyValue::B(v) => query.bind(v), + }; + } + query = query.bind(item_json.as_str()); + } + query.execute(&mut **tx).await.map_err(|e| { + let duplicate = e + .as_database_error() + .and_then(|d| d.code()) + .is_some_and(|code| code == SQLSTATE_UNIQUE_VIOLATION); + if duplicate { + RowInsertError::DuplicateKey(e) + } else { + RowInsertError::Storage(db_err(e)) + } + })?; + + for idx in indexes { + let idx_sks = all_sort_key_info(&idx.key_schema, attribute_definitions); + let projected: Vec<(&Item, Item)> = batch + .iter() + .filter(|(item, _)| item_has_index_keys(item, &idx.key_schema)) + .map(|(item, _)| { + ( + item, + project_item_for_index(item, &idx.key_schema, key_schema, &idx.projection), + ) + }) + .collect(); + let rows: Vec<(&Item, &Item)> = projected.iter().map(|(i, p)| (*i, p)).collect(); + insert_index_rows_multi( + tx, + &idx.table, + &rows, + &idx.key_schema, + key_schema, + &idx_sks, + sort_keys, + ) + .await?; + } + Ok(()) } /// Session-scoped ownership of one job, held for the job's life: a @@ -168,9 +356,9 @@ pub(crate) fn generation_column(generation: u64) -> i64 { i64::try_from(generation).unwrap_or(i64::MAX) } -/// The heartbeat of a legacy restore: refresh its job record while it is -/// RUNNING under this generation. -const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() \ +/// The heartbeat of a restore: refresh its job record while it is RUNNING +/// under this generation. +pub(crate) const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() \ WHERE table_id = $1 AND status = 'RUNNING' AND owner_generation = $2"; /// The advisory-lock key for a restore into `table_name` in `account_id`. @@ -181,7 +369,7 @@ const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() /// `hashtextextended`, so a collision is negligible, and one would only make /// a second restore into the other name wait, never remove a table it should /// not. -fn restore_lock_key(account_id: &str, table_name: &str) -> String { +pub(crate) fn restore_lock_key(account_id: &str, table_name: &str) -> String { format!("{account_id}/{table_name}") } @@ -537,24 +725,14 @@ impl PostgresEngine { use futures::TryStreamExt; let ddb_table = data_table_name(&desc.table_id); - let sort_keys = all_sort_key_info(&desc.key_schema, &desc.attribute_definitions); - let index_rows: Vec<(String, serde_json::Value, serde_json::Value)> = sqlx::query_as( - "SELECT index_id, key_schema, projection FROM indexes WHERE table_id = $1", + let target = RestoreTarget::load( + &self.pool, + &desc.table_id, + &desc.key_schema, + &desc.attribute_definitions, + format!("backup {backup_arn}"), ) - .bind(&desc.table_id) - .fetch_all(&self.pool) - .await - .map_err(db_err)?; - let mut indexes = Vec::with_capacity(index_rows.len()); - for (index_id, ks, proj) in index_rows { - indexes.push(RestoreIndex { - table: index_table_name(&index_id), - key_schema: serde_json::from_value(ks) - .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, - projection: serde_json::from_value(proj) - .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, - }); - } + .await?; // The backup's rows are read from one catalog snapshot in which the // backup is still AVAILABLE. DeleteBackup refuses while this restore @@ -596,10 +774,7 @@ impl PostgresEngine { batch_bytes += item_json.len(); batch.push((item, item_json)); if batch.len() == COPY_BATCH_ITEMS || batch_bytes >= COPY_BATCH_BYTES { - self.write_restore_batch( - &ddb_table, desc, &sort_keys, &indexes, &batch, backup_arn, - ) - .await?; + self.write_restore_batch(&target, &batch).await?; item_count += i64::try_from(batch.len()).unwrap_or(i64::MAX); batch.clear(); batch_bytes = 0; @@ -608,8 +783,7 @@ impl PostgresEngine { drop(rows); snapshot.commit().await.map_err(db_err)?; if !batch.is_empty() { - self.write_restore_batch(&ddb_table, desc, &sort_keys, &indexes, &batch, backup_arn) - .await?; + self.write_restore_batch(&target, &batch).await?; item_count += i64::try_from(batch.len()).unwrap_or(i64::MAX); } @@ -624,18 +798,47 @@ impl PostgresEngine { // and its job record DONE, in one catalog transaction. The table was // created with the transition deferred, so this is the first point it // can become ACTIVE, by ordering rather than by timing. - // - // Only from CREATING, and only while the job record is RUNNING under - // this generation. DeleteTable refuses a restore target, but the - // reconciler can claim one whose owner it found dead (moving it to - // DELETING); that claim wins, and flipping the row back to ACTIVE - // would leave a table whose data tables are being dropped. + if !self + .activate_restore_target(&desc.table_id, generation, item_count, table_size) + .await? + { + // The target was claimed for removal while the copy ran, or the + // job record passed to another claim; the copy's rows go with it. + return Err(StorageError::TableNotFound(format!( + "restore of {backup_arn} into {} did not complete: the table was deleted \ + while it was being restored", + desc.table_name + ))); + } + Ok(()) + } + + /// The end of a restore in one catalog transaction: the job record + /// RUNNING to DONE under `generation`, the table CREATING to ACTIVE with + /// its counts, and every secondary and vector index of the table ACTIVE + /// (the indexes are created ACTIVE with a restore target, so this is the + /// rule stated once rather than a change). `Ok(false)`, with nothing + /// written, when the record is not RUNNING under `generation` or the + /// table is not CREATING. + /// + /// Only from CREATING, and only while the job record is RUNNING under + /// this generation. DeleteTable refuses a restore target, but the + /// reconciler can claim one whose owner it found dead (moving it to + /// DELETING); that claim wins, and flipping the row back to ACTIVE + /// would leave a table whose data tables are being dropped. + pub(crate) async fn activate_restore_target( + &self, + table_id: &str, + generation: u64, + item_count: i64, + table_size: i64, + ) -> Result { let mut tx = self.pool.begin().await.map_err(db_err)?; let done = sqlx::query( "UPDATE table_restores SET status = 'DONE', completed_at = now() \ WHERE table_id = $1 AND status = 'RUNNING' AND owner_generation = $2", ) - .bind(&desc.table_id) + .bind(table_id) .bind(generation_column(generation)) .execute(&mut *tx) .await @@ -649,7 +852,7 @@ impl PostgresEngine { ) .bind(item_count) .bind(table_size) - .bind(&desc.table_id) + .bind(table_id) .execute(&mut *tx) .await .map_err(db_err)? @@ -659,16 +862,26 @@ impl PostgresEngine { }; if activated == 0 { tx.rollback().await.map_err(db_err)?; - // The target was claimed for removal while the copy ran, or the - // job record passed to another claim; the copy's rows go with it. - return Err(StorageError::TableNotFound(format!( - "restore of {backup_arn} into {} did not complete: the table was deleted \ - while it was being restored", - desc.table_name - ))); + return Ok(false); } + sqlx::query( + "UPDATE indexes SET index_status = 'ACTIVE' \ + WHERE table_id = $1 AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + sqlx::query( + "UPDATE vector_indexes SET index_status = 'ACTIVE', backfilling = NULL \ + WHERE table_id = $1 AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; tx.commit().await.map_err(db_err)?; - Ok(()) + Ok(true) } /// RUNNING to FAILED on the job record of `table_id`, conditioned on @@ -696,96 +909,15 @@ impl PostgresEngine { Ok(affected == 1) } - /// Write and commit one batch of restored items: one multi-row INSERT - /// into the base table, then one multi-row INSERT into each secondary - /// index for the batch's items that carry that index's keys. + /// Write and commit one batch of restored items ([`insert_restore_rows`] + /// in one data transaction). async fn write_restore_batch( &self, - ddb_table: &str, - desc: &TableDescription, - sort_keys: &[(&str, ScalarAttributeType)], - indexes: &[RestoreIndex], + target: &RestoreTarget<'_>, batch: &[(Item, String)], - backup_arn: &str, ) -> Result<(), StorageError> { let mut tx = self.data_pool.begin().await.map_err(db_err)?; - let mut cols = vec!["pk".to_owned()]; - cols.extend( - sort_keys - .iter() - .enumerate() - .map(|(i, &(_, t))| sk_column_n(i, t)), - ); - cols.push("item_data".to_owned()); - let width = cols.len(); - // The last column is the item, bound as text and cast in SQL. - let tuples: Vec = (0..batch.len()) - .map(|r| { - let ps: Vec = (1..=width) - .map(|c| { - if c == width { - format!("${}::jsonb", r * width + c) - } else { - format!("${}", r * width + c) - } - }) - .collect(); - format!("({})", ps.join(", ")) - }) - .collect(); - let sql = format!( - "INSERT INTO {ddb_table} ({}) VALUES {}", - cols.join(", "), - tuples.join(", ") - ); - let mut query = sqlx::query(&sql); - for (item, item_json) in batch { - query = query.bind(composite_pk_to_text(item, &desc.key_schema)?); - for &(name, sk_type) in sort_keys { - let value = item.get(name).ok_or_else(|| { - StorageError::Internal(format!( - "backup {backup_arn} has an item without sort key {name}" - )) - })?; - query = match parse_sk(value, sk_type)? { - SortKeyValue::S(v) => query.bind(v), - SortKeyValue::N(v) => query.bind(v), - SortKeyValue::B(v) => query.bind(v), - }; - } - query = query.bind(item_json.as_str()); - } - query.execute(&mut *tx).await.map_err(db_err)?; - - for idx in indexes { - let idx_sks = all_sort_key_info(&idx.key_schema, &desc.attribute_definitions); - let projected: Vec<(&Item, Item)> = batch - .iter() - .filter(|(item, _)| item_has_index_keys(item, &idx.key_schema)) - .map(|(item, _)| { - ( - item, - project_item_for_index( - item, - &idx.key_schema, - &desc.key_schema, - &idx.projection, - ), - ) - }) - .collect(); - let rows: Vec<(&Item, &Item)> = projected.iter().map(|(i, p)| (*i, p)).collect(); - insert_index_rows_multi( - &mut tx, - &idx.table, - &rows, - &idx.key_schema, - &desc.key_schema, - &idx_sks, - sort_keys, - ) - .await?; - } + insert_restore_rows(&mut tx, target, batch).await?; tx.commit().await.map_err(db_err)?; Ok(()) } @@ -804,6 +936,12 @@ impl PostgresEngine { /// DELETING removal, so if a drop below fails, the row stays DELETING with /// its index rows and the control plane finishes the job later. /// + /// The relations dropped are the ones `DeleteTable` drops + /// (`table_storage_relations`): the data table, every secondary index's + /// table and every vector index's table, their ids read in the claiming + /// transaction while the catalog rows still exist (the `tables` row's + /// deletion cascades them away). + /// /// Public for the test that runs it against an ACTIVE table; not an /// interface. #[doc(hidden)] @@ -825,13 +963,17 @@ impl PostgresEngine { .fetch_all(&mut *tx) .await .map_err(db_err)?; + let vector_index_ids: Vec = + sqlx::query_scalar("SELECT index_id FROM vector_indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(&mut *tx) + .await + .map_err(db_err)?; tx.commit().await.map_err(db_err)?; if !claimed { return Ok(false); } - let mut drops = vec![data_table_name(table_id)]; - drops.extend(index_ids.iter().map(|id| index_table_name(id))); - for t in drops { + for t in table_storage_relations(table_id, &index_ids, &vector_index_ids) { sqlx::query(&format!("DROP TABLE IF EXISTS {t}")) .execute(&self.data_pool) .await @@ -845,6 +987,44 @@ impl PostgresEngine { Ok(true) } + /// The body of `remove_restore_target` once its caller holds the target + /// name's advisory lock (`restore_lock_key`): the CREATING-target check, + /// the progress-row delete on the data pool (migration 7; a data + /// database from before it has no `restore_progress`), and the claim and + /// drop through [`Self::abort_restore`]. + pub async fn remove_restore_target_locked( + &self, + key: &extenddb_storage::backup_catalog::RestoreTargetKey, + ) -> Result { + let is_target: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM tables t JOIN table_restores r \ + ON r.table_id = t.table_id WHERE t.table_id = $1 AND t.account_id = $2 \ + AND t.table_name = $3 AND t.table_status = 'CREATING')", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + if !is_target { + return Ok(false); + } + let has_progress: bool = + sqlx::query_scalar("SELECT to_regclass('restore_progress') IS NOT NULL") + .fetch_one(&self.data_pool) + .await + .map_err(db_err)?; + if has_progress { + sqlx::query("DELETE FROM restore_progress WHERE table_id = $1") + .bind(&key.table_id) + .execute(&self.data_pool) + .await + .map_err(db_err)?; + } + self.abort_restore(&key.table_id).await + } + /// The detached task of RestoreTableFromBackup: take the restore lock, /// register the CREATING target with its job record, report the /// registration to the request (which may be gone), then copy. `permit` @@ -926,7 +1106,7 @@ impl PostgresEngine { owner_generation: owner.generation(), }; let mut desc = match self - .create_table_for_restore(&account_id, input, ®istration) + .create_table_for_restore(&account_id, input, RestoreSource::Legacy(®istration)) .await { Ok(desc) => desc, diff --git a/crates/storage-postgres/src/create_table.rs b/crates/storage-postgres/src/create_table.rs index 7e4098a82..61e8928c2 100755 --- a/crates/storage-postgres/src/create_table.rs +++ b/crates/storage-postgres/src/create_table.rs @@ -7,13 +7,14 @@ use extenddb_core::types::{ BillingMode, BillingModeSummary, CreateTableInput, GsiDescription, LsiDescription, ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::RestoreRegistrationRow; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; use crate::PostgresEngine; -/// What a restore target records about where it came from, written in the -/// same catalog transaction that creates it. +/// What a legacy restore target records about where it came from, written +/// in the same catalog transaction that creates it. pub(crate) struct RestoreRegistration<'a> { pub(crate) backup_arn: &'a str, /// Reported as `RestoreSummary.SourceTableArn`. @@ -26,6 +27,18 @@ pub(crate) struct RestoreRegistration<'a> { pub(crate) owner_generation: u64, } +/// Which backup a restore target is registered from, and so which row its +/// registration locks: a legacy restore locks the `backups` row of the +/// in-catalog backup, a store-backed one the `backup_index` row (the lock +/// DeleteBackup's `start_backup_delete` takes). The two paths keep their own +/// locks: a legacy and a store-backed restore are of two different backups +/// and never meet, and a DeleteBackup of a `backup_index` row meets only +/// store-backed registrations. +pub(crate) enum RestoreSource<'a> { + Legacy(&'a RestoreRegistration<'a>), + Manifest(&'a RestoreRegistrationRow), +} + impl PostgresEngine { /// Core implementation of `create_table` (Fix #4: wrapped in a transaction). /// Create a table. When `defer_active` is set (the restore path), the row @@ -54,7 +67,7 @@ impl PostgresEngine { &self, account_id: &str, input: CreateTableInput, - registration: &RestoreRegistration<'_>, + registration: RestoreSource<'_>, ) -> Result { self.create_table_impl_inner(account_id, input, true, Some(registration)) .await @@ -107,7 +120,7 @@ impl PostgresEngine { account_id: &str, input: CreateTableInput, defer_active: bool, - restore: Option<&RestoreRegistration<'_>>, + restore: Option>, ) -> Result { Self::validate_account_id(account_id)?; let table_id = uuid::Uuid::new_v4().to_string(); @@ -149,32 +162,67 @@ impl PostgresEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - if let Some(RestoreRegistration { backup_arn, .. }) = restore { - // FOR UPDATE, not FOR SHARE: two restores of one backup must not - // both pass the CREATING-target check below, so they serialize on - // the backup row (DeleteBackup locks it the same way). - let available: Option = sqlx::query_scalar( - "SELECT 1 FROM backups WHERE backup_arn = $1 AND account_id = $2 \ - AND backup_status = 'AVAILABLE' FOR UPDATE", - ) - .bind(backup_arn) - .bind(account_id) - .fetch_optional(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; - if available.is_none() { - return Err(StorageError::Validation(format!( - "Backup not found: {backup_arn}" - ))); + match &restore { + Some(RestoreSource::Legacy(RestoreRegistration { backup_arn, .. })) => { + // FOR UPDATE, not FOR SHARE: two restores of one backup must not + // both pass the CREATING-target check below, so they serialize on + // the backup row (DeleteBackup locks it the same way). + let available: Option = sqlx::query_scalar( + "SELECT 1 FROM backups WHERE backup_arn = $1 AND account_id = $2 \ + AND backup_status = 'AVAILABLE' FOR UPDATE", + ) + .bind(backup_arn) + .bind(account_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if available.is_none() { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } + // One restore of a backup at a time, as the service: while a + // target restored from it is CREATING the backup is in use. Same + // predicate as DeleteBackup's. + if let Some(table) = + crate::backup_catalog::creating_restore_target(&mut *tx, backup_arn).await? + { + return Err(crate::backup_catalog::backup_in_use(&table, backup_arn)); + } } - // One restore of a backup at a time, as the service: while a - // target restored from it is CREATING the backup is in use. Same - // predicate as DeleteBackup's. - if let Some(table) = - crate::backup_catalog::creating_restore_target(&mut *tx, backup_arn).await? - { - return Err(crate::backup_catalog::backup_in_use(&table, backup_arn)); + Some(RestoreSource::Manifest(registration)) => { + // The `backup_index` row FOR UPDATE: the lock + // `start_backup_delete` takes before its own in-use check, so + // a DeleteBackup and this registration order wholly one + // before the other, and two registrations of one backup + // serialize on the row before the CREATING-target check. + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ + FOR UPDATE", + ) + .bind(account_id) + .bind(®istration.backup.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if status.as_deref() != Some("AVAILABLE") { + return Err(extenddb_storage::backup_catalog::backup_not_found( + ®istration.backup.backup_arn, + )); + } + if let Some(table) = crate::backup_catalog::creating_restore_target( + &mut *tx, + ®istration.backup.backup_arn, + ) + .await? + { + return Err(crate::backup_catalog::backup_in_use( + &table, + ®istration.backup.backup_arn, + )); + } } + None => {} } // Insert table metadata, returning creation timestamp and actual status. @@ -236,24 +284,53 @@ impl PostgresEngine { Err(e) => return Err(StorageError::Internal(e.to_string())), }; - if let Some(r) = restore { - // The job record: RUNNING under the claim's generation, with a - // heartbeat on the catalog's clock, so no CREATING target ever - // exists without an owned record. - sqlx::query( - "INSERT INTO table_restores \ - (table_id, source_backup_arn, source_table_arn, restore_date_time, \ - status, kind, owner_generation, heartbeat_at) \ - VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'legacy', $5, now())", - ) - .bind(&table_id) - .bind(r.backup_arn) - .bind(r.source_table_arn) - .bind(r.restore_date_time) - .bind(crate::backup_engine::generation_column(r.owner_generation)) - .execute(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + // The job record: RUNNING under the claim's generation, with a + // heartbeat on the catalog's clock, so no CREATING target ever + // exists without an owned record. `now()` is this transaction's + // start, not the insert's instant: a registration that waited on the + // `backup_index` row's lock above commits a heartbeat that old, and + // the data DDL after the commit adds to the age the engine's first + // beat (sent before the registration's reply) then refreshes. + match &restore { + Some(RestoreSource::Legacy(r)) => { + sqlx::query( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'legacy', $5, now())", + ) + .bind(&table_id) + .bind(r.backup_arn) + .bind(r.source_table_arn) + .bind(r.restore_date_time) + .bind(crate::backup_engine::generation_column(r.owner_generation)) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Some(RestoreSource::Manifest(r)) => { + sqlx::query( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, store_kind, store_root, store_id, spool_host, \ + owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'manifest', $5, $6, $7, \ + $8, $9, now())", + ) + .bind(&table_id) + .bind(&r.backup.backup_arn) + .bind(&r.source_table_arn) + .bind(r.restore_date_time) + .bind(r.store.store_kind.as_str()) + .bind(&r.store.store_root) + .bind(&r.store.store_id) + .bind(r.spool_host.as_deref()) + .bind(crate::backup_engine::generation_column(r.owner_generation)) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + None => {} } // Insert GSI metadata diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs index 8e031f3e9..f9554ceb5 100755 --- a/crates/storage-postgres/src/lib.rs +++ b/crates/storage-postgres/src/lib.rs @@ -26,6 +26,7 @@ pub mod migrations; mod operations; mod password_hash_store; mod pg_util; +mod restore_engine; mod snapshot_engine; #[doc(hidden)] pub use snapshot_engine::{ @@ -105,6 +106,11 @@ pub use delete_table::test_pause::{ #[cfg(feature = "test-hooks")] #[doc(hidden)] pub use migrations::apply_all_migrations; +/// Pause `sweep_restore_progress` between its two reads, so a test can +/// commit a registration and a progress row in between. +#[cfg(feature = "test-hooks")] +#[doc(hidden)] +pub use restore_engine::sweep_pause; /// The `PostgreSQL` storage backend. /// diff --git a/crates/storage-postgres/src/migrations/mod.rs b/crates/storage-postgres/src/migrations/mod.rs index 1aa4c2ea5..319d03403 100644 --- a/crates/storage-postgres/src/migrations/mod.rs +++ b/crates/storage-postgres/src/migrations/mod.rs @@ -136,6 +136,19 @@ pub fn migration(id: MigrationId) -> Migration { "0006_backup_catalog_index.sql" )], }, + MigrationId::RestoreProgress => Migration::Applies { + // A new table is additive by CONTRIBUTING's step 4, but declaring + // `Kind::Additive` needs the old-binary job, which does not exist + // yet (the paragraph after step 4; `refuse_additive` in + // `crates/bin` pins it). + kind: Kind::Breaking, + convergent: true, + bodies: vec![file_body!( + StoreKind::Data, + "data_migrations", + "0007_restore_progress.sql" + )], + }, } } @@ -158,7 +171,8 @@ pub(crate) const fn runs_in(id: MigrationId) -> RunsIn { | MigrationId::ResourceTagConditionKeys | MigrationId::SortKeyDefinitions | MigrationId::DropContinuousBackups - | MigrationId::BackupCatalogIndex => RunsIn::Transaction, + | MigrationId::BackupCatalogIndex + | MigrationId::RestoreProgress => RunsIn::Transaction, MigrationId::IndexTableBaseKey => RunsIn::Autocommit, } } diff --git a/crates/storage-postgres/src/restore_engine.rs b/crates/storage-postgres/src/restore_engine.rs new file mode 100644 index 000000000..c30666783 --- /dev/null +++ b/crates/storage-postgres/src/restore_engine.rs @@ -0,0 +1,438 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on `PostgreSQL` +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target, the progress record, the ACTIVE flip and the counts. +//! +//! One data transaction per file: the base rows in multi-row INSERTs of +//! `COPY_BATCH_ITEMS` rows or `COPY_BATCH_BYTES` of item text, each +//! secondary index's rows beside them (the `insert_restore_rows` the legacy +//! copy shares), each vector index's rows through the write path's +//! `VectorInsertPlan`, and the file's `restore_progress` row last, so a file +//! commits whole with its record or not at all. The data database has no +//! `tables` row and no job record, so nothing in `bulk_load` reads or +//! conditions on the claim's generation: the fence is the base table's +//! primary key, which refuses a file applied twice, and the claim's +//! liveness, which the engine confirms before each file. Plain INSERTs, so +//! a key the backup holds twice, within a file or across files, fails the +//! file (SQLSTATE 23505) instead of collapsing two items. The progress row +//! is a plain INSERT for a file applied from its start and an upsert +//! (`ON CONFLICT (table_id, file_index) DO UPDATE SET items_applied`) for a +//! file resumed past it, so a row left short of the file's count by a +//! decoder that died after the commit is completed by the resume. +//! +//! `finish_restore` is `activate_restore_target` (the catalog transaction +//! the legacy copy ends with: the record DONE, the table and every index +//! ACTIVE, conditioned on the generation and on CREATING), then the +//! progress rows deleted on the data pool. The delete runs after the commit +//! and cannot fail the restore: its error is logged at WARN and +//! `Finished::Active` returned, since the restore is complete once the flip +//! committed and the rows are the reconciler's `sweep_restore_progress` to +//! remove, as they are after a crash between the two. + +use std::collections::HashSet; + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement}; +use extenddb_storage::backup_catalog::{ + JobClaim, RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::backup_definition::{COPY_BATCH_BYTES, COPY_BATCH_ITEMS}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::PostgresEngine; +use crate::backup_engine::{RestoreTarget, RowInsertError, insert_restore_rows}; +use crate::data::data_table_name; +use crate::data::vector_index::{VectorInsertPlan, fetch_vector_indexes_for_table}; + +fn db_err(e: sqlx::Error) -> StorageError { + StorageError::Internal(format!("Database error: {e}")) +} + +/// The key shape of a CREATING restore target, read from its catalog row. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +impl PostgresEngine { + /// The target's key schema and attribute definitions, when `key` names + /// a CREATING restore target with this table id. + async fn restore_target_shape( + &self, + key: &RestoreTargetKey, + ) -> Result { + let row: Option<(serde_json::Value, serde_json::Value)> = sqlx::query_as( + "SELECT t.key_schema, t.attribute_definitions FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.table_id = $1 AND t.account_id = $2 AND t.table_name = $3 \ + AND t.table_status = 'CREATING'", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (key_schema, attribute_definitions) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + Ok(TargetShape { + key_schema: serde_json::from_value(key_schema) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: serde_json::from_value(attribute_definitions) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }) + } +} + +/// The error for a key the target already holds: the backup carries it +/// twice, within the file or in a file applied before it. The engine +/// prefixes the file's key, so the file is not named here. +fn duplicate_key_error(e: &sqlx::Error) -> StorageError { + StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key twice): {e}" + )) +} + +/// Write `batch` into the base table, every secondary index and every vector +/// index inside the file's transaction, and clear it. +async fn flush_batch( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + target: &RestoreTarget<'_>, + vector_plans: &[VectorInsertPlan<'_>], + batch: &mut Vec<(Item, String)>, +) -> Result { + insert_restore_rows(tx, target, batch) + .await + .map_err(|e| match e { + RowInsertError::DuplicateKey(e) => duplicate_key_error(&e), + RowInsertError::Storage(e) => e, + })?; + for plan in vector_plans { + for (item, _) in batch.iter() { + plan.insert(tx, item).await?; + } + } + let written = u64::try_from(batch.len()).unwrap_or(u64::MAX); + batch.clear(); + Ok(written) +} + +impl RestoreEngine for PostgresEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + _claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + let shape = self.restore_target_shape(target).await?; + let source = format!("data file {}", file.file_index); + let rows = RestoreTarget::load( + &self.pool, + &target.table_id, + &shape.key_schema, + &shape.attribute_definitions, + source.clone(), + ) + .await?; + let vector_metas = fetch_vector_indexes_for_table(&self.pool, &target.table_id).await?; + let vector_plans: Vec> = vector_metas + .iter() + .map(|(meta, _)| { + VectorInsertPlan::new(meta, &shape.key_schema, &shape.attribute_definitions) + }) + .collect(); + + let mut tx = self.data_pool.begin().await.map_err(db_err)?; + let mut batch: Vec<(Item, String)> = Vec::with_capacity(COPY_BATCH_ITEMS); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + while let Some(item) = items.next().await { + let item = item?; + let item_json = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))?; + batch_bytes += item_json.len(); + batch.push((item, item_json)); + if batch.len() == COPY_BATCH_ITEMS || batch_bytes >= COPY_BATCH_BYTES { + applied += flush_batch(&mut tx, &rows, &vector_plans, &mut batch).await?; + batch_bytes = 0; + } + } + if !batch.is_empty() { + applied += flush_batch(&mut tx, &rows, &vector_plans, &mut batch).await?; + } + // The file's record, in the file's transaction. A file applied from + // its start writes a new row: a duplicate `(table_id, file_index)` + // is a file applied twice, which the base table's primary key has + // refused above for any file with items. A file resumed past its + // start (`start_at > 0`) updates the row the earlier run left + // (one with `items_applied` short of the file's count: the + // decoder died after the file's commit and the job's FAILED write + // never landed), so the resume completes the file instead of + // failing on the row's key. + let progress_row = if file.start_at > 0 { + "INSERT INTO restore_progress (table_id, file_index, items_applied) \ + VALUES ($1, $2, $3) \ + ON CONFLICT (table_id, file_index) DO UPDATE \ + SET items_applied = EXCLUDED.items_applied" + } else { + "INSERT INTO restore_progress (table_id, file_index, items_applied) \ + VALUES ($1, $2, $3)" + }; + sqlx::query(progress_row) + .bind(&target.table_id) + .bind(i64::from(file.file_index)) + .bind(i64::try_from(file.start_at.saturating_add(applied)).unwrap_or(i64::MAX)) + .execute(&mut *tx) + .await + .map_err(|e| { + let duplicate = e + .as_database_error() + .and_then(|d| d.code()) + .is_some_and(|code| code == "23505"); + if duplicate { + StorageError::Validation(format!( + "{source} was applied before and its progress recorded: {e}" + )) + } else { + db_err(e) + } + })?; + tx.commit().await.map_err(db_err)?; + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The key names the table; the flip itself is conditioned on the + // table id, CREATING, RUNNING and the generation. + let is_target: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM tables WHERE table_id = $1 AND account_id = $2 \ + AND table_name = $3)", + ) + .bind(&target.table_id) + .bind(&target.account_id) + .bind(&target.table_name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + if !is_target { + return Ok(Finished::NotMatched(format!( + "{} is not a table with id {} in account {}", + target.table_name, target.table_id, target.account_id + ))); + } + let activated = self + .activate_restore_target( + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + ) + .await?; + if !activated { + return Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING", + target.table_name, + claim.generation() + ))); + } + restore_pause_point(RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE).await; + // The flip has committed: the table is ACTIVE and the record DONE, + // so the restore is complete whatever happens to the progress + // rows. A failure here is logged, not returned: an `Err` would make + // the engine fail a finished restore, and the rows are harmless + // (the data database reads them for nothing once the record is + // DONE) and the reconciler's `sweep_restore_progress` removes them. + if let Err(e) = sqlx::query("DELETE FROM restore_progress WHERE table_id = $1") + .bind(&target.table_id) + .execute(&self.data_pool) + .await + { + tracing::warn!( + "restore of table {}: the table is ACTIVE and its record DONE, but the \ + restore_progress rows of table id {} could not be deleted: {e}; the progress \ + rows are the reconciler's to sweep", + target.table_name, + target.table_id + ); + } + Ok(Finished::Active) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data table; the key's account and name are + // checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_shape(key).await?; + let count: i64 = sqlx::query_scalar(&format!( + "SELECT count(*) FROM {}", + data_table_name(&key.table_id) + )) + .fetch_one(&self.data_pool) + .await + .map_err(db_err)?; + Ok(u64::try_from(count).unwrap_or_default()) + }) + } + + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The catalog row first, so a bare id reads no other account's + // table; then the data table's existence (the registration + // commits the catalog transaction before the data DDL). + self.restore_target_shape(key).await?; + let exists: bool = sqlx::query_scalar("SELECT to_regclass($1) IS NOT NULL") + .bind(data_table_name(&key.table_id)) + .fetch_one(&self.data_pool) + .await + .map_err(db_err)?; + Ok(exists) + }) + } + + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec<(i64, i64)> = sqlx::query_as( + "SELECT file_index, items_applied FROM restore_progress WHERE table_id = $1 \ + ORDER BY file_index", + ) + .bind(&key.table_id) + .fetch_all(&self.data_pool) + .await + .map_err(db_err)?; + Ok(rows + .into_iter() + .map(|(index, applied)| { + ( + u32::try_from(index).unwrap_or(u32::MAX), + u64::try_from(applied).unwrap_or_default(), + ) + }) + .collect()) + }) + } + + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // A positive list, candidates before the live set (the trait + // doc): the ids the progress rows name now, then the catalog's + // RUNNING records; a target registered after `running` was + // listed is RUNNING here, and one whose first row lands after + // the first read is not among the candidates. + let named: Vec = + sqlx::query_scalar("SELECT DISTINCT table_id FROM restore_progress") + .fetch_all(&self.data_pool) + .await + .map_err(db_err)?; + if named.is_empty() { + return Ok(0); + } + // A test may interpose here, after the candidate ids are read and + // before the live RUNNING set, to commit a registration and a + // progress row: a row of a record that is RUNNING at the read + // below is protected (the two reads order candidates before the + // live set, the trait doc). + #[cfg(feature = "test-hooks")] + sweep_pause::maybe_pause().await; + let live: Vec = + sqlx::query_scalar("SELECT table_id FROM table_restores WHERE status = 'RUNNING'") + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + let protected: HashSet<&str> = running + .iter() + .chain(live.iter()) + .map(String::as_str) + .collect(); + let stale: Vec = named + .into_iter() + .filter(|id| !protected.contains(id.as_str())) + .collect(); + if stale.is_empty() { + return Ok(0); + } + let removed = + sqlx::query("DELETE FROM restore_progress WHERE table_id = ANY($1::TEXT[])") + .bind(&stale) + .execute(&self.data_pool) + .await + .map_err(db_err)? + .rows_affected(); + Ok(removed) + }) + } +} + +/// A rendezvous a test arms to pause `sweep_restore_progress` between its +/// two reads (the candidate progress-row ids, then the live RUNNING set), +/// so it can commit a registration and a progress row in between and prove +/// that row survives the sweep. `test-hooks` only. +#[cfg(feature = "test-hooks")] +#[doc(hidden)] +pub mod sweep_pause { + use std::sync::{LazyLock, Mutex}; + + use tokio::sync::Notify; + + struct Rendezvous { + arrived: std::sync::Arc, + proceed: std::sync::Arc, + } + + static PAUSE: LazyLock>> = LazyLock::new(|| Mutex::new(None)); + + /// Arm the pause once. Returns `(arrived, proceed)`: the sweep notifies + /// `arrived` when it reaches the point and waits on `proceed`. + #[must_use] + pub fn arm() -> (std::sync::Arc, std::sync::Arc) { + let arrived = std::sync::Arc::new(Notify::new()); + let proceed = std::sync::Arc::new(Notify::new()); + *PAUSE.lock().unwrap() = Some(Rendezvous { + arrived: arrived.clone(), + proceed: proceed.clone(), + }); + (arrived, proceed) + } + + pub(crate) async fn maybe_pause() { + let armed = PAUSE.lock().unwrap().take(); + if let Some(r) = armed { + r.arrived.notify_one(); + r.proceed.notified().await; + } + } +} diff --git a/crates/storage-postgres/tests/restore_job.rs b/crates/storage-postgres/tests/restore_job.rs new file mode 100644 index 000000000..703c265fa --- /dev/null +++ b/crates/storage-postgres/tests/restore_job.rs @@ -0,0 +1,1797 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 +//! Storage-level tests of the store-backed restore on PostgreSQL: the +//! registration under the `backup_index` row's lock (`claim_restore`), the +//! takeover of a dead record (`claim_restore_dead`), the per-file load with +//! its progress row (`bulk_load`), the ACTIVE flip (`finish_restore`), the +//! counts and the progress sweep. The engine's restore job drives these +//! through `BackupCatalog` and `RestoreEngine`; what each must guarantee on +//! this backend is what a test here pins. +//! +//! Each test builds a throwaway catalog and data database and drops them +//! when it ends, passing or failing, as `backup_restore.rs` does; the tests +//! run one at a time. Requires `EXTENDDB_TEST_PG_CONNECTION_STRING`; +//! without it every test reports a skip and passes, unless +//! `EXTENDDB_TEST_PG_REQUIRED` says the job must reach a server. + +use std::time::Duration; + +use extenddb_core::types::{ + CreateTableInput, DistanceFunction, Item, Projection, ProjectionType, TableDescription, + TableStatus, VectorAttribute, VectorIndexSpecification, +}; +use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobStatus, RestoreJobRow, + RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StatusFields, StoreIdentity, StoreKind, + WriteOutcome, backup_in_use_message, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::migrations::Runner; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use extenddb_storage::{BackupCatalog, TableEngine}; +use extenddb_storage_postgres::RESTORE_LOCK_SEED; +use extenddb_storage_postgres::migrations::{PostgresMigrations, PostgresTarget}; +use extenddb_storage_postgres::{PostgresConfig, PostgresEngine}; +use sqlx::PgPool; +use sqlx::postgres::PgPoolOptions; + +const ACCOUNT: &str = "123456789012"; +const REGION: &str = "us-east-1"; +/// A stale window no test's heartbeat crosses on its own. +const STALE: Duration = Duration::from_secs(300); + +// --- the scratch databases -------------------------------------------------------- + +/// A catalog database and a separate data database, the layout `extenddb +/// init` creates, with pgvector installed in the data database when the +/// server offers it. +struct Scratch { + engine: PostgresEngine, + /// The data database. + db: PgPool, + catalog: PgPool, + admin: PgPool, + db_names: [String; 2], + vector: bool, + _guard: DbGuard, + _serial: tokio::sync::MutexGuard<'static, ()>, +} + +/// Drops the scratch databases if the test panics, from the moment they are +/// created, including during setup. +struct DbGuard { + names: Vec, +} + +/// One test at a time: each opens its own pools and claim sessions. +static SERIAL: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + +impl Scratch { + async fn cleanup(self) { + self.db.close().await; + self.catalog.close().await; + drop_databases(&self.admin, &self.db_names).await; + self.admin.close().await; + } +} + +async fn drop_databases(admin: &PgPool, names: &[String]) { + for name in names { + let _ = sqlx::query(&format!("DROP DATABASE IF EXISTS \"{name}\" WITH (FORCE)")) + .execute(admin) + .await; + } +} + +impl Drop for DbGuard { + fn drop(&mut self) { + if !std::thread::panicking() { + return; + } + let (Some(base), names) = (base_conn(), self.names.clone()) else { + return; + }; + let _ = std::thread::spawn(move || { + let Ok(rt) = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + else { + return; + }; + rt.block_on(async { + if let Ok(admin) = PgPoolOptions::new() + .max_connections(1) + .connect(&format!("{base}/postgres")) + .await + { + drop_databases(&admin, &names).await; + admin.close().await; + } + }); + }) + .join(); + } +} + +/// The server's base URL, or `None` to skip. With `EXTENDDB_TEST_PG_REQUIRED` +/// set to any value but `0`, a job that must reach a server fails here +/// instead. +fn base_conn() -> Option { + let conn = std::env::var("EXTENDDB_TEST_PG_CONNECTION_STRING").unwrap_or_default(); + if conn.trim().is_empty() { + let required = + std::env::var("EXTENDDB_TEST_PG_REQUIRED").is_ok_and(|v| !v.is_empty() && v != "0"); + assert!( + !required, + "EXTENDDB_TEST_PG_CONNECTION_STRING is not set, and EXTENDDB_TEST_PG_REQUIRED says \ + this job must reach a PostgreSQL server" + ); + return None; + } + Some(conn.trim_end_matches('/').to_owned()) +} + +/// `true` when the test can run; prints the skip line otherwise. +fn pg_available(test: &str) -> bool { + if base_conn().is_none() { + eprintln!("SKIP {test}: no PostgreSQL"); + return false; + } + true +} + +async fn connect(url: &str) -> PgPool { + PgPoolOptions::new() + .max_connections(3) + .connect(url) + .await + .expect("connect to a scratch database") +} + +async fn scratch() -> Scratch { + let serial = SERIAL.lock().await; + let base = base_conn().expect("caller checks base_conn() first"); + let stem = format!("eddb_rstr_{}", uuid::Uuid::new_v4().simple())[..24].to_owned(); + let (catalog_name, data_name) = (format!("{stem}_c"), format!("{stem}_d")); + let admin = PgPoolOptions::new() + .max_connections(1) + .connect(&format!("{base}/postgres")) + .await + .expect("connect to the postgres maintenance database"); + let guard = DbGuard { + names: vec![catalog_name.clone(), data_name.clone()], + }; + for name in [&catalog_name, &data_name] { + sqlx::query(&format!("CREATE DATABASE \"{name}\"")) + .execute(&admin) + .await + .expect("create a scratch database"); + } + let catalog_url = format!("{base}/{catalog_name}"); + let data_url = format!("{base}/{data_name}"); + // A fresh install, migrated by the runner as `init` does: migration 7 + // creates `restore_progress` in the data database. + let target = PostgresTarget::new(catalog_url.parse().expect("a postgres URL"), &data_name) + .with_data_connection_row(&data_url); + let runner = PostgresMigrations::open_unlocked(&target) + .await + .expect("open the migration runner"); + Runner::new(&runner, "test") + .migrate() + .await + .expect("migrate the scratch databases"); + runner.close().await; + let catalog = connect(&catalog_url).await; + let db = connect(&data_url).await; + let vector: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM pg_available_extensions WHERE name = 'vector')", + ) + .fetch_one(&db) + .await + .expect("read the available extensions"); + if vector { + sqlx::query("CREATE EXTENSION IF NOT EXISTS vector") + .execute(&db) + .await + .expect("create the pgvector extension"); + } else { + eprintln!("NOTE: no pgvector on this server; the vector index cells are skipped"); + } + for key in ["control_plane_delay_seconds", "index_propagation_delay_ms"] { + sqlx::query( + "INSERT INTO settings (key, value) VALUES ($1, '0') \ + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value", + ) + .bind(key) + .execute(&catalog) + .await + .expect("set a delay to zero"); + } + sqlx::query( + "INSERT INTO settings (key, value) VALUES ('data_database_connection_string', $1) \ + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value", + ) + .bind(&data_url) + .execute(&catalog) + .await + .expect("point the catalog at the data database"); + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES ($1, $2)") + .bind(ACCOUNT) + .bind(format!("acct-{stem}")) + .execute(&catalog) + .await + .expect("seed the account row"); + let engine = PostgresEngine::new( + &PostgresConfig { + connection_string: catalog_url.clone(), + pool_size: 6, + max_item_size_bytes: 400_000, + }, + REGION, + ) + .await + .expect("open a PostgresEngine on the scratch databases"); + Scratch { + engine, + db, + catalog, + admin, + db_names: [catalog_name, data_name], + vector, + _guard: guard, + _serial: serial, + } +} + +// --- rows, inputs and items ----------------------------------------------------- + +fn store() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::S3, + store_root: "s3://bucket/extenddb".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// An AVAILABLE `backup_index` row of a backup of `table`, written through +/// the claim as a backup job writes it. +async fn available_backup(s: &Scratch, table: &str) -> BackupRow { + let backup_id = extenddb_storage::util::new_backup_id(); + let backup_arn = extenddb_storage::util::backup_arn(REGION, ACCOUNT, table, &backup_id); + let manifest: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + let row = BackupRow { + backup_id: backup_id.clone(), + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + backup_arn: backup_arn.clone(), + backup_name: "nightly".to_owned(), + status: BackupStatus::Creating, + store: store(), + size_bytes: 0, + item_count: 0, + created_at: extenddb_storage::util::backup_creation_date_time(&backup_id).unwrap(), + completed_at: None, + failure_reason: None, + source_table_details_json: manifest["source_table"].clone(), + owner_generation: 0, + heartbeat_at: 0.0, + }; + let claim = s.engine.claim_backup(&row).await.expect("claim the row"); + let written = claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + item_count: Some(3), + size_bytes: Some(1234), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("the AVAILABLE write"); + assert_eq!(written, WriteOutcome::Matched); + drop(claim); + BackupRow { + status: BackupStatus::Available, + ..row + } +} + +fn registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: 0, + } +} + +/// The target: a composite-key table with a GSI on `g`, an LSI on `l`, and +/// a vector index on `emb` (three dimensions) when the server has pgvector. +fn target_input(name: &str, vector: bool) -> CreateTableInput { + let mut input: CreateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": name, + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }], + "LocalSecondaryIndexes": [{ + "IndexName": "lix", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "l", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "ALL"} + }] + })) + .expect("input"); + if vector { + input.vector_indexes = Some(vec![VectorIndexSpecification { + index_name: "vix".to_owned(), + dimensions: 3, + distance_function: DistanceFunction::Cosine, + vector_attribute: VectorAttribute { + attribute_name: "emb".to_owned(), + }, + search_schema: None, + projection: Some(Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }), + }]); + } + input +} + +/// Item `i` of the fixture: every item has `pk` and `sk`, every second one +/// `g` (the GSI's key), every third one `l` (the LSI's), every fifth one +/// `emb` (the vector). +fn item(i: usize) -> Item { + let mut item = + serde_json::json!({"pk": {"S": format!("p{}", i % 7)}, "sk": {"N": i.to_string()}}); + if i.is_multiple_of(2) { + item["g"] = serde_json::json!({"S": format!("g{}", i % 4)}); + } + if i.is_multiple_of(3) { + item["l"] = serde_json::json!({"S": format!("l{i}")}); + } + if i.is_multiple_of(5) { + item["emb"] = serde_json::json!({"L": [{"N": "0.1"}, {"N": "0.2"}, {"N": "0.3"}]}); + } + serde_json::from_value(item).expect("item") +} + +fn items(range: std::ops::Range) -> Vec { + range.map(item).collect() +} + +fn stream(items: Vec) -> futures::stream::BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +fn key_of(desc: &TableDescription) -> RestoreTargetKey { + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + } +} + +/// Register a restore of `row` into `target` and return the description, its +/// key and the claim. +async fn register( + s: &Scratch, + row: &BackupRow, + target: &str, + spool_host: Option<&str>, +) -> (TableDescription, RestoreTargetKey, Box) { + let (desc, claim) = s + .engine + .claim_restore( + ACCOUNT, + target_input(target, s.vector), + ®istration(row, spool_host), + ) + .await + .expect("register the restore"); + let key = key_of(&desc); + (desc, key, claim) +} + +// --- reads -------------------------------------------------------------------- + +async fn table_status(s: &Scratch, name: &str) -> Option { + sqlx::query_scalar("SELECT table_status FROM tables WHERE account_id = $1 AND table_name = $2") + .bind(ACCOUNT) + .bind(name) + .fetch_optional(&s.catalog) + .await + .expect("table status") +} + +async fn record(s: &Scratch, key: &RestoreTargetKey) -> RestoreJobRow { + s.engine + .read_restore_job(key) + .await + .expect("read the record") + .expect("the record exists") +} + +/// `(status, kind, owner_generation, heartbeat_at, spool_host, completed_at)` +/// of the record, straight from the table. +async fn record_columns( + s: &Scratch, + table_id: &str, +) -> (String, String, i64, f64, Option, Option) { + sqlx::query_as( + "SELECT status, kind, owner_generation, EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8, \ + spool_host, EXTRACT(EPOCH FROM completed_at)::FLOAT8 FROM table_restores \ + WHERE table_id = $1", + ) + .bind(table_id) + .fetch_one(&s.catalog) + .await + .expect("read table_restores") +} + +async fn rows_in(db: &PgPool, physical: &str) -> i64 { + sqlx::query_scalar(&format!("SELECT count(*) FROM {physical}")) + .fetch_one(db) + .await + .expect("count rows") +} + +async fn base_rows(s: &Scratch, table_id: &str) -> i64 { + rows_in(&s.db, &format!("\"_ddb_{table_id}\"")).await +} + +/// `(index_name, index_id, index_status)` of the table's secondary indexes. +async fn indexes(s: &Scratch, table_id: &str) -> Vec<(String, String, String)> { + sqlx::query_as( + "SELECT index_name, index_id, index_status FROM indexes WHERE table_id = $1 \ + ORDER BY index_name", + ) + .bind(table_id) + .fetch_all(&s.catalog) + .await + .expect("read indexes") +} + +async fn index_rows(s: &Scratch, table_id: &str, name: &str) -> i64 { + let id = indexes(s, table_id) + .await + .into_iter() + .find(|(n, _, _)| n == name) + .map(|(_, id, _)| id) + .expect("the index exists"); + rows_in(&s.db, &format!("\"_ddb_{id}\"")).await +} + +/// `(index_id, index_status)` of the table's vector indexes. +async fn vector_indexes(s: &Scratch, table_id: &str) -> Vec<(String, String)> { + sqlx::query_as("SELECT index_id, index_status FROM vector_indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(&s.catalog) + .await + .expect("read vector_indexes") +} + +async fn vector_rows(s: &Scratch, table_id: &str) -> i64 { + let (id, _) = vector_indexes(s, table_id) + .await + .pop() + .expect("the vector index exists"); + rows_in(&s.db, &format!("\"_ddb_vec_{id}\"")).await +} + +/// Whether the data database has a relation named `relation` (the physical +/// name without quotes), by `to_regclass`. +async fn relation_exists(s: &Scratch, relation: &str) -> bool { + let regclass: Option = sqlx::query_scalar("SELECT to_regclass($1)::TEXT") + .bind(format!("\"{relation}\"")) + .fetch_one(&s.db) + .await + .expect("probe the relation"); + regclass.is_some() +} + +async fn progress_rows(s: &Scratch, table_id: &str) -> Vec<(i64, i64)> { + sqlx::query_as( + "SELECT file_index, items_applied FROM restore_progress WHERE table_id = $1 \ + ORDER BY file_index", + ) + .bind(table_id) + .fetch_all(&s.db) + .await + .expect("read restore_progress") +} + +/// The backend pid holding the restore lock on `table`, if any. +async fn lock_holder(s: &Scratch, table: &str) -> Option { + sqlx::query_scalar( + "SELECT pid FROM pg_locks WHERE locktype = 'advisory' AND granted \ + AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) \ + AND ((classid::bigint << 32) | objid::bigint) = hashtextextended($1, $2)", + ) + .bind(format!("{ACCOUNT}/{table}")) + .bind(RESTORE_LOCK_SEED) + .fetch_optional(&s.catalog) + .await + .expect("find the advisory lock holder") +} + +/// Date the record's heartbeat `age` into the past by hand. This proves the +/// takeover's predicate on a stale row, not the window boundary on the +/// database's clock, which `abandoned_restore_is_removed_only_when_its_heartbeat_is_stale` +/// in `backup_restore.rs` crosses. +async fn age_heartbeat(s: &Scratch, table_id: &str, age: Duration) -> f64 { + sqlx::query_scalar( + "UPDATE table_restores SET heartbeat_at = now() - make_interval(secs => $2) \ + WHERE table_id = $1 RETURNING EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8", + ) + .bind(table_id) + .bind(age.as_secs_f64()) + .fetch_one(&s.catalog) + .await + .expect("age the heartbeat") +} + +// --- the registration ----------------------------------------------------------- + +/// `claim_restore` creates the CREATING target with no scheduled transition +/// and its RUNNING `manifest` record carrying the store, the spool host and +/// the claim's generation, in one transaction; the description carries the +/// `RestoreSummary`; the record reads back through `read_restore_job` and +/// `list_restore_jobs`; the claim holds the name's advisory lock and its +/// heartbeat confirms. +#[tokio::test] +async fn claim_restore_registers_the_target_with_its_record() { + if !pg_available("claim_restore_registers_the_target_with_its_record") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + assert_eq!(lock_holder(&s, "Restored").await, None); + let (desc, key, claim) = register(&s, &row, "Restored", Some("host-a:/spool")).await; + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert_eq!( + summary.source_table_arn.as_deref(), + Some(format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music").as_str()) + ); + assert_eq!(summary.restore_date_time, row.created_at); + assert!(summary.restore_in_progress); + assert!( + lock_holder(&s, "Restored").await.is_some(), + "the claim holds the name lock" + ); + + let (status, kind, generation, heartbeat_at, spool_host, completed_at) = + record_columns(&s, &key.table_id).await; + assert_eq!((status.as_str(), kind.as_str()), ("RUNNING", "manifest")); + assert_eq!(generation, i64::try_from(claim.generation()).unwrap()); + assert!(heartbeat_at > 0.0); + assert_eq!(spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!(completed_at, None); + let transition: Option = sqlx::query_scalar( + "SELECT EXTRACT(EPOCH FROM status_transition_at)::FLOAT8 FROM tables WHERE table_id = $1", + ) + .bind(&key.table_id) + .fetch_one(&s.catalog) + .await + .expect("read the transition"); + assert_eq!( + transition, None, + "no scheduled transition: the job flips the table" + ); + + let job = record(&s, &key).await; + assert_eq!(job.target, key); + assert_eq!(job.source_backup_arn, row.backup_arn); + assert_eq!(job.kind, RestoreKind::Manifest); + assert_eq!(job.status, JobStatus::Running); + assert_eq!(job.owner_generation, claim.generation()); + assert_eq!(job.store, Some(store())); + assert_eq!(job.spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((job.file_index, job.items_applied), (None, None)); + assert_eq!( + (job.failure_reason.as_deref(), job.completed_at), + (None, None) + ); + let running = s + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(running, vec![job.clone()]); + assert!( + s.engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list") + .is_empty() + ); + assert_eq!( + s.engine.backup_in_use(&row.key()).await.expect("in use"), + Some("Restored".to_owned()) + ); + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Confirmed); + let heartbeat_now = record_columns(&s, &key.table_id).await.3; + assert!( + heartbeat_now >= heartbeat_at, + "the heartbeat refreshed the row" + ); + + // The indexes a restore target is created with are ACTIVE from the start + // (the flip has nothing to move), and the GSI and LSI data tables exist. + let index_rows = indexes(&s, &key.table_id).await; + assert_eq!(index_rows.len(), 2); + assert!(index_rows.iter().all(|(_, _, status)| status == "ACTIVE")); + assert_eq!(base_rows(&s, &key.table_id).await, 0); + drop(claim); + s.cleanup().await; +} + +/// The ordering DeleteBackup wins: with `start_backup_delete` holding the +/// `backup_index` row FOR UPDATE, a registration waits behind it and, once +/// the row is DELETING, answers `Backup not found` and creates no target. +#[tokio::test] +async fn delete_backup_winning_before_the_registration_leaves_no_target() { + if !pg_available("delete_backup_winning_before_the_registration_leaves_no_target") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + // `start_backup_delete` locks the row FOR UPDATE and then reads + // `table_restores`; a gate on that table holds it with the row lock + // taken. + let mut gate = s.catalog.begin().await.expect("begin the gate"); + sqlx::query("LOCK TABLE table_restores IN ACCESS EXCLUSIVE MODE") + .execute(&mut *gate) + .await + .expect("hold DeleteBackup after its row lock"); + let key = row.key(); + let mut delete = Box::pin(s.engine.start_backup_delete(&key)); + assert!( + tokio::time::timeout(Duration::from_secs(2), &mut delete) + .await + .is_err(), + "DeleteBackup did not reach the gate" + ); + let delete_waits: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM pg_locks \ + WHERE relation = 'table_restores'::regclass AND NOT granted)", + ) + .fetch_one(&s.catalog) + .await + .expect("observe the blocked DeleteBackup"); + assert!( + delete_waits, + "DeleteBackup was not paused after its row lock" + ); + + let input = target_input("Late", s.vector); + let reg = registration(&row, None); + let mut restore = Box::pin(s.engine.claim_restore(ACCOUNT, input, ®)); + assert!( + tokio::time::timeout(Duration::from_millis(700), &mut restore) + .await + .is_err(), + "the registration did not wait for DeleteBackup's row lock" + ); + assert_eq!( + table_status(&s, "Late").await, + None, + "the registration created its target before the row lock" + ); + + gate.commit().await.expect("let DeleteBackup finish"); + assert_eq!( + delete.await.expect("DeleteBackup wins"), + DeleteStart::Deleting + ); + let err = restore + .await + .err() + .expect("the registration finds the backup gone"); + assert!( + extenddb_storage::backup_catalog::is_backup_not_found(&err), + "{err:?}" + ); + assert_eq!( + table_status(&s, "Late").await, + None, + "no target was created" + ); + assert_eq!( + lock_holder(&s, "Late").await, + None, + "the refused registration released the name lock" + ); + assert!( + s.engine + .list_restore_jobs(None) + .await + .expect("list") + .is_empty() + ); + s.cleanup().await; +} + +/// The other ordering: a registration that commits first leaves +/// `start_backup_delete` with `InUse` naming the target, and a second +/// registration of the same backup is `BackupInUse` while the target is +/// CREATING. +#[tokio::test] +async fn a_registration_winning_before_delete_backup_keeps_the_backup_in_use() { + if !pg_available("a_registration_winning_before_delete_backup_keeps_the_backup_in_use") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "First", None).await; + assert_eq!( + s.engine + .start_backup_delete(&row.key()) + .await + .expect("DeleteBackup answers"), + DeleteStart::InUse("First".to_owned()) + ); + let second = s + .engine + .claim_restore( + ACCOUNT, + target_input("Second", s.vector), + ®istration(&row, None), + ) + .await + .err() + .expect("a second restore of the backup is refused"); + // The service's text names the backup, not the target. + assert!( + matches!(&second, StorageError::BackupInUse(m) if *m == backup_in_use_message(&row.backup_arn)), + "{second:?}" + ); + assert_eq!(table_status(&s, "Second").await, None); + assert_eq!(lock_holder(&s, "Second").await, None); + // The row stayed AVAILABLE through both refusals. + let status: String = sqlx::query_scalar("SELECT status FROM backup_index WHERE backup_id = $1") + .bind(&row.backup_id) + .fetch_one(&s.catalog) + .await + .expect("status"); + assert_eq!(status, "AVAILABLE"); + drop(claim); + let _ = key; + s.cleanup().await; +} + +/// A taken name: an ACTIVE table of the name is `TableAlreadyExists`; a +/// CREATING restore target of the name is `TableBeingRestored` naming its +/// source backup, whether the first registration's lock is still held or +/// its owner is gone. +#[tokio::test] +async fn a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress() { + if !pg_available("a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + s.engine + .create_table(ACCOUNT, target_input("Taken", false)) + .await + .expect("create a table of the name"); + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Taken", s.vector), + ®istration(&row, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableAlreadyExists(n) if n == "Taken"), + "{err:?}" + ); + + let (_desc, _key, claim) = register(&s, &row, "Restoring", None).await; + let other = available_backup(&s, "Other").await; + // With the first claim's lock held, the name lock refuses first. + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Restoring", s.vector), + ®istration(&other, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableBeingRestored { table_name, backup_arn } + if table_name == "Restoring" && *backup_arn == row.backup_arn), + "{err:?}" + ); + // With the lock gone (a dead owner), the unique index refuses the same way. + drop(claim); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Restoring").await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "the lock session did not end" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Restoring", s.vector), + ®istration(&other, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableBeingRestored { table_name, .. } if table_name == "Restoring"), + "{err:?}" + ); + s.cleanup().await; +} + +// --- the takeover ------------------------------------------------------------- + +/// `claim_restore_dead` is refused while the owner's lock session lives and +/// while the heartbeat is young, and taken once both are gone: a fresh +/// generation, the claimant's `spool_host` and a refreshed `heartbeat_at` +/// (the database's clock) in one write, so a second taker finds the record +/// live at once; `list_dead_restore_jobs` lists the record with its members +/// only once stale; the old generation's writes are `NotMatched`. +#[tokio::test] +async fn a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat() { + if !pg_available("a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Target", Some("host-a:/spool")).await; + let old_generation = claim.generation(); + // A live owner: the lock is held. + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .is_none(), + "a held lock is a live owner" + ); + // The lock gone (the session ended) but the heartbeat young. + let holder = lock_holder(&s, "Target") + .await + .expect("the owner's session"); + sqlx::query("SELECT pg_terminate_backend($1)") + .bind(holder) + .execute(&s.catalog) + .await + .expect("end the owner's session"); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Target").await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "the lock did not go" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + // The owner's next heartbeat is lost: its session is gone. + assert!( + !matches!(claim.heartbeat().await, Ok(Heartbeat::Confirmed)), + "a terminated claim session cannot confirm" + ); + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .is_none(), + "a young heartbeat is a live owner" + ); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "a young record is not listed" + ); + assert_eq!( + record(&s, &key).await.spool_host.as_deref(), + Some("host-a:/spool") + ); + + // Stale: listed with its members, and taken over. + let aged = age_heartbeat(&s, &key.table_id, Duration::from_secs(3600)).await; + let dead = s.engine.list_dead_restore_jobs(STALE).await.expect("list"); + assert_eq!(dead.len(), 1); + assert_eq!(dead[0].target, key); + assert_eq!(dead[0].kind, RestoreKind::Manifest); + assert_eq!(dead[0].status, JobStatus::Running); + assert_eq!(dead[0].store, Some(store())); + assert_eq!(dead[0].spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!(dead[0].owner_generation, old_generation); + let successor = s + .engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .expect("a stale record with no lock is dead"); + assert_ne!(successor.generation(), old_generation); + let (status, _, generation, heartbeat_at, spool_host, _) = + record_columns(&s, &key.table_id).await; + assert_eq!(status, "RUNNING"); + assert_eq!(generation, i64::try_from(successor.generation()).unwrap()); + assert_eq!( + spool_host.as_deref(), + Some("host-b:/spool"), + "the claimant's spool" + ); + assert!( + heartbeat_at > aged + 3_000.0, + "the takeover refreshed heartbeat_at: {heartbeat_at} vs the aged {aged}" + ); + let db_now: f64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM now())::FLOAT8") + .fetch_one(&s.catalog) + .await + .expect("the database's clock"); + assert!( + (db_now - heartbeat_at).abs() < 5.0, + "the refreshed heartbeat is the database's now: {heartbeat_at} vs {db_now}" + ); + assert!( + lock_holder(&s, "Target").await.is_some(), + "the successor holds the lock" + ); + // A second taker is refused while the successor lives: the lock, and + // the fresh heartbeat (the window a second instance's reconciler would + // otherwise find the record stale in). + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-c:/spool")) + .await + .expect("takeover") + .is_none() + ); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "the taken-over record is live to the listing at once" + ); + // The old generation's writes match nothing: the flip through the old + // claim, and the old claim's status write (its session is gone, so the + // write's outcome is unknown and the re-read shows the successor). + assert!(matches!( + s.engine + .finish_restore(&key, &*claim, 0, 0) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + let old_write = claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("stale".to_owned()), + completed: true, + ..StatusFields::default() + }, + ) + .await; + assert!( + matches!(old_write, Ok(WriteOutcome::Unknown) | Err(_)), + "{old_write:?}" + ); + let reread = claim.read().await.expect("the pool re-read"); + assert_eq!(reread.owner_generation, successor.generation()); + assert_eq!(reread.status, JobStatus::Running); + // The successor's heartbeat confirms and its status write matches; after + // its FAILED write a second write from it is NotMatched (not RUNNING). + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("count".to_owned()), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Done, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched + ); + let job = record(&s, &key).await; + assert_eq!(job.status, JobStatus::Failed); + assert_eq!(job.failure_reason.as_deref(), Some("count")); + assert!(job.completed_at.is_some()); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "a fresh FAILED record is left to its job's removal" + ); + drop(successor); + s.cleanup().await; +} + +// --- the load ------------------------------------------------------------------- + +/// `bulk_load` writes the base rows, the GSI, LSI and vector rows and the +/// file's progress row in one data transaction, across the 500-row batch +/// boundary; `start_at` is added to the recorded total; a duplicate key +/// within a file or against an earlier file fails the file with nothing of +/// it committed, no progress row included; a file whose progress is already +/// recorded fails whole, its index rows included. +#[tokio::test] +async fn bulk_load_writes_each_file_with_its_indexes_and_progress_in_one_transaction() { + if !pg_available("bulk_load_writes_each_file_with_its_indexes_and_progress_in_one_transaction") + { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Loaded", None).await; + + // File 0: 1200 items, past the 500-row batch. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..1200)), + ) + .await + .expect("load file 0"); + assert_eq!(applied, 1200); + assert_eq!(base_rows(&s, &key.table_id).await, 1200); + assert_eq!( + index_rows(&s, &key.table_id, "gix").await, + 600, + "every second item has g" + ); + assert_eq!( + index_rows(&s, &key.table_id, "lix").await, + 400, + "every third item has l" + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 240, + "every fifth item has emb" + ); + } + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 1200)]); + assert_eq!( + s.engine.count_restore_target(&key).await.expect("count"), + 1200 + ); + + // File 1 resumed at item 5 of its 8: the record carries the running total. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 5, + reapplying: false, + }, + stream(items(1205..1208)), + ) + .await + .expect("load file 1"); + assert_eq!(applied, 3); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + + // File 2 holds a key twice: nothing of it lands, no progress row. + let mut twice = items(2000..2010); + twice.push(item(2003)); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 2, + start_at: 0, + reapplying: false, + }, + stream(twice), + ) + .await + .expect_err("a duplicate key fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) + if m.starts_with("holds a primary key the target already has") + && m.contains("the backup holds a key twice")), + "the backend names no file (the engine prefixes the file's key): {err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 3 repeats a key file 0 wrote: the same. + let mut across = items(3000..3004); + across.push(item(7)); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 3, + start_at: 0, + reapplying: false, + }, + stream(across), + ) + .await + .expect_err("a key of an earlier file fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) + if m.starts_with("holds a primary key the target already has")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + index_rows(&s, &key.table_id, "gix").await, + 600 + 1, + "1206 has g" + ); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // A decode or validation failure the engine found mid-stream: the items + // before it are not committed either. + let failing: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::iter( + items(4000..4003) + .into_iter() + .map(Ok) + .chain(std::iter::once(Err(StorageError::Validation( + "key {\"pk\": \"p1\"}, attribute n: 1E126 is not a number".to_owned(), + )))), + )); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 4, + start_at: 0, + reapplying: false, + }, + failing, + ) + .await + .expect_err("an Err item fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) if m.contains("1E126")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 0 again with keys the target lacks: its rows and index rows land + // in the file's transaction, and the progress row's primary key rolls + // the whole file back, index rows included. + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(5000..5004)), + ) + .await + .expect_err("a recorded file is not applied twice"); + assert!( + matches!(&err, StorageError::Validation(m) if m.contains("data file 0")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!(index_rows(&s, &key.table_id, "gix").await, 601); + assert_eq!( + index_rows(&s, &key.table_id, "lix").await, + 400 + 1, + "1206 has l" + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 240 + 1, + "1205 has emb" + ); + } + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 1's row says 8 of its items were applied and the file holds 11 + // (the decoder died after the file's commit and the job's FAILED write + // never landed): the resume starts the file at 8, and the rest completes + // it, the row updated in place instead of refused on its key. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 8, + reapplying: false, + }, + stream(items(1208..1211)), + ) + .await + .expect("a file resumed from a short progress row completes"); + assert_eq!(applied, 3); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 11)], + "the short row is updated to the file's total" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1206); + assert_eq!(index_rows(&s, &key.table_id, "gix").await, 603); + assert_eq!(index_rows(&s, &key.table_id, "lix").await, 402); + if s.vector { + assert_eq!(vector_rows(&s, &key.table_id).await, 242); + } + + // Another account's key, or a wrong table id, reaches no table. + let foreign = RestoreTargetKey { + account_id: "210987654321".to_owned(), + ..key.clone() + }; + assert!(matches!( + s.engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert!(matches!( + s.engine + .bulk_load( + &foreign, + &*claim, + BulkLoadFile { + file_index: 9, + start_at: 0, + reapplying: false, + }, + stream(items(9000..9001)), + ) + .await, + Err(StorageError::TableNotFound(_)) + )); + drop(claim); + s.cleanup().await; +} + +/// `finish_restore` marks the record DONE and the table and every index +/// ACTIVE in one transaction, conditioned on the claim's generation and on +/// CREATING, then removes the progress rows; a second call, or one under a +/// stale generation, is `NotMatched` and writes nothing. The progress sweep +/// keeps a RUNNING target's rows and removes any other's; +/// `remove_restore_target` deletes a target's rows. +#[tokio::test] +async fn finish_restore_flips_the_table_and_every_index_and_clears_the_progress() { + if !pg_available("finish_restore_flips_the_table_and_every_index_and_clears_the_progress") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Done", None).await; + let other_row = available_backup(&s, "Other").await; + let (_desc2, other, other_claim) = register(&s, &other_row, "Still", None).await; + for (k, c) in [(&key, &claim), (&other, &other_claim)] { + s.engine + .bulk_load( + k, + &**c, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + } + // An index a backend left CREATING (none does on this backend; the flip + // states the rule once) is moved with the table. + sqlx::query( + "UPDATE indexes SET index_status = 'CREATING' WHERE table_id = $1 AND index_name = 'gix'", + ) + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("set an index CREATING"); + if s.vector { + sqlx::query("UPDATE vector_indexes SET index_status = 'CREATING' WHERE table_id = $1") + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("set the vector index CREATING"); + } + + // The sweep: `Still`'s rows stay while it is RUNNING; a row of a table + // that is gone goes. + sqlx::query( + "INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ('gone', 0, 5)", + ) + .execute(&s.db) + .await + .expect("an orphan row"); + // A caller whose RUNNING list predates both registrations (the list is + // read before the sweep; a registration and its first file can land in + // between): the sweep protects the RUNNING records it reads itself after + // the candidates, so neither live row goes, only the orphan. + assert_eq!( + s.engine.sweep_restore_progress(&[]).await.expect("sweep"), + 1, + "the orphan row alone, with an empty caller list" + ); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + sqlx::query( + "INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ('gone', 0, 5)", + ) + .execute(&s.db) + .await + .expect("the orphan row again"); + let running: Vec = s + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list") + .into_iter() + .map(|j| j.target.table_id) + .collect(); + assert_eq!(running.len(), 2); + assert_eq!( + s.engine + .sweep_restore_progress(&running) + .await + .expect("sweep"), + 1 + ); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + assert_eq!( + s.engine.read_restore_progress(&key).await.expect("read"), + vec![(0, 30)] + ); + + // A stale generation flips nothing. + let stale_row = available_backup(&s, "Stale").await; + let (_d, stale_key, stale_claim) = register(&s, &stale_row, "StaleOwner", None).await; + let _ = stale_key; + assert!(matches!( + s.engine + .finish_restore(&key, &*stale_claim, 30, 4096) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!(table_status(&s, "Done").await.as_deref(), Some("CREATING")); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + drop(stale_claim); + + assert_eq!( + s.engine + .finish_restore(&key, &*claim, 30, 4096) + .await + .expect("finish"), + Finished::Active + ); + assert_eq!(table_status(&s, "Done").await.as_deref(), Some("ACTIVE")); + let (status, _, _, _, _, completed_at) = record_columns(&s, &key.table_id).await; + assert_eq!(status, "DONE"); + assert!(completed_at.is_some()); + let counts: (i64, i64) = + sqlx::query_as("SELECT item_count, table_size_bytes FROM tables WHERE table_id = $1") + .bind(&key.table_id) + .fetch_one(&s.catalog) + .await + .expect("counts"); + assert_eq!(counts, (30, 4096)); + assert!( + indexes(&s, &key.table_id) + .await + .iter() + .all(|(_, _, status)| status == "ACTIVE"), + "every secondary index is ACTIVE" + ); + if s.vector { + assert!( + vector_indexes(&s, &key.table_id) + .await + .iter() + .all(|(_, status)| status == "ACTIVE"), + "the vector index is ACTIVE" + ); + } + assert!( + progress_rows(&s, &key.table_id).await.is_empty(), + "the progress rows are gone" + ); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + let job = record(&s, &key).await; + assert_eq!(job.status, JobStatus::Done); + assert!(job.completed_at.is_some()); + // The job record's heartbeat is no longer refreshed: not RUNNING. + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); + // A second flip matches nothing. + assert!(matches!( + s.engine + .finish_restore(&key, &*claim, 30, 4096) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + // The restored table reads through the data plane as any ACTIVE table. + let desc = s + .engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "Done".to_owned(), + }, + ) + .await + .expect("describe"); + assert_eq!(desc.table_status, TableStatus::Active); + assert_eq!(desc.item_count, 30); + + // `remove_restore_target` on the other target deletes its progress rows + // with the target. + assert!( + s.engine + .remove_restore_target(&other) + .await + .expect("remove") + ); + assert!(progress_rows(&s, &other.table_id).await.is_empty()); + assert_eq!(table_status(&s, "Still").await, None); + drop(other_claim); + drop(claim); + s.cleanup().await; +} + +/// `remove_restore_target` of a target with a GSI, an LSI and (with pgvector) +/// a vector index drops every relation the target's rows named: the data +/// table, each secondary index's table and the vector index's `_ddb_vec_` +/// table, so a failed vector restore leaves no relation holding its items +/// with no catalog row naming it. +#[tokio::test] +async fn remove_restore_target_drops_the_vector_relation_with_the_data_table() { + if !pg_available("remove_restore_target_drops_the_vector_relation_with_the_data_table") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Vectors", None).await; + s.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + let mut relations: Vec = vec![format!("_ddb_{}", key.table_id)]; + relations.extend( + indexes(&s, &key.table_id) + .await + .into_iter() + .map(|(_, id, _)| format!("_ddb_{id}")), + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 6, + "every fifth item has emb" + ); + relations.extend( + vector_indexes(&s, &key.table_id) + .await + .into_iter() + .map(|(id, _)| format!("_ddb_vec_{id}")), + ); + assert_eq!( + relations.len(), + 4, + "the data table, two index tables, one vector table" + ); + } + for relation in &relations { + assert!( + relation_exists(&s, relation).await, + "{relation} exists before the removal" + ); + } + + assert!(s.engine.remove_restore_target(&key).await.expect("remove")); + assert_eq!(table_status(&s, "Vectors").await, None); + if s.vector { + assert!(vector_indexes(&s, &key.table_id).await.is_empty()); + } + for relation in &relations { + assert!( + !relation_exists(&s, relation).await, + "{relation} is dropped with the target" + ); + } + drop(claim); + s.cleanup().await; +} + +/// A `finish_restore` whose progress delete fails after the flip committed +/// (the data database cannot run the `DELETE`: the `restore_progress` table +/// is renamed under it) is `Active`: the table is ACTIVE and the record DONE, +/// the failure is logged and not returned, the rows are left, and the +/// reconciler's `sweep_restore_progress` removes them once the record is no +/// longer RUNNING. +#[tokio::test] +async fn a_progress_delete_failing_after_the_flip_leaves_the_restore_finished() { + if !pg_available("a_progress_delete_failing_after_the_flip_leaves_the_restore_finished") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Flipped", None).await; + s.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + + sqlx::query("ALTER TABLE restore_progress RENAME TO restore_progress_away") + .execute(&s.db) + .await + .expect("take the progress table away"); + let finished = s.engine.finish_restore(&key, &*claim, 30, 4096).await; + sqlx::query("ALTER TABLE restore_progress_away RENAME TO restore_progress") + .execute(&s.db) + .await + .expect("bring the progress table back"); + assert!( + matches!(finished, Ok(Finished::Active)), + "a progress delete failing after the flip is not the restore's failure: {finished:?}" + ); + assert_eq!(table_status(&s, "Flipped").await.as_deref(), Some("ACTIVE")); + assert_eq!(record(&s, &key).await.status, JobStatus::Done); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 30)], + "the rows the delete could not remove are left" + ); + assert_eq!( + s.engine.sweep_restore_progress(&[]).await.expect("sweep"), + 1, + "the reconciler's sweep removes the DONE target's rows" + ); + assert!(progress_rows(&s, &key.table_id).await.is_empty()); + drop(claim); + s.cleanup().await; +} + +/// Terminating the claim session: the next heartbeat is lost, the status +/// write's outcome is unknown, and the record stays RUNNING for the +/// reconciler once stale. +#[tokio::test] +async fn a_terminated_claim_session_loses_its_next_heartbeat() { + if !pg_available("a_terminated_claim_session_loses_its_next_heartbeat") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Cut", None).await; + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Confirmed); + let holder = lock_holder(&s, "Cut").await.expect("the claim session"); + sqlx::query("SELECT pg_terminate_backend($1)") + .bind(holder) + .execute(&s.catalog) + .await + .expect("terminate"); + let beat = claim.heartbeat().await; + assert!(!matches!(beat, Ok(Heartbeat::Confirmed)), "{beat:?}"); + let write = claim + .set_status( + JobStatus::Running, + JobStatus::Done, + &StatusFields::default(), + ) + .await; + assert!( + matches!(write, Ok(WriteOutcome::Unknown) | Err(_)), + "{write:?}" + ); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + assert_eq!(table_status(&s, "Cut").await.as_deref(), Some("CREATING")); + s.cleanup().await; +} + +/// A legacy restore record is not a takeover candidate: `claim_restore_dead` +/// leaves a stale `kind = legacy` record to the reconciler's legacy duty. +#[tokio::test] +async fn a_legacy_record_is_not_taken_over() { + if !pg_available("a_legacy_record_is_not_taken_over") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Legacyish", None).await; + drop(claim); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Legacyish").await.is_some() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(20)).await; + } + sqlx::query("UPDATE table_restores SET kind = 'legacy' WHERE table_id = $1") + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("make the record legacy"); + age_heartbeat(&s, &key.table_id, Duration::from_secs(3600)).await; + assert!( + s.engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("takeover") + .is_none() + ); + assert_eq!(record(&s, &key).await.kind, RestoreKind::Legacy); + assert_eq!(lock_holder(&s, "Legacyish").await, None); + s.cleanup().await; +} + +/// `sweep_restore_progress` reads the candidate progress-row ids before the +/// live RUNNING set, so a registration that commits and writes a progress +/// row between the two reads keeps that row: the row is not among the +/// candidates read first, and the record is RUNNING in the live set read +/// after. An orphan row with no record is swept in the same call, so the +/// sweep is seen to run. With the two reads swapped the registration's row +/// would be a candidate and not live, and would be deleted. +#[tokio::test] +async fn sweep_keeps_a_progress_row_registered_between_the_two_reads() { + if !pg_available("sweep_keeps_a_progress_row_registered_between_the_two_reads") { + return; + } + let s = scratch().await; + // An orphan progress row (no record names it): a candidate the sweep + // removes, so the sweep is doing work in this cycle. + sqlx::query("INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ('orphan-tid', 0, 5)") + .execute(&s.db) + .await + .expect("seed an orphan progress row"); + + let (arrived, proceed) = extenddb_storage_postgres::sweep_pause::arm(); + let engine = s.engine.clone(); + let sweep = tokio::spawn(async move { engine.sweep_restore_progress(&[]).await }); + + // The sweep has read the candidate ids and is paused before the live + // set: register a RUNNING target and write its progress row now. + arrived.notified().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Late", None).await; + sqlx::query( + "INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ($1, 0, 7)", + ) + .bind(&key.table_id) + .execute(&s.db) + .await + .expect("the registration's progress row"); + proceed.notify_one(); + + let removed = sweep.await.expect("join").expect("sweep"); + assert_eq!(removed, 1, "only the orphan progress row was swept"); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 7)], + "the live registration's progress row stayed" + ); + assert!( + progress_rows(&s, "orphan-tid").await.is_empty(), + "the orphan row went" + ); + drop(claim); + s.cleanup().await; +} diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index 8fa162aff..494434b0c 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -72,9 +72,7 @@ use extenddb_storage::util::{composite_pk_to_text, parse_sk, sk_column_n}; use extenddb_storage::{BackupRestoreSource, JobPermit, LegacyBackupReader, RestoreRequest}; use futures::future::BoxFuture; -use crate::backup_catalog::{ - JobSlots, NOW, SqliteJobClaim, generation_column, legacy_backup_slot_key, -}; +use crate::backup_catalog::{JobSlots, SqliteJobClaim, generation_column, legacy_backup_slot_key}; use crate::create_table::RestoreRegistration; use crate::data::{ BoundValue, all_sort_key_info, data_table_name, index_table_name, insert_index_row_multi, @@ -439,19 +437,16 @@ impl SqliteEngine { // generation, and the table ACTIVE only from CREATING, which // the check above confirmed under the same lock. A record // another claim took over leaves the table to that owner. - let done = sqlx::query(&format!( - "UPDATE table_restores SET status = 'DONE', items_applied = ?, \ - completed_at = {NOW} \ - WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" - )) - .bind(count) - .bind(&desc.table_id) - .bind(generation_column(generation)) - .execute(&mut *tx) - .await - .map_err(db_err)? - .rows_affected(); - if done != 1 { + let activated = crate::restore_engine::activate_restore_target( + &mut tx, + &desc.table_id, + generation, + count, + table_size, + Some(count), + ) + .await?; + if !activated.record { tx.rollback().await.map_err(db_err)?; return Err(StorageError::TableNotFound(format!( "restore of {backup_arn} into {} did not complete: its job record \ @@ -459,17 +454,6 @@ impl SqliteEngine { desc.table_name ))); } - sqlx::query( - "UPDATE tables SET item_count = ?, table_size_bytes = ?, \ - table_status = 'ACTIVE', status_transition_at = NULL \ - WHERE table_id = ? AND table_status = 'CREATING'", - ) - .bind(count) - .bind(table_size) - .bind(&desc.table_id) - .execute(&mut *tx) - .await - .map_err(db_err)?; tx.commit().await.map_err(db_err)?; return Ok(count); }; @@ -3699,7 +3683,7 @@ mod restore_tests { #[tokio::test] async fn a_restart_ages_an_interrupted_restore_so_the_reconciler_removes_it() { use extenddb_storage::BackupCatalog; - use extenddb_storage::backup_catalog::{RestoreTargetKey, UNOWNED_GENERATION}; + use extenddb_storage::backup_catalog::{RestoreKind, RestoreTargetKey, UNOWNED_GENERATION}; let engine = engine().await; indexed_table(&engine, "src").await; let backup = engine @@ -3743,7 +3727,7 @@ mod restore_tests { .expire_restore_job_heartbeats() .await .expect("expire"); - assert_eq!(aged, vec!["dst".to_owned()]); + assert_eq!(aged, vec![("dst".to_owned(), RestoreKind::Legacy)]); let (heartbeat, record_generation): (String, i64) = sqlx::query_as( "SELECT heartbeat_at, owner_generation FROM table_restores WHERE table_id = ?", ) @@ -5891,7 +5875,7 @@ mod restore_tests { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind, RestoreTargetKey}; use super::*; - use crate::backup_catalog::restore_slot_key; + use crate::backup_catalog::legacy_restore_slot_key; const STALE: Duration = Duration::from_secs(300); @@ -5937,7 +5921,7 @@ mod restore_tests { let desc = start_restore(&engine, "dst", &backup.backup_arn) .await .expect("registers"); - let slot = restore_slot_key(&desc.table_id); + let slot = legacy_restore_slot_key(&desc.table_id); assert!( engine.job_slots.holds(&slot), "the restore holds its slot from registration" @@ -6157,7 +6141,7 @@ mod restore_tests { .await .expect("registers"); let key = key_of(&desc); - let slot = restore_slot_key(&desc.table_id); + let slot = legacy_restore_slot_key(&desc.table_id); // Hold the copy between two batches and write the record FAILED // under its own generation (the write lock is the test's, as a diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs index 562f94b0b..49db0f280 100644 --- a/crates/storage-sqlite/src/backup_catalog.rs +++ b/crates/storage-sqlite/src/backup_catalog.rs @@ -17,7 +17,12 @@ //! ([`BackupCatalog::claim_dead_backup`]) reads no stale window here, and //! every RUNNING restore job record of the stopped process is aged at startup //! so the reconciler's first cycle finds it -//! ([`SqliteEngine::expire_restore_job_heartbeats`]). +//! ([`SqliteEngine::expire_restore_job_heartbeats`]). A store-backed +//! restore's claim holds the slot of its target name +//! ([`restore_slot_key`]) the same way: the registration takes it before the +//! target exists, so two registrations into one name exclude each other in +//! this process, and a RUNNING `manifest` record whose slot no task holds is +//! taken over at once by [`BackupCatalog::claim_restore_dead`]. //! //! Every write a job makes to its row goes through its claim and carries the //! claim's `owner_generation`, so a row a successor took over matches no @@ -43,11 +48,13 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::JobPermit; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -114,12 +121,21 @@ pub(crate) fn legacy_backup_slot_key(backup_arn: &str) -> String { format!("legacy-backup:{backup_arn}") } -/// The slot key of a restore's hold on its job record, by target table id. -/// Held for the restore's life ([`SqliteJobClaim::for_restore`]), so the -/// reconciler never judges a running restore of this process by its -/// heartbeat, which queues behind every writer holding the write lock. -pub(crate) fn restore_slot_key(table_id: &str) -> String { - format!("restore:{table_id}") +/// The slot key of an in-catalog restore's hold on its job record, by +/// target table id. Held for the restore's life +/// ([`SqliteJobClaim::for_restore`]), so the reconciler never judges a +/// running restore of this process by its heartbeat, which queues behind +/// every writer holding the write lock. +pub(crate) fn legacy_restore_slot_key(table_id: &str) -> String { + format!("legacy-restore:{table_id}") +} + +/// The slot key of a store-backed restore's claim on its target name: taken +/// before the target exists, so two registrations into one name in this +/// process exclude each other before either reaches the catalog, and held +/// by the claim for the job's life. +pub(crate) fn restore_slot_key(account_id: &str, table_name: &str) -> String { + format!("restore:{account_id}/{table_name}") } /// An `owner_generation` as the catalog stores it (`INTEGER`); generations @@ -152,7 +168,7 @@ impl SqliteJobClaim { /// A claim on a restore job record, born with the registration that /// writes `generation` on the row. The permit is the job's slot in the /// engine's `max_concurrent_jobs`, freed with the claim. The in-process - /// slot ([`restore_slot_key`]) is the SQLite rule of design 4(e): a + /// slot ([`legacy_restore_slot_key`]) is the SQLite rule of design 4(e): a /// RUNNING record whose slot a task of this process holds is alive /// whatever its heartbeat says, so a heartbeat delayed behind a long /// writer (an UpdateTable adding a GSI to a large table holds the write @@ -168,7 +184,7 @@ impl SqliteJobClaim { ) -> Self { Self { engine: engine.clone(), - _slot: JobSlots::take(&engine.job_slots, restore_slot_key(table_id)), + _slot: JobSlots::take(&engine.job_slots, legacy_restore_slot_key(table_id)), _permit: Some(permit), generation, record: JobRecord::Restore { @@ -177,6 +193,27 @@ impl SqliteJobClaim { } } + /// A claim on a store-backed restore's job record, born with the + /// registration that writes `generation` on the row, holding the + /// target name's slot ([`restore_slot_key`]) for the job's life. The + /// engine's job holds the permit; none is held here. + pub(crate) fn for_manifest_restore( + engine: &SqliteEngine, + table_id: &str, + generation: u64, + slot: JobSlot, + ) -> Self { + Self { + engine: engine.clone(), + _slot: Some(slot), + _permit: None, + generation, + record: JobRecord::Restore { + table_id: table_id.to_owned(), + }, + } + } + /// Heartbeat the record every `period` until a heartbeat is not /// confirmed; returns why. Never returns while the claim is confirmed. pub(crate) async fn heartbeat_loop(&self, period: Duration) -> StorageError { @@ -313,15 +350,15 @@ impl JobClaim for SqliteJobClaim { .await } JobRecord::Restore { table_id } => { + // The record's progress (`file_index`, `items_applied`) + // is `bulk_load`'s write; a status write leaves it. sqlx::query(&format!( "UPDATE table_restores SET status = ?, \ - items_applied = COALESCE(?, items_applied), \ failure_reason = COALESCE(?, failure_reason), \ completed_at = CASE WHEN ? THEN {NOW} ELSE completed_at END \ WHERE table_id = ? AND owner_generation = ? AND status = ?" )) .bind(to.as_str()) - .bind(fields.item_count) .bind(fields.failure_reason.as_deref()) .bind(fields.completed) .bind(table_id) @@ -530,10 +567,36 @@ fn zero_window_clause(window: Duration, column: &str) -> String { } } -/// A restore job record as the two listings select it. +/// The columns of one restore job record joined with its table, in the +/// order [`RestoreJobTuple`] reads them; every read of a record selects +/// them. +const RESTORE_ROW_COLUMNS: &str = "r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ + r.kind, r.status, r.owner_generation, r.heartbeat_at, r.store_kind, r.store_root, \ + r.store_id, r.spool_host, r.file_index, r.items_applied, r.failure_reason, r.completed_at"; + +/// One restore job record as the driver reads it. #[allow(clippy::type_complexity)] -fn restore_job_row( - ( +type RestoreJobTuple = ( + String, + String, + String, + String, + String, + String, + i64, + String, + Option, + Option, + Option, + Option, + Option, + Option, + Option, + Option, +); + +fn restore_job_from_tuple(t: RestoreJobTuple) -> Result { + let ( table_id, account_id, table_name, @@ -542,8 +605,27 @@ fn restore_job_row( status, owner_generation, heartbeat_at, - ): (String, String, String, String, String, String, i64, String), -) -> Result { + store_kind, + store_root, + store_id, + spool_host, + file_index, + items_applied, + failure_reason, + completed_at, + ) = t; + let store = match (store_kind, store_root, store_id) { + (Some(kind), Some(store_root), Some(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root, + store_id, + }), + _ => None, + }; Ok(RestoreJobRow { target: RestoreTargetKey { account_id, @@ -555,6 +637,12 @@ fn restore_job_row( status: JobStatus::parse(&status)?, owner_generation: u64::try_from(owner_generation).unwrap_or_default(), heartbeat_at: timestamp_epoch_seconds(&heartbeat_at), + store, + spool_host, + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason, + completed_at: completed_at.as_deref().map(timestamp_epoch_seconds), }) } @@ -562,28 +650,34 @@ impl SqliteEngine { /// Every RUNNING restore job record at startup was owned by the process /// that stopped: one process serves a file, so a restart is proof of /// death. Set each record's heartbeat to the epoch, so the reconciler's - /// first cycle fails it and removes its target without waiting the stale - /// window. Returns the names of the targets whose record was aged (one - /// already at the epoch is not reported again). Runs before the server - /// takes requests, so no restore of this process can be in flight. - pub(crate) async fn expire_restore_job_heartbeats(&self) -> Result, StorageError> { + /// first cycle acts on it (a store-backed record is taken over and + /// resumed, a legacy one failed and its target removed) without waiting + /// the stale window. Returns the name and kind of each target whose + /// record was aged (one already at the epoch is not reported again). + /// Runs before the server takes requests, so no restore of this process + /// can be in flight. + pub(crate) async fn expire_restore_job_heartbeats( + &self, + ) -> Result, StorageError> { let _writer = self.write_lock.lock().await; let mut tx = self .pool .begin_with("BEGIN IMMEDIATE") .await .map_err(db_err)?; - let names: Vec = sqlx::query_scalar( + let rows: Vec<(String, String)> = sqlx::query_as( "UPDATE table_restores SET heartbeat_at = '1970-01-01T00:00:00.000Z' \ WHERE status = 'RUNNING' AND heartbeat_at > '1970-01-01T00:00:00.000Z' \ RETURNING COALESCE((SELECT table_name FROM tables t \ - WHERE t.table_id = table_restores.table_id), table_id)", + WHERE t.table_id = table_restores.table_id), table_id), kind", ) .fetch_all(&mut *tx) .await .map_err(db_err)?; tx.commit().await.map_err(db_err)?; - Ok(names) + rows.into_iter() + .map(|(name, kind)| Ok((name, RestoreKind::parse(&kind)?))) + .collect() } /// Remove the maintenance claim, whoever holds it: the server's startup @@ -993,23 +1087,11 @@ impl BackupCatalog for SqliteEngine { if !is_target { return Ok(false); } - // Progress rows belong to a later release's restore path; a - // catalog without that table has none to delete. - let has_progress: bool = sqlx::query_scalar( - "SELECT EXISTS(SELECT 1 FROM sqlite_master \ - WHERE type = 'table' AND name = 'restore_progress')", - ) - .fetch_one(&self.pool) - .await - .map_err(db_err)?; - if has_progress { - let _writer = self.write_lock.lock().await; - sqlx::query("DELETE FROM restore_progress WHERE table_id = ?") - .bind(&key.table_id) - .execute(&self.pool) - .await - .map_err(db_err)?; - } + // A store-backed restore's progress lives on the `table_restores` + // row (`file_index`, `items_applied`); SQLite has + // no `restore_progress` table (migration 7 is `NotApplicable` + // here), so the record goes with the target below and there are + // no rows to delete. self.abort_restore(&key.table_id).await }) } @@ -1028,24 +1110,13 @@ impl BackupCatalog for SqliteEngine { // a task still on its own failure path to finish it. Both ages // are measured on the database's clock (`julianday('now')`); a // zero window selects by status alone (`zero_window_clause`). - #[allow(clippy::type_complexity)] - let rows: Vec<( - String, - String, - String, - String, - String, - String, - i64, - String, - )> = sqlx::query_as(&format!( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, r.heartbeat_at \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE (r.status = 'RUNNING' AND {}) \ - OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ - AND {}) \ - ORDER BY t.account_id, t.table_name", + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (r.status = 'RUNNING' AND {}) \ + OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ + AND {}) \ + ORDER BY t.account_id, t.table_name", zero_window_clause(stale_after, "r.heartbeat_at"), zero_window_clause(stale_after, "COALESCE(r.completed_at, r.heartbeat_at)"), )) @@ -1057,21 +1128,25 @@ impl BackupCatalog for SqliteEngine { // A RUNNING record whose in-process slot a task holds is skipped, // however old its heartbeat: one process serves the file, so the // slot is the liveness rule and the heartbeat only what a restart - // ages (design 4(e)). A row this build cannot read (a kind or - // status no release writes) is logged and skipped, as ListBackups - // skips a row it cannot read, so one such row does not stop the - // cycle. + // ages (design 4(e)); an in-catalog restore holds its table id's + // slot, a store-backed one its target name's. A row this build + // cannot read (a kind or status no release writes) is logged and + // skipped, as ListBackups skips a row it cannot read, so one such + // row does not stop the cycle. Ok(rows .into_iter() .filter_map(|row| { let (table_id, account_id, table_name, status) = (row.0.clone(), row.1.clone(), row.2.clone(), row.5.clone()); if status == JobStatus::Running.as_str() - && self.job_slots.holds(&restore_slot_key(&table_id)) + && (self.job_slots.holds(&legacy_restore_slot_key(&table_id)) + || self + .job_slots + .holds(&restore_slot_key(&account_id, &table_name))) { return None; } - match restore_job_row(row) { + match restore_job_from_tuple(row) { Ok(job) => Some(job), Err(e) => { tracing::warn!( @@ -1090,26 +1165,164 @@ impl BackupCatalog for SqliteEngine { Box::pin(async move { // By status alone: a record stamped in this statement's instant, // or ahead of the database's clock, is RUNNING too. - #[allow(clippy::type_complexity)] - let rows: Vec<( - String, - String, - String, - String, - String, - String, - i64, - String, - )> = sqlx::query_as( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, r.heartbeat_at \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name", - ) + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.status = 'RUNNING' ORDER BY t.account_id, t.table_name" + )) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() + }) + } + + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let table_name = input.table_name.clone(); + // The name's slot first, before the target exists (module doc): + // a slot already held is another store-backed restore into this + // name in flight in this process. It is refused as the name is + // taken: TableBeingRestored naming its source once its CREATING + // row is visible, else as CreateTable would report it. + let Some(slot) = + JobSlots::take(&self.job_slots, restore_slot_key(account_id, &table_name)) + else { + return Err(self.name_taken_error(account_id, &table_name).await); + }; + let generation = new_generation(); + let registration = RestoreRegistrationRow { + owner_generation: generation, + ..registration.clone() + }; + // The registration transaction (`BEGIN IMMEDIATE` under the + // engine write lock): the `backup_index` row's status, the + // in-use check, the CREATING target and its RUNNING record. + let mut desc = match self + .create_table_for_manifest_restore(account_id, input, ®istration) + .await + { + Ok(desc) => desc, + Err(e) => { + drop(slot); + return Err(e); + } + }; + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(registration.backup.backup_arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let claim = + SqliteJobClaim::for_manifest_restore(self, &desc.table_id, generation, slot); + Ok((desc, Box::new(claim) as Box)) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + _stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The slot is the liveness test, as for `claim_dead_backup`: a + // job of this process that holds the name's slot is alive + // whatever its heartbeat says, and a record without a holder is + // dead at once (a restart frees every slot, and ages every + // RUNNING record so the reconciler lists it). Taking the slot is + // the test and the exclusion of a second taker in one step. + let Some(slot) = JobSlots::take( + &self.job_slots, + restore_slot_key(&key.account_id, &key.table_name), + ) else { + return Ok(None); + }; + let generation = new_generation(); + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + // The generation, the claimant's spool and a fresh heartbeat in + // one write (the trait doc says why a restore record is refreshed + // where a backup row is not; here the slot already excludes a + // second taker of this process, and the beat keeps the record + // out of the reconciler's dead listing). A legacy record is + // never taken over: the reconciler fails and removes it. + let taken = sqlx::query(&format!( + "UPDATE table_restores SET owner_generation = ?, spool_host = ?, \ + heartbeat_at = {NOW} \ + WHERE table_id = ? AND status = 'RUNNING' AND kind = 'manifest' \ + AND EXISTS(SELECT 1 FROM tables t WHERE t.table_id = table_restores.table_id \ + AND t.account_id = ? AND t.table_name = ?)" + )) + .bind(generation_column(generation)) + .bind(spool_host) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + tx.commit().await.map_err(db_err)?; + if taken != 1 { + drop(slot); + return Ok(None); + } + Ok(Some(Box::new(SqliteJobClaim::for_manifest_restore( + self, + &key.table_id, + generation, + slot, + )) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let row: Option = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.table_id = ? AND t.account_id = ? AND t.table_name = ?" + )) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + row.map(restore_job_from_tuple).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (?1 IS NULL OR r.status = ?1) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(status.map(JobStatus::as_str)) .fetch_all(&self.pool) .await .map_err(db_err)?; - rows.into_iter().map(restore_job_row).collect() + rows.into_iter().map(restore_job_from_tuple).collect() }) } diff --git a/crates/storage-sqlite/src/create_table.rs b/crates/storage-sqlite/src/create_table.rs index 509e73086..1b183037a 100644 --- a/crates/storage-sqlite/src/create_table.rs +++ b/crates/storage-sqlite/src/create_table.rs @@ -16,6 +16,7 @@ use extenddb_core::types::{ BillingMode, BillingModeSummary, CreateTableInput, GsiDescription, LsiDescription, ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::RestoreRegistrationRow; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; @@ -35,7 +36,57 @@ pub(crate) struct RestoreRegistration<'a> { pub owner_generation: u64, } +/// Which backup a restore target is created for, and so which row the +/// registration re-checks in its write transaction: an in-catalog `backups` +/// row (the legacy reader's restore, `kind = legacy`) or a `backup_index` row +/// of a backup in the store (`kind = manifest`, with the store, the spool +/// host and the claim's generation on the record). +pub(crate) enum RestoreSource<'a> { + Legacy(&'a RestoreRegistration<'a>), + Manifest(&'a RestoreRegistrationRow), +} + +impl RestoreSource<'_> { + fn backup_arn(&self) -> &str { + match self { + Self::Legacy(r) => r.backup_arn, + Self::Manifest(r) => &r.backup.backup_arn, + } + } +} + impl SqliteEngine { + /// The error for a restore into a name another restore holds the slot + /// of: `TableBeingRestored` naming that restore's backup once its + /// CREATING row is committed, else `TableAlreadyExists`, as CreateTable + /// reports a taken name. Reads committed state through the pool; a read + /// failure is "not a restore target": an error is being produced + /// already. + pub(crate) async fn name_taken_error( + &self, + account_id: &str, + table_name: &str, + ) -> StorageError { + let restoring: Option = sqlx::query_scalar( + "SELECT r.source_backup_arn FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.account_id = ? AND t.table_name = ? AND t.table_status = 'CREATING'", + ) + .bind(account_id) + .bind(table_name) + .fetch_optional(&self.pool) + .await + .ok() + .flatten(); + match restoring { + Some(backup_arn) => StorageError::TableBeingRestored { + table_name: table_name.to_owned(), + backup_arn, + }, + None => StorageError::TableAlreadyExists(table_name.to_owned()), + } + } + /// Create a table. When `defer_active` is set (the restore path), the row /// is written `CREATING` with **no** scheduled transition, so the /// background control-plane worker cannot flip it to `ACTIVE` while the @@ -61,8 +112,33 @@ impl SqliteEngine { input: CreateTableInput, registration: &RestoreRegistration<'_>, ) -> Result { - self.create_table_impl_inner(account_id, input, true, Some(registration)) - .await + self.create_table_impl_inner( + account_id, + input, + true, + Some(RestoreSource::Legacy(registration)), + ) + .await + } + + /// Create a store-backed restore's target and register the restore in + /// the write transaction that also re-checks, on the backup's + /// `backup_index` row, that it is AVAILABLE and has no CREATING target. + /// The job record is born RUNNING, `kind = manifest`, with the + /// registration's store, spool host and generation. + pub(crate) async fn create_table_for_manifest_restore( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result { + self.create_table_impl_inner( + account_id, + input, + true, + Some(RestoreSource::Manifest(registration)), + ) + .await } async fn create_table_impl_inner( @@ -70,7 +146,7 @@ impl SqliteEngine { account_id: &str, input: CreateTableInput, defer_active: bool, - restore: Option<&RestoreRegistration<'_>>, + restore: Option>, ) -> Result { Self::validate_account_id(account_id)?; // D1: every writer holds the engine write lock. This method runs two @@ -153,7 +229,8 @@ impl SqliteEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - if let Some(RestoreRegistration { backup_arn, .. }) = restore { + if let Some(source) = &restore { + let backup_arn = source.backup_arn(); // A restore into a name that is itself a CREATING restore target // is refused as such before the backup is checked, so a second // restore of the same backup into the same name is @@ -175,19 +252,45 @@ impl SqliteEngine { backup_arn: source_backup_arn, }); } - let available: bool = sqlx::query_scalar( - "SELECT EXISTS(SELECT 1 FROM backups WHERE backup_arn = ? \ - AND account_id = ? AND backup_status = 'AVAILABLE')", - ) - .bind(backup_arn) - .bind(account_id) - .fetch_one(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; - if !available { - return Err(StorageError::Validation(format!( - "Backup not found: {backup_arn}" - ))); + match source { + RestoreSource::Legacy(_) => { + let available: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM backups WHERE backup_arn = ? \ + AND account_id = ? AND backup_status = 'AVAILABLE')", + ) + .bind(backup_arn) + .bind(account_id) + .fetch_one(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if !available { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } + } + RestoreSource::Manifest(registration) => { + // The `backup_index` row's status, in the write + // transaction `start_backup_delete` reads it in: the + // engine write lock and `BEGIN IMMEDIATE` are the lock + // on the row, so a DeleteBackup and this registration + // order wholly one before the other, and two + // registrations of one backup serialize before the + // CREATING-target check below. + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = ? AND backup_id = ?", + ) + .bind(account_id) + .bind(®istration.backup.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if status.as_deref() != Some("AVAILABLE") { + return Err(extenddb_storage::backup_catalog::backup_not_found( + backup_arn, + )); + } + } } // One restore of a backup at a time, as the service: while a // target restored from it is CREATING the backup is in use. Same @@ -258,24 +361,51 @@ impl SqliteEngine { return Err(StorageError::TableAlreadyExists(input.table_name.clone())); } - if let Some(registration) = restore { - sqlx::query(&format!( - "INSERT INTO table_restores \ - (table_id, source_backup_arn, source_table_arn, restore_date_time, \ - status, kind, owner_generation, heartbeat_at) \ - VALUES (?, ?, ?, ?, 'RUNNING', 'legacy', ?, {})", - crate::backup_catalog::NOW - )) - .bind(&table_id) - .bind(registration.backup_arn) - .bind(registration.source_table_arn) - .bind(format_epoch_seconds(registration.restore_date_time)) - .bind(crate::backup_catalog::generation_column( - registration.owner_generation, - )) - .execute(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + match &restore { + Some(RestoreSource::Legacy(registration)) => { + sqlx::query(&format!( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, 'RUNNING', 'legacy', ?, {})", + crate::backup_catalog::NOW + )) + .bind(&table_id) + .bind(registration.backup_arn) + .bind(registration.source_table_arn) + .bind(format_epoch_seconds(registration.restore_date_time)) + .bind(crate::backup_catalog::generation_column( + registration.owner_generation, + )) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Some(RestoreSource::Manifest(registration)) => { + sqlx::query(&format!( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, store_kind, store_root, store_id, spool_host, \ + owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, 'RUNNING', 'manifest', ?, ?, ?, ?, ?, {})", + crate::backup_catalog::NOW + )) + .bind(&table_id) + .bind(®istration.backup.backup_arn) + .bind(®istration.source_table_arn) + .bind(format_epoch_seconds(registration.restore_date_time)) + .bind(registration.store.store_kind.as_str()) + .bind(®istration.store.store_root) + .bind(®istration.store.store_id) + .bind(registration.spool_host.as_deref()) + .bind(crate::backup_catalog::generation_column( + registration.owner_generation, + )) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + None => {} } // GSI / LSI metadata. diff --git a/crates/storage-sqlite/src/data/vector_index.rs b/crates/storage-sqlite/src/data/vector_index.rs index 02bc09836..80acca8cd 100644 --- a/crates/storage-sqlite/src/data/vector_index.rs +++ b/crates/storage-sqlite/src/data/vector_index.rs @@ -126,7 +126,7 @@ fn base_key_binds( } /// Column names for the base key, in key-schema order. -fn base_key_columns( +pub(crate) fn base_key_columns( base_key_schema: &[KeySchemaElement], attr_defs: &[AttributeDefinition], ) -> Vec { diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs index c7ec54c23..d1a0cb5dc 100644 --- a/crates/storage-sqlite/src/lib.rs +++ b/crates/storage-sqlite/src/lib.rs @@ -38,6 +38,9 @@ pub mod migrations; mod number_key; mod operations; mod password_hash_store; +mod restore_engine; +#[cfg(test)] +mod restore_job_tests; mod schema; mod serve_lock; mod snapshot_engine; @@ -328,18 +331,28 @@ fn sqlite_server_components_factory( // A restore the previous run left mid-copy: one process serves this // file, so its owner is gone. Age every RUNNING job record so the - // engine's reconciler fails it and removes its target on its first - // cycle rather than after the stale window. Before the server takes - // requests, so no restore of this process can be in flight. A backup - // the previous run left CREATING is reported to the reconciler by + // engine's reconciler acts on it on its first cycle rather than + // after the stale window: a store-backed (`manifest`) record is + // taken over and resumed, an in-catalog (`legacy`) one is failed and + // its target removed. Before the server takes requests, so no + // restore of this process can be in flight. A backup the previous + // run left CREATING is reported to the reconciler by // `abandoned_creating_backups` without a step here: no task of this // process holds its slot. match engine.expire_restore_job_heartbeats().await { - Ok(names) => { - for name in &names { + Ok(aged) => { + for (name, kind) in &aged { + let duty = match kind { + extenddb_storage::backup_catalog::RestoreKind::Manifest => { + "the reconciler resumes it (a store-backed restore)" + } + extenddb_storage::backup_catalog::RestoreKind::Legacy => { + "the reconciler removes it (an in-catalog restore)" + } + }; tracing::warn!( "table '{name}': its restore did not finish before the last shutdown; \ - the reconciler removes it" + {duty}" ); } } diff --git a/crates/storage-sqlite/src/migrations/mod.rs b/crates/storage-sqlite/src/migrations/mod.rs index e784fa82c..e94ab8fe5 100644 --- a/crates/storage-sqlite/src/migrations/mod.rs +++ b/crates/storage-sqlite/src/migrations/mod.rs @@ -118,6 +118,10 @@ pub fn migration(id: MigrationId) -> Migration { m0006_backup_catalog_index::SOURCE, )], }, + MigrationId::RestoreProgress => Migration::NotApplicable( + "the restore progress of a store-backed restore is on table_restores (file_index, \ + items_applied) on SQLite; the table is PostgreSQL's data database's", + ), } } diff --git a/crates/storage-sqlite/src/migrations/released_tests.rs b/crates/storage-sqlite/src/migrations/released_tests.rs index fe9bb5b09..321d2340d 100644 --- a/crates/storage-sqlite/src/migrations/released_tests.rs +++ b/crates/storage-sqlite/src/migrations/released_tests.rs @@ -399,11 +399,13 @@ async fn a_catalog_an_older_release_wrote_to_is_reported_and_reconverged() { next_second().await; let report = reconverge(&db).await.expect("reconverge"); - // Every convergent migration ran again; migration 2, which is not - // convergent, is named as skipped. + // Every convergent migration that applies on SQLite ran again (7, + // not applicable here, has nothing to run); migration 2, which is + // not convergent, is named as skipped. let convergent = MigrationId::ALL .into_iter() - .filter(|id| super::migration(*id).convergent()) + .map(super::migration) + .filter(|m| m.convergent() && !m.bodies().is_empty()) .count(); assert_eq!(report.applied.len(), convergent, "{trace}: {report:?}"); let skipped = format!("Skipped {}:", MigrationId::ResourceTagConditionKeys); diff --git a/crates/storage-sqlite/src/migrations/tests.rs b/crates/storage-sqlite/src/migrations/tests.rs index 82b3d3b80..90efc0d67 100644 --- a/crates/storage-sqlite/src/migrations/tests.rs +++ b/crates/storage-sqlite/src/migrations/tests.rs @@ -270,7 +270,14 @@ async fn a_fresh_database_is_current_after_one_run_and_seeds_nothing() { "SELECT COUNT(*) FROM extenddb_migrations WHERE outcome = 'applied'" ) .await, - i64::from(LATEST) + i64::from(LATEST) - 1, + "every id but 7 (the PostgreSQL progress table) applies on SQLite" + ); + assert_eq!( + scalar(&db, "SELECT outcome FROM extenddb_migrations WHERE id = 7") + .await + .as_deref(), + Some("not_applicable") ); assert_eq!(version_row(&db).await, Some(LATEST.to_string())); assert_eq!( diff --git a/crates/storage-sqlite/src/restore_engine.rs b/crates/storage-sqlite/src/restore_engine.rs new file mode 100644 index 000000000..0b11818da --- /dev/null +++ b/crates/storage-sqlite/src/restore_engine.rs @@ -0,0 +1,562 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on SQLite +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target, the progress record, the ACTIVE flip and the count. +//! +//! One write transaction (`BEGIN IMMEDIATE` under the engine write lock) per +//! batch of at most [`RESTORE_BATCH_ITEMS`] items or [`RESTORE_BATCH_BYTES`] +//! of serialized item text (a batch closes at the first item that takes its +//! `item_data` text past the bound, so the batch term of a restore's memory +//! is bounded in bytes, not only in items): the base rows (plain INSERTs, so +//! a key the backup holds twice, within a file or across files, fails the +//! file instead of collapsing two items), each secondary index's rows and +//! each vector index's rows beside them, and last the record's progress, +//! `UPDATE table_restores SET file_index, items_applied` conditioned on the +//! claim's generation and on RUNNING: a batch commits whole with its +//! progress or not at all, and a batch whose record another owner took +//! rolls back and ends the file as a lost claim. Other writers on the server +//! wait for a batch rather than the whole file. A resume reads the record's +//! `(file_index, items_applied)` and the engine skips that many items of the +//! file before the stream starts. An empty file records its progress too +//! (`items_applied = start_at`), so a resume does not re-read it. +//! +//! The restore's own buffer is this one batch of `(Item, String)`, bounded +//! by [`RESTORE_BATCH_BYTES`] whatever the table's indexes are. A table with +//! secondary or vector indexes nonetheless restores in more resident memory +//! than one without (about 90 MiB over idle at 100,000 200-byte items with +//! two full-projection GSIs against 43 MiB with none), and the excess is +//! SQLite's own: its per-connection page cache (`PRAGMA cache_size = +//! -32000`, 32 MiB) holds the index tables' b-tree pages, which a load of +//! scattered index keys dirties across the whole tree, and glibc's +//! per-arena free lists hold freed blocks (the figure falls by about a third +//! under `MALLOC_ARENA_MAX=1`). The excess is bounded by `cache_size` and +//! grows only sub-linearly with the table (about 90 MiB at 100,000 items, +//! 175 MiB at 300,000), not with the restore's buffers; a smaller batch does +//! not lower it (measured: the peak is unchanged at a third of the item +//! cap), because the cache persists across batches. The published restore +//! figures are for a table with no secondary index and state that shape. +//! +//! A job's heartbeat on SQLite (`SqliteJobClaim::heartbeat`) is an `UPDATE` +//! under the same engine write lock, so a beat waits for the batch that +//! holds the lock and for every writer queued ahead of it: other jobs' +//! batches, item writes, the GSI backfill, the TTL sweep, a table's drop. +//! The bench record (`docs/design/15-backup-restore-bench-pr9.md`) puts one +//! batch at about 1.1 s for items of 27.7 MB (24 of them, one per batch, +//! restored in 26.8 s with their download and decode) and about 0.27 s for +//! 10,000 items of 200 bytes (1,000,000 in 26.8 s). A beat that has not +//! returned within one heartbeat period (15 s at the least) ends the job as +//! a lost claim: it stops with its record RUNNING and makes no further +//! table access, its slot is freed, and the reconciler's next cycle takes +//! the record over and resumes it from the last committed batch. At the +//! default four jobs a beat waits a few seconds at most; many jobs of large +//! items, or the drop of a large table, can hold it past a period, which +//! costs a restart of the file from its last batch and a lost-claim log +//! line, not data. +//! +//! `finish_restore` is [`activate_restore_target`], the one transaction the +//! legacy copy ends with too: the record DONE conditioned on the generation +//! and on RUNNING, the table ACTIVE conditioned on CREATING, and every +//! secondary and vector index of the table ACTIVE. + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement, Projection}; +use extenddb_storage::backup_catalog::{ + JobClaim, RESTORE_SQLITE_MID_BATCH, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +pub(crate) use extenddb_storage::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; +use extenddb_storage::util::{composite_pk_to_text, parse_sk, sk_column_n}; +use extenddb_storage::vector_lifecycle::VectorIndexMeta; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::SqliteEngine; +use crate::backup_catalog::{NOW, generation_column}; +use crate::data::vector_index::{ + VectorRowConflict, base_key_columns, fetch_vector_indexes_for_table, insert_vector_row, +}; +use crate::data::{ + BoundValue, all_sort_key_info, data_table_name, index_table_name, insert_index_row_multi, + item_has_index_keys, project_item_for_index, sk_bound, +}; +use crate::sqlite_util::is_unique_violation; + +fn db_err(e: sqlx::Error) -> StorageError { + StorageError::Internal(format!("Database error: {e}")) +} + +/// The error for a key the target already holds: the backup carries it +/// twice, within the file or in a file applied before it. The engine +/// prefixes the file's key, so the file is not named here. +fn duplicate_key_error(e: &sqlx::Error) -> StorageError { + StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key twice): {e}" + )) +} + +/// The key shape of a CREATING restore target, read from its catalog row. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +/// A secondary index of the target, as the load writes it. +struct TargetIndex { + table: String, + key_schema: Vec, + projection: Projection, +} + +/// What `bulk_load` writes per item and per batch, read once per file. +struct LoadPlan { + insert_sql: String, + shape: TargetShape, + indexes: Vec, + vector_key_columns: Vec, + file: BulkLoadFile, + generation: u64, + /// The file as an error names it. + source: String, +} + +/// What [`activate_restore_target`] matched. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Activated { + /// The record was RUNNING under the generation and is DONE. + pub record: bool, + /// The table was CREATING and is ACTIVE. + pub table: bool, +} + +/// The end of a restore, inside the caller's write transaction: the record +/// DONE (`completed_at` now; `items_applied` left as the progress stands +/// unless the caller gives a total) only while it is RUNNING under +/// `generation`; the table ACTIVE with its counts only from CREATING; every +/// secondary and vector index of the table ACTIVE (`backfilling` cleared on +/// the vector rows, which a restore target creates CREATING with the table). +/// Reports what each conditional write matched; the caller rolls back when +/// it needed a match it did not get. +pub(crate) async fn activate_restore_target( + tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>, + table_id: &str, + generation: u64, + item_count: i64, + table_size_bytes: i64, + items_applied: Option, +) -> Result { + let record = sqlx::query(&format!( + "UPDATE table_restores SET status = 'DONE', \ + items_applied = COALESCE(?, items_applied), completed_at = {NOW} \ + WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" + )) + .bind(items_applied) + .bind(table_id) + .bind(generation_column(generation)) + .execute(&mut **tx) + .await + .map_err(db_err)? + .rows_affected() + == 1; + if !record { + return Ok(Activated { + record, + table: false, + }); + } + let table = sqlx::query( + "UPDATE tables SET item_count = ?, table_size_bytes = ?, \ + table_status = 'ACTIVE', status_transition_at = NULL \ + WHERE table_id = ? AND table_status = 'CREATING'", + ) + .bind(item_count) + .bind(table_size_bytes) + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)? + .rows_affected() + == 1; + sqlx::query("UPDATE indexes SET index_status = 'ACTIVE' WHERE table_id = ? AND index_status <> 'ACTIVE'") + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)?; + sqlx::query( + "UPDATE vector_indexes SET index_status = 'ACTIVE', backfilling = NULL \ + WHERE table_id = ? AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)?; + Ok(Activated { record, table }) +} + +impl SqliteEngine { + /// The target's key schema and attribute definitions, when `key` names + /// a CREATING restore target with this table id. + async fn restore_target_shape( + &self, + key: &RestoreTargetKey, + ) -> Result { + let row: Option<(String, String)> = sqlx::query_as( + "SELECT t.key_schema, t.attribute_definitions FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.table_id = ? AND t.account_id = ? AND t.table_name = ? \ + AND t.table_status = 'CREATING'", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (key_schema, attribute_definitions) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + Ok(TargetShape { + key_schema: serde_json::from_str(&key_schema) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: serde_json::from_str(&attribute_definitions) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }) + } + + /// Everything a file's writes need that does not change between batches. + async fn load_plan( + &self, + key: &RestoreTargetKey, + file: BulkLoadFile, + generation: u64, + ) -> Result { + let shape = self.restore_target_shape(key).await?; + let sort_keys = all_sort_key_info(&shape.key_schema, &shape.attribute_definitions); + let mut cols = vec!["pk".to_owned()]; + cols.extend( + sort_keys + .iter() + .enumerate() + .map(|(i, &(_, t))| sk_column_n(i, t)), + ); + cols.push("item_data".to_owned()); + let data_table = data_table_name(&key.table_id); + let insert_sql = format!( + "INSERT INTO {data_table} ({}) VALUES ({})", + cols.join(", "), + vec!["?"; cols.len()].join(", ") + ); + let index_rows: Vec<(String, String, String)> = sqlx::query_as( + "SELECT index_id, key_schema, projection FROM indexes WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + let mut indexes = Vec::with_capacity(index_rows.len()); + for (index_id, ks, proj) in index_rows { + indexes.push(TargetIndex { + table: index_table_name(&index_id), + key_schema: serde_json::from_str(&ks) + .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, + projection: serde_json::from_str(&proj) + .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, + }); + } + let vector_key_columns = base_key_columns(&shape.key_schema, &shape.attribute_definitions); + Ok(LoadPlan { + insert_sql, + shape, + indexes, + vector_key_columns, + file, + generation, + source: format!("data file {}", file.file_index), + }) + } + + /// One batch: the base rows, the index rows, the vector rows and the + /// record's progress in one write transaction under the write lock. + /// `applied` is the file's running total before this batch, `start_at` + /// included; each item comes with its `item_data` text, serialized when + /// the batch was collected. `Ok(false)` when the record was not RUNNING + /// under the plan's generation: nothing of the batch is committed. + async fn write_restore_batch( + &self, + key: &RestoreTargetKey, + plan: &LoadPlan, + applied: u64, + batch: &mut Vec<(Item, String)>, + ) -> Result { + let batch_len = batch.len(); + let source = plan.source.as_str(); + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let shape = &plan.shape; + let sort_keys = all_sort_key_info(&shape.key_schema, &shape.attribute_definitions); + // The vector indexes, read inside the transaction as the write path + // does (the id names the data table). + let vector_metas: Vec = + fetch_vector_indexes_for_table(&mut tx, &key.table_id).await?; + for (item, item_json) in batch.drain(..) { + let mut values = vec![BoundValue::Text(composite_pk_to_text( + &item, + &shape.key_schema, + )?)]; + for &(name, sk_type) in &sort_keys { + let value = item.get(name).ok_or_else(|| { + StorageError::Validation(format!( + "{source} holds an item without its sort key {name}" + )) + })?; + values.push(sk_bound(&parse_sk(value, sk_type)?)); + } + values.push(BoundValue::Text(item_json)); + let mut q = sqlx::query(&plan.insert_sql); + for v in values { + q = crate::data::bind_bound!(q, v); + } + q.execute(&mut *tx).await.map_err(|e| { + if is_unique_violation(&e) { + duplicate_key_error(&e) + } else { + db_err(e) + } + })?; + for idx in &plan.indexes { + if !item_has_index_keys(&item, &idx.key_schema) { + continue; + } + let idx_sks = all_sort_key_info(&idx.key_schema, &shape.attribute_definitions); + let projected = project_item_for_index( + &item, + &idx.key_schema, + &shape.key_schema, + &idx.projection, + ); + insert_index_row_multi( + &mut tx, + &idx.table, + &item, + &projected, + &idx.key_schema, + &shape.key_schema, + &idx_sks, + &sort_keys, + ) + .await?; + } + for meta in &vector_metas { + insert_vector_row( + &mut tx, + &key.table_id, + meta, + &item, + &shape.key_schema, + &shape.attribute_definitions, + &plan.vector_key_columns, + VectorRowConflict::Fail, + ) + .await?; + } + } + // The progress, last and in the same transaction, conditioned on the + // generation and on RUNNING: a record another owner took rolls the + // batch back with it. + let total = applied.saturating_add(u64::try_from(batch_len).unwrap_or(u64::MAX)); + let matched = sqlx::query( + "UPDATE table_restores SET file_index = ?, items_applied = ? \ + WHERE table_id = ? AND owner_generation = ? AND status = 'RUNNING'", + ) + .bind(i64::from(plan.file.file_index)) + .bind(i64::try_from(total).unwrap_or(i64::MAX)) + .bind(&key.table_id) + .bind(generation_column(plan.generation)) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + if matched != 1 { + tx.rollback().await.map_err(db_err)?; + return Ok(false); + } + tx.commit().await.map_err(db_err)?; + Ok(true) + } +} + +impl RestoreEngine for SqliteEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + let generation = claim.generation(); + let plan = self.load_plan(target, file, generation).await?; + let mut batch: Vec<(Item, String)> = Vec::with_capacity(RESTORE_BATCH_ITEMS.min(1024)); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + let mut batches_written = 0u32; + let mut ended = false; + while !ended { + // Collect a batch, closed at the count cap or at the first + // item whose text takes it past the byte bound; an `Err` + // item ends the file with that error, and the batch in hand + // is never written. + while batch.len() < RESTORE_BATCH_ITEMS && batch_bytes < RESTORE_BATCH_BYTES { + match items.next().await { + Some(item) => { + let item = item?; + let item_json = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))?; + batch_bytes = batch_bytes.saturating_add(item_json.len()); + batch.push((item, item_json)); + } + None => { + ended = true; + break; + } + } + } + if batch.is_empty() && batches_written > 0 { + break; + } + if batches_written > 0 { + restore_pause_point(RESTORE_SQLITE_MID_BATCH).await; + } + let batch_len = batch.len(); + let written = self + .write_restore_batch( + target, + &plan, + file.start_at.saturating_add(applied), + &mut batch, + ) + .await?; + if !written { + return Err(StorageError::Internal(format!( + "claim lost: the job record of {} is not RUNNING under generation \ + {generation}; the batch was rolled back", + target.table_name + ))); + } + applied += u64::try_from(batch_len).unwrap_or(u64::MAX); + batches_written += 1; + batch_bytes = 0; + } + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The key names the table; the flip itself is conditioned on the + // table id, CREATING, RUNNING and the generation. + let is_target: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM tables WHERE table_id = ? AND account_id = ? \ + AND table_name = ?)", + ) + .bind(&target.table_id) + .bind(&target.account_id) + .bind(&target.table_name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + if !is_target { + return Ok(Finished::NotMatched(format!( + "{} is not a table with id {} in account {}", + target.table_name, target.table_id, target.account_id + ))); + } + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let activated = activate_restore_target( + &mut tx, + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + None, + ) + .await?; + if !(activated.record && activated.table) { + tx.rollback().await.map_err(db_err)?; + return Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING", + target.table_name, + claim.generation() + ))); + } + tx.commit().await.map_err(db_err)?; + Ok(Finished::Active) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data table; the key's account and name are + // checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_shape(key).await?; + let count: i64 = sqlx::query_scalar(&format!( + "SELECT COUNT(*) FROM {}", + data_table_name(&key.table_id) + )) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + Ok(u64::try_from(count).unwrap_or_default()) + }) + } + + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The catalog row first, so a bare id reads no other account's + // table; then the data table's existence (the registration + // commits the catalog transaction before the data DDL). + // `data_table_name` is a quoted identifier; `sqlite_master.name` + // holds it unquoted. + self.restore_target_shape(key).await?; + let quoted = data_table_name(&key.table_id); + let name = quoted.trim_matches('"'); + let exists: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = ?", + ) + .bind(name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + Ok(exists > 0) + }) + } +} diff --git a/crates/storage-sqlite/src/restore_job_tests.rs b/crates/storage-sqlite/src/restore_job_tests.rs new file mode 100644 index 000000000..146904e27 --- /dev/null +++ b/crates/storage-sqlite/src/restore_job_tests.rs @@ -0,0 +1,1537 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Storage-level tests of the store-backed restore on SQLite: the +//! registration under the write lock (`claim_restore`), the takeover of a +//! record whose slot no task holds (`claim_restore_dead`), the per-batch load +//! with its progress write (`bulk_load`), the ACTIVE flip (`finish_restore`) +//! and the count. The engine's restore job drives these through +//! `BackupCatalog` and `RestoreEngine`; what each must guarantee on this +//! backend is what a test here pins. Every test builds its own in-memory +//! engine. + +use std::sync::Arc; +use std::time::Duration; + +use extenddb_core::types::{ + AttributeValue, CreateTableInput, DistanceFunction, Item, Projection, ProjectionType, + TableDescription, TableStatus, VectorAttribute, VectorIndexSpecification, +}; +use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobStatus, RestoreJobRow, + RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StatusFields, StoreIdentity, StoreKind, + WriteOutcome, backup_in_use_message, is_backup_not_found, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use extenddb_storage::{BackupCatalog, JobPermit, TableEngine}; +use futures::StreamExt; +use futures::stream::BoxStream; + +use crate::SqliteEngine; +use crate::backup_catalog::{SqliteJobClaim, restore_slot_key}; +use crate::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; + +const ACCOUNT: &str = "123456789012"; +const REGION: &str = "us-east-1"; +/// A stale window no test's heartbeat crosses on its own. +const STALE: Duration = Duration::from_secs(300); + +// --- the engine, rows, inputs and items ------------------------------------------- + +async fn engine() -> SqliteEngine { + let engine = SqliteEngine::new(":memory:", 2, REGION, 409_600) + .await + .expect("engine"); + crate::migrations::apply_for_tests(&engine.pool).await; + for (key, value) in [ + ("control_plane_delay_seconds", "0"), + ("index_propagation_delay_ms", "0"), + ] { + sqlx::query( + "INSERT INTO settings (key, value) VALUES (?, ?) \ + ON CONFLICT(key) DO UPDATE SET value = excluded.value", + ) + .bind(key) + .bind(value) + .execute(&engine.pool) + .await + .expect("setting"); + } + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES (?, 'default')") + .bind(ACCOUNT) + .execute(&engine.pool) + .await + .expect("account"); + engine +} + +fn store() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/var/lib/extenddb/backups".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// An AVAILABLE `backup_index` row of a backup of `table`, written through +/// the claim as a backup job writes it. +async fn available_backup(engine: &SqliteEngine, table: &str) -> BackupRow { + let backup_id = extenddb_storage::util::new_backup_id(); + let backup_arn = extenddb_storage::util::backup_arn(REGION, ACCOUNT, table, &backup_id); + let manifest: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + let row = BackupRow { + backup_id: backup_id.clone(), + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + backup_arn, + backup_name: "nightly".to_owned(), + status: BackupStatus::Creating, + store: store(), + size_bytes: 0, + item_count: 0, + created_at: extenddb_storage::util::backup_creation_date_time(&backup_id).unwrap(), + completed_at: None, + failure_reason: None, + source_table_details_json: manifest["source_table"].clone(), + owner_generation: 0, + heartbeat_at: 0.0, + }; + let claim = engine.claim_backup(&row).await.expect("claim the row"); + let written = claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + item_count: Some(3), + size_bytes: Some(1234), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("the AVAILABLE write"); + assert_eq!(written, WriteOutcome::Matched); + drop(claim); + BackupRow { + status: BackupStatus::Available, + ..row + } +} + +fn registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: 0, + } +} + +/// The target: a composite-key table with a GSI on `g`, an LSI on `l` and, +/// when asked, a vector index on `emb` (three dimensions). +fn target_input(name: &str, vector: bool) -> CreateTableInput { + let mut input: CreateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": name, + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }], + "LocalSecondaryIndexes": [{ + "IndexName": "lix", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "l", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "ALL"} + }] + })) + .expect("input"); + if vector { + input.vector_indexes = Some(vec![VectorIndexSpecification { + index_name: "vix".to_owned(), + dimensions: 3, + distance_function: DistanceFunction::Cosine, + vector_attribute: VectorAttribute { + attribute_name: "emb".to_owned(), + }, + search_schema: None, + projection: Some(Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }), + }]); + } + input +} + +/// Item `i` of the fixture: every item has `pk` and `sk`, every second one +/// `g` (the GSI's key), every third one `l` (the LSI's), every fifth one +/// `emb` (the vector). +fn item(i: usize) -> Item { + let mut item = + serde_json::json!({"pk": {"S": format!("p{}", i % 7)}, "sk": {"N": i.to_string()}}); + if i.is_multiple_of(2) { + item["g"] = serde_json::json!({"S": format!("g{}", i % 4)}); + } + if i.is_multiple_of(3) { + item["l"] = serde_json::json!({"S": format!("l{i}")}); + } + if i.is_multiple_of(5) { + item["emb"] = serde_json::json!({"L": [{"N": "0.1"}, {"N": "0.2"}, {"N": "0.3"}]}); + } + serde_json::from_value(item).expect("item") +} + +fn items(range: std::ops::Range) -> Vec { + range.map(item).collect() +} + +fn stream(items: Vec) -> BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +fn file(index: u32, start_at: u64) -> BulkLoadFile { + BulkLoadFile { + file_index: index, + start_at, + reapplying: false, + } +} + +fn key_of(desc: &TableDescription) -> RestoreTargetKey { + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + } +} + +/// Register a restore of `row` into `target` and return the description, its +/// key and the claim. +async fn register( + engine: &SqliteEngine, + row: &BackupRow, + target: &str, + vector: bool, + spool_host: Option<&str>, +) -> (TableDescription, RestoreTargetKey, Box) { + let (desc, claim) = engine + .claim_restore( + ACCOUNT, + target_input(target, vector), + ®istration(row, spool_host), + ) + .await + .expect("register the restore"); + let key = key_of(&desc); + (desc, key, claim) +} + +/// A claim on the record under `generation`, as a job whose record passed to +/// another owner still holds one. +fn stale_claim(engine: &SqliteEngine, table_id: &str, generation: u64) -> SqliteJobClaim { + SqliteJobClaim::for_restore(engine, table_id, generation, JobPermit::detached()) +} + +// --- reads -------------------------------------------------------------------- + +async fn table_status(engine: &SqliteEngine, name: &str) -> Option { + sqlx::query_scalar("SELECT table_status FROM tables WHERE account_id = ? AND table_name = ?") + .bind(ACCOUNT) + .bind(name) + .fetch_optional(&engine.pool) + .await + .expect("table status") +} + +async fn record(engine: &SqliteEngine, key: &RestoreTargetKey) -> RestoreJobRow { + engine + .read_restore_job(key) + .await + .expect("read the record") + .expect("the record exists") +} + +/// `(status, kind, owner_generation, heartbeat_at, spool_host, file_index, +/// items_applied, completed_at)` of the record, straight from the table. +type RecordColumns = ( + String, + String, + i64, + String, + Option, + Option, + Option, + Option, +); + +async fn record_columns(engine: &SqliteEngine, table_id: &str) -> RecordColumns { + sqlx::query_as( + "SELECT status, kind, owner_generation, heartbeat_at, spool_host, file_index, \ + items_applied, completed_at FROM table_restores WHERE table_id = ?", + ) + .bind(table_id) + .fetch_one(&engine.pool) + .await + .expect("read table_restores") +} + +async fn rows_in(engine: &SqliteEngine, physical: &str) -> i64 { + sqlx::query_scalar(&format!("SELECT COUNT(*) FROM {physical}")) + .fetch_one(&engine.pool) + .await + .expect("count rows") +} + +async fn base_rows(engine: &SqliteEngine, table_id: &str) -> i64 { + rows_in(engine, &crate::data::data_table_name(table_id)).await +} + +/// `(index_name, index_id, index_status)` of the table's secondary indexes. +async fn indexes(engine: &SqliteEngine, table_id: &str) -> Vec<(String, String, String)> { + sqlx::query_as( + "SELECT index_name, index_id, index_status FROM indexes WHERE table_id = ? \ + ORDER BY index_name", + ) + .bind(table_id) + .fetch_all(&engine.pool) + .await + .expect("read indexes") +} + +async fn index_rows(engine: &SqliteEngine, table_id: &str, name: &str) -> i64 { + let id = indexes(engine, table_id) + .await + .into_iter() + .find(|(n, _, _)| n == name) + .map(|(_, id, _)| id) + .expect("the index exists"); + rows_in(engine, &crate::data::index_table_name(&id)).await +} + +/// `(index_id, index_status, backfilling)` of the table's vector indexes. +async fn vector_indexes( + engine: &SqliteEngine, + table_id: &str, +) -> Vec<(String, String, Option)> { + sqlx::query_as( + "SELECT index_id, index_status, backfilling FROM vector_indexes WHERE table_id = ?", + ) + .bind(table_id) + .fetch_all(&engine.pool) + .await + .expect("read vector_indexes") +} + +async fn vector_rows(engine: &SqliteEngine, table_id: &str) -> i64 { + let (id, _, _) = vector_indexes(engine, table_id) + .await + .pop() + .expect("the vector index exists"); + rows_in(engine, &crate::data::vector_table_name(table_id, &id)).await +} + +async fn progress(engine: &SqliteEngine, table_id: &str) -> (Option, Option) { + let (_, _, _, _, _, file_index, items_applied, _) = record_columns(engine, table_id).await; + (file_index, items_applied) +} + +async fn tables_named(engine: &SqliteEngine, name: &str) -> i64 { + sqlx::query_scalar("SELECT COUNT(*) FROM tables WHERE account_id = ? AND table_name = ?") + .bind(ACCOUNT) + .bind(name) + .fetch_one(&engine.pool) + .await + .expect("count tables") +} + +// --- the registration ----------------------------------------------------------- + +/// `claim_restore` takes the name's slot and creates the CREATING target with +/// no scheduled transition and its RUNNING `manifest` record carrying the +/// store, the spool host and the claim's generation in one transaction; the +/// description carries the `RestoreSummary`; the record reads back through +/// `read_restore_job` and `list_restore_jobs`; the heartbeat confirms and +/// refreshes the row; the secondary indexes are ACTIVE from creation and the +/// vector index CREATING with its table; dropping the claim frees the slot. +#[tokio::test] +async fn claim_restore_registers_the_target_with_its_record() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let slot = restore_slot_key(ACCOUNT, "Restored"); + assert!(!engine.job_slots.holds(&slot)); + let (desc, key, claim) = register(&engine, &row, "Restored", true, Some("host-a:/spool")).await; + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert_eq!( + summary.source_table_arn.as_deref(), + Some(format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music").as_str()) + ); + assert_eq!(summary.restore_date_time, row.created_at); + assert!(summary.restore_in_progress); + assert!( + engine.job_slots.holds(&slot), + "the claim holds the name's slot" + ); + + let (status, kind, generation, heartbeat_at, spool_host, file_index, items_applied, done) = + record_columns(&engine, &key.table_id).await; + assert_eq!((status.as_str(), kind.as_str()), ("RUNNING", "manifest")); + assert_eq!(generation, i64::try_from(claim.generation()).unwrap()); + assert!(heartbeat_at.starts_with("20"), "{heartbeat_at}"); + assert_eq!(spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((file_index, items_applied, done), (None, None, None)); + let transition: Option = + sqlx::query_scalar("SELECT status_transition_at FROM tables WHERE table_id = ?") + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("read the transition"); + assert_eq!( + transition, None, + "no scheduled transition: the job flips the table" + ); + + let job = record(&engine, &key).await; + assert_eq!(job.target, key); + assert_eq!(job.source_backup_arn, row.backup_arn); + assert_eq!(job.kind, RestoreKind::Manifest); + assert_eq!(job.status, JobStatus::Running); + assert_eq!(job.owner_generation, claim.generation()); + assert_eq!(job.store, Some(store())); + assert_eq!(job.spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((job.file_index, job.items_applied), (None, None)); + assert_eq!( + (job.failure_reason.as_deref(), job.completed_at), + (None, None) + ); + let running = engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(running, vec![job.clone()]); + assert_eq!( + engine.list_restore_jobs(None).await.expect("list all"), + vec![job.clone()] + ); + assert!( + engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list done") + .is_empty() + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + Some("Restored".to_owned()), + "the backup is in use while its target is CREATING" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "a fresh record is not dead" + ); + + tokio::time::sleep(Duration::from_millis(20)).await; + assert_eq!( + claim.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + let after = record(&engine, &key).await; + assert!( + after.heartbeat_at > job.heartbeat_at, + "{} > {}", + after.heartbeat_at, + job.heartbeat_at + ); + assert_eq!( + after.owner_generation, + claim.generation(), + "the heartbeat left the rest of the record" + ); + let read = claim.read().await.expect("read through the claim"); + assert_eq!(read.status, JobStatus::Running); + assert_eq!(read.owner_generation, claim.generation()); + + for (name, _, status) in indexes(&engine, &key.table_id).await { + assert_eq!(status, "ACTIVE", "secondary index {name} is created ACTIVE"); + } + let vectors = vector_indexes(&engine, &key.table_id).await; + assert_eq!(vectors.len(), 1); + assert_eq!( + (vectors[0].1.as_str(), vectors[0].2), + ("CREATING", None), + "the vector index is CREATING with its table and not a backfill" + ); + + drop(claim); + assert!( + !engine.job_slots.holds(&slot), + "dropping the claim frees the slot" + ); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); +} + +/// DeleteBackup winning: `start_backup_delete` moves the row to DELETING and a +/// registration after it finds the backup gone, creates no target, keeps no +/// slot and writes no record. The two also serialize when they contend on +/// the write lock: started together behind it, the outcomes are one of the +/// two consistent pairs and never a target beside a DELETING row. +#[tokio::test] +async fn delete_backup_winning_before_the_registration_leaves_no_target() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + assert_eq!( + engine + .start_backup_delete(&row.key()) + .await + .expect("delete"), + DeleteStart::Deleting + ); + let refused = engine + .claim_restore( + ACCOUNT, + target_input("Restored", false), + ®istration(&row, None), + ) + .await + .err() + .expect("the registration finds the backup gone"); + assert!( + is_backup_not_found(&refused), + "the registration finds the backup gone: {refused:?}" + ); + assert_eq!(tables_named(&engine, "Restored").await, 0, "no target"); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")) + ); + assert!( + engine + .list_restore_jobs(None) + .await + .expect("list") + .is_empty(), + "no record" + ); + + // Contended: both behind the write lock, released together. + let engine = Arc::new(engine); + for attempt in 0..6 { + let row = available_backup(&engine, &format!("Music{attempt}")).await; + let name = format!("Restored{attempt}"); + let held = engine.write_lock.lock().await; + let registering = tokio::spawn({ + let engine = Arc::clone(&engine); + let row = row.clone(); + let name = name.clone(); + async move { + engine + .claim_restore( + ACCOUNT, + target_input(&name, false), + ®istration(&row, None), + ) + .await + } + }); + let deleting = tokio::spawn({ + let engine = Arc::clone(&engine); + let key = row.key(); + async move { engine.start_backup_delete(&key).await } + }); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(!registering.is_finished() && !deleting.is_finished()); + drop(held); + let registered = tokio::time::timeout(Duration::from_secs(10), registering) + .await + .expect("the registration ends") + .expect("no panic"); + let deleted = tokio::time::timeout(Duration::from_secs(10), deleting) + .await + .expect("the delete ends") + .expect("no panic") + .expect("start_backup_delete"); + match (registered, deleted) { + (Ok((desc, claim)), DeleteStart::InUse(table)) => { + assert_eq!(table, name, "attempt {attempt}: the registration won"); + assert_eq!( + table_status(&engine, &name).await.as_deref(), + Some("CREATING") + ); + drop(claim); + assert!( + engine + .remove_restore_target(&key_of(&desc)) + .await + .expect("remove") + ); + } + (Err(e), DeleteStart::Deleting) => { + assert!(is_backup_not_found(&e), "attempt {attempt}: {e:?}"); + assert_eq!(tables_named(&engine, &name).await, 0, "attempt {attempt}"); + } + (registered, deleted) => panic!( + "attempt {attempt}: the two outcomes are not one ordering: {:?} / {deleted:?}", + registered.map(|(d, _)| d.table_status) + ), + } + } +} + +/// The registration winning: the backup is in use (`start_backup_delete` +/// answers the target's name), a second registration of the backup into +/// another name is `BackupInUse` naming the target and leaves no slot, and +/// the row stays AVAILABLE. +#[tokio::test] +async fn a_registration_winning_before_delete_backup_keeps_the_backup_in_use() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, _key, _claim) = register(&engine, &row, "First", false, None).await; + assert_eq!( + engine + .start_backup_delete(&row.key()) + .await + .expect("delete"), + DeleteStart::InUse("First".to_owned()) + ); + let second = engine + .claim_restore( + ACCOUNT, + target_input("Second", false), + ®istration(&row, None), + ) + .await + .err() + .expect("a second registration of the backup is refused"); + match second { + // The service's text names the backup, not the target. + StorageError::BackupInUse(m) => assert_eq!(m, backup_in_use_message(&row.backup_arn)), + other => panic!("{other:?}"), + } + assert_eq!(tables_named(&engine, "Second").await, 0); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Second"))); + let arn = extenddb_core::backup_format::BackupArn::parse(&row.backup_arn).unwrap(); + assert_eq!( + engine + .read_backup_row(&arn) + .await + .expect("read") + .expect("the row") + .status, + BackupStatus::Available + ); +} + +/// A taken name: an existing table is `TableAlreadyExists`; a CREATING +/// restore target is `TableBeingRestored` naming its backup, both while its +/// claim holds the slot and after the claim is gone (the committed row). +#[tokio::test] +async fn a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + engine + .create_table_impl(ACCOUNT, target_input("Plain", false), false) + .await + .expect("a table"); + let taken = engine + .claim_restore( + ACCOUNT, + target_input("Plain", false), + ®istration(&row, None), + ) + .await + .err() + .expect("the name is taken"); + assert!( + matches!(&taken, StorageError::TableAlreadyExists(n) if n == "Plain"), + "{taken:?}" + ); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Plain"))); + + let other = available_backup(&engine, "Other").await; + let (_desc, _key, claim) = register(&engine, &row, "Target", false, None).await; + async fn refuse(engine: &SqliteEngine, other: &BackupRow) -> StorageError { + engine + .claim_restore( + ACCOUNT, + target_input("Target", false), + ®istration(other, None), + ) + .await + .err() + .expect("the name is a restore target") + } + // While the first claim holds the slot: refused on the slot, the + // committed CREATING row naming the backup. + match refuse(&engine, &other).await { + StorageError::TableBeingRestored { + table_name, + backup_arn, + } => { + assert_eq!(table_name, "Target"); + assert_eq!(backup_arn, row.backup_arn, "the slot held"); + } + other => panic!("slot held: {other:?}"), + } + // The claim gone, the target still CREATING: refused in the + // registration transaction on the committed row. + drop(claim); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Target"))); + match refuse(&engine, &other).await { + StorageError::TableBeingRestored { + table_name, + backup_arn, + } => { + assert_eq!(table_name, "Target"); + assert_eq!( + backup_arn, row.backup_arn, + "the slot free, the row committed" + ); + } + other => panic!("slot free: {other:?}"), + } + assert!( + !engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Target")), + "a refused registration returns the slot" + ); +} + +// --- the takeover --------------------------------------------------------------- + +/// `claim_restore_dead`: refused while the owner's claim holds the slot, +/// whatever the heartbeat says; a record whose slot no task holds is taken at +/// once, fresh heartbeat or not, with a fresh generation, the claimant's +/// `spool_host` and `heartbeat_at` refreshed to the database's clock (an +/// aged record is live to the dead listing at once); a third taker is +/// refused; the old generation's heartbeat is lost and its writes match +/// nothing; the successor's FAILED write matches and leaves the progress; a +/// DONE record, a legacy record and a missing one are `None` and hold no +/// slot. +#[tokio::test] +async fn a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, Some("host-a:/a")).await; + let old_generation = claim.generation(); + engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..10))) + .await + .expect("a first file"); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(10))); + assert!( + engine + .claim_restore_dead(&key, STALE, Some("host-b:/b")) + .await + .expect("claim") + .is_none(), + "a live claim's slot refuses the takeover" + ); + // The dead owner's last beat, an hour ago: the record is in the dead + // listing until the takeover refreshes it. + sqlx::query( + "UPDATE table_restores SET heartbeat_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', \ + '-1 hour') WHERE table_id = ?", + ) + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("age the record"); + let before = record_columns(&engine, &key.table_id).await; + assert_eq!(before.4.as_deref(), Some("host-a:/a")); + + drop(claim); + assert_eq!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .len(), + 1, + "the aged record is listed before the takeover" + ); + let successor = engine + .claim_restore_dead(&key, STALE, Some("host-b:/b")) + .await + .expect("claim") + .expect("a record whose slot no task holds is dead at once"); + assert_ne!(successor.generation(), old_generation); + let after = record_columns(&engine, &key.table_id).await; + assert_eq!(after.0, "RUNNING"); + assert_eq!( + after.2, + i64::try_from(successor.generation()).unwrap(), + "the successor's generation" + ); + assert!( + after.3 > before.3, + "the takeover refreshed heartbeat_at: {} was {}", + after.3, + before.3 + ); + let age_secs: f64 = sqlx::query_scalar( + "SELECT (julianday('now') - julianday(heartbeat_at)) * 86400.0 FROM table_restores \ + WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("the record's age"); + assert!( + age_secs.abs() < 5.0, + "the refreshed heartbeat is the database's now: {age_secs} s old" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "the taken-over record is live to the listing at once" + ); + assert_eq!( + after.4.as_deref(), + Some("host-b:/b"), + "the claimant's spool" + ); + assert_eq!( + (after.5, after.6), + (Some(0), Some(10)), + "the progress stays" + ); + assert!( + engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")) + ); + assert!( + engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("claim") + .is_none(), + "a third taker is refused" + ); + + let old = stale_claim(&engine, &key.table_id, old_generation); + assert_eq!(old.heartbeat().await.expect("beat"), Heartbeat::Lost); + assert_eq!( + old.set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("old owner".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("write"), + WriteOutcome::NotMatched + ); + assert_eq!( + engine + .finish_restore(&key, &old, 10, 100) + .await + .expect("finish"), + Finished::NotMatched(format!( + "the job record of Restored is not RUNNING under generation {old_generation}, or \ + the table is not CREATING" + )) + ); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + item_count: Some(99), + failure_reason: Some("target_not_empty".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + let failed = record_columns(&engine, &key.table_id).await; + assert_eq!(failed.0, "FAILED"); + assert!(failed.7.is_some(), "the FAILED write stamps completed_at"); + assert_eq!( + (failed.5, failed.6), + (Some(0), Some(10)), + "a status write leaves the progress; item_count is not items_applied" + ); + let read = successor.read().await.expect("read"); + assert_eq!(read.status, JobStatus::Failed); + assert_eq!(read.failure_reason.as_deref(), Some("target_not_empty")); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched, + "a FAILED record is not RUNNING" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "a fresh FAILED record is not listed yet" + ); + drop(successor); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")), + "dropping the successor frees the slot" + ); + assert!( + engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("claim") + .is_none(), + "a FAILED record is not taken over" + ); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")), + "a refused takeover holds no slot" + ); + let listed = engine + .list_restore_jobs(Some(JobStatus::Failed)) + .await + .expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!( + listed[0].failure_reason.as_deref(), + Some("target_not_empty") + ); + assert_eq!(listed[0].spool_host.as_deref(), Some("host-b:/b")); + assert!(listed[0].completed_at.is_some()); + + // A legacy record is never taken over: the reconciler fails and removes it. + let legacy = engine + .create_table_impl(ACCOUNT, target_input("Legacy", false), true) + .await + .expect("a CREATING table"); + sqlx::query(&format!( + "INSERT INTO table_restores (table_id, source_backup_arn, source_table_arn, \ + restore_date_time, status, kind, owner_generation, heartbeat_at) \ + VALUES (?, 'arn:legacy', 'arn:aws:dynamodb:us-east-1:123456789012:table/Music', {NOW}, \ + 'RUNNING', 'legacy', 7, '1970-01-01T00:00:00.000Z')", + NOW = crate::backup_catalog::NOW + )) + .bind(&legacy.table_id) + .execute(&engine.pool) + .await + .expect("a legacy record"); + let legacy_key = key_of(&legacy); + assert!( + engine + .claim_restore_dead(&legacy_key, STALE, None) + .await + .expect("claim") + .is_none(), + "a legacy record is not taken over" + ); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Legacy"))); + let dead = engine.list_dead_restore_jobs(STALE).await.expect("dead"); + assert_eq!(dead.len(), 1, "{dead:?}"); + assert_eq!(dead[0].kind, RestoreKind::Legacy); + assert_eq!(dead[0].target, legacy_key); + assert_eq!( + (dead[0].store.clone(), dead[0].spool_host.clone()), + (None, None) + ); + let missing = RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "Nothing".to_owned(), + table_id: "no-such-id".to_owned(), + }; + assert!( + engine + .claim_restore_dead(&missing, STALE, None) + .await + .expect("claim") + .is_none() + ); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Nothing")) + ); + assert!( + engine + .read_restore_job(&missing) + .await + .expect("read") + .is_none() + ); +} + +// --- the load ------------------------------------------------------------------- + +/// `bulk_load` writes the base rows, every secondary index's rows and the +/// vector rows across the batch boundary, recording `(file_index, +/// items_applied)` on the record; a file resumed at `start_at` records the +/// running total; a key the backup holds twice, within a file or against an +/// earlier file, fails the file with nothing of its batch committed and the +/// progress untouched; an `Err` item fails the file the same way; another +/// account's key is refused before any write. +#[tokio::test] +async fn bulk_load_writes_each_batch_with_its_indexes_and_progress() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + let n = RESTORE_BATCH_ITEMS + 500; + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..n))) + .await + .expect("load file 0"); + assert_eq!(applied as usize, n); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + n / 2 + ); + assert_eq!( + index_rows(&engine, &key.table_id, "lix").await as usize, + n.div_ceil(3) + ); + assert_eq!(vector_rows(&engine, &key.table_id).await as usize, n / 5); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(n as i64)) + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count") as usize, + n + ); + let listed = record(&engine, &key).await; + assert_eq!( + (listed.file_index, listed.items_applied), + (Some(0), Some(n as u64)) + ); + + // A file resumed after five of its items: the record carries the total. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(1, 5), stream(items(n..n + 3))) + .await + .expect("load the rest of file 1"); + assert_eq!(applied, 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + + // A key twice within a file. + let mut twice = items(n + 3..n + 6); + twice.push(item(n + 4)); + let e = engine + .bulk_load(&key, claim.as_ref(), file(2, 0), stream(twice)) + .await + .expect_err("a key twice fails the file"); + match &e { + StorageError::Validation(m) => { + assert!( + m.starts_with("holds a primary key the target already has"), + "the backend names no file (the engine prefixes the file's key): {m}" + ); + assert!(m.contains("the backup holds a key twice"), "{m}"); + } + other => panic!("{other:?}"), + } + assert_eq!( + base_rows(&engine, &key.table_id).await as usize, + n + 3, + "nothing of the failed batch committed" + ); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + (n + 3).div_ceil(2), + "no index row of the failed batch" + ); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // A key of file 0 in file 3. + let mut across = items(n + 6..n + 8); + across.push(item(1)); + let e = engine + .bulk_load(&key, claim.as_ref(), file(3, 0), stream(across)) + .await + .expect_err("a key of an earlier file fails the file"); + assert!( + matches!(&e, StorageError::Validation(m) if m.starts_with("holds a primary key the target already has")), + "{e:?}" + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // An `Err` item mid-stream: the error, nothing committed. + let failing: BoxStream<'static, Result> = Box::pin( + futures::stream::iter(items(n + 10..n + 12).into_iter().map(Ok)).chain( + futures::stream::once(async { + Err(StorageError::Validation( + "item cannot be restored".to_owned(), + )) + }), + ), + ); + let e = engine + .bulk_load(&key, claim.as_ref(), file(4, 0), failing) + .await + .expect_err("an Err item fails the file"); + assert!( + matches!(&e, StorageError::Validation(m) if m == "item cannot be restored"), + "{e:?}" + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // Another account's key names no restore target of this account. + let foreign = RestoreTargetKey { + account_id: "111111111111".to_owned(), + ..key.clone() + }; + assert!(matches!( + engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert!(matches!( + engine + .bulk_load(&foreign, claim.as_ref(), file(5, 0), stream(items(0..1))) + .await, + Err(StorageError::TableNotFound(_)) + )); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); +} + +/// A resume from a mid-file `items_applied`: the engine hands the rest of the +/// file with `start_at` set, and exactly those items land, the record's total +/// becoming the file's count. +#[tokio::test] +async fn a_resume_from_a_mid_file_progress_applies_exactly_the_rest() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, None).await; + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..60))) + .await + .expect("the first run's batch"); + assert_eq!(applied, 60); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(60))); + // The successor: `start_at` 60, the stream from item 60. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 60), stream(items(60..100))) + .await + .expect("the resumed rest of the file"); + assert_eq!(applied, 40, "exactly the rest"); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(100))); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 100); + assert_eq!(index_rows(&engine, &key.table_id, "gix").await, 50); + assert_eq!(index_rows(&engine, &key.table_id, "lix").await, 34); + + // An empty file (a resume whose `start_at` is the file's whole count, + // or a data file with no items) records its progress as complete. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(1, 0), stream(Vec::new())) + .await + .expect("an empty file"); + assert_eq!(applied, 0); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(1), Some(0)), + "an empty file's progress is recorded" + ); + let applied = engine + .bulk_load(&key, claim.as_ref(), file(2, 7), stream(Vec::new())) + .await + .expect("a file resumed at its end"); + assert_eq!(applied, 0); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(2), Some(7)), + "`start_at` is the recorded total of an empty rest" + ); +} + +/// A batch closes at the first item whose text takes it past +/// `RESTORE_BATCH_BYTES`, long before the count cap: a file of 100 items of +/// about 350 KB each (35 MB of text against the 32 MiB bound) is written in +/// two transactions, the first committed with its progress before the +/// stream hands over its last item. +#[tokio::test] +async fn a_batch_closes_at_the_byte_bound_before_the_count_cap() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, None).await; + const N: usize = 100; + const TEXT: usize = 350_000; + const { + assert!(N < RESTORE_BATCH_ITEMS); + // 100 items cross the byte bound once. + assert!(N * TEXT > RESTORE_BATCH_BYTES && (N - 2) * TEXT < 2 * RESTORE_BATCH_BYTES); + } + let big = |i: usize| -> Item { + let mut it = item(i); + it.insert("v".to_owned(), AttributeValue::S("x".repeat(TEXT))); + it + }; + // The stream reads the record when asked for the last item: by then the + // byte bound has closed and committed the first batch. + let pool = engine.pool.clone(); + let table_id = key.table_id.clone(); + let seen = Arc::new(std::sync::Mutex::new(None)); + let seen_by_stream = Arc::clone(&seen); + let bounded: BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let pool = pool.clone(); + let table_id = table_id.clone(); + let seen = Arc::clone(&seen_by_stream); + async move { + if i >= N { + return None; + } + if i == N - 1 { + let progress: (Option, Option) = sqlx::query_as( + "SELECT file_index, items_applied FROM table_restores WHERE table_id = ?", + ) + .bind(&table_id) + .fetch_one(&pool) + .await + .expect("progress"); + *seen.lock().expect("seen") = Some(progress); + } + Some((Ok(big(i)), i + 1)) + } + })); + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), bounded) + .await + .expect("load"); + assert_eq!(applied as usize, N); + let (file_index, items_applied) = seen + .lock() + .expect("seen") + .expect("the stream read the record at its last item"); + assert_eq!( + file_index, + Some(0), + "the first batch committed with its progress" + ); + let first_batch = items_applied.expect("the first batch's total") as usize; + assert!( + 0 < first_batch && first_batch < N, + "the first batch closed at the byte bound with {first_batch} of {N} items" + ); + assert!( + first_batch * TEXT < RESTORE_BATCH_BYTES + TEXT + 1_000 * N, + "the batch closed at the first item past the bound: {first_batch} items" + ); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(N as i64)) + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, N); +} + +/// A claim lost between two batches of one file: the second batch's progress +/// write matches no record, the batch rolls back whole (base, index and +/// vector rows), and `bulk_load` fails as a lost claim; the first batch and +/// its progress stand. +#[tokio::test] +async fn a_lost_claim_mid_file_rolls_its_batch_back() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + let n = RESTORE_BATCH_ITEMS + 200; + // The stream hands the first batch, then takes the record away before + // the first item of the second. + let pool = engine.pool.clone(); + let table_id = key.table_id.clone(); + let taken: BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let pool = pool.clone(); + let table_id = table_id.clone(); + async move { + if i >= n { + return None; + } + if i == RESTORE_BATCH_ITEMS { + sqlx::query( + "UPDATE table_restores SET owner_generation = owner_generation + 1 \ + WHERE table_id = ?", + ) + .bind(&table_id) + .execute(&pool) + .await + .expect("another owner takes the record"); + } + Some((Ok(item(i)), i + 1)) + } + })); + let e = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), taken) + .await + .expect_err("the lost claim ends the file"); + assert!( + matches!(&e, StorageError::Internal(m) if m.starts_with("claim lost")), + "{e:?}" + ); + assert_eq!( + base_rows(&engine, &key.table_id).await as usize, + RESTORE_BATCH_ITEMS, + "the second batch's base rows rolled back" + ); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + RESTORE_BATCH_ITEMS / 2, + "the second batch's index rows rolled back" + ); + assert_eq!( + vector_rows(&engine, &key.table_id).await as usize, + RESTORE_BATCH_ITEMS / 5, + "the second batch's vector rows rolled back" + ); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(RESTORE_BATCH_ITEMS as i64)), + "the first batch's progress stands" + ); + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); +} + +// --- the flip ------------------------------------------------------------------- + +/// `finish_restore`: a stale generation is `NotMatched` and changes nothing; +/// the claim's flips the record DONE with `completed_at` and the progress +/// left, the table ACTIVE with its counts and no transition, and every +/// secondary and vector index ACTIVE in one transaction; the heartbeat is +/// lost after DONE; a second flip is `NotMatched`; the backup is no longer +/// in use; `describe_table` reports ACTIVE with the count. +/// `remove_restore_target` of a CREATING target removes it with its vector +/// data table. +#[tokio::test] +async fn finish_restore_flips_the_table_and_every_index() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..30))) + .await + .expect("load"); + // A secondary index left CREATING by hand, beside the vector index the + // creation left CREATING; the vector row's `backfilling` set as a backfill + // in progress leaves it, so the flip's `backfilling = NULL` is observed + // (an ACTIVE row may not carry it: `chk_vector_active_has_no_backfilling`). + sqlx::query( + "UPDATE indexes SET index_status = 'CREATING' WHERE table_id = ? AND index_name = 'gix'", + ) + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("a CREATING index"); + sqlx::query("UPDATE vector_indexes SET backfilling = 1 WHERE table_id = ?") + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("a backfilling vector index"); + assert_eq!( + vector_indexes(&engine, &key.table_id) + .await + .iter() + .map(|(_, status, backfilling)| (status.as_str(), *backfilling)) + .collect::>(), + vec![("CREATING", Some(true))], + "the fixture's one vector index is CREATING and backfilling before the flip" + ); + + let stale = stale_claim(&engine, &key.table_id, claim.generation() + 1); + assert!(matches!( + engine + .finish_restore(&key, &stale, 30, 3000) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); + assert_eq!(record_columns(&engine, &key.table_id).await.0, "RUNNING"); + + assert_eq!( + engine + .finish_restore(&key, claim.as_ref(), 30, 3000) + .await + .expect("finish"), + Finished::Active + ); + let (status, _, _, _, _, file_index, items_applied, completed_at) = + record_columns(&engine, &key.table_id).await; + assert_eq!(status, "DONE"); + assert!(completed_at.is_some(), "DONE stamps completed_at"); + assert_eq!( + (file_index, items_applied), + (Some(0), Some(30)), + "the DONE write leaves the progress" + ); + let table: (String, i64, i64, Option) = sqlx::query_as( + "SELECT table_status, item_count, table_size_bytes, status_transition_at \ + FROM tables WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("the table row"); + assert_eq!(table, ("ACTIVE".to_owned(), 30, 3000, None)); + for (name, _, status) in indexes(&engine, &key.table_id).await { + assert_eq!(status, "ACTIVE", "every secondary index is ACTIVE ({name})"); + } + for (id, status, backfilling) in vector_indexes(&engine, &key.table_id).await { + assert_eq!( + (status.as_str(), backfilling), + ("ACTIVE", None), + "every vector index is ACTIVE with `backfilling` cleared ({id})" + ); + } + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); + assert!(matches!( + engine + .finish_restore(&key, claim.as_ref(), 30, 3000) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + None + ); + let described = engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "Restored".to_owned(), + }, + ) + .await + .expect("describe"); + assert_eq!(described.table_status, TableStatus::Active); + assert_eq!(described.item_count, 30); + let done = engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list"); + assert_eq!(done.len(), 1); + assert!(done[0].completed_at.is_some()); + assert!( + matches!( + engine.count_restore_target(&key).await, + Err(StorageError::TableNotFound(_)) + ), + "an ACTIVE table is no longer a CREATING restore target" + ); + + // Removal of another target: the table, its record, its index and + // vector data tables. + let other = available_backup(&engine, "Other").await; + let (desc, other_key, other_claim) = register(&engine, &other, "Gone", true, None).await; + engine + .bulk_load( + &other_key, + other_claim.as_ref(), + file(0, 0), + stream(items(0..10)), + ) + .await + .expect("load"); + let (vector_id, _, _) = vector_indexes(&engine, &other_key.table_id) + .await + .pop() + .unwrap(); + let data_tables = || async { + sqlx::query_scalar::<_, i64>( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name IN (?, ?)", + ) + .bind(crate::data::data_table_name(&desc.table_id).trim_matches('"')) + .bind(crate::data::vector_table_name(&desc.table_id, &vector_id).trim_matches('"')) + .fetch_one(&engine.pool) + .await + .expect("sqlite_master") + }; + assert_eq!(data_tables().await, 2); + drop(other_claim); + assert!( + engine + .remove_restore_target(&other_key) + .await + .expect("remove") + ); + assert_eq!(tables_named(&engine, "Gone").await, 0); + assert_eq!( + data_tables().await, + 0, + "the data and vector tables are dropped" + ); + assert!( + engine + .read_restore_job(&other_key) + .await + .expect("read") + .is_none() + ); + assert!( + !engine + .remove_restore_target(&other_key) + .await + .expect("remove again") + ); +} diff --git a/crates/storage-sqlite/tests/backup_job_panic.rs b/crates/storage-sqlite/tests/backup_job_panic.rs index d366de136..14bd52ed8 100644 --- a/crates/storage-sqlite/tests/backup_job_panic.rs +++ b/crates/storage-sqlite/tests/backup_job_panic.rs @@ -285,7 +285,7 @@ async fn a_job_task_that_panics_leaves_a_row_the_next_cycle_resolves_without_a_r // One reconciler cycle in the same process: the slot no task holds makes // the row dead at once, and without a manifest it is FAILED. let mut state = ReconcilerState::with_shutdown(shutdown.child_token()); - let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) .await .expect("a reconciler cycle"); assert_eq!(report.store_healthy, Some(true), "{report:?}"); @@ -326,7 +326,7 @@ async fn a_job_task_that_panics_leaves_a_row_the_next_cycle_resolves_without_a_r Ok(answer) => panic!("DescribeBackup of the FAILED row answered {}", answer.body), } // A second cycle has nothing left to resolve. - let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) .await .expect("a second cycle"); assert!(report.creating_resolved.is_empty(), "{report:?}"); diff --git a/crates/storage-sqlite/tests/restore_job_resume.rs b/crates/storage-sqlite/tests/restore_job_resume.rs new file mode 100644 index 000000000..2b1a5dd81 --- /dev/null +++ b/crates/storage-sqlite/tests/restore_job_resume.rs @@ -0,0 +1,491 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! A store-backed restore on SQLite whose owner died mid-file is resumed by +//! one reconciler cycle in the same process, from the record's +//! `(file_index, items_applied)`, and ends ACTIVE with the backup's item +//! count (design 4(e) and 5.3 step 5 on this backend: a RUNNING `manifest` +//! record whose in-process slot no task holds is dead at once, and the +//! progress on the record is where the resume starts). +//! +//! The engine is built the way the server builds it (the backend's server +//! components factory, its workers spawned through the runtime hooks), the +//! store is a real filesystem store, and the backup is taken through the +//! engine's `dispatch` with data files small enough to make several. The +//! dead owner is played at the storage level: `claim_restore`, `bulk_load` +//! of the first file and of half of the second, the claim dropped (its slot +//! freed) and the record's heartbeat aged as the server's restart ages it. +//! The cycle takes the record over, the resume loads the rest, and the table +//! reaches ACTIVE with every item once; a second cycle finds nothing. + +use std::os::unix::fs::DirBuilderExt; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +use extenddb_backup_store::FilesystemStore; +use extenddb_core::backup_format::{ + BackupArn, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, data_file_key, +}; +use extenddb_core::limits::LimitsConfig; +use extenddb_core::types::Item; +use extenddb_engine::reconciler::{self, ReconcilerState}; +use extenddb_engine::{ + BackupRuntime, ConfiguredStore, JobSettings, OperationContext, check_configured_store, dispatch, +}; +use extenddb_storage::backup_catalog::{ + JobStatus, RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StoreKind, UNOWNED_GENERATION, +}; +use extenddb_storage::restore_engine::BulkLoadFile; +use extenddb_storage::server_components::ServerComponentsOptions; +use extenddb_storage::{ + CancellationToken, RestoreTableOverrides, WorkerContext, + backup_definition::restore_input_from_manifest, +}; +use extenddb_storage_sqlite::SqliteConfig; +use serde_json::json; + +const REGION: &str = "us-east-1"; +const ITEMS: usize = 300; + +/// A scratch directory under the cargo target directory, removed on drop. +struct Scratch(PathBuf); + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +async fn call( + ctx: &OperationContext, + operation: &str, + body: serde_json::Value, +) -> serde_json::Value { + dispatch(operation, body, ctx, "127.0.0.1:0") + .await + .unwrap_or_else(|e| panic!("{operation}: {e:?}")) + .body +} + +async fn wait_table_status(ctx: &OperationContext, table: &str, status: &str) -> serde_json::Value { + let deadline = std::time::Instant::now() + Duration::from_secs(30); + loop { + let desc = call(ctx, "DescribeTable", json!({"TableName": table})).await; + if desc["Table"]["TableStatus"] == status { + return desc; + } + assert!( + std::time::Instant::now() < deadline, + "{table} did not reach {status}: {desc}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } +} + +async fn scan_count(ctx: &OperationContext, table: &str, index: Option<&str>) -> usize { + let mut count = 0; + let mut start: Option = None; + loop { + let mut body = json!({"TableName": table, "Select": "COUNT"}); + if let Some(index) = index { + body["IndexName"] = json!(index); + } + if let Some(key) = &start { + body["ExclusiveStartKey"] = key.clone(); + } + let page = call(ctx, "Scan", body).await; + count += usize::try_from(page["Count"].as_u64().expect("Count")).unwrap(); + match page.get("LastEvaluatedKey") { + Some(key) if !key.is_null() => start = Some(key.clone()), + _ => return count, + } + } +} + +/// The items of data file `index` of the backup under `root`, decoded as the +/// engine's load decodes them. +fn file_items( + root: &std::path::Path, + arn: &BackupArn, + manifest: &ExtenddbManifest, + index: usize, +) -> Vec { + let entry = &manifest.data_files[index]; + // Data file keys count from one; `BulkLoadFile::file_index` from zero. + assert_eq!(entry.key, data_file_key(u32::try_from(index + 1).unwrap())); + let compressed = std::fs::File::open(root.join(arn.store_prefix()).join(&entry.key)) + .expect("the data file in the store"); + let mut reader = manifest + .reader_prehashed(entry, compressed) + .expect("a reader"); + let mut items = Vec::new(); + while let Some(item) = reader.next_item().expect("an item") { + items.push(item); + } + assert_eq!(items.len() as u64, entry.item_count); + items +} + +fn stream( + items: Vec, +) -> futures::stream::BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_dead_restore_is_resumed_by_one_cycle_from_its_recorded_progress() { + let scratch = Scratch( + PathBuf::from(env!("CARGO_TARGET_TMPDIR")) + .join(format!("restore-job-resume-{}", uuid::Uuid::new_v4())), + ); + std::fs::create_dir_all(&scratch.0).expect("scratch directory"); + let db_path = scratch.0.join("catalog.sqlite"); + let root = scratch.0.join("backups"); + std::fs::DirBuilder::new() + .mode(0o700) + .create(&root) + .expect("root"); + + // The engine as the server builds it. + let config = SqliteConfig { + path: db_path.to_str().expect("UTF-8 path").to_owned(), + pool_size: 4, + }; + let components = (extenddb_storage_sqlite::backend().server_components)(&config, REGION, { + let mut options = ServerComponentsOptions::default(); + options.bootstrap_if_uninitialized = true; + options.release = "test"; + options.max_concurrent_jobs = 4; + options + }) + .await + .unwrap_or_else(|e| panic!("server components: {e:?}")); + let storage = components.engine; + let shutdown = CancellationToken::new(); + let (_layer, reload_handle) = + tracing_subscriber::reload::Layer::new(tracing_subscriber::EnvFilter::new("info")); + let worker_ctx = WorkerContext { + metrics: Arc::new(extenddb_core::metrics::MetricsCollector::new()), + catalog_store: components.catalog_store.clone(), + reload_handle, + config_log_level: "info".to_owned(), + shutdown: shutdown.clone(), + }; + let hooks = components + .runtime_hooks + .expect("the SQLite backend has runtime hooks"); + let workers = hooks.spawn_workers(&worker_ctx).await; + let catalog = sqlx::SqlitePool::connect(&format!("sqlite://{}", db_path.display())) + .await + .expect("a second connection to the catalog"); + for (key, value) in [ + ("control_plane_delay_seconds", "0.05"), + ("index_propagation_delay_ms", "0"), + ] { + sqlx::query("INSERT OR REPLACE INTO settings (key, value) VALUES (?, ?)") + .bind(key) + .bind(value) + .execute(&catalog) + .await + .expect("setting"); + } + let account_id: String = + sqlx::query_scalar("SELECT value FROM settings WHERE key = 'default_account_id'") + .fetch_one(&catalog) + .await + .expect("the bootstrapped account"); + + // A healthy filesystem store, its identity file written by the check + // the server runs at startup; data files of about 4 KB so the backup + // has several. + let store = Arc::new( + FilesystemStore::open(&root) + .await + .expect("filesystem store"), + ); + let configured = ConfiguredStore::checking( + StoreKind::Filesystem, + root.to_str().expect("UTF-8 path").to_owned(), + store, + None, + ); + let outcome = check_configured_store(&configured, &*storage, "sqlite").await; + assert!(outcome.is_ok(), "the store's first check: {outcome:?}"); + configured.record_check(outcome); + let stale_job_after = Duration::from_secs(300); + let runtime = Arc::new( + BackupRuntime::new(4) + .with_stale_job_after(stale_job_after) + .with_store(configured) + .with_source("sqlite", "test") + .with_job_settings(JobSettings { + data_file_target_bytes: 4096, + ..JobSettings::default() + }) + .with_limits(Arc::new(LimitsConfig::default())) + .with_shutdown(shutdown.child_token()), + ); + let ctx = OperationContext { + storage: storage.clone(), + backup: runtime.clone(), + limits: Arc::new(LimitsConfig::default()), + region: Arc::from(REGION), + account_id: Arc::from(account_id.as_str()), + import_paths: Arc::from(Vec::new()), + export_paths: Arc::from(Vec::new()), + authorized_key_info: Default::default(), + auth_cache: extenddb_auth::AuthCacheRegistry::empty(), + table_key_info_lookup: None, + }; + + call( + &ctx, + "CreateTable", + json!({ + "TableName": "Music", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }] + }), + ) + .await; + wait_table_status(&ctx, "Music", "ACTIVE").await; + for i in 0..ITEMS { + let mut item = json!({ + "pk": {"S": format!("p{}", i % 7)}, + "sk": {"N": i.to_string()}, + "note": {"S": format!("item {i} of the fixture, padded to make the files small {}", "x".repeat(40))} + }); + if i % 2 == 0 { + item["g"] = json!({"S": format!("g{}", i % 4)}); + } + call(&ctx, "PutItem", json!({"TableName": "Music", "Item": item})).await; + } + + // The backup, to the store, AVAILABLE. + let details = call( + &ctx, + "CreateBackup", + json!({"TableName": "Music", "BackupName": "nightly"}), + ) + .await; + let arn_text = details["BackupDetails"]["BackupArn"] + .as_str() + .expect("an ARN") + .to_owned(); + let deadline = std::time::Instant::now() + Duration::from_secs(60); + loop { + let described = call(&ctx, "DescribeBackup", json!({"BackupArn": arn_text})).await; + if described["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" { + break; + } + assert!( + std::time::Instant::now() < deadline, + "the backup did not become AVAILABLE: {described}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(20)).await; + } + let arn = BackupArn::parse(&arn_text).expect("a backup ARN"); + let row = storage + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + let manifest_bytes = std::fs::read(root.join(arn.store_prefix()).join(EXTENDDB_MANIFEST_FILE)) + .expect("manifest"); + let manifest = ExtenddbManifest::from_json_bytes(&manifest_bytes).expect("a manifest"); + assert!( + manifest.data_files.len() >= 3, + "the backup has several data files: {}", + manifest.data_files.len() + ); + let total: u64 = manifest.data_files.iter().map(|f| f.item_count).sum(); + assert_eq!(total as usize, ITEMS); + + // The dead owner: the registration, file 0 whole, half of file 1, then + // gone (the claim dropped frees the slot; the heartbeat aged as the + // server's restart ages every RUNNING record). + let input = restore_input_from_manifest( + &manifest.source_table, + "Restored", + &RestoreTableOverrides::default(), + ); + let registration = RestoreRegistrationRow { + backup: row.key(), + source_table_arn: manifest.source_table.table_arn.clone(), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: None, + owner_generation: UNOWNED_GENERATION, + }; + let (desc, claim) = storage + .claim_restore(&account_id, input, ®istration) + .await + .expect("register the restore"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let file0 = file_items(&root, &arn, &manifest, 0); + let file1 = file_items(&root, &arn, &manifest, 1); + let half = file1.len() / 2; + assert!(half > 0); + storage + .bulk_load( + &key, + claim.as_ref(), + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(file0.clone()), + ) + .await + .expect("file 0"); + storage + .bulk_load( + &key, + claim.as_ref(), + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + stream(file1[..half].to_vec()), + ) + .await + .expect("half of file 1"); + let dead_generation = claim.generation(); + drop(claim); + let before = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_eq!(before.status, JobStatus::Running); + assert_eq!(before.kind, RestoreKind::Manifest); + assert_eq!( + (before.file_index, before.items_applied), + (Some(1), Some(half as u64)) + ); + assert_eq!( + storage.count_restore_target(&key).await.expect("count") as usize, + file0.len() + half + ); + sqlx::query( + "UPDATE table_restores SET heartbeat_at = '1970-01-01T00:00:00.000Z' WHERE table_id = ?", + ) + .bind(&key.table_id) + .execute(&catalog) + .await + .expect("age the heartbeat as a restart does"); + let desc_before = call(&ctx, "DescribeTable", json!({"TableName": "Restored"})).await; + assert_eq!(desc_before["Table"]["TableStatus"], "CREATING"); + assert!( + desc_before["Table"]["RestoreSummary"]["RestoreInProgress"] + .as_bool() + .unwrap_or(false), + "{desc_before}" + ); + + // One cycle: the record's slot no task holds makes it dead at once; the + // resume is spawned through the runtime. + let mut state = ReconcilerState::with_shutdown(shutdown.child_token()); + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a reconciler cycle"); + assert_eq!(report.store_healthy, Some(true), "{report:?}"); + assert_eq!( + report.manifest_restores_resumed, + vec!["Restored".to_owned()], + "{report:?}" + ); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + assert!(report.legacy_restores_removed.is_empty(), "{report:?}"); + let taken = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_ne!( + taken.owner_generation, dead_generation, + "the takeover minted a generation" + ); + assert_eq!(taken.spool_host, None, "a filesystem store has no spool"); + let deadline = std::time::Instant::now() + Duration::from_secs(60); + while runtime.running_jobs() > 0 { + assert!( + std::time::Instant::now() < deadline, + "the resumed restore did not end" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + + // The table ACTIVE with every item once; the record DONE at the last + // file's count; the GSI answering. + let desc_after = wait_table_status(&ctx, "Restored", "ACTIVE").await; + assert_eq!( + desc_after["Table"]["ItemCount"], + json!(0), + "the restored table's ItemCount is 0 at ACTIVE, as the service reports it" + ); + assert!( + desc_after["Table"].get("RestoreSummary").is_none(), + "{desc_after}" + ); + assert_eq!(scan_count(&ctx, "Restored", None).await, ITEMS); + assert_eq!(scan_count(&ctx, "Restored", Some("gix")).await, ITEMS / 2); + assert_eq!(scan_count(&ctx, "Music", None).await, ITEMS); + let done = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_eq!(done.status, JobStatus::Done); + assert!(done.completed_at.is_some()); + let last = manifest.data_files.len() - 1; + assert_eq!( + (done.file_index, done.items_applied), + ( + Some(u32::try_from(last).unwrap()), + Some(manifest.data_files[last].item_count) + ), + "the progress names the last file complete" + ); + assert_eq!(done.owner_generation, taken.owner_generation); + assert_eq!( + storage.backup_in_use(&row.key()).await.expect("in use"), + None + ); + + // A second cycle has nothing left to resume. + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a second cycle"); + assert!(report.manifest_restores_resumed.is_empty(), "{report:?}"); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + + catalog.close().await; + shutdown.cancel(); + for worker in workers { + let _ = tokio::time::timeout(Duration::from_secs(5), worker).await; + } +} diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs index 96766a6de..5831f0c52 100644 --- a/crates/storage/src/backup_catalog.rs +++ b/crates/storage/src/backup_catalog.rs @@ -28,9 +28,9 @@ use std::time::Duration; use extenddb_core::backup_format::{BackupArn, SourceTableSchema}; use extenddb_core::types::{ - BackupDescription, BackupDetails, BackupSummary, BillingMode, GlobalSecondaryIndexInfo, - LocalSecondaryIndexInfo, ProvisionedThroughput, SourceTableDetails, SourceTableFeatureDetails, - TableDescription, + BackupDescription, BackupDetails, BackupSummary, BillingMode, CreateTableInput, + GlobalSecondaryIndexInfo, LocalSecondaryIndexInfo, ProvisionedThroughput, SourceTableDetails, + SourceTableFeatureDetails, TableDescription, }; use futures::future::BoxFuture; use futures::stream::BoxStream; @@ -70,12 +70,13 @@ pub fn pool_size_for_jobs(max_concurrent_jobs: usize) -> u32 { jobs.saturating_mul(2).saturating_add(10) } -/// How often a running job refreshes the heartbeat of the record it owns, -/// on every backend, until a later release wires `[backup] -/// heartbeat_interval_secs` to the jobs. `BackupConfig::validate` floors -/// `stale_job_after_secs` on this period, so a configured interval shorter -/// than it cannot make the reconciler declare a live job dead between two -/// heartbeats. +/// The floor of a job's heartbeat period on every backend: the store-backed +/// jobs run at `max([backup] heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)` +/// (`BackupConfig::effective_heartbeat_interval_secs`, which the server +/// hands the jobs), the in-catalog paths and the maintenance claim at the +/// constant. `BackupConfig::validate` floors `stale_job_after_secs` on this +/// period, so a configured interval shorter than it cannot make the +/// reconciler declare a live job dead between two heartbeats. pub const JOB_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(15); /// The bound a job refusal names, so the text says which one was met @@ -616,21 +617,152 @@ impl RestoreKind { } } -/// A restore job record the reconciler acts on: RUNNING with a dead owner, -/// or FAILED with its target still CREATING ([`BackupCatalog::list_dead_restore_jobs`]). +/// A restore job record: the row [`BackupCatalog::claim_restore`] writes +/// beside a CREATING target, as [`BackupCatalog::read_restore_job`] and +/// [`BackupCatalog::list_restore_jobs`] return it, and the one the +/// reconciler acts on (RUNNING with a dead owner, or FAILED with its target +/// still CREATING, [`BackupCatalog::list_dead_restore_jobs`]). #[derive(Debug, Clone, PartialEq)] pub struct RestoreJobRow { pub target: RestoreTargetKey, pub source_backup_arn: String, pub kind: RestoreKind, - /// RUNNING or FAILED; the reconciler reads it to decide whether the - /// FAILED write is still to be made. + /// RUNNING, DONE or FAILED; the reconciler reads it to decide whether + /// the FAILED write is still to be made. pub status: JobStatus, pub owner_generation: u64, /// Epoch seconds, on the database's clock. pub heartbeat_at: f64, + /// The store the job reads its backup from; `None` on a legacy record, + /// which reads the catalog. + pub store: Option, + /// `:` of the instance that downloads the backup + /// into its spool; `None` for a filesystem store, which is read in + /// place, and on a legacy record. A takeover rewrites it to the + /// claimant's. + pub spool_host: Option, + /// The last data file whose progress the job recorded, as the backend + /// records it (SQLite and Cassandra write it per batch, MongoDB per + /// file; PostgreSQL keeps its progress in the data database and leaves + /// this `None`). + pub file_index: Option, + /// Items of `file_index` applied so far, where the backend records + /// mid-file progress (SQLite and Cassandra). + pub items_applied: Option, + pub failure_reason: Option, + /// Epoch seconds, on the database's clock; set by the DONE and FAILED + /// writes. + pub completed_at: Option, +} + +impl RestoreJobRow { + /// The row as the four backends' `list_dead_restore_jobs` built it + /// before the restore job existed: the six members the reconciler's + /// duties on legacy and dead records read, the rest `None`. + #[must_use] + pub fn reconciler_row( + target: RestoreTargetKey, + source_backup_arn: String, + kind: RestoreKind, + status: JobStatus, + owner_generation: u64, + heartbeat_at: f64, + ) -> Self { + Self { + target, + source_backup_arn, + kind, + status, + owner_generation, + heartbeat_at, + store: None, + spool_host: None, + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + } + } +} + +/// What [`BackupCatalog::claim_restore`] returns: the CREATING target's +/// description with its `RestoreSummary`, and the claim on its record. +pub type RestoreRegistered = (TableDescription, Box); + +/// What a restore's registration writes beside the target table it creates +/// (the design's 5.3 step 4): the backup the job reads, the provenance the +/// target's `RestoreSummary` reports, the store, the claimant's spool and +/// the claim's generation. The record is born RUNNING and `kind = manifest`. +#[derive(Debug, Clone, PartialEq)] +pub struct RestoreRegistrationRow { + pub backup: BackupKey, + /// Reported as `RestoreSummary.SourceTableArn`. + pub source_table_arn: String, + /// Reported as `RestoreSummary.RestoreDateTime`, epoch seconds: the + /// backup's creation time, as the service reports it. + pub restore_date_time: f64, + pub store: StoreIdentity, + /// `:` when the job downloads into a spool (an S3 + /// store); `None` for a filesystem store. + pub spool_host: Option, + /// The claim's fencing token, written on the record with the target. + pub owner_generation: u64, +} + +/// The names of the pause points inside the backends' restore writes, one +/// spelling each: the backend passes its constant to +/// [`restore_pause_point`], and the kill-point harness names the same +/// constant in `EXTENDDB_BACKUP_PAUSE`. The engine's own kill points are its +/// `KillPoint` enum; these four live here because the backends cannot see +/// the engine. +/// +/// SQLite's `bulk_load`: between two batches of one file, after the first +/// batch's transaction committed with its progress. +pub const RESTORE_SQLITE_MID_BATCH: &str = "restore_sqlite_mid_batch"; +/// MongoDB's `bulk_load`: after the file's last `insert_many` returned, +/// before the record's `file_index` write. +pub const RESTORE_MONGODB_AFTER_INSERT_MANY: &str = "restore_mongodb_after_insert_many"; +/// Cassandra's `bulk_load`: after a batch's writes, before its progress +/// lightweight transaction. +pub const RESTORE_CASSANDRA_BEFORE_PROGRESS: &str = "restore_cassandra_before_progress"; +/// PostgreSQL's `finish_restore`: after the ACTIVE flip committed, before +/// the `restore_progress` rows are deleted. +pub const RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE: &str = "restore_postgres_before_progress_delete"; + +/// A pause point inside a backend's `bulk_load` or its progress write, for +/// the binary-level kill-point harness: with `test-hooks` it pauses while +/// `EXTENDDB_BACKUP_PAUSE` names `name` (one of the constants above), until +/// the file `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists, logging `restore +/// job reached kill point ` as the engine's own kill points do (one +/// harness drives both); without the feature it is a no-op. +#[cfg(feature = "test-hooks")] +pub async fn restore_pause_point(name: &str) { + if std::env::var("EXTENDDB_BACKUP_PAUSE").ok().as_deref() != Some(name) { + return; + } + let release = std::env::var("EXTENDDB_BACKUP_PAUSE_RELEASE").ok(); + tracing::warn!( + "restore job reached kill point {name}; paused until {} exists", + release + .as_deref() + .unwrap_or("no release file (the pause does not end)") + ); + loop { + if let Some(release) = &release + && std::path::Path::new(release).exists() + { + tracing::warn!("restore job released from kill point {name}"); + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } } +/// [`restore_pause_point`] without `test-hooks`: nothing. +#[cfg(not(feature = "test-hooks"))] +#[allow(clippy::unused_async)] +pub async fn restore_pause_point(_name: &str) {} + /// A legacy CREATING backup row whose create died. #[derive(Debug, Clone, PartialEq, Eq)] pub struct AbandonedLegacyBackup { @@ -884,6 +1016,24 @@ pub trait BackupCatalog: Send + Sync { key: &'a RestoreTargetKey, ) -> BoxFuture<'a, Result>; + /// [`Self::remove_restore_target`] for the `extenddb backup + /// remove-target` command, which runs with no claim of its own: a + /// backend whose owner holds a lock that outlives its heartbeat + /// (PostgreSQL's per-session advisory lock) takes that lock here first, + /// so the removal and a reconciler's takeover of the same target exclude + /// each other and a frozen owner whose session still holds the lock + /// keeps its target; `false` when the lock is held. The default is + /// [`Self::remove_restore_target`]: the job's own failure path and the + /// reconciler's resume already hold the claim when they remove, so the + /// lock lives only on this variant, and the other backends fence the + /// takeover on the record's generation and heartbeat alone. + fn remove_restore_target_exclusive<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + self.remove_restore_target(key) + } + /// The restore job records the reconciler acts on, two kinds: RUNNING /// records whose `heartbeat_at` is older than `stale_after` on the /// database's clock (the owner is dead by every backend's rule; the @@ -917,7 +1067,9 @@ pub trait BackupCatalog: Send + Sync { /// nothing and the live owner keeps its record. The write sets a fresh /// `owner_generation`, fencing the dead job's writes out. `false` when the /// record was not RUNNING under those values or is gone. The reconciler's - /// write for a dead legacy restore, which is never resumed. + /// write for a dead legacy restore, which is never resumed, and for a + /// manifest restore it cannot resume (a store other than the configured + /// one, a maintenance claim held). fn fail_restore_job<'a>( &'a self, key: &'a RestoreTargetKey, @@ -926,6 +1078,92 @@ pub trait BackupCatalog: Send + Sync { reason: &'a str, ) -> BoxFuture<'a, Result>; + /// A store-backed restore's registration (the design's 5.3 step 4): + /// create the target table CREATING with no scheduled transition and + /// its RUNNING `kind = manifest` job record, `registration`'s store, + /// `spool_host` and generation on it, under the + /// lock on the backup's `backup_index` row that + /// [`Self::start_backup_delete`] takes, re-checking there that the row + /// is AVAILABLE and that no CREATING target of the backup exists. The + /// catalog rows commit first and the target's data tables (on MongoDB + /// its collections and index metadata) are created after the commit, in + /// the order [`RestoreEngine::restore_target_ready`]'s doc states per + /// backend, so a kill right after the commit leaves a RUNNING record + /// over a target with no data table; the reconciler's resume checks + /// `restore_target_ready` before it resumes a record. A + /// DeleteBackup and a registration of one backup therefore order wholly + /// one before the other. Returns the CREATING description, its + /// `RestoreSummary` filled from `registration`, and the claim the job + /// heartbeats and writes its status through; the record is born owned, + /// as [`Self::claim_backup`]'s row is. The record's `heartbeat_at` is + /// the database's clock as the backend stamps it at the insert: on + /// PostgreSQL `now()` is the registration transaction's start, so a + /// registration that waited on the `backup_index` row's lock, or whose + /// data DDL after the commit took long, hands back a record whose + /// heartbeat is that much older than its return; the engine's first beat, + /// sent before the registration's reply, refreshes it. + /// + /// # Errors + /// + /// `Validation("Backup not found: ")` ([`is_backup_not_found`]) + /// when the row is missing or not AVAILABLE; `BackupInUse` when a target + /// of the backup is CREATING; `TableAlreadyExists` when the name is + /// taken, or `TableBeingRestored` when it is a CREATING restore target; + /// `LimitExceeded` when the backend admits no further job. + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result>; + + /// Take over a RUNNING manifest record whose owner is dead (the design's + /// 4(e)): a fresh generation, the claimant's `spool_host` and + /// `heartbeat_at` set to the database's clock, in one conditional write, + /// so a second taker finds the record live at once. The refresh is where + /// this takeover differs from [`Self::claim_dead_backup`]: the backup + /// takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale + /// window counts from it, while nothing waits on a restore record's old + /// heartbeat after a takeover (the FAILED-target duty ages on + /// `completed_at`, which every FAILED write of a resume stamps, and the + /// maintenance claim waits on RUNNING, not on the beat); without the + /// refresh a second reconciler's cycle landing before the successor's + /// first beat (one period after the takeover) would take the same record + /// again. Dead is `heartbeat_at` older than `stale_after` on the + /// database's clock; on PostgreSQL the owner's advisory lock on the + /// target name is tried first, so a stopped owner whose session lives + /// keeps its record, as [`Self::claim_dead_backup`] says; on SQLite the + /// name's in-process slot is the test instead and `stale_after` is not + /// read: a record whose slot no task of this process holds is dead at + /// once (a restart frees every slot and ages every RUNNING record), and + /// one whose slot a task holds is alive whatever its heartbeat says. + /// `None` when the record is not RUNNING, is gone, or has a live owner; + /// the old generation's writes find [`WriteOutcome::NotMatched`] from + /// then on. On Cassandra the takeover also re-applies the target's + /// `active_restores` entry on the backup's row when it is absent (a + /// crash between the target's DDL and the registration), so the backup + /// stays in use for the resumed load. + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>>; + + /// The job record of the target `key` names, of whatever status; `None` + /// when the table is not a restore target with this table id. + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>>; + + /// Every job record with `status`, or of every status for `None`: the + /// reconciler's listing of RUNNING targets for its sweeps. + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>>; + /// The distinct `store_id`s the catalog's rows record for a root, of /// every status: the store identity check reads them before it knows an /// id. Empty when no row names the root. diff --git a/crates/storage/src/backup_definition.rs b/crates/storage/src/backup_definition.rs index b8f014f09..db7f5247a 100644 --- a/crates/storage/src/backup_definition.rs +++ b/crates/storage/src/backup_definition.rs @@ -24,6 +24,7 @@ //! alter the meaning of backups already on disk. A reader that finds a version //! it does not know refuses the restore rather than guessing. +use extenddb_core::backup_format::SourceTableSchema; use extenddb_core::types::{ AttributeDefinition, BillingMode, CreateTableInput, GsiInput, KeySchemaElement, KeyType, LsiInput, OnDemandThroughput, ProvisionedThroughput, StreamSpecification, @@ -437,6 +438,75 @@ impl BackupRestoreSource { } } +/// What the engine answers a `RestoreTableFromBackup` with, beside the +/// target's description: the three members of the `RestoreSummary`, from a +/// legacy backup's [`BackupRestoreSource`] or a store-backed backup's +/// manifest. +#[derive(Debug, Clone, PartialEq)] +pub struct RestoreSource { + pub backup_arn: String, + pub source_table_arn: String, + /// Epoch seconds; reported as `RestoreSummary.RestoreDateTime`, as the + /// service does. + pub backup_creation_date_time: f64, +} + +impl From<&BackupRestoreSource> for RestoreSource { + fn from(source: &BackupRestoreSource) -> Self { + Self { + backup_arn: source.backup_arn.clone(), + source_table_arn: source.source_table_arn.clone(), + backup_creation_date_time: source.backup_creation_date_time, + } + } +} + +/// The restore target's `CreateTableInput` for a store-backed backup: the +/// manifest's `source_table` with the request's billing, throughput and SSE +/// overrides applied, by the rule [`BackupTableDefinition::apply_to`] applies +/// to a legacy definition (one rule for both paths: the GSI throughput +/// defaulting, the on-demand normalization, `TableClass` not restored). The +/// engine applies the index and on-demand overrides to the result and +/// validates it with `validate_create_table`. `vector_indexes` is the +/// manifest's list when it is non-empty (a target without vector support has +/// refused it before this is called), else `None`; the recorded SSE +/// description maps back to the specification the backends derive it from +/// (an `ENABLED` description was an enabled `KMS` specification). Streams, +/// TTL and tags are not restored, as the service does not. +#[must_use] +pub fn restore_input_from_manifest( + source: &SourceTableSchema, + target_table_name: &str, + overrides: &RestoreTableOverrides, +) -> CreateTableInput { + let mut input = CreateTableInput { + table_name: target_table_name.to_owned(), + key_schema: source.key_schema.clone(), + attribute_definitions: source.attribute_definitions.clone(), + ..Default::default() + }; + let definition = BackupTableDefinition { + version: BACKUP_DEFINITION_VERSION, + billing_mode: crate::backup_catalog::billing_mode_name(source.billing_mode).to_owned(), + provisioned_throughput: source.provisioned_throughput.clone(), + global_secondary_indexes: source.global_secondary_indexes.clone(), + local_secondary_indexes: source.local_secondary_indexes.clone(), + table_class: source.table_class.clone(), + sse_specification: source + .sse_description + .as_ref() + .filter(|sse| sse.status == "ENABLED") + .map(|_| serde_json::json!({"Enabled": true, "SSEType": "KMS"})), + on_demand_throughput: source.on_demand_throughput.clone(), + vector_index_names: Vec::new(), + time_to_live: None, + stream_specification: None, + }; + definition.apply_to(&mut input, overrides); + input.vector_indexes = source.vector_indexes.clone().filter(|v| !v.is_empty()); + input +} + /// What a backend is asked to do for one `RestoreTableFromBackup`, after the /// engine has resolved and validated it. #[derive(Debug, Clone, PartialEq)] diff --git a/crates/storage/src/lib.rs b/crates/storage/src/lib.rs index 79f8665a0..96bed67b0 100755 --- a/crates/storage/src/lib.rs +++ b/crates/storage/src/lib.rs @@ -23,6 +23,7 @@ pub mod migrations; pub mod operations; pub mod password; pub mod policy_migration; +pub mod restore_engine; pub mod secret_cipher; pub mod server_components; pub mod settings_store; @@ -41,7 +42,10 @@ pub use backend::{Backend, BackendAlreadySet, backend_name, set_backend, try_bac pub use backup_catalog::{ BackupCatalog, JobClaim, JobPermit, LegacyBackupReader, MaintenanceClaim, }; -pub use backup_definition::{BackupRestoreSource, RestoreRequest, RestoreTableOverrides}; +pub use backup_definition::{ + BackupRestoreSource, RestoreRequest, RestoreSource, RestoreTableOverrides, +}; +pub use restore_engine::{BulkLoadFile, Finished, RestoreEngine}; pub use snapshot::{SnapshotEngine, SnapshotInfo, TableSnapshot}; pub use transact::{IdempotencyKey, TransactGetOp, TransactWriteOp}; @@ -727,6 +731,7 @@ pub trait StorageEngine: + StreamEngine + BackupCatalog + LegacyBackupReader + + RestoreEngine + SnapshotEngine + WorkerStore + Send @@ -734,7 +739,7 @@ pub trait StorageEngine: { } -// Blanket implementation: any type implementing all 8 traits is a StorageEngine +// Blanket implementation: any type implementing all 9 traits is a StorageEngine impl StorageEngine for T where T: TableEngine + DataEngine @@ -742,6 +747,7 @@ impl StorageEngine for T where + StreamEngine + BackupCatalog + LegacyBackupReader + + RestoreEngine + SnapshotEngine + WorkerStore + Send diff --git a/crates/storage/src/migrations/registry.rs b/crates/storage/src/migrations/registry.rs index 5814e60aa..d6371ebe8 100644 --- a/crates/storage/src/migrations/registry.rs +++ b/crates/storage/src/migrations/registry.rs @@ -73,6 +73,15 @@ migration_ids! { BackupCatalogIndex = 6, "backup_catalog_index", "create the backup catalog index, the restore job fields, the maintenance claim \ record and the catalog id"; + /// Creates `restore_progress` in the PostgreSQL data database, where a + /// store-backed restore records each data file it applied; the other + /// backends keep that progress on the job record and have nothing to do. + /// Declared `Kind::Breaking` on PostgreSQL although a new table is + /// additive by CONTRIBUTING's step 4: declaring `Kind::Additive` needs + /// the old-binary job, which does not exist yet. + RestoreProgress = 7, "restore_progress", + "create the restore progress table of store-backed restores in the PostgreSQL data \ + database"; } /// Highest id that has shipped in a release. Bodies at or below it are frozen: diff --git a/crates/storage/src/restore_engine.rs b/crates/storage/src/restore_engine.rs new file mode 100644 index 000000000..650b5b534 --- /dev/null +++ b/crates/storage/src/restore_engine.rs @@ -0,0 +1,169 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore: how a backend takes the +//! items of one verified data file into a CREATING target, records how far +//! it got, flips the target ACTIVE, and counts what it holds. +//! +//! The engine (`crates/engine/src/backup/restore.rs`) owns the job: it +//! downloads and verifies every file, decodes and validates the items, and +//! hands them to [`RestoreEngine::bulk_load`] one file at a time. The +//! backend owns the writes: the base rows, the index rows and the vector +//! rows in its own batches and transactions, the progress record its +//! resume reads, and the ACTIVE flip conditioned on the claim. Progress is +//! recorded inside `bulk_load`, never by a separate call, because the +//! backends record it at different points (PostgreSQL per file in the data +//! database, SQLite, MongoDB and Cassandra per batch) and only the +//! write that applied the items knows when they are durable. + +use extenddb_core::types::Item; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::backup_catalog::{JobClaim, RestoreTargetKey}; +use crate::error::StorageError; + +/// Items per write batch of the SQLite and Cassandra `bulk_load` (SQLite's +/// write transaction, Cassandra's run of upserts before a progress write): +/// the count cap of the batch term of a restore's memory (the design's +/// section 7), and the unit other writers wait for on SQLite. +pub const RESTORE_BATCH_ITEMS: usize = 10_000; + +/// Bytes of serialized item text per write batch of the SQLite and Cassandra +/// `bulk_load`: a batch closes at the first item that takes its `item_data` +/// text past this, so the batch term of a restore's memory is bounded in +/// bytes as well as in items (10,000 items of the 400 KB limit would be +/// 3.9 GiB). Measured on the JSON text each backend writes as the row's +/// `item_data`, not on the service's `item_size_bytes`. +pub const RESTORE_BATCH_BYTES: usize = 32 * 1024 * 1024; + +/// Which data file a [`RestoreEngine::bulk_load`] call carries, and where in +/// it the items start. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BulkLoadFile { + /// The file's index in the manifest's `data_files`. + pub file_index: u32, + /// Items of this file already applied by an earlier run, which the + /// engine has skipped: the stream holds the rest. The backend records + /// `start_at` plus the items it applied as the file's running total. + pub start_at: u64, + /// The file is the one a resumed job may have applied in part without a + /// record of it (a backend that records progress per file and died + /// after some of the file's writes). A backend whose writes are plain + /// inserts may suppress duplicate-key failures for this file only; + /// elsewhere a duplicate key is the backup's and fails the file. + pub reapplying: bool, +} + +/// What [`RestoreEngine::finish_restore`] did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Finished { + /// The record is DONE and the table and every index are ACTIVE. + Active, + /// The record is DONE and the table waits for its index backfill; the + /// backend's worker flips it ACTIVE once every index has left CREATING + /// (MongoDB). + Pending, + /// The record was not RUNNING under the claim's generation, or the + /// table was not a CREATING restore target: nothing was written. Carries + /// the backend's account of what it found. + NotMatched(String), +} + +/// The data-plane operations of a store-backed restore, implemented by each +/// backend beside [`crate::backup_catalog::BackupCatalog`]. +pub trait RestoreEngine: Send + Sync { + /// Write one verified data file's items, from `file.start_at` on, into + /// the CREATING target: the base rows and the rows of every secondary + /// and vector index the target has, in the backend's own batches, and + /// the file's progress as the backend records it (the design's 5.3 step + /// 5), under `claim`'s generation where the backend's record carries it. + /// Returns the items it applied from the stream. + /// + /// A duplicate primary key, within the file or against rows an earlier + /// file wrote, fails the file (the backup holds a key twice), except as + /// [`BulkLoadFile::reapplying`] says. An `Err` item of the stream (a + /// decode or validation failure the engine found) fails the file with + /// that error, and nothing of a failed file is recorded as applied where + /// the backend writes the file in one transaction. + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result>; + + /// The end of a restore (the design's 5.3 step 6): the record RUNNING to + /// DONE and the table and every index CREATING to ACTIVE with + /// `item_count` and `size_bytes`, in one write where the backend can, + /// conditional on the claim's generation and on the record RUNNING and + /// the table CREATING ([`Finished::Active`]); or DONE with the ACTIVE + /// flip left to the backend's index backfill ([`Finished::Pending`]). + /// [`Finished::NotMatched`] when the record or the table was not the + /// claim's. A backend that keeps progress outside the record removes it + /// after the flip. + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result>; + + /// The items the target's base table holds: the count check every + /// restore ends with (equal to the manifest's total, else the backup + /// holds a key twice), and the emptiness check a resume with no recorded + /// progress makes. + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result>; + + /// Whether the target's data tables exist as the registration leaves + /// them: the reconciler's check before it resumes a dead owner's record. + /// Every backend commits the record before the target's data DDL + /// (PostgreSQL and SQLite create the data table after the catalog + /// transaction, MongoDB the collections and the index metadata after the + /// insert's commit, Cassandra the data table after its registration + /// LWT), so an owner killed in between leaves a RUNNING record over a + /// target with no table, or on MongoDB with its index metadata missing; + /// `false` names that record, which the reconciler fails and removes + /// instead of resuming. `Err` when `key` is not a CREATING restore + /// target with this table id, or the read failed. + fn restore_target_ready<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result>; + + /// The progress rows a backend keeps outside its job record, as + /// `(file_index, items_applied)` pairs: PostgreSQL's `restore_progress` + /// rows in the data database. Empty by default: the other backends + /// carry the progress on the record ([`crate::backup_catalog::RestoreJobRow`]). + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + let _ = key; + Box::pin(async { Ok(Vec::new()) }) + } + + /// Remove progress rows of targets not among `running` (the table ids + /// of RUNNING restore targets, read from the catalog): the reconciler's + /// sweep of rows a crash left behind where the progress lives apart + /// from the record. The candidates are a positive list: the backend + /// reads the distinct table ids the progress rows name first, then the + /// catalog's RUNNING records, and deletes only the rows of ids in the + /// first set that are in neither the second nor `running`; a target + /// whose first row lands after the first read is never a candidate, and + /// one registered before it is RUNNING at the second read, so a row of + /// a live job is never deleted. Returns the rows removed; `0` by + /// default. + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxFuture<'a, Result> { + let _ = running; + Box::pin(async { Ok(0) }) + } +} diff --git a/devtools/backup-kill-points.py b/devtools/backup-kill-points.py index fff70ed63..3480201c8 100755 --- a/devtools/backup-kill-points.py +++ b/devtools/backup-kill-points.py @@ -274,7 +274,7 @@ def main() -> None: # # kill-points.py restore \ # --bin [--items N] [--work DIR] [--second-reach] \ -# [--remove-target] [--vector] [--source-bin ] +# [--vector] [--source-bin ] # # `pg_terminate_backend` is not a pause point of its own: the server is # paused at `restore_after_file_commit`, the session holding the claim's @@ -302,12 +302,15 @@ def main() -> None: # the job logs its release), and the server is SIGKILLed when the job # reaches the point again. For `restore_mongodb_after_insert_many` and # `restore_cassandra_before_progress` that puts the kill after an earlier -# file's or batch's progress is recorded, so the resume runs the -# re-application path (MongoDB: the one unrecorded file re-applied with -# E11000 suppressed; Cassandra: at most one batch re-applied by the upsert) -# and the outcome asserted is ACTIVE with the exact count, where the -# first-reach kill (no progress recorded, items written) asserts -# `target_not_empty`. +# batch's progress is recorded, so the resume continues from that progress +# (MongoDB: each batch commits with its progress, nothing is re-applied; +# Cassandra: at most one batch re-applied by the upsert). The first-reach +# kill of `restore_cassandra_before_progress` leaves the first batch's items +# with no progress recorded; the resume finds the count at or below the +# backup's total and re-applies the first file with upserts (the engine's +# `target_not_empty` rule fails only a count above the total). Every kill +# point's outcome is therefore ACTIVE with the exact count, except the +# tampered-file point, whose target is removed. # # The services are local (PostgreSQL 55432, MongoDB 27117, Cassandra 9042). # Every server is stopped by the PID this script started, never by a @@ -851,58 +854,14 @@ def store_id() -> str: _say(f"backup {arn} adopted by the {target.backend} target (prefix copied, backup_index row seeded)") -def _remove_target(dep: Deployment, ddb, target: str, arn: str) -> None: - """The operator's remedy for a `target_not_empty` target: `extenddb backup - remove-target --account
    ` run beside the live server (MongoDB - and Cassandra take the maintenance claim; SQLite would need the server - stopped), its output asserted, then the target gone (`DescribeTable` - ResourceNotFound) and `DeleteBackup` of the backup allowed.""" - from botocore.exceptions import ClientError - - # DeleteTable's refusal shows the stuck state the command is for. - try: - ddb.delete_table(TableName=target) - raise AssertionError("DeleteTable of the target_not_empty target was accepted") - except ClientError as e: - assert e.response["Error"]["Code"] == "ResourceInUseException", e.response["Error"] - with_ = [dep.bin, "backup", "remove-target", "--account", ACCOUNT, target, "--config", dep.config] - r = subprocess.run(with_, cwd=dep.work, capture_output=True, text=True, timeout=300) - with open(os.path.join(dep.work, "remove-target.out"), "w") as f: - f.write(r.stdout + r.stderr) - assert r.returncode == 0, f"remove-target exited {r.returncode}: {r.stdout}{r.stderr}" - for needle in REMOVE_TARGET_LINES(target): - assert needle in r.stdout, f"remove-target output lacks {needle!r}:\n{r.stdout}{r.stderr}" - try: - ddb.describe_table(TableName=target) - raise AssertionError("the target is still described after remove-target") - except ClientError as e: - assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] - ddb.delete_backup(BackupArn=arn) - _say(f"{dep.backend}: remove-target removed {target}; DescribeTable ResourceNotFound, DeleteBackup allowed" - f" ({' | '.join(l for l in r.stdout.splitlines() if l.strip())[:300]})") - - -def REMOVE_TARGET_LINES(target: str) -> list[str]: - """The lines `backup remove-target` prints for a removed FAILED target.""" - return [ - "=== extenddb backup remove-target ===", - 'Taking the maintenance claim as "extenddb backup remove-target on ', - f"removed the restore target {target} (account {ACCOUNT}, table id ", - "was FAILED (target_not_empty)", - "The maintenance claim is released.", - ] - - def _run_restore_point(backend: str, point: str, binary: str, items: int, work: str, second_reach: bool = False, source_bin: str | None = None, - remove_target: bool = False, vector: bool = False) -> int: + vector: bool = False) -> int: """One restore kill point on one backend. Returns 0 on pass, 1 on a failed assertion, 2 on setup error. With `second_reach` the SIGKILL is taken at the point's second reach (module comment). On Cassandra the backup is taken on a SQLite source (`source_bin`) and adopted by the - target (module comment). With `remove_target` a first-reach run of the - two no-progress points goes on to `extenddb backup remove-target` and - asserts the target gone and the backup deletable. With `vector` the + target (module comment). With `vector` the source table carries a vector index and the tamper cell on PostgreSQL asserts the removed target's data table and vector relation gone.""" import boto3 # noqa: F401 @@ -1101,24 +1060,14 @@ def _run_restore_point(backend: str, point: str, binary: str, items: int, work: # removes the target, so give it more than one window. deadline = time.monotonic() + 240 - # These two points pause before the first file's/batch's progress is - # recorded, so the kill leaves inserted items with no recorded - # progress. On resume the engine correctly applies the no-progress - # safety rule (4(e)/5.3 step 7): a non-empty target with no progress - # is FAILED `target_not_empty` and left for the operator, never blindly - # re-applied. (The "re-apply the one unrecorded file/batch with - # suppression" path needs an EARLIER recorded file, which a one-shot - # pause on the first reach cannot set up; `--second-reach` runs it.) - if point in ("restore_mongodb_after_insert_many", "restore_cassandra_before_progress") and not second_reach: - line = _log_has(dep, "target_not_empty", timeout=120) - assert line, f"{point}: the resume did not apply the target_not_empty rule" - st = ddb.describe_table(TableName=target)["Table"]["TableStatus"] - assert st == "CREATING", f"{point}: target is {st}, expected it left CREATING for the operator" - _say(f"{backend}/{point}: resume found no progress and a non-empty target -> " - f"FAILED target_not_empty, target left for the operator ({line.strip()[-90:]})") - if remove_target: - _remove_target(dep, ddb, target, arn) - return 0 + # `restore_cassandra_before_progress` at its first reach pauses + # before the first batch's progress is recorded, so the kill leaves + # the batch's items with no recorded progress. The resume counts the + # target, finds no more items than the backup's total, and re-applies + # the first file with upserts (the engine fails `target_not_empty` + # only above the total), so the outcome below is ACTIVE with the exact + # count, as for every other point. (MongoDB commits each batch with + # its progress, so its first reach leaves nothing unrecorded.) want_gone = point == "restore_between_failed_and_removal" while True: @@ -1181,8 +1130,6 @@ def _restore_main(argv) -> None: p.add_argument("--second-reach", action="store_true", help="SIGKILL at the point's second reach (after an earlier progress record)") p.add_argument("--source-bin", help="a SQLite build that takes the backup a Cassandra target restores") - p.add_argument("--remove-target", action="store_true", - help="after a first-reach target_not_empty outcome, run `extenddb backup remove-target` and assert the target gone") p.add_argument("--vector", action="store_true", help="give the source table a vector index; restore_between_failed_and_removal on postgres then asserts the removed target's vector relation gone") a = p.parse_args(argv) @@ -1192,7 +1139,7 @@ def _restore_main(argv) -> None: raise SystemExit(f"{a.point} does not apply to {a.backend}") sys.exit(_run_restore_point(a.backend, a.point, os.path.abspath(a.bin), a.items, os.path.abspath(a.work), second_reach=a.second_reach, source_bin=a.source_bin, - remove_target=a.remove_target, vector=a.vector)) + vector=a.vector)) # Dispatch `kill-points.py restore ...` to the restore orchestrator, every diff --git a/devtools/backup-restore-matrix b/devtools/backup-restore-matrix new file mode 100755 index 000000000..589c52479 --- /dev/null +++ b/devtools/backup-restore-matrix @@ -0,0 +1,974 @@ +#!/usr/bin/env python3 +"""One cell of the cross-backend restore matrix, at the binary level. + +A cell proves that a backup taken on the source backend restores, through the +store, into the target backend: the restored table has the source's items, +key schema, secondary indexes, throughput, SSE and (where the target supports +them) vector indexes, and its class is STANDARD. The target's catalog has no +`backup_index` row for the source's backup until `backup sync --adopt-store` +runs (PR 10); the cell stands in for that command by copying the prefix under +a root the target owns and seeding the row at the storage level, so it proves +the restore and not the adoption. `matrix-nightly.yml` runs the twelve plain +cells and the duplicate-key and vector cells through this committed script. + + backup-restore-matrix --source --target + [--items N] [--vector] [--duplicate-key] [--work DIR] + [--bin-sqlite PATH] [--bin-postgres PATH] [--bin-mongodb PATH] [--bin-cassandra PATH] + +Starts a source server from the backend's binary with a filesystem store at +its own root, creates a composite-key table with a GSI and an LSI (and, with +--vector, a vector index), writes --items items carrying every attribute +type, takes a backup and waits for AVAILABLE. Starts the target server with +its own empty root (it writes its own extenddb-store.json), copies the +source's backup prefix under the target root, seeds the target's +backup_index row at the storage level with the target's store id (what +`backup sync --adopt-store` will do in PR 10; the cell proves the restore, +not the adoption), restores, waits for ACTIVE, compares the item set and the +description (key schema, indexes, throughput, SSE; the table class STANDARD), +queries each GSI, and tears both servers down by PID and drops their +databases. With --duplicate-key the copied prefix is replaced by a hand-built +unordered backup whose two data files share one key, and the cell passes +when the restore fails (the target gone, no table) instead. + +The two servers connect to PostgreSQL at 127.0.0.1:55432 (postgres/devpass), +MongoDB at 127.0.0.1:27117 and Cassandra at 127.0.0.1:9042, the coordinates +`matrix-nightly.yml` publishes its services on. Each server is an `init` with +its port patched, `[backup]` with short intervals and `serve --foreground`, +the test credentials provisioned through devtools/provision-test-credentials. +The MongoDB and Cassandra deployments use the backends' fixed names (the +`extenddb_catalog` and `extenddb_data` databases; the `extenddb_catalog` and +`extenddb_account_` keyspaces) and drop them in `init` and +`teardown`, so two cells with a MongoDB or Cassandra role, or a cell beside +another process on those names, destroy each other's catalogs: run such cells +one at a time, as the nightly does. The MongoDB and Cassandra roles need +`pymongo` and `cassandra-driver` (requirements-matrix.txt). + +Logs go under --work (default `$TMPDIR/backup-restore-matrix/`). +Exit 0 when the cell passes, 1 on an assertion failure or a setup error. +""" +from __future__ import annotations + +import argparse +import base64 +import gzip +import hashlib +import json +import os +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import time +import urllib.parse + +import boto3 +import requests +import urllib3 +from botocore.auth import SigV4Auth +from botocore.awsrequest import AWSRequest +from botocore.config import Config +from botocore.exceptions import ClientError + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +HERE = os.path.dirname(os.path.abspath(__file__)) +# The script lives at /devtools/backup-restore-matrix; the repo root +# is its parent. A caller overrides the binaries with --bin-. +REPO = os.path.abspath(os.path.join(HERE, "..")) +ACCOUNT = "123456789012" +REGION = "us-east-1" +ADMIN_PASSWORD = "TestPass1!" +PG = {"host": "127.0.0.1", "port": "55432", "user": "postgres", "password": "devpass"} +MONGODB_URI = "mongodb://127.0.0.1:27117/?directConnection=true" + +DEFAULT_BINS = { + "sqlite": os.path.join(REPO, "target", "rel-sqlite", "release", "extenddb"), + "postgres": os.path.join(REPO, "target", "rel-postgres", "release", "extenddb"), # or target/release + "mongodb": os.path.join(REPO, "target", "rel-mongodb", "release", "extenddb"), + "cassandra": os.path.join(REPO, "target", "rel-cassandra", "release", "extenddb"), +} + + +def say(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +class CellFailed(Exception): + pass + + +def check(cond: bool, msg: str) -> None: + if not cond: + raise CellFailed(msg) + + +def sqlite_module(): + """pysqlite3 when installed (an old sqlite3 library bundled with the + interpreter cannot read the catalog), else the standard module.""" + try: + import pysqlite3 as sqlite3 # type: ignore + except ImportError: + import sqlite3 # type: ignore + return sqlite3 + + +def free_port() -> int: + import socket + + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +# --- deployments ----------------------------------------------------------------- + + +class Deployment: + """One server: its binary, work directory, config, store root, PID and + the credentials provisioned on it.""" + + def __init__(self, backend: str, binary: str, work: str, role: str): + self.backend = backend + self.bin = binary + self.work = os.path.join(work, role) + self.role = role + os.makedirs(self.work, exist_ok=True) + self.config = os.path.join(self.work, "extenddb.toml") + self.root = os.path.join(self.work, "root") + self.port = free_port() + self.endpoint = f"https://127.0.0.1:{self.port}" + self.tag = f"mx{int(time.time()) % 1_000_000}{role[0]}" + self.proc: subprocess.Popen | None = None + self.creds: dict[str, str] = {} + + # -- init and config -- + + def init(self) -> None: + env = dict(os.environ, EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD) + os.makedirs(self.root, mode=0o700, exist_ok=True) + if self.backend == "mongodb": + # The MongoDB backend uses fixed database names + # (extenddb_catalog / extenddb_data), so a cell drops them before + # it inits and after it tears down, and a MongoDB source and a + # MongoDB target never run at once (the source stops before the + # target starts). + self.mongo_drop_databases() + with open(self.config, "w") as f: + f.write( + f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\n' + f'connection_string = "{MONGODB_URI}"\n' + ) + cmd = [self.bin, "init", "--backend", "mongodb", "--config", self.config, + "--overwrite"] + with open(os.path.join(self.work, "init.log"), "w") as log: + r = subprocess.run(cmd, cwd=self.work, env=env, stdout=log, stderr=subprocess.STDOUT) + if r.returncode != 0: + print(open(os.path.join(self.work, "init.log")).read()) + raise SystemExit(f"init of the {self.role} failed") + # `init` regenerates the config without the port, run_dir and + # backup sections the cell needs; write the whole config, as + # devtools/run-mongodb-tests does. + home = os.path.expanduser("~") + with open(self.config, "w") as f: + f.write( + f'[server]\nbind_addr = "127.0.0.1"\nport = {self.port}\n' + f'region = "{REGION}"\nrun_dir = "{self.work}/run"\n\n' + f'[server.tls]\ncert_path = "{home}/.extenddb/tls/cert.pem"\n' + f'key_path = "{home}/.extenddb/tls/key.pem"\n\n' + f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\n' + f'connection_string = "{MONGODB_URI}"\n\n' + f'[auth]\nprovider = "builtin"\n\n' + f'[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\n\n' + f'[backup.store]\nkind = "filesystem"\n' + f'[backup.store.filesystem]\npath = "{self.root}"\n' + ) + return + if self.backend == "sqlite": + cmd = [self.bin, "init", "--backend", "sqlite", "--sqlite-path", + os.path.join(self.work, "db.sqlite"), "--config", self.config] + elif self.backend == "postgres": + cmd = [self.bin, "init", "--pg-host", PG["host"], "--pg-port", PG["port"], + "--pg-user", PG["user"], "--pg-pass", PG["password"], + "--data-db", f"{self.tag}_data", "--catalog-db", f"{self.tag}_catalog", + "--extenddb-user", f"{self.tag}_app", "--config", self.config] + elif self.backend == "cassandra": + cmd = [self.bin, "init", "--backend", "cassandra", + "--cassandra-contact-points", "127.0.0.1:9042", + "--cassandra-user", "cassandra", "--cassandra-pass", "cassandra", + "--config", self.config] + else: + raise NotImplementedError(f"deployment of a {self.backend} server") + with open(os.path.join(self.work, "init.log"), "w") as log: + r = subprocess.run(cmd, cwd=self.work, env=env, stdout=log, stderr=subprocess.STDOUT) + if r.returncode != 0: + print(open(os.path.join(self.work, "init.log")).read()) + raise SystemExit(f"init of the {self.role} failed") + text = open(self.config).read() + text, n = re.subn(r"(?m)^[ \t]*#?[ \t]*port[ \t]*=[ \t]*\d+.*$", f"port = {self.port}", text, count=1) + assert n == 1, "no port line in the config" + text = re.sub(r'(?m)^run_dir = .*$', f'run_dir = "{self.work}/run"', text) + os.makedirs(self.root, mode=0o700, exist_ok=True) + text += ( + "\n[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\n" + f'\n[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{self.root}"\n' + ) + open(self.config, "w").write(text) + + def mongo_drop_databases(self) -> None: + """Drop the MongoDB backend's fixed catalog and data databases and wait + until each drop has completed. On a replica set `dropDatabase` is + two-phase: the database stays drop-pending until the drop is + majority-committed, a collection created meanwhile fails with + `DatabaseDropPending` (code 215) or vanishes with the database when + the drop lands, and a server started over a pending drop then answers + `ResourceNotFoundException` to writes into a table it reported + ACTIVE. The probe below creates a collection in each database until + the server accepts it, then drops the collection (a collection drop + leaves no pending database).""" + from pymongo import MongoClient + from pymongo.errors import OperationFailure + + client = MongoClient(MONGODB_URI) + try: + for name in ("extenddb_catalog", "extenddb_data"): + client.drop_database(name) + deadline = time.monotonic() + 120 + for name in ("extenddb_catalog", "extenddb_data"): + while True: + try: + client[name].create_collection("drop_probe") + except OperationFailure as e: + if e.code == 215 and time.monotonic() < deadline: + time.sleep(0.25) + continue + raise + client[name].drop_collection("drop_probe") + break + finally: + client.close() + + # -- the process -- + + def start(self, log_name: str = "serve.log", extra_env: dict | None = None) -> None: + env = dict(os.environ, **(extra_env or {})) + self.log = open(os.path.join(self.work, log_name), "a") + self.proc = subprocess.Popen( + [self.bin, "serve", "--foreground", "--config", self.config], + cwd=self.work, env=env, stdout=self.log, stderr=subprocess.STDOUT, + ) + for _ in range(300): + try: + if requests.get(f"{self.endpoint}/health", verify=False, timeout=2).status_code == 200: + break + except requests.RequestException: + pass + if self.proc.poll() is not None: + break + time.sleep(0.2) + else: + self.stop() + raise SystemExit(f"the {self.role} server did not start; see {self.log.name}") + if self.proc.poll() is not None: + raise SystemExit(f"the {self.role} server exited at start; see {self.log.name}") + say(f"{self.role} ({self.backend}) pid {self.proc.pid} on {self.port}") + + def stop(self) -> None: + """SIGTERM by PID, then SIGKILL if it outlives the drain.""" + if self.proc is None or self.proc.poll() is not None: + return + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=60) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait(timeout=10) + say(f"{self.role} pid {self.proc.pid} stopped (exit {self.proc.returncode})") + + def teardown(self) -> None: + self.stop() + if self.backend == "postgres": + psql(f"DROP DATABASE IF EXISTS {self.tag}_data WITH (FORCE)") + psql(f"DROP DATABASE IF EXISTS {self.tag}_catalog WITH (FORCE)") + psql(f"DROP ROLE IF EXISTS {self.tag}_app") + elif self.backend == "mongodb": + self.mongo_drop_databases() + elif self.backend == "cassandra": + # The Cassandra target server used the default keyspace prefix + # `extenddb`; drop its catalog and this account's data keyspace. + # The names are fixed, so another cell or process on the same + # node loses its catalog here (module docstring). + try: + from cassandra.auth import PlainTextAuthProvider + from cassandra.cluster import Cluster + + cluster = Cluster( + ["127.0.0.1"], port=9042, + auth_provider=PlainTextAuthProvider("cassandra", "cassandra"), + ) + session = cluster.connect() + for ks in ("extenddb_catalog", f"extenddb_account_{ACCOUNT}"): + session.execute(f"DROP KEYSPACE IF EXISTS {ks}") + cluster.shutdown() + except Exception as e: # noqa: BLE001 + print(f"cassandra teardown: {e}") + + # -- credentials and clients -- + + def provision(self) -> None: + env = dict(os.environ, EXTENDDB_TEST_ENDPOINT=self.endpoint, EXTENDDB_ADMIN_USER="admin", + EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD) + r = subprocess.run([sys.executable, os.path.join(REPO, "devtools", "provision-test-credentials")], + env=env, capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"provisioning on the {self.role} failed: {r.stderr}") + for line in r.stdout.splitlines(): + m = re.match(r"export (\w+)=(.*)", line.strip()) + if m: + self.creds[m.group(1)] = m.group(2) + for key, value in [("control_plane_delay_seconds", "0.05"), ("index_propagation_delay_ms", "0")]: + subprocess.run([self.bin, "settings", "--config", self.config, "set", key, value], + cwd=self.work, capture_output=True) + + def client(self): + return boto3.client( + "dynamodb", endpoint_url=self.endpoint, region_name=REGION, verify=False, + aws_access_key_id=self.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=self.creds["AWS_SECRET_ACCESS_KEY"], + config=Config(retries={"max_attempts": 10, "mode": "standard"}, read_timeout=120), + ) + + def signed_post(self, operation: str, body: dict) -> requests.Response: + """A hand-signed request for an operation botocore's model lacks + (CreateTable with VectorIndexes, DescribeTable's VectorIndexes, + SearchVectors).""" + body_bytes = json.dumps(body).encode() + headers = {"X-Amz-Target": f"DynamoDB_20120810.{operation}", + "Content-Type": "application/x-amz-json-1.0"} + frozen = boto3.Session( + aws_access_key_id=self.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=self.creds["AWS_SECRET_ACCESS_KEY"], + ).get_credentials().get_frozen_credentials() + req = AWSRequest(method="POST", url=self.endpoint, data=body_bytes, headers=headers) + SigV4Auth(frozen, "dynamodb", REGION).add_auth(req) + return requests.post(self.endpoint, data=body_bytes, headers=dict(req.headers), verify=False) + + def store_id(self) -> str: + return json.load(open(os.path.join(self.root, "extenddb-store.json")))["store_id"] + + +def psql(statement: str, database: str = "postgres", tuples: bool = False) -> str: + env = dict(os.environ, PGPASSWORD=PG["password"]) + cmd = ["psql", "-h", PG["host"], "-p", PG["port"], "-U", PG["user"], "-d", database, "-v", "ON_ERROR_STOP=1", "-q"] + if tuples: + cmd.append("-tA") + r = subprocess.run(cmd + ["-c", statement], env=env, capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"psql failed: {r.stderr}\n{statement}") + return r.stdout.strip() + + +# --- the table, the items, the backup -------------------------------------------- + +TABLE = "MatrixSource" +TARGET = "MatrixTarget" + + +def table_input(name: str, vector: bool) -> dict: + body = { + "TableName": name, + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"}, + ], + "BillingMode": "PROVISIONED", + "ProvisionedThroughput": {"ReadCapacityUnits": 7, "WriteCapacityUnits": 9}, + "GlobalSecondaryIndexes": [{ + "IndexName": "by-g", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "ALL"}, + "ProvisionedThroughput": {"ReadCapacityUnits": 3, "WriteCapacityUnits": 4}, + }], + "LocalSecondaryIndexes": [{ + "IndexName": "by-l", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "l", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "KEYS_ONLY"}, + }], + "SSESpecification": {"Enabled": True, "SSEType": "KMS"}, + } + if vector: + # Vector indexes are accepted on on-demand tables only. + body["BillingMode"] = "PAY_PER_REQUEST" + del body["ProvisionedThroughput"] + del body["GlobalSecondaryIndexes"][0]["ProvisionedThroughput"] + body["VectorIndexes"] = [{ + "IndexName": "by-emb", "Dimensions": 3, "DistanceFunction": "COSINE", + "VectorAttribute": {"AttributeName": "emb"}, "Projection": {"ProjectionType": "ALL"}, + }] + return body + + +def item(i: int, vector: bool) -> dict: + """Item `i` with every attribute type; every second item carries the GSI + key, every third the LSI key, every fifth the vector.""" + it = { + "pk": {"S": f"p{i % 11:03d}"}, + "sk": {"N": str(i)}, + "s": {"S": f"text {i} \u00e9\u2028"}, + "n": {"N": f"{i}.25"}, + "big": {"N": "12345678901234567890123456789012345678"}, + "b": {"B": bytes([i % 256, 0, 255, 7])}, + "ss": {"SS": [f"a{i}", "b", "c"]}, + "ns": {"NS": ["1", "2.5", str(-i)]}, + "bs": {"BS": [b"\x00\x01", bytes([i % 256])]}, + "bool": {"BOOL": i % 2 == 0}, + "null": {"NULL": True}, + "list": {"L": [{"S": "x"}, {"N": "1"}, {"M": {"k": {"BOOL": False}}}]}, + "map": {"M": {"nested": {"M": {"deep": {"L": [{"NULL": True}]}}}, "n": {"N": "3"}}}, + } + if i % 2 == 0: + it["g"] = {"S": f"g{i % 4}"} + if i % 3 == 0: + it["l"] = {"S": f"l{i:05d}"} + if vector and i % 5 == 0: + it["emb"] = {"L": [{"N": "0.1"}, {"N": str(0.01 * (i % 50))}, {"N": "0.3"}]} + return it + + +def create_and_fill(d: Deployment, items: int, vector: bool) -> None: + body = table_input(TABLE, vector) + r = d.signed_post("CreateTable", body) + check(r.status_code == 200, f"CreateTable on the source: {r.status_code} {r.text}") + c = d.client() + wait_active(d, TABLE) + reqs = [] + for i in range(items): + reqs.append({"PutRequest": {"Item": item(i, vector)}}) + if len(reqs) == 25 or i == items - 1: + pending = {TABLE: reqs} + while pending: + pending = c.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + reqs = [] + say(f"source: {items} items written") + + +def wait_active(d: Deployment, name: str, timeout: float = 600) -> dict: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + desc = d.signed_post("DescribeTable", {"TableName": name}) + if desc.status_code == 200: + table = desc.json()["Table"] + indexes = (table.get("GlobalSecondaryIndexes") or []) + (table.get("VectorIndexes") or []) + if table["TableStatus"] == "ACTIVE" and all(x.get("IndexStatus", "ACTIVE") == "ACTIVE" for x in indexes): + return table + time.sleep(0.2) + raise CellFailed(f"{name} did not become ACTIVE on the {d.role} within {timeout} s") + + +def take_backup(d: Deployment) -> tuple[str, dict]: + c = d.client() + arn = c.create_backup(TableName=TABLE, BackupName="matrix")["BackupDetails"]["BackupArn"] + deadline = time.monotonic() + 600 + while True: + desc = c.describe_backup(BackupArn=arn)["BackupDescription"] + status = desc["BackupDetails"]["BackupStatus"] + if status == "AVAILABLE": + say(f"source: backup {arn} AVAILABLE ({desc['BackupDetails']['BackupSizeBytes']} bytes)") + return arn, desc + check(status == "CREATING", f"the backup is {status}") + check(time.monotonic() < deadline, "the backup did not become AVAILABLE") + time.sleep(0.2) + + +def copy_prefix(source_prefix: str, target_prefix: str, target_root: str) -> None: + """Copy the backup's prefix under the target root, every directory + 0700 and every file 0600 (the store refuses a root others can read).""" + rel = os.path.relpath(os.path.dirname(target_prefix), target_root) + path = target_root + for part in rel.split(os.sep): + path = os.path.join(path, part) + if not os.path.isdir(path): + os.mkdir(path, 0o700) + os.chmod(path, 0o700) + shutil.copytree(source_prefix, target_prefix) + for dirpath, dirnames, filenames in os.walk(target_prefix): + os.chmod(dirpath, 0o700) + for name in filenames: + os.chmod(os.path.join(dirpath, name), 0o600) + + +def prefix_of(root: str, arn: str) -> str: + backup_id = arn.rsplit("/", 1)[1] + table = arn.split(":table/")[1].split("/")[0] + return os.path.join(root, ACCOUNT, table, backup_id) + + +# --- a hand-built unordered backup with one key in two files ----------------------- + + +def write_duplicate_key_prefix(prefix: str, manifest: dict, vector: bool) -> dict: + """Replace the data files under `prefix` with two unordered files that + share the key of item 0, and rewrite the manifest and its sidecar; every + file written here is 0600 and the data directory 0700, as `copy_prefix` + leaves the rest of the prefix.""" + data_dir = os.path.join(prefix, "data") + shutil.rmtree(data_dir) + os.makedirs(data_dir, mode=0o700) + os.chmod(data_dir, 0o700) + files = [] + total = 0 + for index, rows in enumerate([[item(i, vector) for i in range(0, 6)], [item(i, vector) for i in range(6, 12)] + [item(0, vector)]], start=1): + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":"), default=b64).encode() + b"\n" for r in rows) + import io + + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + compressed = buffer.getvalue() + key = f"data/{index:06d}.json.gz" + with open(os.path.join(prefix, key), "wb") as f: + f.write(compressed) + os.chmod(os.path.join(prefix, key), 0o600) + files.append({ + "key": key, "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": len(lines), "key_range": None, + }) + total += len(compressed) + manifest = dict(manifest) + manifest["item_order"] = "unordered" + manifest["data_files"] = files + manifest["backup_size_bytes"] = total + manifest["source_table"] = dict(manifest["source_table"], item_count=13) + text = json.dumps(manifest, separators=(",", ":")).encode() + with open(os.path.join(prefix, "extenddb-manifest.json"), "wb") as f: + f.write(text) + os.chmod(os.path.join(prefix, "extenddb-manifest.json"), 0o600) + with open(os.path.join(prefix, "extenddb-manifest.sha256"), "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(os.path.join(prefix, "extenddb-manifest.sha256"), 0o600) + write_ddb_manifests(prefix, files, total) + return manifest + + +def write_ddb_manifests(prefix: str, files: list[dict], size_bytes: int) -> None: + """Rewrite the two DynamoDB export manifests for the data files now under + `prefix`, so the prefix is a whole backup in the format's sense (one + `md5Checksum` and one `itemCount` per file, agreeing with the + extenddb manifest by construction): `manifest-files.json` is one line per + file, `manifest-summary.json` keeps the source's document with the new + item count and size.""" + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + with open(os.path.join(prefix, "manifest-files.json"), "w") as f: + for entry in files: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(os.path.join(prefix, "manifest-files.json"), 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in files) + summary["billedSizeBytes"] = size_bytes + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def b64(value): + if isinstance(value, bytes): + return base64.b64encode(value).decode() + raise TypeError(type(value)) + + +# --- the seed ------------------------------------------------------------------ + + +def seed_row(target: Deployment, arn: str, manifest: dict, size_bytes: int, item_count: int) -> None: + """Insert the target's `backup_index` row for the copied prefix, as + `backup sync` will (PR 10): AVAILABLE, the target's store identity, the + manifest's source table, `created_at` from the backup id.""" + backup_id = arn.rsplit("/", 1)[1] + table = arn.split(":table/")[1].split("/")[0] + created_ms = int(backup_id.split("-")[0]) + # `created_at` as the job writes it (the id's millisecond, six fraction + # digits on SQLite); `completed_at` and `heartbeat_at` are the seed's own + # instant, distinct from `created_at` as the job's AVAILABLE write and + # its last beat are. + created_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(created_ms / 1000)) + f".{created_ms % 1000:03d}000Z" + now_ms = int(time.time() * 1000) + now_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(now_ms / 1000)) + f".{now_ms % 1000:03d}Z" + store_root = os.path.realpath(target.root) + source_table = json.dumps(manifest["source_table"], separators=(",", ":")) + if target.backend == "postgres": + catalog = f"{target.tag}_catalog" + psql( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " + "source_table_details_json, owner_generation, heartbeat_at) VALUES " + f"('{ACCOUNT}', '{backup_id}', '{table}', '{arn}', 'matrix', 'AVAILABLE', 'filesystem', " + f"'{store_root}', '{target.store_id()}', {size_bytes}, {item_count}, " + f"to_timestamp({created_ms} / 1000.0), now(), '{source_table.replace(chr(39), chr(39) * 2)}'::jsonb, 0, now())", + database=catalog, + ) + elif target.backend == "sqlite": + conn = sqlite_module().connect(os.path.join(target.work, "db.sqlite"), timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " + "source_table_details_json, owner_generation, heartbeat_at) VALUES (?, ?, ?, ?, 'matrix', 'AVAILABLE', " + "'filesystem', ?, ?, ?, ?, ?, ?, ?, 0, ?)", + (ACCOUNT, backup_id, table, arn, store_root, target.store_id(), size_bytes, item_count, + created_iso, now_iso, source_table, now_iso), + ) + conn.commit() + finally: + conn.close() + elif target.backend == "mongodb": + import datetime + + from bson import Int64 + from pymongo import MongoClient + + created = datetime.datetime.fromtimestamp(created_ms / 1000, tz=datetime.timezone.utc) + now = datetime.datetime.fromtimestamp(now_ms / 1000, tz=datetime.timezone.utc) + client = MongoClient(MONGODB_URI) + try: + client["extenddb_catalog"]["backup_index"].insert_one({ + "account_id": ACCOUNT, + "backup_id": backup_id, + "table_name": table, + "backup_arn": arn, + "backup_name": "matrix", + "status": "AVAILABLE", + "store_kind": "filesystem", + "store_root": store_root, + "store_id": target.store_id(), + "size_bytes": Int64(size_bytes), + "item_count": Int64(item_count), + "created_at": created, + "completed_at": now, + "source_table_details_json": manifest["source_table"], + "owner_generation": Int64(0), + "heartbeat_at": now, + "lock_seq": Int64(0), + }) + finally: + client.close() + elif target.backend == "cassandra": + # The catalog-index row (`store_kind` set) in `backups_by_arn`, plus + # its `backup_index_by_status` entry; DescribeTable and the restore + # re-read the authoritative row, so the two list rows are optional. + import datetime + + from cassandra.auth import PlainTextAuthProvider + from cassandra.cluster import Cluster + + st = manifest["source_table"] + created_dt = datetime.datetime.utcfromtimestamp(created_ms / 1000) + tct = st.get("table_creation_date_time") + table_created_dt = ( + datetime.datetime.utcfromtimestamp(tct) if tct else created_dt + ) + now_dt = datetime.datetime.utcnow() + cluster = Cluster( + ["127.0.0.1"], port=9042, + auth_provider=PlainTextAuthProvider("cassandra", "cassandra"), + ) + session = cluster.connect() + try: + insert = session.prepare( + "INSERT INTO extenddb_catalog.backups_by_arn (account_id, backup_arn, backup_id, " + "backup_name, table_id, table_name, table_arn, backup_status, backup_type, " + "backup_size_bytes, item_count, key_schema, attribute_definitions, billing_mode, " + "table_created_at, created_at, store_kind, store_root, store_id, " + "source_table_details_json, owner_generation, heartbeat_at) " + "VALUES (?, ?, ?, 'matrix', ?, ?, ?, 'AVAILABLE', 'USER', ?, ?, ?, ?, ?, ?, ?, " + "'filesystem', ?, ?, ?, 0, ?)" + ) + session.execute( + insert, + ( + ACCOUNT, arn, backup_id, st["table_id"], table, st["table_arn"], + size_bytes, item_count, + json.dumps(st["key_schema"], separators=(",", ":")), + json.dumps(st["attribute_definitions"], separators=(",", ":")), + str(st.get("billing_mode", "PAY_PER_REQUEST")), + table_created_dt, created_dt, store_root, target.store_id(), + source_table, now_dt, + ), + ) + status_entry = session.prepare( + "INSERT INTO extenddb_catalog.backup_index_by_status (store_root, backup_status, " + "backup_arn, account_id, store_kind, store_id) " + "VALUES (?, 'AVAILABLE', ?, ?, 'filesystem', ?)" + ) + session.execute(status_entry, (store_root, arn, ACCOUNT, target.store_id())) + finally: + cluster.shutdown() + else: + raise NotImplementedError(f"seed_row for {target.backend}") + say(f"target: backup_index row seeded for {arn} (store {target.store_id()})") + + +# --- comparison --------------------------------------------------------------- + + +def normalize(it: dict) -> str: + def norm(v): + if isinstance(v, dict): + if len(v) == 1: + (k, inner), = v.items() + if k in ("SS", "NS"): + return {k: sorted(inner)} + if k == "BS": + return {k: sorted(base64.b64encode(x).decode() for x in inner)} + if k == "B": + return {k: base64.b64encode(inner).decode()} + if k == "N": + return {k: str(inner)} + return {k: norm(x) for k, x in v.items()} + if isinstance(v, list): + return [norm(x) for x in v] + return v + return json.dumps(norm(it), sort_keys=True) + + +def scan_all(d: Deployment, name: str, index: str | None = None) -> list[str]: + c = d.client() + out, kwargs = [], {"TableName": name} + if index: + kwargs["IndexName"] = index + while True: + page = c.scan(**kwargs) + out.extend(normalize(i) for i in page["Items"]) + if "LastEvaluatedKey" not in page: + return sorted(out) + kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"] + + +def compare_descriptions(source: dict, target: dict) -> None: + def keys(x): + return [(k["AttributeName"], k["KeyType"]) for k in x["KeySchema"]] + check(keys(source) == keys(target), "key schema differs") + check(sorted((a["AttributeName"], a["AttributeType"]) for a in source["AttributeDefinitions"]) + == sorted((a["AttributeName"], a["AttributeType"]) for a in target["AttributeDefinitions"]), "attribute definitions differ") + for kind in ("GlobalSecondaryIndexes", "LocalSecondaryIndexes"): + s = {g["IndexName"]: (keys(g), g["Projection"]["ProjectionType"]) for g in source.get(kind) or []} + t = {g["IndexName"]: (keys(g), g["Projection"]["ProjectionType"]) for g in target.get(kind) or []} + check(s == t, f"{kind} differ: {s} vs {t}") + def units(x): + pt = x.get("ProvisionedThroughput") or {} + return (pt.get("ReadCapacityUnits", 0), pt.get("WriteCapacityUnits", 0)) + for g in target.get("GlobalSecondaryIndexes") or []: + sg = next(x for x in source["GlobalSecondaryIndexes"] if x["IndexName"] == g["IndexName"]) + check(units(g) == units(sg), f"GSI {g['IndexName']} throughput differs: {units(g)} vs {units(sg)}") + check(units(source) == units(target), f"table throughput differs: {units(source)} vs {units(target)}") + check((source.get("BillingModeSummary") or {}).get("BillingMode", "PROVISIONED") + == (target.get("BillingModeSummary") or {}).get("BillingMode", "PROVISIONED"), "billing mode differs") + check((source.get("SSEDescription") or {}).get("Status") == (target.get("SSEDescription") or {}).get("Status"), "SSE differs") + check((target.get("TableClassSummary") or {}).get("TableClass", "STANDARD") == "STANDARD", "the target's table class is not STANDARD") + sv = {v["IndexName"]: (v["Dimensions"], v["DistanceFunction"]) for v in source.get("VectorIndexes") or []} + tv = {v["IndexName"]: (v["Dimensions"], v["DistanceFunction"]) for v in target.get("VectorIndexes") or []} + check(sv == tv, f"vector indexes differ: {sv} vs {tv}") + + +# --- the cell ------------------------------------------------------------------ + + +def run(args) -> int: + cell_tag = f"{args.source}-{args.target}" + ("-vector" if args.vector else "") + ("-dup" if args.duplicate_key else "") + work = args.work or os.path.join(tempfile.gettempdir(), "backup-restore-matrix", cell_tag) + work = os.path.abspath(work) + shutil.rmtree(work, ignore_errors=True) + os.makedirs(work) + bins = {b: os.path.abspath(getattr(args, f"bin_{b}") or DEFAULT_BINS[b]) for b in DEFAULT_BINS} + for role, backend in (("source", args.source), ("target", args.target)): + check(os.path.isfile(bins[backend]), f"no {backend} binary at {bins[backend]} for the {role}") + source = Deployment(args.source, bins[args.source], work, "source") + target = Deployment(args.target, bins[args.target], work, "target") + rc = 0 + try: + source.init() + source.start() + source.provision() + create_and_fill(source, args.items, args.vector) + source_desc = wait_active(source, TABLE) + arn, backup_desc = take_backup(source) + source_items = scan_all(source, TABLE) + source_prefix = prefix_of(source.root, arn) + manifest = json.load(open(os.path.join(source_prefix, "extenddb-manifest.json"))) + source.stop() + + target.init() + target.start() + target.provision() + check(os.path.isfile(os.path.join(target.root, "extenddb-store.json")), "the target wrote no identity file") + target_prefix = prefix_of(target.root, arn) + copy_prefix(source_prefix, target_prefix, target.root) + if args.duplicate_key: + manifest = write_duplicate_key_prefix(target_prefix, manifest, args.vector) + say("target: the prefix now holds an unordered backup with one key in two files") + seed_row(target, arn, manifest, manifest["backup_size_bytes"], manifest["source_table"]["item_count"]) + + c = target.client() + described = c.describe_backup(BackupArn=arn)["BackupDescription"] + check(described["BackupDetails"]["BackupStatus"] == "AVAILABLE", "the seeded row does not describe AVAILABLE") + started = time.monotonic() + + if args.vector and args.target == "mongodb": + # A manifest with vector indexes into a backend without vector + # support is refused before any table is created (section 1, + # 5.3 step 2). + try: + c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + err = e.response["Error"] + check(err.get("Code") == "ValidationException", + f"the refusal is a ValidationException: {err}") + check("vector" in err.get("Message", "").lower(), + f"the refusal names vector indexes: {err}") + probe = target.signed_post("DescribeTable", {"TableName": TARGET}) + check(probe.status_code == 400 and "ResourceNotFound" in probe.text, + f"a table was created despite the refusal: {probe.status_code} {probe.text}") + say(f"target: the vector manifest was refused before any table: {err['Message'][:160]}") + print("CELL PASSED (vector manifest refused)") + return 0 + raise CellFailed("a vector manifest into MongoDB was not refused") + + # A vector manifest is refused into a backend without vector support + # (Cassandra), synchronously and before any table is created. + if args.vector and args.target == "cassandra": + try: + c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + err = e.response["Error"] + check( + err.get("Code") == "ValidationException" and "vector index" in err.get("Message", ""), + f"the vector manifest was not refused with a vector ValidationException: {err}", + ) + probe = target.signed_post("DescribeTable", {"TableName": TARGET}) + check( + probe.status_code == 400 and "ResourceNotFound" in probe.text, + f"a table was created for a refused vector restore: {probe.status_code} {probe.text}", + ) + say("target: the vector manifest was refused before any table was created") + print("CELL PASSED (vector manifest refused)") + return 0 + raise CellFailed("the vector manifest into Cassandra was not refused") + + try: + resp = c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + raise CellFailed(f"RestoreTableFromBackup refused: {e.response['Error']}") from e + desc = resp["TableDescription"] + check(desc["TableStatus"] == "CREATING", f"the restore answered {desc['TableStatus']}") + summary = desc.get("RestoreSummary") or {} + check(summary.get("SourceBackupArn") == arn and summary.get("RestoreInProgress") is True, + f"the response carries no RestoreSummary: {summary}") + say(f"target: restore of {arn} into {TARGET} registered CREATING") + + if args.duplicate_key: + deadline = time.monotonic() + 600 + while time.monotonic() < deadline: + r = target.signed_post("DescribeTable", {"TableName": TARGET}) + if r.status_code == 400 and "ResourceNotFound" in r.text: + break + # Cassandra removes a failed restore target in two writes + # (CREATING to DELETING, then the drop), so DELETING is a + # transient state on the way out, not a reached ACTIVE. + check( + r.status_code != 200 + or r.json()["Table"]["TableStatus"] in ("CREATING", "DELETING"), + f"a backup with a duplicated key reached {r.json()['Table']['TableStatus']}", + ) + time.sleep(0.5) + else: + raise CellFailed("the target of the duplicated-key backup was not removed") + # The job logs its FAILED line just after it removes the target, + # so poll the log for a short window rather than reading it the + # instant the table disappears. + log_deadline = time.monotonic() + 15 + failed = [] + while time.monotonic() < log_deadline: + log = open(os.path.join(target.work, "serve.log")).read() + failed = [l for l in log.splitlines() if "FAILED" in l and TARGET in l] + if failed: + break + time.sleep(0.25) + check(bool(failed), "the server log does not record the FAILED restore") + check(any("holds a key twice" in l or "the manifest lists" in l for l in failed), + f"the failure names neither the duplicate key nor the count: {failed[-1][:300]}") + say(f"target: {failed[-1].split(': FAILED', 1)[-1][:200]}") + say(f"target: the duplicated-key restore failed and the target is gone after {time.monotonic() - started:.1f} s") + print("CELL PASSED (duplicate key refused)") + return 0 + + target_desc = wait_active(target, TARGET) + say(f"target: {TARGET} ACTIVE after {time.monotonic() - started:.1f} s") + check(target_desc.get("RestoreSummary") is None, "an ACTIVE table still reports a RestoreSummary") + target_items = scan_all(target, TARGET) + check(len(target_items) == args.items, f"the target holds {len(target_items)} items, the source wrote {args.items}") + check(target_items == source_items, "the item sets differ") + compare_descriptions(source_desc, target_desc) + for gsi in target_desc.get("GlobalSecondaryIndexes") or []: + name = gsi["IndexName"] + rows = c.query(TableName=TARGET, IndexName=name, + KeyConditionExpression="g = :g", ExpressionAttributeValues={":g": {"S": "g0"}})["Items"] + expected = sum(1 for i in range(args.items) if i % 2 == 0 and i % 4 == 0) + check(len(rows) == expected, f"Query on {name} returned {len(rows)} items, expected {expected}") + lsi_rows = c.query(TableName=TARGET, IndexName="by-l", + KeyConditionExpression="pk = :p", ExpressionAttributeValues={":p": {"S": "p000"}})["Items"] + expected_l = sum(1 for i in range(args.items) if i % 11 == 0 and i % 3 == 0) + check(len(lsi_rows) == expected_l, f"Query on by-l returned {len(lsi_rows)} items, expected {expected_l}") + if args.vector: + r = target.signed_post("SearchVectors", {"TableName": TARGET, "IndexName": "by-emb", + "SearchVector": [{"N": "0.1"}, {"N": "0.0"}, {"N": "0.3"}], + "TopK": 3}) + check(r.status_code == 200, f"SearchVectors on the target: {r.status_code} {r.text}") + results = r.json().get("SearchResults") or [] + check(len(results) == 3, f"SearchVectors returned {len(results)} results, expected 3") + # The index strips the vector from the projected item; the items it + # answers are the ones that carried one (every fifth). + keys = [int(x["Item"]["sk"]["N"]) for x in results] + check(all(k % 5 == 0 for k in keys), f"SearchVectors answered items that carried no vector: {keys}") + check(described["SourceTableDetails"]["ItemCount"] == args.items, "DescribeBackup's item count differs from the items written") + say("target: item set, description, GSI and LSI queries match the source") + print("CELL PASSED") + return 0 + except CellFailed as e: + print(f"CELL FAILED: {e}") + return 1 + finally: + for d in (target, source): + try: + d.teardown() + except Exception as e: # noqa: BLE001 + print(f"teardown of the {d.role}: {e}") + print(f"logs under {work}") + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--source", required=True, choices=["sqlite", "postgres", "mongodb"]) + p.add_argument("--target", required=True, choices=["sqlite", "postgres", "mongodb", "cassandra"]) + p.add_argument("--items", type=int, default=1200) + p.add_argument("--vector", action="store_true") + p.add_argument("--duplicate-key", action="store_true") + p.add_argument("--work") + for b in DEFAULT_BINS: + p.add_argument(f"--bin-{b}") + sys.exit(run(p.parse_args())) + + +if __name__ == "__main__": + main() diff --git a/devtools/run-mongodb-tests b/devtools/run-mongodb-tests index 8c0f5db9f..57eaba215 100755 --- a/devtools/run-mongodb-tests +++ b/devtools/run-mongodb-tests @@ -16,6 +16,13 @@ # --mongo-port PORT Host port to publish mongo on (default: 27021) # --output DIR Directory for logs and generated config # (default: /tmp/run-mongodb-tests-) +# --backup-store KIND filesystem (default): the server gets a filesystem +# [backup.store] under the output directory and the +# suites read EXTENDDB_TEST_BACKUP_STORE=filesystem; +# none: no [backup.store], the in-catalog backup path, +# EXTENDDB_TEST_BACKUP_STORE=none. The legacy LSI +# fallback cell of tests/test_mongodb_backup_restore.py +# runs only under none. # --keep Do not tear down the mongo container / server # on exit (useful for post-run inspection) # --mongo-only Start the replica set, print its URI, and exit without @@ -24,15 +31,12 @@ # crate's snapshot tests). Tear down with # docker rm -f extenddb-runtests-mongo-. # --backup-store-root DIR -# Configure a filesystem backup store at DIR (created -# 0700 if missing; a fresh directory per run, since the -# store's identity file names the catalog that owns it -# and this runner initializes a new catalog each time): -# CreateBackup is then the store-backed job, which the -# backup suites run with -# EXTENDDB_TEST_BACKUP_STORE=filesystem. Without it the -# server has no [backup.store] and CreateBackup is the -# in-catalog create (EXTENDDB_TEST_BACKUP_STORE=none). +# The filesystem store's root (default: backups/ under +# the output directory), created 0700 if missing; a +# fresh directory per run, since the store's identity +# file names the catalog that owns it and this runner +# initializes a new catalog each time. Implies +# --backup-store filesystem. # --rust-live Also run the Rust live tests of the storage-mongodb # crate against this runner's replica set. They are # `#[ignore]`d, because they need a replica set, and @@ -47,6 +51,7 @@ # devtools/run-mongodb-tests -- --pytest --filter test_put_item # devtools/run-mongodb-tests --keep -- --pytest --filter test_rfc0003 # devtools/run-mongodb-tests --rust-live -- --rust --rust-integration +# devtools/run-mongodb-tests --backup-store none -- --pytest --filter test_mongodb_backup_restore # # Prerequisites: # - Docker running @@ -71,6 +76,7 @@ KEEP=false MONGO_ONLY=false RUST_LIVE=false BACKUP_STORE_ROOT="" +BACKUP_STORE=filesystem RUN_TESTS_ARGS=() while [[ $# -gt 0 ]]; do @@ -78,12 +84,18 @@ while [[ $# -gt 0 ]]; do --port) BIND_PORT="$2"; shift 2 ;; --mongo-port) MONGO_PORT="$2"; shift 2 ;; --output) OUTPUT_DIR="$2"; shift 2 ;; + --backup-store) + case "$2" in + filesystem|none) BACKUP_STORE="$2" ;; + *) echo "error: --backup-store takes filesystem or none, not: $2" >&2; exit 1 ;; + esac + shift 2 ;; --keep) KEEP=true; shift ;; --mongo-only) MONGO_ONLY=true; KEEP=true; shift ;; --rust-live) RUST_LIVE=true; shift ;; - --backup-store-root) BACKUP_STORE_ROOT="$2"; shift 2 ;; + --backup-store-root) BACKUP_STORE_ROOT="$2"; BACKUP_STORE=filesystem; shift 2 ;; --) shift; RUN_TESTS_ARGS=("$@"); break ;; - -h|--help) sed -n '5,55p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -h|--help) sed -n '5,60p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "error: unknown option: $1" >&2; exit 1 ;; esac done @@ -146,7 +158,7 @@ cleanup() { if $KEEP; then echo "" echo "=== --keep set; leaving mongo + extenddb running ===" - echo " server log (startup): $SERVER_LOG" + echo " server log: $SERVER_LOG" echo " mongo container: $CONTAINER_NAME (port $MONGO_PORT)" echo " tear down manually: $BINARY stop --config $CONFIG; docker rm -f $CONTAINER_NAME" return @@ -233,7 +245,38 @@ EXTENDDB_ADMIN_PASSWORD="$ADMIN_PASSWORD" "$BINARY" init \ # `init` regenerates the config with default port 18443 and no # `[import]` / `[export]` sections. Rewrite with the port + defaults the # integration suite needs. This overwrite falls away once `extenddb init` -# grows flags for these knobs. +# grows flags for these knobs. With `--backup-store filesystem` (the +# default) the `[backup.store]` is a filesystem root under the output +# directory: CreateBackup is the store-backed job and RestoreTableFromBackup +# the store-backed restore, which the MongoDB restore cells +# (tests/test_mongodb_backup_restore.py) reach through the root +# (EXTENDDB_TEST_BACKUP_ROOT) and the server's log (EXTENDDB_TEST_SERVER_LOG); +# the suites read EXTENDDB_TEST_BACKUP_STORE=filesystem to expect that path. +# With `--backup-store none` the section is absent, CreateBackup writes the +# in-catalog format and the suites read EXTENDDB_TEST_BACKUP_STORE=none. +BACKUP_ROOT="${BACKUP_STORE_ROOT:-$OUTPUT_DIR/backups}" +STORE_SECTION="" +if [[ "$BACKUP_STORE" == "filesystem" ]]; then + if [[ -z "$BACKUP_STORE_ROOT" ]]; then + # A root left by an earlier run of a reused --output directory + # carries that run's `extenddb-store.json`, which names that run's + # catalog; the container is recreated every run, so the new catalog's + # id differs and the server would find the store "owned by another + # catalog". Start from an empty root. A root the caller named is + # theirs to clear: it is created and made owner-only, nothing more. + rm -rf "$BACKUP_ROOT" + fi + mkdir -p "$BACKUP_ROOT" + chmod 0700 "$BACKUP_ROOT" + echo " backup store: filesystem at $BACKUP_ROOT" + STORE_SECTION=" +[backup.store] +kind = \"filesystem\" + +[backup.store.filesystem] +path = \"$BACKUP_ROOT\" +" +fi cat > "$CONFIG" <> "$CONFIG" <"$SERVER_LOG" 2>&1 +# The server runs in the foreground in the background of this shell, with its +# PID file written so `extenddb stop` (the cleanup) can signal it, and its +# whole log in SERVER_LOG: the restore cells read the job's log lines there, +# where a daemonized server's would go to syslog. +"$BINARY" serve --foreground --write-pid-file --config "$CONFIG" >"$SERVER_LOG" 2>&1 & for i in $(seq 1 30); do if curl -sk "https://127.0.0.1:$BIND_PORT/health" >/dev/null 2>&1; then @@ -347,6 +379,18 @@ export EXTENDDB_TEST_MONGODB_CONTAINER="$CONTAINER_NAME" # physical index lifecycle after exercising the DynamoDB API. export EXTENDDB_TEST_MONGODB_CONTAINER="$CONTAINER_NAME" +# Where the server writes its backups, which run-tests requires for --pytest +# and the backup suites read to pick their branch (filesystem: the +# store-backed job and restore; none: the in-catalog format). The MongoDB +# restore cells rewrite a backup's prefix under the store's root and read the +# restore job's log lines from the server's log; without a store there is no +# root and those cells skip, while the legacy LSI fallback cell runs. +export EXTENDDB_TEST_BACKUP_STORE="$BACKUP_STORE" +if [[ "$BACKUP_STORE" == "filesystem" ]]; then + export EXTENDDB_TEST_BACKUP_ROOT="$BACKUP_ROOT" +fi +export EXTENDDB_TEST_SERVER_LOG="$SERVER_LOG" + # Some tests read EXTENDDB_CONFIG to invoke `extenddb settings` against # the running instance. Without this, they fall back to `./extenddb.toml` # in the repo root, which is a stale dev config that points at a @@ -399,6 +443,24 @@ if $RUST_LIVE; then echo " ✗ MongoDB backup engine live Rust tests failed" RC=1 fi + + + # The restore fence cells need the `test-hooks` feature (the GSI backfill + # gate `bulk_load` honours is compiled only under it), and no other job + # builds that feature, so the takeover fence has no CI coverage without + # this step. Run them one at a time against the same replica set. + echo "" + echo "=== MongoDB restore fence cells (storage-mongodb, test-hooks) ===" + if cargo test -p extenddb-storage-mongodb --lib --features test-hooks -- \ + --ignored --test-threads=1 \ + backup_engine::live_tests::a_takeover_during_bulk_load_fences_the_file_index_write \ + backup_engine::live_tests::live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_calls \ + backup_engine::live_tests::live_bulk_load_records_each_batch_and_a_resume_continues_from_items_applied; then + echo " ✓ MongoDB restore fence cells passed" + else + echo " ✗ MongoDB restore fence cells failed" + RC=1 + fi fi exit $RC diff --git a/docs/design/15-backup-restore-bench-pr9.md b/docs/design/15-backup-restore-bench-pr9.md new file mode 100644 index 000000000..24e190dd1 --- /dev/null +++ b/docs/design/15-backup-restore-bench-pr9.md @@ -0,0 +1,310 @@ +# PR 9 restore memory and duration record (2026-11-02; the index, 60,000-item, input-order and 3M-row runs added the same day) + +This is the record the design cites for the restore (section 7, revision item +30 of `15-backup-restore.md`) and the admin guide's `max_concurrent_jobs` row +quotes. The scripts (`bench-restore-rss.sh`, `kill-points.py`) and the raw +samples (`/proc//status` every 200 ms per run, the servers' logs) are +kept beside the PR outside the repository, on the measuring host; the figures +below are what they produced. The backup side's record is +`15-backup-restore-bench-pr8.md`. + +## Method + +The release binary (`cargo build --release -p extenddb --no-default-features +--features sqlite` and `--features postgres`, built from the PR's final tree) +running `serve --foreground` with a filesystem backup store on local disk. +Each run: a fresh deployment (`init`, the port patched, `[backup] +max_concurrent_jobs = 2`), the table loaded through the wire +(`kill-points.py create-table`: BatchWriteItem from 16 threads, composite +key, no GSI; or `put-worst` for the worst shape), `CreateBackup` and the wait +for `AVAILABLE`, then the server stopped and started again so the process's +high-water mark covers the restore alone; `RestoreTableFromBackup` into a new +table; `VmHWM` read right before the request (the idle figure) and once +`DescribeTable` reports `ACTIVE` (the peak), sampled every 200 ms in between; +"over idle" is the difference. The duration is from the request to `ACTIVE` +as the client saw it; the job's own line gives the same figure within 0.2 s. + +The worst shape is #394's worst-expansion item (`data_file.rs`'s line-cap +table): a list of 2,262 number sets whose members are `1.0E-130`, 409 KB as +the service sizes an item, 27.7 MB as a data-file line, about 34 MB as a +decoded `Item`; each `worst` table holds 1,000 small items and then `N` +such items. `big1200` is 1,200 items of a 350,000-byte string under one +hash key, written into one data file (`data_file_target_bytes` raised past +the table), the shape on which the SQLite batch was measured on the PR's first +build, before it had a byte bound. The 100k and 1M tables are 200-byte `S` pads under a +composite key and have no secondary index, and their data files are in key +order (the server's own backup of a table loaded through the wire is +`key_ordered`); every figure in the first table below is for that shape, a +table with secondary indexes restores in more memory on SQLite (the second +table), and an unordered backup of the same items restores in more memory and +time on SQLite (the input-order table). The 60,000-item runs are the same items +and key, no index: the shape the differences document's in-catalog restore +timings were measured on. + +Host: 16 cores (Xeon Platinum 8124M), 30 GiB, Linux 5.10 (amzn2int), glibc +2.26; PostgreSQL 16.15 in a container on loopback (55432); the SQLite the +binary bundles. What was not held constant: the host ran other work during +the runs (the four hooks builds during the first six runs, with the load +average between 8 and 13 on 16 cores; a MongoDB kill-point run and the +matrix cells beside the later ones, load 2 to 7), so the durations are upper +bounds and comparable only within one back-to-back pair; the peaks are the +process's own and do not depend on the host's load. Everything is on +loopback and on a filesystem store: nothing here says what a network database +or an S3 store's download does to the duration, and the spool's disk is not +in these figures. + +## The figures + +| backend | shape | data files | idle `VmHWM` | peak `VmHWM` | peak − idle | `ACTIVE` after | downloaded | +|---|---|---|---|---|---|---|---| +| SQLite | worst 12 | 4 | 22.1 MiB | 870.5 MiB | **848 MiB** | 17.7 s | 3,793,555 B | +| SQLite | worst 24 | 8 | 20.4 MiB | 869.2 MiB | **848 MiB** | 26.8 s | 7,567,821 B | +| SQLite | worst 48 | 16 | 22.1 MiB | 1,035.1 MiB | **1,013 MiB** | 57.0 s | 15,116,357 B | +| SQLite | worst 24, `MALLOC_ARENA_MAX=1` | 8 | 20.2 MiB | 412.9 MiB | **392 MiB** | 20.5 s | 7,567,821 B | +| PostgreSQL | worst 24 | 8 | 21.4 MiB | 799.8 MiB | **778 MiB** | 22.5 s | 7,567,821 B | +| PostgreSQL | worst 24, `MALLOC_ARENA_MAX=1` | 8 | 21.4 MiB | 448.8 MiB | **427 MiB** | 22.7 s | 7,567,821 B | +| SQLite | big1200 (1,200 × 350 KB, one file) | 1 | 20.8 MiB | 291.5 MiB | **270 MiB** | 3.1 s | 450,922 B | +| PostgreSQL | big1200 | 1 | 21.4 MiB | 177.8 MiB | **156 MiB** | 3.3 s | 450,922 B | +| SQLite | 100k × 200 B | 1 | 21.9 MiB | 60.2 MiB | **38 MiB** | 2.9 s | 366,631 B | +| SQLite | 1M × 200 B, key-ordered lines | 4 | 21.2 MiB | 110.4 MiB | **89 MiB** | 26.8 s | 3,017,864 B | +| PostgreSQL | 100k × 200 B | 1 | 21.4 MiB | 32.0 MiB | **10.7 MiB** | 1.1 s | 366,631 B | +| PostgreSQL | 1M × 200 B, key-ordered lines | 4 | 21.5 MiB | 37.8 MiB | **16.2 MiB** | 9.5 s | 3,017,864 B | + +The SQLite 1M run's high-water mark over time: 60 MiB at 2.7 s (the 100k +run's peak), rising to 87 MiB by 20.8 s as the load went on, then 110 MiB in +the last second, at the count and the flip. The growth during the load is the +write connection's page cache filling (`PRAGMA cache_size = -32000`, +`crates/storage-sqlite/src/sqlite_util.rs`: 31 MiB per connection, bounded by +the pool, as the backup record says of its 45 MiB); the final 23 MiB is the +`COUNT(*)` over the base table: the counting connection's page cache filling +with the table's b-tree pages, bounded by the same `cache_size` (32 MiB) and +measured at the same size at 3M rows (22 to 28 MiB, the input-order table +below), so it does not grow with the table past the cache. On PostgreSQL the 1M run's high-water mark reached 90% of its final +value within 3 s and the count added nothing visible. 1M − 100k: +51 MiB on +SQLite (+27 of them during the load), +5.5 MiB on PostgreSQL. Neither +materializes the table (1M items of 200 bytes are over 200 MB of items). + +## The terms the code bounds + +Per running restore, times `max_concurrent_jobs`: + +- the parsed manifest, for the whole job (one entry per data file); +- the decoder's line, at most `MAX_LINE_BYTES` (32 MiB, `data_file.rs`), and + the deserializer's intermediate value for it (the backup record's "one + row's parse", `r + 2d`: 109 MiB at the line cap); +- the decoded items of up to `LOAD_CHUNK_BYTES` (32 MiB, + `crates/engine/src/backup/restore/load.rs`) of data-file line text per + chunk, the chunk closing at the first item that takes it to the bound (so at + most the bound plus one line, 64 MiB of text at the cap) or at + `LOAD_CHUNK_ITEMS` (1,000) items, at four chunk positions: two in the + channel to `bulk_load`, the one being filled and the one being drained. The + bound counts the bytes the decoder read for the chunk's items, the unit the + backup side budgets in, not the service's `item_size_bytes` (which is 68 + times smaller for the worst shape and bounded nothing). `bulk_load` is one + data file at a time, so the chunks in flight never hold more than one file's + items: at most the source's `data_file_target_bytes` plus one line of text + (64 + 32 MiB at the defaults), a knob the restoring host does not set; +- the backend's write batch: on SQLite and Cassandra at most + `RESTORE_BATCH_BYTES` (32 MiB, `crates/storage/src/restore_engine.rs`) of + serialized item text plus one item, or `RESTORE_BATCH_ITEMS` (10,000) + items when they are smaller, held beside the decoded items of that text + (SQLite collects `(Item, item_json)`; Cassandra serializes once to measure + and again to write); on PostgreSQL at most `COPY_BATCH_BYTES` (4 MiB, + `crates/storage/src/backup_definition.rs`) of item text plus one item, or + `COPY_BATCH_ITEMS` (500) rows, as `Vec<(Item, String)>` (the item and its + JSON text, two copies), one transaction per file; on MongoDB 1,000 + documents per `insert_many`, bounded in bytes only by the data file + (`bulk_load` is per file), since the driver splits a call by the server's + message bound on its own; +- on SQLite, the `COUNT(*)` after the last file: the counting connection's + page cache (`PRAGMA cache_size = -32000`, 32 MiB), which the count fills + with the base table's b-tree pages and which stops at that setting whatever + the table holds (22 to 28 MiB measured at 1M and at 3M rows, the input-order + table below). + +None of the terms grows with the table: the chunk and batch terms grow with +the item, which `max_item_size_bytes` bounds, the line cap bounds the line, +and `cache_size` bounds the count's pages. These are the restore's own terms; a table with +secondary or vector indexes adds, on SQLite, the backend's own per-connection +page cache for the index b-trees (below), which is not a buffer of the +restore and which the batch bound does not cover. The decoded `Item` of the worst shape is about 1.2 times +its line, so at the line cap the chunk positions hold up to 4 × 64 MiB × 1.2 ≈ +307 MiB of decoded items when a file holds that much, the SQLite or Cassandra +batch up to 32 MiB of text plus about 77 MiB of decoded items, and the +PostgreSQL batch 4 MiB plus one 32 MiB item twice. The measured worst-shape +peaks are above those terms under glibc's default allocator for the reason +the backup record gives: the items' strings are allocated on one thread (the +decoder's blocking thread) and freed on another (the backend's write), and +the per-thread arenas keep that memory, so the high-water mark grows with the +number of large items a restore reads (848 MiB at 12 and 24 items, 1,013 MiB +at 48 on SQLite, with no plateau measured) and varies between runs. Under +`MALLOC_ARENA_MAX=1` the peak is the terms' and did not grow from 12 to 24 +items on the backup side; here it is 392 MiB (SQLite) and 427 MiB +(PostgreSQL) at 24 items. + +Before the bounds (the figures measured the same way on the PR's first +build, kept beside this record): SQLite worst 12, +24 and 48 at 616, 891 and 925 MiB, worst 24 under one arena 370 MiB; +PostgreSQL worst 24 at 861 MiB, under one arena 428 MiB; the 1,200 × 350 KB +table on SQLite 435 MiB over idle (`VmRSS`, sampled every 100 ms in the +process that had taken the backup) against 127 MiB on PostgreSQL, the whole +file's items in one batch. With the batch bounded at 32 MiB of text the same +table holds 270 MiB on SQLite (`VmHWM` from a fresh start, so not the same +figure's method; the one-file table of 420 MB of items now loads in batches +of about 95 items) and 156 MiB on PostgreSQL. The worst-shape SQLite figures +with the default arenas moved with the allocator between the two series (616 +to 848 at 12 items, 925 to 1,013 at 48), as the backup record's series did; +the one-arena figures, which measure the terms, are 370 to 392 MiB on SQLite +and 428 to 427 MiB on PostgreSQL. + +## Tables with secondary indexes + +The same 200-byte items under the same composite key with two GSIs +(`by-g0`, `by-g1`, hash key `g` with 97 values, range key `sk`, `ALL` +projection), measured the same way (a fresh start, `VmHWM`), after every GSI +reported `ACTIVE` and with each GSI's `Scan` count checked against the item +count (equal in every run). "head" is the build of the tree before the +spool-sweep order, the `finish_restore` re-read and the `remove-target` +command were added, "final" the build with them; nothing in the restore's +load changed between the two, and the runs differ as the allocator's runs do. + +| backend | shape | build | idle `VmHWM` | peak `VmHWM` | peak − idle | `ACTIVE` after | load average at start | +|---|---|---|---|---|---|---|---| +| SQLite | 100k × 200 B, two GSIs | head | 21.4 MiB | 164.8 MiB | **143 MiB** | 10.2 s | 2.9 | +| SQLite | 100k × 200 B, two GSIs | final | 21.3 MiB | 246.4 MiB | **225 MiB** | 12.3 s | 12.1 (release builds running) | +| SQLite | 100k × 200 B, two GSIs | final, repeated | 21.1 MiB | 164.1 MiB | **143 MiB** | 13.9 s | 1.8 | +| SQLite | 300k × 200 B, two GSIs | head | 22.0 MiB | 187.1 MiB | **165 MiB** | 51.7 s | 8.1 | +| SQLite | 300k × 200 B, two GSIs | final | 21.7 MiB | 193.5 MiB | **172 MiB** | 51.9 s | 6.1 | +| PostgreSQL | 100k × 200 B, two GSIs | head | 21.4 MiB | 50.8 MiB | **29.3 MiB** | 9.9 s | 8.4 | +| PostgreSQL | 100k × 200 B, two GSIs | final | 23.3 MiB | 46.2 MiB | **22.9 MiB** | 6.2 s | 4.6 | +| PostgreSQL | 300k × 200 B, two GSIs | head | 21.3 MiB | 52.1 MiB | **30.8 MiB** | 19.7 s | 4.8 | +| PostgreSQL | 300k × 200 B, two GSIs | final | 24.0 MiB | 57.6 MiB | **33.5 MiB** | 21.4 s | 9.7 | + +With two GSIs the SQLite restore holds three to six times the no-index +figure at 100k (143 to 225 MiB in three runs against 38 and 43 MiB) and +grows from 100k to 300k (143 to 165 and 172 MiB); PostgreSQL holds 23 to +34 MiB against 10.7 MiB and grows little. In the SQLite runs the load itself +climbs to about 105 to 125 MiB of `VmRSS` before the last batch and the final +`COUNT(*)` and flip add the rest (50 MiB in the 143 MiB runs, 120 MiB in the +225 MiB run). The term is SQLite's, not the restore's: the restore's own +buffer is the one batch of `(Item, item_json)` bounded by `RESTORE_BATCH_BYTES` +whatever the indexes are, and the extra resident memory is the write +connection's page cache (`PRAGMA cache_size = -32000`, 32 MiB per connection, +`crates/storage-sqlite/src/sqlite_util.rs`) holding the index tables' b-tree +pages, which a load of scattered index keys dirties across the whole tree, +plus glibc's per-arena free lists (the figure falls by about a third under +`MALLOC_ARENA_MAX=1`: 72 MiB against 119 MiB at 100k in another run on the +same shape). It is bounded by `cache_size` per connection and grows +sub-linearly with the table, not with the restore's buffers; a smaller batch +for an indexed table was built and measured (91.7 MiB against 92.7 MiB over +idle at 100k in that pair of runs: the cache persists across batches, so +smaller transactions do not lower the peak) and not kept. The published +no-index figures stand for the no-index shape; size a SQLite host that +restores indexed tables for the no-index figure plus about 32 MiB of page +cache per running restore, and read the run-to-run spread above as the +allocator's. + +## Input order and the count term + +The 1M figures above are for the server's own backup, whose data files are +in key order. The same items (200-byte `S` pads, composite key, no index) +as an unordered backup, the prefix built by hand with the hash keys cycling +(`pk = i % 1000`, the shape of every MongoDB-sourced backup and every +multi-part-key table), and once more in key order (`pk = i // 1000`, 1,000 +items per hash key), at 1M and at 3M rows, on two builds of this PR: +"committed", the pushed head's, and "final", the build after the last +corrections (the PostgreSQL vector-relation drop on a removed target, the +Cassandra progress re-read, the job's first beat before the reply; nothing in +the SQLite load changed between the two). Sampled every 50 ms. +"count + flip" is the resident set's growth in the restore's last stretch +(the peak less the `VmRSS` one second before `ACTIVE`), which is the final +`COUNT(*)` and the flip; the host's load average at the start is beside each +run (the hooks builds and other suites ran beside the SQLite runs). + +| backend | shape | build | peak − idle | count + flip | `ACTIVE` after | load | +|---|---|---|---|---|---|---| +| SQLite | 1M, key-ordered lines | final | **98.1 MiB** | 21.9 MiB | 56.3 s | 16.2 | +| SQLite | 1M, unordered lines | final | **163.5 MiB** | 13.6 MiB | 90.3 s | 18.0 | +| SQLite | 1M, unordered lines | committed | **144.4 MiB** | 13.4 MiB | 66.2 s | 3.1 | +| SQLite | 3M, key-ordered lines | final | **109.3 MiB** | 27.6 MiB | 102.7 s | 19.1 | +| SQLite | 3M, key-ordered lines | committed | **116.4 MiB** | 22.9 MiB | 84.9 s | 4.5 | +| SQLite | 3M, unordered lines | final | **173.5 MiB** | 12.8 MiB | 264.9 s | 10.9 | +| SQLite | 3M, unordered lines | committed | **170.8 MiB** | 14.4 MiB | 274.7 s | 6.0 | +| PostgreSQL | 1M, unordered lines | final | **16.4 MiB** | 0.1 MiB | 17.5 s | 5.6 | + +Two things the table shows. The input order is a term of SQLite's, not of +the restore's: an unordered backup of 1M items holds 1.5 to 1.8 times the +key-ordered figure (144 to 164 MiB against 87 to 98) and takes two to three +times as long, because scattered inserts dirty pages across the whole base +b-tree where ordered inserts append to its right edge, and the write +connection's page cache (`cache_size`, 32 MiB) plus the allocator's arenas +hold the difference; the restore's own buffers (the chunk and the batch) are +the same in both runs, and PostgreSQL does not depend on the order (16.2 MiB +ordered, 16.4 unordered). The ordered 1M run on this binary reproduced the +first table's figure (98 against 89 MiB over idle; 86.9 MiB in another run +of the same shape on the committed build). And the count's term does not grow +with the table: 22 to 28 MiB at 1M and at 3M rows in key order, 13 to 14 MiB +unordered at both sizes, every figure under the counting connection's +`cache_size` (32 MiB), which is what bounds it; from 1M to 3M rows the whole +peak moved by +11 MiB key-ordered and +7 to +10 MiB unordered while the +table tripled. The durations are upper bounds: the SQLite runs on the final +build ran under a load average of 11 to 19 (the ordered 1M run took 56 s +against 26.8 s on the quieter host of the first table). + +## The 60,000-item shape, store-backed + +The differences document's "How `RestoreTableFromBackup` runs" row quotes +response and `ACTIVE` times measured on an earlier build's in-catalog restore +of a 60,000-item table with no secondary index. The same shape (60,000 items +of 200-byte `S` pads under a composite key, no index) restored from a +filesystem store on the final binaries, the host's load average 5.4 to 7.6 +with other runs beside it: + +| backend | response | `ACTIVE` after | peak − idle | data files / bytes | +|---|---|---|---|---| +| SQLite | 0.016 s | 1.85 s | 38.3 MiB | 1 / 360,532 | +| PostgreSQL | 0.051 s | 0.67 s | 9.3 MiB | 1 / 360,532 | +| MongoDB | 0.055 s | 3.74 s (the job's DONE at 1.8 s; the backfill worker's flip after) | 30.1 MiB | 1 / 360,532 | + +Cassandra was not measured on this shape (its own CreateBackup is in-catalog, +so a store-backed Cassandra restore needs a backup taken elsewhere and +adopted, which the kill-point and matrix runs do at 12,000 items). The +in-catalog figures the differences document quotes (response 0.032, 0.012, +0.053, 0.151 s; `ACTIVE` after 1.5, 2.1, 3.6, 11.2 s on PostgreSQL, SQLite, +MongoDB, Cassandra) were measured on another build and host and are not +comparable run for run; the store-backed response includes the sidecar and +manifest reads, and its `ACTIVE` time the download and the per-file +verification. + +## The sizing sentence + +Size a host for about 450 MiB per running restore of a table of large items +under `MALLOC_ARENA_MAX=1` on SQLite or PostgreSQL (392 and 427 MiB measured +at 24 items of 27.7 MB), times `max_concurrent_jobs`; under the default +arenas the peak grows with the number of large items a restore reads and no +bound was measured (1,013 MiB over idle at 48 such items on SQLite is the +largest seen; 778 MiB at 24 on PostgreSQL). A restore of 200-byte items holds +under 100 MiB on SQLite and under 20 MiB on PostgreSQL at 1M items when the +table has no secondary index and the backup's lines are in key order (the +server's own backups of hash-only and single-range-key tables); an unordered +backup of the same items (every MongoDB-sourced backup and every +multi-part-key table) held 144 to 164 MiB over idle on SQLite at 1M and 171 to +174 MiB at 3M (the input-order section), PostgreSQL 16 MiB either way; with two GSIs SQLite held 143 to 225 MiB at +100k and 165 to 172 MiB at 300k (the section above), PostgreSQL 23 to 34 MiB. A +restore is not "much less" than a backup of the same table: the backup +record's worst-shape figures are 466 to 1,281 MiB under the default arenas +and 452 to 481 MiB under one arena. + +## Durations (the RTO baseline the testing summary names) + +On this host, loopback, filesystem store: 1M items of 200 bytes in key order +restore in 9.5 s on PostgreSQL and 26.8 s on SQLite (100k: 1.1 s and 2.9 s; +60k: 0.67 s and 1.85 s, and 3.74 s on MongoDB); the same 1M items as an +unordered backup in 17.5 s on PostgreSQL and 66 to 90 s on SQLite, and 3M +items on SQLite in 85 to 103 s key-ordered and 265 to 275 s unordered (the +input-order section, on a host whose load average ran between 4 and 19); with two GSIs 100k in 6.2 to 9.9 s on +PostgreSQL and 10.2 to 13.9 s on SQLite and 300k in 19.7 to 21.4 s and 51.7 +to 51.9 s, on a host whose load average ran between 1.8 and 12; 24 items of +27.7 MB in 22.5 s on PostgreSQL and 26.8 s on SQLite; 1,200 items of 350 KB in +3.3 s and 3.1 s. The backup side's 1M figures are in the PR 8 record (36 s on +PostgreSQL under a loaded host, 12.5 s on SQLite). diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 51f66ecd8..d7631bcd5 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -61,9 +61,9 @@ A data file line (one `{"Item": ...}` object and its newline) is at most 32 MiB `backup_creation_date_time` is the instant `open_table_snapshot` returned, read just after the snapshot pinned, and it is the instant encoded in the backup id: CreateBackup opens the snapshot first (after any wait behind an in-flight UpdateTable), then mints the id from that millisecond (5.1 step 2), so the ARN, the manifest and the catalog's `created_at` carry one time. -A restore rejects a backup whose data files contain the same primary key twice, across files as well as within one. For key-ordered backups the manifest's `key_range` check rejects overlap before any file is downloaded and the per-file key-range check proves the rest, on every target. For unordered backups a plain `INSERT` on PostgreSQL and SQLite fails on a duplicate and so does MongoDB's `_id` unique index (E11000), but a Cassandra write is an upsert and detects nothing, so the general rule is a count: after the last file the restore counts the target's items and requires the count to equal the sum of the manifest's `item_count`; a shortfall is a duplicate and fails the restore. +A restore rejects a backup whose data files contain the same primary key twice, across files as well as within one. For key-ordered backups the manifest's `key_range` check rejects overlap before any file is downloaded and the per-file key-range check proves the rest, on every target. For unordered backups a plain `INSERT` on PostgreSQL and SQLite fails on a duplicate and so does MongoDB's `_id` unique index (E11000, suppressed only on the one unrecorded file a resume re-applies, 5.3 step 5), but a Cassandra write is an upsert and detects nothing, so the general rule is a count: after the last file of every backup, key-ordered or not, the restore counts the target's items (`count_restore_target`, section 3) and requires the count to equal the sum of the manifest's `item_count`; a shortfall is a duplicate and fails the restore. -- Numbers are DynamoDB JSON strings, so the 38-digit precision survives the file. A restore into MongoDB of an `N` sort-key value beyond Decimal128's 34 significant digits fails with ValidationException and the restore reports FAILED (see 5.3); it never rounds. Other `N` values are stored as strings there and keep their 38 digits. +- Numbers are DynamoDB JSON strings, so the 38-digit precision survives the file. A restore into MongoDB of an `N` sort-key value beyond Decimal128's 34 significant digits fails in `bulk_load` with the write path's ValidationException text, which becomes the restore's FAILED reason (5.3 step 7: no wire exception, the client sees the target disappear); it never rounds. Other `N` values are stored as strings there and keep their 38 digits, so a 38-digit non-key number restores intact. ### 2. Backup store abstraction @@ -82,7 +82,7 @@ Three store facts that rules of sections 4 and 5 rest on, and the one method a l - Cassandra: `open_table_snapshot` returns `Unsupported`. Cassandra has no `SnapshotEngine`, so the engine routes CreateBackup there to the legacy create (`LegacyBackupReader::create`, the #384 in-catalog path) whether or not `[backup.store]` is configured, the path every backend takes while no store is configured (section 7); the off-host path is never taken there. That stays until PR 14, by which time either a token-range `SnapshotEngine` with a documented non-point-in-time `SnapshotInfo.kind` (`cassandra_token_scan`) has been accepted as a design change to this section, or Cassandra backups are documented as unavailable. RFC-0003 (backend acceptance criteria) has no backup criterion today; the criterion this trait implies ("a backend that cannot fix an instant across a table returns `Unsupported` rather than a best-effort stream") is to be proposed there in a separate docs change, not in PR 5. The interfaces later PRs add to `extenddb-storage`, each with its contract; the signatures and the row keys are in the section of the PR that adds them (Phasing): -- `bulk_load(target_table_id, claim, file_index, start_at, items) -> Result`, a new `StorageEngine` method each backend owns (PR 9a), writes the items of one verified data file from item `start_at` on, and their index rows, into a CREATING restore target, records progress as 5.3 step 5 specifies for the backend under the job's `claim` (so a progress write conditioned on the generation has it) and returns how many items it applied. There is one spelling of that signature; 5.3 step 5 calls it. +- `RestoreEngine` (PR 9a), a trait in `StorageEngine`'s bound (`StorageEngine` has no methods of its own) that each backend implements: `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> Result` writes the items of one verified data file from item `start_at` on, and their index rows, into a CREATING restore target, records progress in the write that applied the items as 5.3 step 5 specifies for the backend (the job's `claim` carries the generation the record's writes are conditioned on) and returns how many items it applied; `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }` is the ACTIVE flip of 5.3 step 6; `count_restore_target(target)` is the count of section 1; `read_restore_progress` and `sweep_restore_progress(running)` read and sweep PostgreSQL's `restore_progress` rows (defaults elsewhere). There is one spelling of each signature, in PR 9a's section; 5.3 step 5 calls `bulk_load`. - The engine is backend-agnostic (main's `crates/engine` reaches every backend through the `extenddb_storage` traits), and nothing in sections 4 and 5 that touches a catalog row (the claim, the heartbeat, the conditional status writes, the store-scoped listings, the restore job record) can be written in it. `BackupCatalog`, a trait in `extenddb-storage` that PR 6 adds and each backend implements, is how the engine reaches the catalog index of section 4; PRs 8, 9a and 10 add methods to the same trait as their sections need them. Section 4's "Authority" line names this trait beside the engine module that binds it. - `LegacyBackupReader` is a trait in `extenddb-storage` with `describe`, `list`, `restore` and `delete`, plus `create`, which every backend keeps until PR 14: the engine calls it for CreateBackup when no `[backup.store]` is configured (section 7), logging a deprecation warning that names `[backup.store]` on each call, and on Cassandra always, since it has no `SnapshotEngine` (the decision before PR 14 settles whether that stays), and a legacy create takes a `max_concurrent_jobs` permit like any job, returning `LimitExceededException` when none is free (5.1 step 1), since PR 6 removes the slot semaphores #384's creates wait on; each backend implements it with a frozen copy of its #384 code for reading and decoding the legacy shape (not calls through trait methods later PRs remove), and the engine holds `Option>`. PR 14 removes the trait. @@ -92,13 +92,13 @@ The invariants and state machines every later PR honours, by lettered part; the (a) The backup row. The engine keeps a catalog row per backup, in `backup_index` (PR 6's schema), as the fast index for Describe and List, and as the state machine record; `status` is CREATING, AVAILABLE, DELETING, DELETED or FAILED. Status mapping to the API: CREATING and AVAILABLE as themselves; DELETING is reported as DELETED by DescribeBackup and DeleteBackup (a client retrying DeleteBackup sees the terminal status) and is not listed; DELETED and FAILED rows are not described (BackupNotFoundException, which is what the live service returns for a deleted backup) and not listed. FAILED and DELETED rows are tombstones the reconciler owns. Transitions and who performs them: CreateBackup inserts the row CREATING under the job's claim (5.1 step 2); the job writes AVAILABLE after the manifest, or FAILED, each conditional on its generation and on `status = CREATING` (5.1 steps 4, 5); the reconciler resolves a CREATING row whose owner is dead to AVAILABLE if a complete, verified manifest exists in the configured store, else to FAILED, under the same condition (5.1 step 6); DeleteBackup moves an AVAILABLE row to DELETING under the row's lock and to DELETED once the prefix lists empty (5.2); the reconciler finishes a DELETING row, deletes the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs` and of a DELETED row that still lists objects, and removes a FAILED or DELETED row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs`. -(b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, per-batch progress, `failure_reason` and `completed_at` (PR 6's schema). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `restore_jobs_total{outcome="failed"}`. +(b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, its progress (`file_index` and `items_applied`: per batch on SQLite and Cassandra, `file_index` per file on MongoDB; PostgreSQL records progress in `restore_progress` rows of the data database instead, 5.3 step 5), `failure_reason` and `completed_at` (PR 6's schema; `RestoreJobRow` carries every member). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `RestoreJobs{failed}` (section 9). (c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10), and the previous owner then finds the store unhealthy and stops acting on it. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. The probe needs an AVAILABLE row, and until one exists the guard has a gap: two instances with host-local roots at one path and a copied `extenddb-store.json` (one id in both files, so the identity check passes on each) are not told apart, the dead-owner duty of one fails a killed instance's complete backup under the other directory (no manifest in its own) and the killed instance's restart deletes that backup's prefix, the only copy (measured on PostgreSQL, PR 8's review; revision item 32). The rule is stated in the admin guide: one shared root from the first backup on, and the identity file is never copied between roots. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. (d) Identities. The ARN encodes partition, region, account, table, and backup id, and the key prefix is derived from account, table and id deterministically by `BackupArn::store_prefix`, under the configured store's prefix, and no row stores it, so restore, sync, purge and the reconciler derive it the same way and `describe_backup(arn)` works from the store alone after a sync. The engine, the writer and sync all parse and format ARNs through one implementation, `BackupArn` in `extenddb_core::backup_format` (#394); the engine keeps #384's region handling: it resolves a backup under the ARN as written and under the ARN re-spelled in its own region, and refuses a partition other than its own. `backup_creation_date_time` and `created_at` are the id's instant, so the ARN, the manifest, the API's reported time and the ListBackups cursor agree by construction. One row is addressed by `BackupKey` (`account_id`, `backup_id`, `backup_arn`): PostgreSQL, SQLite and MongoDB key `backup_index` by (`account_id`, `backup_id`), Cassandra keys `backups_by_arn` by (`account_id`, `backup_arn`); no method takes a bare backup id, and the engine treats a row whose `account_id`, `table_name` or `backup_arn` (after that re-spelling) differs from the request's as absent. One target is addressed by `RestoreTargetKey` (`account_id`, `table_name`, `table_id`), the table id guarding against a table re-created under the name. -(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. +(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING backup row and every RUNNING store-backed restore record at startup (the restart ages every RUNNING restore's heartbeat to the epoch), and at runtime a row or record whose in-process slot no task holds (on SQLite the store-backed restore's claim holds the `restore:/
    ` slot from before the target exists to the task's end; the legacy restore holds no slot, and a legacy and a store-backed restore into one name exclude each other through the name's `tables` row under the write transaction (`BEGIN IMMEDIATE`: the CREATING-target check, then the unique name)); on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A dead RUNNING store-backed restore record is taken over through `claim_restore_dead(key, stale_after, spool_host)` under the same rules (PostgreSQL's name lock and SQLite's slot refuse a taker outright; MongoDB and Cassandra condition on the stale heartbeat, Cassandra's lightweight transaction also on the heartbeat it read and on `kind = manifest`), which writes the claimant's `spool_host` with the fresh generation and never takes over a legacy record; the successor resumes the restore from the recorded progress (5.3 step 7). A restore record's takeover differs from a backup row's in one write: `claim_restore_dead` sets `heartbeat_at` to the database's clock in the same conditional write as the generation and `spool_host` (PostgreSQL `now()`, SQLite `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, MongoDB `$$NOW`, Cassandra `toTimestamp(now())`), so a second instance's reconciler, whose cycle lands before the successor's first beat (one `heartbeat_interval_secs` after the takeover), finds the record live and is refused. The backup takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale window counts from the owner's last beat; nothing waits on a restore record's old heartbeat after a takeover (the FAILED-target duty ages on `COALESCE(completed_at, heartbeat_at)` and every FAILED write of a resume stamps `completed_at`; the maintenance claim waits on RUNNING, not on the beat). Per backend, the second taker is refused by: PostgreSQL the successor's advisory lock on the target name, then the fresh heartbeat; SQLite the successor's in-process slot (one process serves the file), and the record the startup step had aged is no longer listed as dead once the takeover refreshed its heartbeat; MongoDB the `$expr` staleness term of the takeover's own filter against the refreshed `heartbeat_at`; Cassandra the takeover's serial read, which sees the refreshed `restore_heartbeat_at` and stops before its lightweight transaction. So the two-taker argument of the backup takeover is, for a restore, "a second taker finds the record live"; the belt under it is that every progress write is fenced by the generation (a first taker's next write would match nothing and its job end with no table access) and that a resume's first file is `reapplying`, which covers a partial file on MongoDB and which Cassandra's upserts collapse, so a later change to either mechanism (a progress write without the generation, `reapplying` on a later file) is seen to matter here. The takeover on Cassandra also re-applies the target's `active_restores[table_id] = name` entry on the backup's row when it is absent (one serial read, then `UPDATE ... IF backup_status = 'AVAILABLE'`): the registration writes the target's row and its DDL before the entry, so an owner that died between the two left a resumable record whose backup nothing held in use, and `DeleteBackup` would have succeeded under the resumed load. A backup no longer AVAILABLE keeps no entry; the resumed load fails on the missing prefix and the target is removed. The engine's first confirmed beat of a fresh restore is the return of the one heartbeat the task sends right after `claim_restore` returns and before the reply (5.3 step 4), and of a resumed restore the instant `claim_restore_dead` returned, taken by the reconciler beside the takeover and handed to the job; a takeover's `heartbeat_at` was stamped inside that write, so the instant is at or after the beat it stands for by the write's own duration, and the catalog reads between a takeover and the resumed job's start count against the two-period window rather than widen it. The registration's own `heartbeat_at` stamp is the catalog's clock at some point of the registration, not its end: on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL commits a stale `heartbeat_at`; the job's first beat, sent before the reply, refreshes it. Two readers judge a RUNNING record's owner by the heartbeat alone and need that beat: the reconciler under a maintenance claim (a dead record is failed and its target removed, or failed `target_not_empty`) and `extenddb backup remove-target`; the lock fences only `claim_restore_dead`. A first beat that finds the claim lost, or fails, ends the task before the reply: the record is failed with reason "the claim was lost before the job started" (a write that matches nothing when the claim is another's) and the request is answered `InternalServerError`. What bounds a registration's duration per backend: PostgreSQL holds the `backup_index` row's `FOR UPDATE` lock and the target name's advisory lock through the registration, so `claim_restore_dead` is refused however long it takes; SQLite holds the in-process `restore:/
    ` slot from before the target exists; MongoDB runs the registration in a transaction bounded by `transactionLifetimeLimitSeconds` (default 60 s); Cassandra's registration writes are lightweight transactions bounded by the coordinator's `write_request_timeout_in_ms` and `cas_contention_timeout_in_ms` (Cassandra defaults 2 s and 1 s), and the manifest registration runs no heartbeat during the target's DDL; a registration whose DDL outlasts the stale window can lose its record to a takeover, and its refusal path then conditions the target's removal on its own generation and removes nothing (the registration answers an error naming the lost claim, and the record's owner finishes the restore). A registration that took two periods does not end the job at its first check, since the two-period rule counts from the first beat's return. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. PostgreSQL's lock rule applies to the resume's takeover alone: the two writes the reconciler makes to a dead RUNNING manifest record without resuming it, FAILED `OWNER_DEAD` (the target removed) or `target_not_empty` (the target left, 5.3 step 7) under a maintenance claim and FAILED `OTHER_STORE`, test the heartbeat alone on every backend, because the operator's command waits on the first and the second is a configuration this design forbids (4(c): instances of one catalog share one root), so neither waits for a stopped owner's session to end: both go through `fail_restore_job`, which conditions on RUNNING alone after `list_dead_restore_jobs` selected the record by heartbeat age, and on PostgreSQL they do not try the owner's advisory lock, so a stopped owner whose session lives keeps its record on the resume path and loses it on these two. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. While the server holds no serving registration (PostgreSQL, MongoDB and Cassandra) it does what the module doc of `crates/server/src/registration.rs` says: it refuses new requests, stops its background workers, the reconciler among them, and starts a new set once it registers again, and the work requests started goes on. Two points that doc leaves to this design: a reconciler cycle running at the stop spawns no resume once its own token has fired (the record is left for the next set's reconciler, which starts from a fresh state), and the backup and restore jobs, resumes included, run on the jobs' token, a child of the server's shutdown token and not of the workers', so a job running when the registration is lost goes on writing the catalog and the store until it ends or the server stops, beside a migration that may already have taken the catalog. Gating the running jobs on the registration (a job stopping while the server is unregistered) is GitHub issue #755. (f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup job checks the claim before its permit and so before its row exists (5.1 step 2; the in-catalog create and restore the same, revision items 29 and 32), the restore job as 5.3 step 4 states, and each refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the command's wait lists the rows again each time it waits, a job whose row lands after the command's claim is seen by the wait, which waits for it; a job that checked before the claim was taken runs to its end, as a job started before the command would. @@ -106,7 +106,7 @@ The invariants and state machines every later PR honours, by lettered part; the (h) Trust boundary of sync: a principal that can write under an account's prefix can forge a backup for that account. Checksums do not help, they are forgeable by the same principal. The deployment guide states that write access to an account's prefix is backup-administrator authority for that account, and gives per-account IAM prefix policies. A server-side MAC over manifests is not in v1; if hostile writers with bucket access are in scope for a deployment, the operator restricts write access rather than relying on ExtendDB to detect forgeries. -(i) Authority: the catalog migration PR 6 adds (migration id 6, `backup_catalog_index`, on every backend, through the numbered runner that PostgreSQL and SQLite use and #681 put Cassandra and MongoDB on; PR 9a's `restore_progress` is id 7), the `BackupCatalog` trait of section 3 that each backend implements, and the engine's `backup` module (`crates/engine/src/backup/`), which binds it; this section is the specification they implement. The mixed-version rule is PR 6's section. +(i) Authority: the catalog migration PR 6 adds (migration id 6, `backup_catalog_index`, on every backend, through the numbered runner that PostgreSQL and SQLite use and #681 put Cassandra and MongoDB on; PR 9a's `restore_progress` is id 7), the `BackupCatalog` trait of section 3 that each backend implements, and the engine's `backup` module (`crates/engine/src/backup/`), which binds it; this section is the specification they implement. The mixed-version rule is PR 6's section. PR 9 adds migration id 7, `restore_progress` (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL only (`data_migrations/0007_restore_progress.sql`), declared `NotApplicable` on SQLite, MongoDB and Cassandra, which keep a store-backed restore's progress on its job record (SQLite's `table_restores` row, MongoDB's `restore_job` field, Cassandra's catalog `tables` row) (PR 9a's section). ### 5. Operations @@ -124,19 +124,19 @@ Per operation, the contract: the request checks in their order, the error each r #### 5.2 DescribeBackup, ListBackups, DeleteBackup Served from the catalog index with the legacy fallback. DescribeBackup reads the row by the ARN's account and backup id (`BackupCatalog::read_backup_row`) and returns BackupNotFoundException unless the row's `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, as 5.3 step 1 does. ListBackups implements the full request as #384 implements it (the members, the (`created_at`, `backup_arn`) order and the cursor rules with #384's two exceptions), except that `TableName` may be a table name or a table ARN, as the service accepts it (measured on DynamoDB) and as main's CreateBackup already does; the differences doc's `ListBackups` paging row changes with PR 6 (section 10). -DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock, so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; the store calls run under the store's check bound and in-flight rule (section 7); the request answers the description with BackupStatus DELETED once the DELETING write has committed, whatever the deletion does (every reader reports the backup deleted from that write on, and the service has no counterpart for a failed object deletion); if a step fails the row stays DELETING, the failure is logged, and the reconciler finishes the deletion later (section 4) (revision item 30). +DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a store-backed restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock (the legacy registration keeps the `backups` row's), so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; the store calls run under the store's check bound and in-flight rule (section 7); the request answers the description with BackupStatus DELETED once the DELETING write has committed, whatever the deletion does (every reader reports the backup deleted from that write on, and the service has no counterpart for a failed object deletion); if a step fails the row stays DELETING, the failure is logged, and the reconciler finishes the deletion later (section 4) (revision item 30). #### 5.3 RestoreTableFromBackup -The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the manifest parse, as DynamoDB does; the job runs steps 3, 5 and 6. On MongoDB the request's wait for step 4 is bounded (`RESTORE_REGISTRATION_TIMEOUT`, 60 s, against a connection pool that has stopped handing out connections): past it the request answers `InternalServerError` while the detached task goes on, so the target can appear after the error, holding the backup in use and the name until the copy ends, and DescribeTable is where the client finds it (the differences doc's row on how `RestoreTableFromBackup` runs states it for operators). The request's own awaits are the checks of step 1 in their order, the permit, the manifest read and the validation of step 2; the step 4 transaction and the spawn of the job run together in a detached task whose result the request awaits, so a client disconnect cannot commit the table and the RUNNING row without an owner. The table exists from the response onwards, so no second restore or CreateTable can take the name while files download, but it accepts no reads or writes until every file has verified and loaded. +The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the manifest parse, as DynamoDB does; the job runs steps 3, 5 and 6. On every backend the request's wait for step 4 is bounded (`RESTORE_REGISTRATION_TIMEOUT`, 60 s, the engine's constant; a connection pool that has stopped handing out connections is the case that meets it): past it the request answers `InternalServerError` while the detached task goes on, so the target can appear after the error, holding the backup in use and the name until the load ends, and DescribeTable is where the client finds it (the differences doc's row on how `RestoreTableFromBackup` runs states it for operators). There are two detaches. Once the request is authenticated and authorized, its whole body (the checks of step 1 in their order, the permit, the manifest read and the validation of step 2, then step 4) runs in a task of its own, so a client disconnect after authorization cancels nothing (#384's rule); inside it, the step 4 transaction and the spawn of the job run together in a second detached task whose result the request awaits under the 60 s bound, so the bound can return while the registration goes on and a client disconnect cannot commit the table and the RUNNING row without an owner. The table exists from the response onwards, so no second restore or CreateTable can take the name while files download, but it accepts no reads or writes until every file has verified and loaded. -1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name's syntax; the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`; then the target name under the account (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target), after the backup is known to exist and before its in-use rule and the overrides are checked, the order the service uses (a restore of a missing backup into an existing table, into a table CreateTable is still creating or into a CREATING restore target is `BackupNotFoundException`, an in-use backup into an existing table `TableAlreadyExistsException` and a backup into its own CREATING target `TableInUseException`, all measured); then, for a row, a backup in use (the rule of 5.2) returns `BackupInUseException` naming the backup, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is freed within the 30 s wait of 5.1). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`. +1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name's syntax; the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`; then the target name under the account (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target), after the backup is known to exist and before its in-use rule and the overrides are checked, the order the service uses (a restore of a missing backup into an existing table, into a table CreateTable is still creating or into a CREATING restore target is `BackupNotFoundException`, an in-use backup into an existing table `TableAlreadyExistsException` and a backup into its own CREATING target `TableInUseException`, all measured); then, for a row, a backup in use (the rule of 5.2) returns `BackupInUseException` naming the backup, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is freed within the 30 s wait of 5.1). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). The request's sidecar and manifest reads collect at most 4 KiB and 16 MiB; an object over its bound behind an AVAILABLE row is a `ValidationException` naming a corrupt prefix, as a bad sidecar is, and a resume meeting one is FAILED with the same reason. A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`. 2. Validate that the target backend supports every feature in `source_table` that a restore reproduces (a non-empty `vector_indexes` into a backend without vector support is the one case, section 1); refuse before creating anything otherwise. Unlike #384 (`ensure_single_part_base_key`), this path restores multi-part-key tables (as `unordered` backups, section 1): `bulk_load` keys each row and derives its index rows with the functions the backend's item write path uses (section 3), so a restored multi-part-key table is laid out as a natively created one. -3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place and no copy is made. A transport error during a download is retried with backoff for up to 15 minutes (a constant, not a knob in v1), the row kept RUNNING and heartbeated, before the restore fails (step 7); a checksum or size mismatch is not retried and fails the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section. -4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 29); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle. -5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed`, apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through `bulk_load(target_table_id, claim, file_index, start_at, items)` (section 3), with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise. Each backend writes base rows and index rows together and records progress so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL and SQLite, and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra (PR 9a's section: the batch sizes and progress records). The restored table has no streams and no TTL, so there is nothing to bypass. -6. (Job.) After all files, and for an unordered backup after the count of section 1 matched, the job flips the target to ACTIVE conditional on the table still being CREATING as a restore target and on the job's `owner_generation`; on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE. MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction, and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today. -7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with only its spool removed, because the claim holder owns the table. DescribeTable then returns ResourceNotFound. Crash mid-restore: the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (section 4(e)) and resumes from the recorded progress, reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not, and resuming nothing while the configured store is unhealthy; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). Shutdown: a restore job stops at its next batch without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule. +3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place, with no spool: a hashing `get` pass over every file before any load, then a second `get` of each file as it is loaded, hashed again through `ExtenddbManifest::reader`, so a file replaced between the passes fails in the load; on a filesystem store the load's second `get` hands items to `bulk_load` as they decode and checks the digest at the end of the stream, so the items of a file replaced between the two passes may be written to the CREATING target (committed, on a backend that commits per batch) before the end-of-stream check fails the file and the failure path removes the target; the store root's owner-only mode (section 2) is the guard. A `StoreError::Transport` during a download (at the `get` or mid-body) is retried for up to `DOWNLOAD_RETRY_BOUND`, 15 minutes (a constant, not a knob in v1), after `min(2^n s, 60 s)` plus up to a second of jitter, the row kept RUNNING and heartbeated, before the restore fails (step 7); the bound counts from the job's first download attempt, one clock for every file (`download_all` takes it once and every file's retries count against it), so a store that stays down costs a restore the bound once, not the bound per file; the failure names the file whose attempt crossed it; a filesystem store's `get` errors are `Io` and fail at once, so the retry covers S3 stores; `NotFound`, a denial, and a checksum or size mismatch are not retried and fail the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. On an S3 store the manifest and its sidecar are written to the spool entry before the first download, each as a data file is (`.partial`, synced, renamed), so a crash during the write leaves the earlier document or none and never a truncated one; a resume that read the pair from the spool does not write it again, and a spooled pair that fails verification (the sidecar, the parse, the record's ARN) is replaced by the store's, with one WARN line. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section. +4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the `backup_index` row's lock that DeleteBackup's `start_backup_delete` takes (5.2: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. The task then sends one heartbeat through the claim and takes its return as the instant the heartbeat's two-period rule counts from (`heartbeat.rs`), checks the maintenance claim, and replies: a first beat that finds the claim lost, or fails, runs step 7 with reason "the claim was lost before the job started" and the request returns `InternalServerError` (4(e)); a held maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 29) runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return the `TableDescription` the registration produced, with TableStatus CREATING and its `RestoreSummary`. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle. +5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed` from the spool and `ExtenddbManifest::reader` in place (a filesystem store's file, hashed again as it is read), apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through one `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items)` (section 3) over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB) of data-file line text (the unit the backup side budgets in; a chunk closes at the first item that takes it to the bound, so it holds at most the bound plus one line) through a channel of two chunks, with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise, and `reapplying` true for the first file a resume meets and false after. Each backend writes base rows and index rows together and records progress in the write that applied the items, so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL (a `restore_progress` row per file in the data database, fenced by the base table's primary key and the progress row's own key rather than by the generation, which fences only the record's writes in the catalog) and SQLite (the record's `(file_index, items_applied)` per batch), and by idempotent re-application of the one unrecorded file or batch on MongoDB (the record's `file_index` per file; E11000 suppressed only on the `reapplying` file) and Cassandra (the record's progress per batch under a lightweight transaction; `put_restored_item` is an upsert). The batches: PostgreSQL one data transaction per file, the base rows in multi-row INSERTs closed at 500 rows or 4 MiB of item text (`COPY_BATCH_ITEMS`, `COPY_BATCH_BYTES`), the progress row last in the transaction; SQLite one write transaction per batch, closed at 10,000 items or at the first item that takes the batch's serialized `item_data` text past 32 MiB (`RESTORE_BATCH_ITEMS`, `RESTORE_BATCH_BYTES` in `extenddb_storage::restore_engine`), the record's progress last in the transaction; MongoDB `insert_many(ordered = false)` calls of 1,000 documents (the driver splits a call by the server's message and write-batch bounds; 1,000 is the unit a half-applied file is re-applied in), the record's `file_index` written after the file; Cassandra upserts in batches closed at 10,000 items or 32 MiB of serialized item text, the same two constants, each batch followed by the progress lightweight transaction; an empty file records its progress on every backend: `items_applied = start_at` on PostgreSQL, SQLite and Cassandra, the `file_index` alone on MongoDB, whose record carries no `items_applied` (PR 9a's section). Liveness inside a file: the feeder checks at every chunk boundary that the heartbeat confirmed the claim within two periods and ends the stream as a lost claim otherwise; on MongoDB and Cassandra `bulk_load` reads the record before each write batch (MongoDB one `find_one` on the target's `_id`, `table_id`, `restore_job.status` and `restore_job.owner_generation` before each `insert_many`; Cassandra a lightweight transaction on the record, the heartbeat refreshed `IF` RUNNING under the generation, before each batch of writes, after reading the coordinator's clock as the batch's write timestamp) and ends the file as a lost claim when the record is not RUNNING under its generation, so a writer whose record another instance took over while it stalled lands at most the one batch it was in, and on Cassandra that batch lands `USING TIMESTAMP` the reading taken before the check, below every write and delete the successor made since, so it changes nothing (the PR 9a load paragraph); on Cassandra an unknown outcome of the progress lightweight transaction (a write timeout or failure, a lost connection) is decided by a `LOCAL_SERIAL` re-read of the record: RUNNING under the generation and carrying the values written is applied; RUNNING under the generation with the earlier values is one retry, whose own unknown outcome is re-read the same way, and a second unknown outcome the record does not carry fails the restore; anything else is the lost claim. A stalled owner whose record a successor took over, flipped ACTIVE and whose clients then wrote, lands nothing that outlives the successor's state: on MongoDB the batch's `insert_many` and its progress commit in one transaction whose first write is the claim's fence, so a batch that reaches the server after the takeover is refused whole; on Cassandra the batch's writes carry the timestamp read before its claim check, below every write and delete the successor and its clients made since, so the cells land and lose to the later ones and a key a client deleted stays deleted (the stalled writer's progress write then finds the record not its own and the load stops). The rule on Cassandra needs the ExtendDB hosts' clocks to agree, as the version scheme already does. PostgreSQL's window stays as stated: nothing in its `bulk_load` conditions on the generation; the base table's primary key and the claim's liveness, confirmed before each file, are its fence, the advisory lock holds the record through a stall while the owner's session lives, and the window that remains is a session the database server dropped during a stall longer than the stale window (an `idle_session_timeout`, not a default): the stalled file's transaction can commit after a takeover, and the successor's own load of that file then fails on the duplicate key and the restore is FAILED and its target removed, a lost restore and not wrong data. The restored table has no streams and no TTL, so there is nothing to bypass. +6. (Job.) After all files, and after the count of section 1 matched, the job flips the target to ACTIVE through `finish_restore` (section 3), conditional on the table still being CREATING as a restore target and on the job's `owner_generation` (`NotMatched` otherwise: an error line and no table access); on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE (`Active`, never `Pending`, there). A `finish_restore` that returns an error is a conditional write whose outcome is unknown and is handled as the status writes are: the record is re-read through the claim; DONE under the job's generation is the write landed and the restore complete (one WARN naming the error); RUNNING under it is the write not taken, sent once more at most and its own error re-read the same way; anything else, a second error included, is the DONE write that matched no record, after which the job makes no table access and the reconciler resumes the record once stale. `finish_restore` writes the manifest's `source_table.table_size_bytes` as the restored table's size until the statistics pass rewrites it; `DescribeTable` of a restored table reports the source table's size, not the compressed size of the backup's files. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE (on Cassandra this holds for a resumed restore's re-applied batch because the write path rewrites an unchanged item's index rows instead of deleting them, #424, the PR 9a load paragraph). MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction (`finish_restore` returns `Pending`), and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today. +7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with no further table access and its spool entry left to the sweep, because the claim holder owns the table and may be reading that entry (the entry is keyed by table id, so a same-host successor shares it). DescribeTable then returns ResourceNotFound. Crash mid-restore: once the server's shutdown has begun (the jobs' token is cancelled) the reconciler takes no dead record over and writes no FAILED on the maintenance or other-store arm: the record is left as it is for the next instance's reconciler, since a resume spawned on the way out would stop at its first check and leave the record with a fresh heartbeat, a second stale window for the next instance to wait. Otherwise the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (`claim_restore_dead`, section 4(e)) and resumes from the recorded progress (a dead record met under a maintenance claim is failed through the unclaimed RUNNING-only write before its store is looked at: `target_not_empty` with its target left for `remove-target` (which runs under the same claim) when it recorded no progress and its target holds more items than the backup's row counts, else `OWNER_DEAD` with its target removed; the command's wait is satisfied either way, the record being no longer RUNNING; a record of another store is failed `OTHER_STORE` below, both arms on the heartbeat alone on every backend, 4(e); a resume takes a job permit and is skipped to the next cycle without one; a resume with no recorded progress into a target holding more items than the backup's total (the backup row's `item_count`) is FAILED `target_not_empty` and its target left for `extenddb backup remove-target --account
    ` (section 4's duty list), which ships with PR 9, the rule for a catalog restored from a dump that names a restore which finished and was written to since; a target holding no more than the total with no recorded progress is a first data file applied in part (on Cassandra, which records a batch's progress after its writes, a crash before the first progress record, the first 10,000 items or 32 MiB, leaves this; MongoDB commits each batch with its progress and leaves nothing unrecorded) and is resumed with the first file re-applied under `reapplying`, every backend's load being idempotent for such a file; the takeover after one instance's fifth of a record (`RESTORE_TAKEOVERS`, counted in the reconciler's state, so per instance and reset by a restart) fails it `owner dead: taken over 5 times by one instance without the restore ending` and removes its target as `OWNER_DEAD` does, so a record whose every resume ends RUNNING again (a progress read or the final count that fails every time, a load that stops with the record RUNNING) is not retaken every stale window for as long as the server runs), reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not (`OTHER_STORE`, the target left to the FAILED-target duty), and resuming nothing while the configured store is unhealthy; on SQLite at once after a restart, by the slot rule; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). The reconciler's sweep of `restore_progress` rows (PostgreSQL) is a positive list: the backend reads the distinct table ids the progress rows name, then the catalog's RUNNING records, and deletes only the rows of ids in the first set that are in neither the second nor the reconciler's own RUNNING list; a target whose first row lands after the first read is never a candidate, and one registered before it is RUNNING at the second read, so a live job's row is never deleted. Shutdown: a restore job stops at its next chunk without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule; it records `RestoreJobs{cancelled}` and no worker metric (section 9), where a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` and leaves the spool entry to the sweep: the reconciler's spool sweep lists the spool directory first and the RUNNING records second, and removes only the entries it listed that no RUNNING record of this host's `spool_host` names, so a restore whose registration commits and whose entry opens between the two reads is never a candidate; an entry its own job removed between the listing and the removal is done, not an error. Restore overrides. #384 applies every override the operation models as the service applies it (`crates/engine/src/backup/overrides.rs`) and refuses only `VectorIndexOverride`, which the botocore model this project targets does not carry, with a typed ValidationException. The v1 contract keeps exactly that: the new path applies the overrides at step 4 to the manifest's `source_table` before the target is created, through the same `overrides` module, and `VectorIndexOverride` keeps the refusal. The differences and limits docs' override rows stay as they are. @@ -169,7 +169,7 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 29). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 31): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary or vector indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 33), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 29). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, the root's calls run one at a time so that calls started before the first reached its bound wait for it under the bound and then have the bound for their own call, rather than parking a thread each, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. @@ -185,7 +185,7 @@ Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job k ### 9. Observability -Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`, and `/metrics` emits them under those names as JSON (`render_metrics`, `docs/design/06-component-server.md`: the endpoint is not Prometheus exposition format; the deployment guide tells operators to convert); the snake-case spellings above are the names a converter would give them, not a rendering the server does. PR 8 adds `BackupJobs` (one count per job end, `operation` = the outcome, `succeeded`, `failed` or `cancelled`), `BackupJobDuration` (the job's duration in microseconds, like every duration the collector records, with the same `operation`), `BackupBytesUploaded` (bytes put to the store), `BackupSnapshotAge` (seconds since the snapshot instant, set at each confirmed heartbeat) and `BackupStoreErrors` (`operation` = the store call that failed: the job's `put` and `delete_prefix`, and the reconciler's `get` and `list` of the manifest probe and the deleting duties and its `delete_prefix`, an absent object not counted), with `QuerySource::BackupJob`; `BackupBytesUploaded` is recorded for a job that ended AVAILABLE, and `WorkerErrorCount` under `BackupJob` for a FAILED job and not for one the shutdown cancelled (which records `BackupJobs` with `cancelled` and neither worker metric); the restore variants are PR 9a's. The existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the reconciler like the other workers and by the backup job at its end under `BackupJob`. +Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `restore_bytes_downloaded_total`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`, and `/metrics` emits them under those names as JSON (`render_metrics`, `docs/design/06-component-server.md`: the endpoint is not Prometheus exposition format; the deployment guide tells operators to convert); the snake-case spellings above are the names a converter would give them, not a rendering the server does. PR 8 adds `BackupJobs` (one count per job end, `operation` = the outcome, `succeeded`, `failed` or `cancelled`), `BackupJobDuration` (the job's duration in microseconds, like every duration the collector records, with the same `operation`), `BackupBytesUploaded` (bytes put to the store), `BackupSnapshotAge` (seconds since the snapshot instant, set at each confirmed heartbeat) and `BackupStoreErrors` (`operation` = the store call that failed: the job's `put` and `delete_prefix`, from PR 9a a restore's `get`, and the reconciler's `get` and `list` of the manifest probe and the deleting duties and its `delete_prefix`, an absent object not counted), with `QuerySource::BackupJob`; `BackupBytesUploaded` is recorded for a job that ended AVAILABLE, and `WorkerErrorCount` under `BackupJob` for a FAILED job and not for one the shutdown cancelled (which records `BackupJobs` with `cancelled` and neither worker metric); the restore variants are PR 9a's: `RestoreJobs` (`operation` = the outcome), `RestoreJobDuration` and `RestoreBytesDownloaded`, with `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job: the request's and a resume's two document reads and each download attempt the store failed, retried (a transport error) or not (an absent file, a refusal, an I/O error); a digest mismatch and the spool's own errors are not the store's. `RestoreBytesDownloaded` counts the bytes of each data-file download attempt as they arrived, verified or not; the manifest and sidecar reads and a filesystem store's second read of each file are not counted. `RestoreJobs{failed}` counts the job's FAILED writes and the ends at which it learnt the record is another's; the reconciler's own FAILED writes (`target_not_empty`, `OTHER_STORE`, `OWNER_DEAD` under a maintenance claim) are counted in its cycle summary line and `CycleReport`, not in `RestoreJobs`. `WorkerErrorCount{RestoreJob}` accompanies each of those job ends (the maintenance refusal before the job ran, a FAILED write, a lost claim, a DONE write that matched no record); a job the shutdown stopped records `RestoreJobs{cancelled}` and no worker metric; `WorkerLastSuccess{RestoreJob}` is recorded on a restore that reached DONE. The existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the reconciler like the other workers and by the backup job at its end under `BackupJob`. ### 10. Documentation @@ -200,7 +200,9 @@ Which document and which row each PR changes; what each row says is in the ownin | differences doc: vector-backup row | PR 8 (CreateBackup) and PR 9a (the restore's refusal) | | differences doc: `ListBackups` paging row (table ARN in `TableName`) | PR 6 | | differences doc: multi-part-key row | PR 9a | -| differences doc: restore-interruption row | PR 8 | +| differences doc: restore-interruption row | PR 8 (the reconciler) and PR 9a (the store-backed resume per backend, SIGTERM, the kill-point outcomes) | +| differences doc: how `RestoreTableFromBackup` runs (the 60 s registration bound on every backend; the measured durations in that row are the base's in-catalog figures), the DeleteBackup and concurrent-restore rows (the `backup_index` lock per backend) | PR 9a | +| differences doc: new row for the MongoDB Decimal128 sort-key rule | PR 9a | | differences doc: Import and Export rows | PR 13 | | differences doc: new row for CreateBackup of a table holding an `N` value #618 let in | PR 8 | | differences doc: where-backups-live and backup-creation rows (the store-backed, asynchronous create) | PR 8 | @@ -210,7 +212,14 @@ Which document and which row each PR changes; what each row says is in the ownin | admin guide: the `[backup]` rows (`data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs` with the memory figures) and the `[backup.store]` section (the store-backed create, the job's two log lines, the unhealthy-store refusal) | PR 8 | | deployment guide: the service unit's `MALLOC_ARENA_MAX=1` line | PR 8 | | `docs/design/README.md` row 15b (the measurement record) | PR 8 | +| `docs/design/15-backup-restore-bench-pr9.md` (the restore memory and duration record) and `docs/design/README.md` row 15c | PR 9a | | usage guide: "backups not supported" line | PR 8 | +| usage guide: the backups row's RestoreTableFromBackup ✓ (from the store into every backend) | PR 9a | +| upgrade manual: the restore of a store-backed backup lands, `kind = "disabled"` no longer needed for it, `spool_path` for an S3 store, migration 7 and `extenddb migrate`, the rollback paragraph's progress-row sentence | PR 9a | +| admin guide: the `spool_path` row (what the spool holds, its disk cost, the sweep), the `max_concurrent_jobs` row's restore memory terms, `backup list` lists no restore | PR 9a | +| limits doc: the concurrent-restores row (the registration bound, the spool term, Cassandra's `COUNT(*)`) and the backup-contents row (restores into any backend, vector where supported, multi-part keys) | PR 9a | +| `docs/schema/postgres.txt`: the `restore_progress` table | PR 9a | +| `.github/workflows/matrix-nightly.yml` and `devtools/backup-restore-matrix` (the nightly's script) | PR 9b | | upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 (and PR 8 for the store-backed backups a restored catalog no longer lists) | | upgrade manual: CreateBackup without `[backup.store]`, `kind = "disabled"`, the SQLite default root, shared store roots and `--adopt-store` | PR 7 (and PR 10 for `--adopt-store`) | | upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 | @@ -273,7 +282,7 @@ store_kind, store_root, store_id the store the restore reads, recorded at 5.3 owner_generation the same claim as a backup job (5.1 step 3); replaces #384's restore_operation_id (MongoDB) and restore_owner (Cassandra, where a non-key column cannot be renamed: migration 6 adds restore_owner_generation beside restore_owner, which PR 14 drops) heartbeat_at replaces #384's restore_heartbeat / restore_heartbeat_at spool_host the host whose spool holds the downloaded files (5.3 step 3) -file_index, items_applied per-batch progress on SQLite and Cassandra (5.3 step 5); unused on PostgreSQL (see `restore_progress`) and MongoDB (per-file record) +file_index, items_applied the restore's progress: per batch on SQLite and Cassandra, `file_index` per file on MongoDB (`items_applied` unused there); unused on PostgreSQL, whose progress is `restore_progress` rows in the data database (migration 7, 5.3 step 5) failure_reason, completed_at ``` PR 6's migration backfills the new fields on the registrations that exist at migration time, on all four backends: `kind = legacy`, `status = DONE` where the table is not CREATING, and where it is (a restore the upgrade's stop interrupted), `status = RUNNING` with `heartbeat_at` set to the epoch (1970-01-01T00:00:00Z), not NULL, on every backend, so every backend's staleness test selects them (the column is NOT NULL after the migration on PostgreSQL, so its test has no `IS NULL` arm), and a sentinel `owner_generation` (0) that no job holds; the first reconciler pass fails those rows as legacy restores (section 3) and removes their targets (`remove_restore_target`), which is what #384's startup sweeps did for such a target. @@ -368,53 +377,76 @@ Documentation rows it corrects: limits doc backup rows (the knobs); upgrade manu ### PR 8: CreateBackup job, worker and reconciler Goal: the CreateBackup job of 5.1 on the filesystem store, with the reconciler and its full duty list, the byte budget on the snapshot streams and the item validator. Its line names the interfaces and carries the acceptance: -- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, in the page query on PostgreSQL and while the rows are pulled on SQLite, PostgreSQL's cursor replaced by the three-layer keyset page under the same snapshot (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, the create half of an off-host round trip of a vector table added (CreateBackup succeeds, DescribeBackup reaches AVAILABLE with the table's features, ListBackups lists it, and the restore answers the ValidationException naming the store until PR 9a adds the restore half) and `test_create_backup_of_a_vector_table_is_refused` expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which every CI pytest job that runs the suite sets (`filesystem` on the PostgreSQL job, which appends a filesystem store after `init`, and on the SQLite job, the defaulted root; `s3` on the SQLite job against moto; `none` on the MongoDB and Cassandra jobs) and `devtools/run-tests --pytest` refuses to run without (5.1 step 1); the restore cells of `tests/test_backup_restore_fidelity.py` skip on a server with a store, naming PR 9a, until its restore job lands; FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend with a `SnapshotEngine` (PostgreSQL, SQLite, MongoDB; Cassandra keeps the legacy create and its existing cells); SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs waits for the slot (up to 30 s, item 28) and is answered once the first job ended, `LimitExceededException` when the wait outlives the bound; a 1M-item run records duration and peak RSS, and the 1M run's peak must be within 112 MiB of the 100k run's (the budget, the line cap and the channels; on every backend but `sqlite-memory`, whose open copies the table; on MongoDB a 500k run stands in for the 1M run, its load alone taking 27 minutes through the wire); the memory test with #394's worst-expansion items, with the worst lines placed after a page of small rows, has its peak over idle under 1.2 times the sum of section 7's terms at the measured row (the decoded rows in the page and the pipeline, one row's parse, the line buffer, the chunks and SQLite's run-ahead) with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one SQLite binary and at 12 rows on PostgreSQL, and its peak with glibc's default arenas recorded beside it at the same counts, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows a PostgreSQL page reads past its cut are at most `n - 1`, measured by `EXPLAIN (ANALYZE, BUFFERS)` on a 1,000-small-then-60-worst table (4 read, 1 past the cut at the floor) and pinned by `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips on a 300-row table with nine 27 MB rows); `EXPLAIN` of the page query on the six key shapes a table can have (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation) shows `Limit` over one `Index Scan` on the primary key and no other scan of the relation where the planner's estimate is above `n`, and the bitmap scan with a `Sort` near the end of a small table (PostgreSQL 16), both pinned by a test and stated in the module doc; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on PostgreSQL and SQLite (the two backends `tests/test_backup_job.py`'s deployment fixture starts; the fixture's MongoDB deployment destroys and initializes the catalog of a replica set of the suite's own, since the database names are fixed, and the `mongodb-backup-store` job runs the suite and the backup selection against a filesystem store) (an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`; UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone); with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS is recorded against the terms of section 7 in each, proving the encoder streams into the upload (the incompressible run's 68 MB file passes through a 4 MiB chunk window) and that the byte budget and the channel bound hold at the largest line (under `MALLOC_ARENA_MAX=1` the worst-expansion run's peak does not grow with the number of worst rows, 452, 480 and 455 MiB over idle at 12, 24 and 48 rows on the final SQLite binary; with glibc's default arenas it does, 707, 700 and 1,281 MiB in the same series, the allocator's retention and not the pipeline's terms, which the one-arena series shows do not grow). Depends on PR 4, 5, 6, 7. +- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, in the page query on PostgreSQL and while the rows are pulled on SQLite, PostgreSQL's cursor replaced by the three-layer keyset page under the same snapshot (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, the create half of an off-host round trip of a vector table added (CreateBackup succeeds, DescribeBackup reaches AVAILABLE with the table's features, ListBackups lists it, and the restore answers the ValidationException naming the store until PR 9a adds the restore half) and `test_vector_table_backup_refused_in_catalog_and_restored_from_store` (named `test_create_backup_of_a_vector_table_is_refused` until PR 9 added the restore half) expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which every CI pytest job that runs the suite sets (`filesystem` on the PostgreSQL job, which appends a filesystem store after `init`, and on the SQLite job, the defaulted root; `s3` on the SQLite job against moto; `filesystem` on the MongoDB jobs, whose `devtools/run-mongodb-tests` serves a filesystem root under its output directory from PR 9 on, where it was `none` before; `none` on the Cassandra job) and `devtools/run-tests --pytest` refuses to run without (5.1 step 1); the restore cells of `tests/test_backup_restore_fidelity.py` skip on a server with a store, naming PR 9a, until its restore job lands; FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend with a `SnapshotEngine` (PostgreSQL, SQLite, MongoDB; Cassandra keeps the legacy create and its existing cells); SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs waits for the slot (up to 30 s, item 28) and is answered once the first job ended, `LimitExceededException` when the wait outlives the bound; a 1M-item run records duration and peak RSS, and the 1M run's peak must be within 112 MiB of the 100k run's (the budget, the line cap and the channels; on every backend but `sqlite-memory`, whose open copies the table; on MongoDB a 500k run stands in for the 1M run, its load alone taking 27 minutes through the wire); the memory test with #394's worst-expansion items, with the worst lines placed after a page of small rows, has its peak over idle under 1.2 times the sum of section 7's terms at the measured row (the decoded rows in the page and the pipeline, one row's parse, the line buffer, the chunks and SQLite's run-ahead) with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one SQLite binary and at 12 rows on PostgreSQL, and its peak with glibc's default arenas recorded beside it at the same counts, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows a PostgreSQL page reads past its cut are at most `n - 1`, measured by `EXPLAIN (ANALYZE, BUFFERS)` on a 1,000-small-then-60-worst table (4 read, 1 past the cut at the floor) and pinned by `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips on a 300-row table with nine 27 MB rows); `EXPLAIN` of the page query on the six key shapes a table can have (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation) shows `Limit` over one `Index Scan` on the primary key and no other scan of the relation where the planner's estimate is above `n`, and the bitmap scan with a `Sort` near the end of a small table (PostgreSQL 16), both pinned by a test and stated in the module doc; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on PostgreSQL and SQLite (the two backends `tests/test_backup_job.py`'s deployment fixture started first; the fixture's MongoDB deployment destroys and initializes the catalog of a replica set of the suite's own, since the database names are fixed, and the `mongodb-backup-store` job runs the suite and the backup selection against a filesystem store) (an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`; UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone); with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS is recorded against the terms of section 7 in each, proving the encoder streams into the upload (the incompressible run's 68 MB file passes through a 4 MiB chunk window) and that the byte budget and the channel bound hold at the largest line (under `MALLOC_ARENA_MAX=1` the worst-expansion run's peak does not grow with the number of worst rows, 452, 480 and 455 MiB over idle at 12, 24 and 48 rows on the final SQLite binary; with glibc's default arenas it does, 707, 700 and 1,281 MiB in the same series, the allocator's retention and not the pipeline's terms, which the one-arena series shows do not grow). Depends on PR 4, 5, 6, 7. Invariants honoured: 4(a) (every transition of the backup row a job or the reconciler performs), 4(c) (the reconciler acts only on rows of the configured store and on none while it is unhealthy), 4(e) (the claim, the fencing of every status write, the FAILED-write gate before any delete), 4(f) (the duties that run under a maintenance claim), 5.1 as written, section 2's three store facts, section 7's memory bound and pool rule, and the Drawbacks' #618 regression. Interfaces it adds: `extenddb_core::validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits)` (5.1 step 4; the key argument is a schema every table has, multi-part keys included, each key attribute checked with the format's own `check_key_value`; `RestorableLimits` is built from `&LimitsConfig`: the item size, the hash and range key sizes and the attribute name size), `RestorableItemReason::{AttributeNameTooLong, LineTooLong}` (the latter carrying `{ cap }`), `SHOWN_VALUE_BYTES` (4096, the bound on each text an error carries other than a key value) and `SHOWN_KEY_VALUE_BYTES` (12,288, the bound on a key value), `extenddb_core::limits::DEFAULT_MAX_ITEM_SIZE_BYTES` (the default item limit, the feeder's threshold), `extenddb_core::backup_format::encode_item_line_len_bounded(item, cap)` (the line's length up to `cap`, measured without a buffer), `extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES` with `fetch_bytes()`, `record_page_bytes` and `record_rows_past_cut` (counters under `test-hooks`), `extenddb_storage::snapshot::source_table_schema` (5.1 step 2), the job (`crates/engine/src/backup/job.rs` with `job/{writer,heartbeat,failure}.rs`) and the reconciler's new duties (`crates/engine/src/backup/reconciler.rs`) in the engine's backup module, the server handing the engine the shutdown token, the metrics collector, the job settings (`JobSettings`) and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, the metrics of section 9, and on `BackupCatalog`: `claim_dead_backup(key, stale_after)` (the takeover of 4(e); `claim_backup` inserts and takes over nothing), `list_backup_rows_older_than(status, store, clock, age)` (`StaleClock::{Heartbeat, Completed}`, the backend's clock; a row with `completed_at` unset is never older on the completed clock), `sweep_restore_registrations` (Cassandra's `active_restores` sweep, a default empty body elsewhere) and `StatusFields.source_table_details_json` (the AVAILABLE write carries the manifest's `source_table`). The request's detached task opens the snapshot, claims the row, checks the maintenance claim and returns the response through a one-shot channel, then continues as the job with the same owned values, so nothing but the response crosses a channel between the request and the job (the pipeline's item and chunk channels and its blocking encoder are the job's own); the runtime's task tracker holds every job, and the server's shutdown cancels and awaits the jobs' failure paths (`BackupRuntime::drain_jobs`, a 30 s bound) before it drains its workers. It is needed because four write paths store an `N` value such as `1E126` or `abc` unvalidated on main (#618): BatchWriteItem, TransactWriteItems, UpdateItem on an absent item (which creates the item from its `Key`) and ImportTable, the list #394's `key_order.rs` doc gives. So that one run finds every offending item, the first refusal stops encoding and uploading but not validation: the job reads and checks the rest of the snapshot, logs each refused item (its key, attribute path and reason, with the backup ARN) up to 1000 items and counts the rest, and then fails the backup (step 5) with a FAILED reason that gives the number of refused items and the first one, naming #618 only when that item's refusal is an `N` value; so no backup that no restore can load is reported AVAILABLE. The byte budget (section 7 states the bound): PR 8 replaces #395's cursor on PostgreSQL with pages bounded by bytes in the query, each page a keyset `SELECT` in the same `REPEATABLE READ` transaction, ordered by the data table's primary key and starting after the last key of the page before, in three layers (the innermost takes `n` rows after the key under `LIMIT`, since a window function runs before `LIMIT`; the middle adds the running byte sum and the rows' count; the outer keeps the rows whose predecessors' bytes are under `SNAPSHOT_FETCH_BYTES` (64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a configuration key in v1) and orders them), so a page holds at least one row and at most the budget plus one, and the stream ends when the inner query returned fewer rows than asked and the cut dropped none of them; `n` starts at 16 and is then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows (`RowBoundMemory`), never under 4 and never over `DEFAULT_FETCH_BATCH`; the rows read past the cut (at most `n - 1`) are read by the server and dropped by the outer layer, and each costs the server one serialization of its item text and two spills of it (the window functions buffer the `n` rows before the outer layer drops them), so a page costs the server `n` rows of text, not the rows it returns: sized from the average, a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned; sized from the largest row it asks for 4 (208 MiB, 1.1 s); the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, at most once per row bound's worth of small rows; the key columns come back under qualified aliases so the next cursor is the row's own typed key, and the outer `ORDER BY` states a guarantee the executor gives anyway (a mutant without it passes every behavioural test, and the query text pins it); the snapshot, the lock and which tables stream `key_ordered` are unchanged. On SQLite a page stops at `DEFAULT_FETCH_BATCH` rows or after the row that reaches `SNAPSHOT_FETCH_BYTES` of item text, whichever comes first, and holds at least one row; the rows are pulled one at a time, so the stream pulls none past the cut, while the driver's worker thread reads at most five rows ahead of the pull (sqlx's `row_buffer_size`, set on the pool to four from its default of 50, which read every row of a 24-row worst-shape table ahead of the first page, plus the row the worker holds while its `send` waits; 160 MiB at the line cap), and the stream ends only when a page is short of the row bound and no cut happened. The cut rule exists in SQL on PostgreSQL and in Rust on SQLite, so the check that no page holds more than the budget plus one row lives in `extenddb_storage::snapshot_conformance`, which both run; a MongoDB batch is already bounded by the server's 16 MiB reply. -Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds the create half of an off-host round trip of a vector table beside it (PR 9a adds the restore half), and `test_create_backup_of_a_vector_table_is_refused` (`tests/test_backup_restore_fidelity.py`) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs a suite creating backups sets it (the Rust integration suite reads it the same way, `tests/rust/src/backup_restore.rs`): it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. +Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds the create half of an off-host round trip of a vector table beside it (PR 9a adds the restore half), and `test_vector_table_backup_refused_in_catalog_and_restored_from_store` (`tests/test_backup_restore_fidelity.py`; `test_create_backup_of_a_vector_table_is_refused` until PR 9) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs a suite creating backups sets it (the Rust integration suite reads it the same way, `tests/rust/src/backup_restore.rs`): it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. -State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (the resume is PR 9a's, with the restore job; this release's reconciler logs such a record once; once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target
    ` (PR 10), the one command that removes a FAILED restore's CREATING target and its record, through `remove_restore_target`, under the maintenance claim like the other writing commands, since DeleteTable refuses a restore target (5.3 step 6); a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove `restore_progress` rows whose table is not a CREATING restore target and spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3), both from PR 9a, which adds the records they read (the duty that removed a CREATING restore target with no job row is withdrawn: on every backend the job record is the registration itself, so no such state exists after migration 6); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. +State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (the resume is PR 9a's, with the restore job, through `claim_restore_dead`, 4(e); once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target --account
    ` (PR 10's command, shipped with PR 9: on MongoDB and Cassandra a crash before the first progress record, the whole first data file on MongoDB and the first 10,000 items or 32 MiB on Cassandra, leaves written items and no progress, so the ordinary crash there ends in this state and the release that ships the restore ships its remedy, item 33; until it runs, DeleteTable refuses the target, 5.3 step 6, and DeleteBackup and a second restore of the backup are `BackupInUseException`, 5.2). `extenddb backup remove-target --account
    ` removes a restore target DeleteTable refuses (a CREATING table) and its restore record, under the maintenance claim like the other writing commands: a FAILED record's target whatever the reason (`target_not_empty` included), a RUNNING record's target whose owner is dead (the backend's `list_dead_restore_jobs` names it under `stale_job_after_secs` on the database's clock; on SQLite under the serve lock the window is zero and every RUNNING record is listed), and a legacy record's target the same way; it refuses a RUNNING record whose owner is live ("stop it or wait for it"), a DONE record and a table no record names (an ACTIVE table is one or the other), the latter two as "not a restore target". It needs no store, prints the claim it took and one line naming what it removed and the record's state, exits 1 with the refusal otherwise, and logs the removal under `extenddb::audit` with the account, the table and the backup's ARN. The removal is `remove_restore_target`, section 3; a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove `restore_progress` rows whose table is not a CREATING restore target and spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3), both from PR 9a, which adds the records they read (the duty that removed a CREATING restore target with no job row is withdrawn: on every backend the job record is the registration itself, so no such state exists after migration 6); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. Kill points (testing summary): between a data file upload and the manifest; between the three documents and the manifest (`before_manifest`); between the manifest put and the AVAILABLE update; SIGTERM during a backup. Main tests that must pass unchanged: #384's backup tests other than the vector tests named above. Documentation rows it corrects: the differences doc's backup consistency row (its MongoDB half, "not point-in-time" today, which the snapshot read concern of section 3 makes one; its Cassandra half stays true), its vector-backup row ("Backing up and restoring a table that has vector indexes": supported on the off-host path from PR 8 and 9a, refused only into a target without vector support, CreateBackup on the legacy create, without `[backup.store]` or on Cassandra, still refused as on main, the legacy reader's behaviour kept for legacy backups; 5.1 step 1), its restore-interruption row (one reconciler with the duties above in place of the per-backend sweeps), a new row for CreateBackup of a table holding an `N` value #618 let in (the backup fails on the off-host path, where main backs it up and restores it; 5.1 step 4), the limits doc's line cap and file size rows, the usage guide's "backups not supported" line; upgrade manual: on the off-host path a backup of a table holding an `N` value the wire refuses (#618) fails where main backs it up, which meets file-backed SQLite without a `[backup]` section and any deployment once a store is configured (5.1 step 4), the reason reaches only the operator (`backup list --failed`, and the log, which names each offending item's key and attribute path with the backup ARN a tenant's support case can cite), and the way out is to rewrite each named item. ### PR 9a: restore job Goal: the restore job of 5.3 on the same backend over a filesystem store. Its line names the interfaces and carries the acceptance: -- PR 9a: restore job per 5.3, same backend, filesystem store (PR 8 leaves it the engine's `check_catalog_row_for_restore`, which answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" for a row of the configured store, the 24 restore cells of `tests/test_backup_restore_fidelity.py` and `tests/test_auth_permissions.py`'s restore-authorization cell skipped on a server with a store, and the vector wire test's restore assertion, all four to replace with the restore): request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target_table_id, claim, file_index, start_at, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL migration 7 for `restore_progress`, reserved here) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `RestorableLimits` (item, hash key, range key and attribute name sizes; `N` normalization included); items read through `ExtenddbManifest::reader_prehashed`; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write, DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB via duplicate-key suppression); SIGKILL at each kill point of the testing summary leaves a table that either finishes with the right count or is gone (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other; a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers. Depends on PR 8. +- PR 9a: restore job per 5.3, same backend, filesystem store (shipped in PR 9; the engine's `check_catalog_row_for_restore` no longer refuses a `backup_index` row with "Backup is in a backup store; this release restores in-catalog backups only", the 24 restore cells of `tests/test_backup_restore_fidelity.py` and `tests/test_auth_permissions.py`'s restore-authorization cell run on a server with a store, and the vector wire test's restore asserts the round trip into a vector-capable target and the refusal into one without): request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL migration 7 for `restore_progress`) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `RestorableLimits` (item, hash key, range key and attribute name sizes; `N` normalization included); items read through `ExtenddbManifest::reader_prehashed` from the spool and `ExtenddbManifest::reader` in place; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write (`remove_restore_target` on PostgreSQL drops the relations `DeleteTable` drops (`table_storage_relations`): the data table, every secondary index's table and every vector index's `_ddb_vec_` table, their ids read in the claiming transaction), DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB's duplicate-key suppression is proved by `restore_mongodb_after_insert_many` at its second reach, since a kill after a file's commit resumes MongoDB at the next file with nothing of it written; the Cassandra points run with a SQLite source, since a Cassandra server's own backup is in-catalog, item 33); SIGKILL at each kill point of the testing summary leaves a table that finishes with the right count (the base table's and every GSI's), is gone, or, at the first reach of the two points that pause before a first progress record (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), is FAILED `target_not_empty` for `extenddb backup remove-target`, which removes it (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB and on Cassandra two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other (on Cassandra the registration's lightweight transaction, `IF backup_status = 'AVAILABLE' AND active_restores = null`, is linearized on the backup's partition, so the second finds `active_restores` non-null; the in-use pre-read is advisory); a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers; a 1M-item restore on PostgreSQL and SQLite records its duration and peak memory (the record above: 9.5 s and 26.8 s, 16 and 89 MiB over idle), the MongoDB and Cassandra figures being PR 11's. Depends on PR 8. Invariants honoured: 4(b) (the restore record and its transitions), 4(d) (every target by `RestoreTargetKey`), 4(e) (the claim, the FAILED-write gate before the target is removed, the PostgreSQL stale window), 4(f) (the job checks the maintenance claim after its row exists), 5.2 (the registration under the backup row's lock), 5.3 as written, section 1's duplicate rule and vector refusal, and the Goals' fidelity list. -Interfaces it adds: `bulk_load` (section 3) and the restore methods of `BackupCatalog`: +Interfaces it adds: `RestoreEngine` (section 3) and the restore methods of `BackupCatalog`: ```rust - // Restore job record (section 4). PR 9a. One target is addressed by `RestoreTargetKey` (`account_id`, - // `table_name`, `table_id`): MongoDB's `tables` `_id` and Cassandra's `tables` primary key are - // (`account_id`, `table_name`), and the table id guards against a table re-created under the name. - fn claim_restore(&self, row: &RestoreJobRow) -> BoxFuture, StorageError>>; // same contract as claim_backup - // at creation it is 5.3 step 4's registration: the target, the job row, and the re-check under the backup row's - // lock that the row is AVAILABLE and not in use (BackupNotFound or BackupInUse otherwise, no target left); - // a takeover writes the claimant's `spool_host` in the same conditional write that changes the generation - fn read_restore_job(&self, key: &RestoreTargetKey) -> BoxFuture, StorageError>>; - fn list_restore_jobs(&self, status: Option) -> BoxStream>; - fn record_progress(&self, claim: &dyn JobClaim, file_index: u32, items_applied: u64) -> BoxFuture>; - // the per-file (or per-batch) progress write of 5.3 step 5, conditioned on the claim's generation - fn count_restore_target(&self, key: &RestoreTargetKey) -> BoxFuture>; - // a storage-level count of a CREATING table: Scan is refused on one, so the count is a backend method + // RestoreEngine, a trait in StorageEngine's bound (PR 9). StorageEngine has no + // methods of its own; bulk_load, finish_restore and the counters live here. + fn bulk_load<'a>(&'a self, target: &'a RestoreTargetKey, claim: &'a dyn JobClaim, + file: BulkLoadFile, items: BoxStream<'a, Result>) + -> BoxFuture<'a, Result>; // BulkLoadFile { file_index, start_at, reapplying } + // applies one data file with its secondary- and vector-index rows, recording progress in the + // same write; `reapplying` lets MongoDB suppress E11000 on the one unrecorded file a resume meets + fn finish_restore<'a>(&'a self, target: &'a RestoreTargetKey, claim: &'a dyn JobClaim, + item_count: u64, size_bytes: u64) -> BoxFuture<'a, Result>; + // Finished { Active, Pending (MongoDB: the worker flips the table), NotMatched(reason) } + fn count_restore_target<'a>(&'a self, key: &'a RestoreTargetKey) -> BoxFuture<'a, Result>; + fn read_restore_progress<'a>(&'a self, key: &'a RestoreTargetKey) // default Ok(vec![]) + -> BoxFuture<'a, Result, StorageError>>; + fn sweep_restore_progress<'a>(&'a self, running: &'a [String]) // default Ok(0) + -> BoxFuture<'a, Result>; + + // Restore job record on BackupCatalog (section 4). PR 9. A target is addressed by RestoreTargetKey + // (account_id, table_name, table_id); the table id guards a name re-created under it. + fn claim_restore<'a>(&'a self, account_id: &'a str, input: CreateTableInput, + registration: &'a RestoreRegistrationRow) + -> BoxFuture<'a, Result<(TableDescription, Box), StorageError>>; + // 5.3 step 4's registration: create the CREATING target and the RUNNING record under the backend's + // own generation (registration.owner_generation is UNOWNED_GENERATION), the re-check under the + // backup_index row's lock that it is AVAILABLE and not in use (BackupNotFound / BackupInUse else); + // returns the description the request answers + fn claim_restore_dead<'a>(&'a self, key: &'a RestoreTargetKey, stale_after: Duration, + spool_host: Option<&'a str>) + -> BoxFuture<'a, Result>, StorageError>>; + // the restore analogue of claim_dead_backup: take over a RUNNING manifest record whose heartbeat is + // stale, writing the claimant's spool_host, a fresh generation and heartbeat_at = now() in one write (4(e)) + fn read_restore_job<'a>(&'a self, key: &'a RestoreTargetKey) + -> BoxFuture<'a, Result, StorageError>>; + fn list_restore_jobs<'a>(&'a self, status: Option) + -> BoxFuture<'a, Result, StorageError>>; + // remove_restore_target and fail_restore_job are PR 6's, used by the job's and the reconciler's failure + // paths; RestoreJobRow carries store, spool_host, file_index, items_applied, failure_reason, completed_at. ``` -`count_restore_target` exists because a CREATING table accepts no reads (5.3), so the count of section 1 cannot be a Scan. The job holds the same claim as a backup job (5.1 step 3), keyed `/
    ` on PostgreSQL (the key and seed #384's `try_restore_lock` already uses, so a legacy restore and a new restore into one name exclude each other through the lock as well as the name), the row itself on MongoDB and Cassandra, and heartbeats on the same timer. -The spool and the resume (5.3 step 3): A file that verifies gets a `.verified` marker in the spool, written after the file's `fsync` and renamed into place, so a spooled file without its marker (a crash mid-download) is re-downloaded on resume. The spool is per instance: the `restore_jobs` row records `spool_host` as `:`, a successor whose `spool_host` differs has an empty spool and downloads everything again (its claim writes its own `spool_host` in the conditional write that takes the row, so the sweep of section 4 never removes the spool of the job that owns the row), and two instances must not share a `spool_path` (startup holds a `.lock` file in the spool with an exclusive `flock`, the serve-lock mechanism SQLite already uses, and refuses a spool whose lock another live process holds). A resume with the same `spool_host` reads the manifest and sidecar from the spool; the sweep of section 4 removes only entries of this `spool_host`. - The restore spool holds plaintext items while a restore runs; it is created `0700` with `0600` files, lives outside the store root, and is removed when the job ends. -The load (5.3 step 5): PostgreSQL: multi-row INSERT in 500-row batches with the GSI, LSI and vector-index rows in the same transaction, not through the propagation queue (section 3), one data transaction per file. The catalog and the data live in different databases, so the per-file progress record is a row in the data database (`restore_progress(table_id, file_index)`, data migration 005, reserved for PR 9a next to the catalog numbers in PR 1 and PR 6), written in the file's own transaction; the catalog job row holds only the terminal status. A file is therefore applied exactly once. SQLite: 10k-item batches, each its own transaction holding the batch's index rows, with the progress record `(file_index, items_applied)` committed in each batch's transaction; MongoDB: `insert_many(ordered=false)` per file, the progress record written after the insert, a resumed restore re-applying the one file whose record is missing with E11000 duplicate-key errors ignored on that file only (any other write error fails the restore), then `countDocuments` checked against the cumulative `item_count` (section 1); Cassandra: #384's restore write path (`put_restored_item`, eight in flight) in 10k-item batches, the progress record written after each batch with a lightweight transaction conditioned on the job's `owner_generation`, so a crash re-applies at most one batch, which is safe because the target accepts no other writes and a put of an identical item changes nothing. The table's `restore_progress` rows live in the data database on PostgreSQL (step 5), so they cannot be in that transaction; they are deleted after it commits, and a crash between the two leaves orphan rows for an ACTIVE table, which are harmless because the reconciler reads progress only for a CREATING restore target and removes progress rows whose table is not one (section 4). The other order, progress rows first, would make a crash between the two resume the restore from file 0 and fail a finished restore on the first duplicate key. +There is no `record_progress`: each backend records progress inside `bulk_load`, in the write that applied the items (item 33). `count_restore_target` exists because a CREATING table accepts no reads (5.3), so the count of section 1 cannot be a Scan. The job holds the same claim as a backup job (5.1 step 3), keyed `/
    ` on PostgreSQL (the key and seed #384's `try_restore_lock` already uses, so a legacy restore and a new restore into one name exclude each other through the lock as well as the name), the in-process slot `restore:/
    ` on SQLite (4(e)), the row itself on MongoDB and Cassandra, and heartbeats on the same timer. +The spool and the resume (5.3 step 3): A file is downloaded to `restore-
    /.json.gz.partial` (`` the file's index zero-padded to six digits, `000003.json.gz`), synced, renamed to `.json.gz` and then given its `.verified` marker, created and synced after the rename, so a spooled file without its marker (a crash mid-download) is re-downloaded on resume and one with it is read through `reader_prehashed`. `.verified` holds the file's sha256; a file is verified for a run only while its marker names the digest the manifest in hand records, so a replaced manifest has no old file accepted. A spool entry is a job's to reuse only while the record named this host before the job's claim: a resume whose record moved to another host and back removes its own old entry before use, and a job that learnt the record is another's (its claim lost, a DONE or FAILED write that matched no record) leaves its entry to the sweep rather than removing it from under the record's owner. The spool is per instance: the `restore_jobs` row records `spool_host` as `:`, a successor whose `spool_host` differs has an empty spool and downloads everything again (its claim writes its own `spool_host` in the conditional write that takes the row, and the sweep of section 4 lists the spool before the records, 5.3 step 7, so it never removes the spool of the job that owns the row), and two instances must not share a `spool_path` (startup holds a `.lock` file in the spool with an exclusive `flock`, the serve-lock mechanism SQLite already uses, and refuses a spool whose lock another live process holds). A resume with the same `spool_host` reads the manifest and sidecar from the spool; the sweep of section 4 removes only entries of this `spool_host`. The restore spool holds plaintext items while a restore runs; it is created `0700` with `0600` files, lives outside the store root, and its entry is removed at every end of the job but the shutdown's, which keeps it for the resume, and the ends at which the job learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record), which leave it to the sweep because a same-host successor may be reading it (5.3 step 7). +The load (5.3 step 5): PostgreSQL: multi-row INSERT in batches of 500 rows or 4 MiB of item text, whichever is reached first, with the GSI, LSI and vector-index rows in the same transaction, not through the propagation queue (section 3), one data transaction per file. The catalog and the data live in different databases, so the per-file progress record is a row in the data database (`restore_progress(table_id, file_index, items_applied)`, migration 7 (a StoreKind::Data body on PostgreSQL only; NotApplicable on SQLite, MongoDB and Cassandra)), written in the file's own transaction; the catalog job row holds only the terminal status. A file is therefore applied exactly once. On PostgreSQL a file's `restore_progress` row is a plain INSERT when the file is applied from its start and an upsert on `(table_id, file_index)` when it is resumed past its start, so a row a decoder panic left short of the file's item count (its progress committed, the job's FAILED write lost to a second fault) is completed by the resume rather than failing the restore on the row's primary key. SQLite: batches of at most 10,000 items or `RESTORE_BATCH_BYTES` (32 MiB) of serialized item text (the batch closes at the first item that takes it past the bound), each its own `BEGIN IMMEDIATE` transaction under the engine write lock holding the batch's base, secondary-index and vector-index rows, with the progress record `(file_index, items_applied)` committed in each batch's transaction conditioned on the generation and on RUNNING (a record another owner took rolls the batch back); MongoDB: `insert_many(ordered=false)` in 1000-document calls at `w: majority` per file, base documents only (the backfill worker fills the index collections), the record's `file_index` written after the insert conditioned on the generation and on RUNNING, a resumed restore re-applying the one file whose record is missing (`reapplying`) with E11000 duplicate-key errors ignored on that file only (any other write error fails the restore), then `count_documents` checked against the cumulative `item_count` (section 1); the re-application needs an earlier file's record, so a crash before the first file's record (anywhere in the first file) is not re-applied: the resume finds written items and no progress and answers `target_not_empty` (5.3 step 7, `extenddb backup remove-target`); Cassandra: each item's base row and every GSI and LSI row in one logged batch at a write timestamp read before the batch's claim check (`put_restored_item_at`, no lightweight transaction, eight in flight; the timestamp is the fence: a batch a stalled writer lands after a takeover carries a timestamp below every write and delete its successor made, so Cassandra keeps the successor's cells and tombstones; the base row's `version` is the timestamp, so a later ordinary write's version is above it, ADR-0027), in batches of at most 10,000 items or `RESTORE_BATCH_BYTES` (32 MiB) of serialized item text, the progress record written after each batch with a lightweight transaction conditioned on the job's `owner_generation` and on RUNNING, so a crash after the first batch's record re-applies at most one batch, and a crash before it leaves the first batch's items with no progress, resumed with the first file re-applied (step 7), which is safe because the target accepts no other writes while CREATING and a put of an identical item rewrites the same base and index rows: a PutItem or UpdateItem of an item whose secondary-index keys do not change rewrites its GSI and LSI rows rather than deleting them (#424: the write path's per-index delete-and-insert in one LOGGED BATCH shared the batch's timestamp, and a tombstone wins a tie, so the two statements left the row deleted, GitHub issue #742; the write path skips the DELETE when the old and new items produce the same index row and lets the INSERT rewrite it, and stamps every index row with its base row's `version`, ADR-0027, so a re-applied restore batch, a resume re-applying at most the one batch it was in, rewrites the same base and index rows under a later stamp and a Query on each GSI sees every restored item as soon as the target reports ACTIVE); a crash before the first batch's record (in the first 10,000 items or 32 MiB) leaves the batch's items with no progress, and the resume, finding no more items than the backup's total, re-applies the first file with the same upserts (5.3 step 7; a count above the total is `target_not_empty`). The table's `restore_progress` rows live in the data database on PostgreSQL (step 5), so they cannot be in that transaction; they are deleted after it commits, and a crash between the two leaves orphan rows for an ACTIVE table, which are harmless because the reconciler reads progress only for a CREATING restore target and removes progress rows whose table is not one (section 4). On PostgreSQL the `restore_progress` rows are deleted on the data pool after the ACTIVE flip has committed; that delete cannot fail the restore (its error is logged and `RestoreTableFromBackup` still finishes ACTIVE), because the table is ACTIVE and the record DONE once the flip commits, and the reconciler's `sweep_restore_progress` removes any rows the delete left, exactly as it does after a crash between the flip and the delete. The other order, progress rows first, would make a crash between the two resume the restore from file 0 and fail a finished restore on the first duplicate key. State transitions it owns: RUNNING at registration (5.3 step 4); RUNNING to DONE with the ACTIVE flip, on MongoDB DONE then the worker's flip (5.3 step 6); RUNNING to FAILED with the target removed through `remove_restore_target` only after the FAILED write matched (5.3 step 7); the resume of a dead RUNNING row, on PostgreSQL only after the stale window (5.3 step 7). Kill points (testing summary): every restore kill point, the fenced-out restore job and the `pg_terminate_backend` case; SIGTERM during a restore. Main tests that must pass unchanged: #384's override tests (`crates/engine/src/backup/tests.rs`), `table_class_is_not_restored` and `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused`; the two orderings over a `backup_index` row are PR 9a's own new tests (deferred from PR 6). Documentation rows it corrects: the differences doc's vector-backup row (the restore's refusal narrowed to a target without vector support) and its multi-part-key row (restored, 5.3 step 2); the differences and limits docs' override rows stay (5.3). ### PR 9b: cross-backend matrix -- PR 9b: cross-backend matrix (twelve cells, nightly: PostgreSQL, SQLite and MongoDB sources into those three and Cassandra), #384's override tests run against the new path per 5.3, MongoDB target refuses a vector manifest, an `N` sort-key value beyond 34 significant digits into MongoDB fails with ValidationException, and a 38-digit non-key number into MongoDB restores intact; the Cassandra-target cells include an unordered backup with a duplicated key. Depends on PR 9a. +- PR 9b: cross-backend matrix (shipped in PR 9): the twelve plain cells (PostgreSQL, SQLite and MongoDB sources into PostgreSQL, SQLite, MongoDB and Cassandra targets), the `--duplicate-key` cells into every target (an unordered backup carrying one key in two files fails into every target, into Cassandra by the count alone) and the `--vector` cells (PostgreSQL and SQLite sources restored into PostgreSQL and SQLite with `SearchVectors` answering, refused before any table into MongoDB and Cassandra), run by `devtools/backup-restore-matrix`: a cell takes a backup on the source server, copies its prefix under a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10; the seed stands in for it until then), so the cell proves the restore and not the adoption, then restores and compares the item sets, the descriptions and the index queries; `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs every cell nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do. #384's override tests run against the new path per 5.3; the MongoDB target refuses a vector manifest before any table, an `N` sort-key value beyond 34 significant digits into MongoDB fails the restore with the write path's ValidationException text as its reason and the target gone, and a 38-digit non-key number into MongoDB restores intact (`tests/test_mongodb_backup_restore.py`). Depends on PR 9a. ### PR 10: `backup sync`, its options, `remove-target`, `migrate-legacy` Goal: the catalog-loss recovery and the maintenance commands of section 4. Its line names the interfaces and carries the acceptance: -- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, `extenddb backup remove-target
    ` (section 4), the SQLite serve lock on every command that writes and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler or sync on either resolves or deletes a row because its own root lacks the bytes; `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. +- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, the SQLite serve lock on every command that writes (`extenddb backup remove-target --account
    `, section 4, shipped with PR 9 rather than here, item 33: a crash before the first progress record on MongoDB or Cassandra leaves a `target_not_empty` target that no other command removes) and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler or sync on either resolves or deletes a row because its own root lacks the bytes; `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. Invariants honoured: 4(c) (store identity, scoping, adoption only through `--adopt-store`), 4(f) (every writing command takes the maintenance claim), 4(g) (what sync adopts and never adopts; the opt-ins), 4(d) (the ARN checked against the path and `source_table`), and 4(a)'s rule that no deletion duty acts on the bytes of a row the operator forgot. Interfaces it adds: @@ -427,7 +459,7 @@ Interfaces it adds: // listing reads each row's `backups_by_arn` row for the rest of the row fn rewrite_store(&self, keys: &[BackupKey], store: &StoreIdentity) -> BoxFuture>; // `--move-store`, PR 10 ``` -`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
    //` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts; the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. +`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
    //` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts (the `backup_index` row PR 9b's matrix seeds by hand is the row this adoption writes); the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. `migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`; a command's take tries the exclusive lock a few times over about 10 ms on its connection, so a job's one-statement shared lock does not refuse it. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, and so is the holder of any maintenance claim row, which such a command takes over whatever its heartbeat says; before the wait it runs the reconciler's dead-owner resolutions itself, as the claim's holder (the cycle takes the held branch without reading the claim back through the zero window) (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 29 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. @@ -436,7 +468,7 @@ Documentation rows it corrects: upgrade manual: on SQLite the backup CLI command ### PR 11: CI and documentation -- PR 11: CI and documentation: drop the `not import_export` filters, the nightly schedule for the matrix, and the documentation per section 10 with RPO and RTO numbers from the PR 8 and 9a runs on a stated host class. Depends on PR 3, 9a and 9b. +- PR 11: CI and documentation: drop the `not import_export` filters; measure the 1M-item restore on MongoDB and Cassandra and the 1M-item vector cell on PostgreSQL and SQLite (the testing summary's two open cells; the nightly schedule for the matrix shipped with PR 9); the documentation per section 10 with RPO and RTO numbers from the PR 8, PR 9 and these runs on a stated host class. Depends on PR 3, 9a and 9b. Documentation it writes (section 10's table): New `docs/manuals/14-backup-and-restore.md`: strategy, RPO and RTO statements per store kind (RPO equals backup interval, no PITR; RTO measured in the acceptance tests and published), configuration, IAM policy example, restore drill procedure, catalog loss recovery with `backup sync`, migrating legacy backups, what a restore preserves and what it does not (the table from the readiness discussion), moving a backup to or from Amazon DynamoDB with ImportTable and export. The IAM and S3-compatible-server material points at `crates/backup-store/README.md` rather than copying it; the admin guide's database names; ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded; the admin and upgrade manuals' backup sections rewritten. Acceptance: the statements the series defers to the deployment guide, each stated in the manual: a filesystem store keeps the same-host failure domain, said in the first sentence of the backup section, with S3 as the production recommendation; encryption at rest is the store's job; a pickup agent runs as the server's user or as root, and a root under a storage administrator's directory, an NFS mount owned by `nobody` or a group-writable parent without the sticky bit is refused; a bind mount or a filesystem without Unix permission semantics is outside what the store can observe; an `AbortIncompleteMultipartUpload` lifecycle rule, and for a versioned bucket the `NoncurrentVersionExpiration` and `ExpiredObjectDeleteMarker` rules; the least-privilege IAM policy example; a PostgreSQL snapshot retains every dead tuple written during the backup and the SQLite WAL grows by the write volume during it, with the sizing figures the manual measures; the claim connection's keepalive bound (about a minute after the loss of the owner's host) and `pg_terminate_backend` on the claim's session as the way to end it sooner; a transaction-mode pooler is unsupported for the claim; the SDK's `trace` logs carry the session token and are not enabled in production; write access to an account's prefix is backup-administrator authority for that account, with per-account IAM prefix policies; MongoDB's `minSnapshotHistoryWindowInSeconds` as the operator setting a slow backup needs; a root that reports `0700` while enforcing nothing. @@ -466,28 +498,28 @@ Phase 3, later (v1.x): - Unit: manifest serde and golden files, key mapping and ARN parsing, store conformance. - Integration, every backend, every PR: round trip with full configuration preservation, composite keys, GSIs and LSIs, vector indexes where supported, legacy fallback, PITR surface. - Consistency: concurrent-writer snapshot test on every backend. -- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with `devtools/backup-kill-points.sh` and its API half `devtools/backup-kill-points.py` (a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store; the clippy job lints the feature so the pause code compiles on every push), not by a pytest cell; the restore kill points extend the SIGKILL harness the migration tests already use. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after a file's transaction commits and before the next file starts; MongoDB between `insert_many` and the progress write (the path that exercises E11000 suppression and the count check); Cassandra between a batch's last write and its progress write; SQLite mid-batch; PostgreSQL between the ACTIVE flip and the progress-row deletion; MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip, where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7; a fenced-out restore job that resumes after a successor claimed the row (its FAILED write matches zero rows and the successor's table survives); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). +- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with `devtools/backup-kill-points.sh` and its API half `devtools/backup-kill-points.py` (a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store; the clippy job lints the feature so the pause code compiles on every push), not by a pytest cell; the restore kill points are run the same way with `devtools/backup-kill-points.py` (every point of every backend runs in the nightly matrix workflow's four `restore-kill-points-` jobs, one per backend, each built with `extenddb-app/backup-test-hooks` and starting its backend as the matrix job does, the Cassandra job building a plain SQLite binary as the source beside its hooks binary: a server built with `extenddb-app/backup-test-hooks`, whose feature chain pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks` so each backend's own pause point compiles in without the server naming a backend crate, a backup of enough items for two data files, a restore paused at the named point by `EXTENDDB_BACKUP_PAUSE`, `kill -9` by PID, a restart, and the outcome asserted: the exact count after the resume, of the base table and of the GSI `by-g` every item of the fixture carries, or the target gone for a failing restore), on every backend the point applies to (item 33 lists the cells as run on the build before this round, when the first reach of the Cassandra point below ended `target_not_empty`); a Cassandra kill point takes its backup on a SQLite source and seeds the target's row as the matrix does, Cassandra's own CreateBackup being in-catalog (section 3); the Cassandra point that pauses before a batch's progress record (`restore_cassandra_before_progress`) is run at its first reach (a crash before the first progress record leaves the first batch's items with no progress; the resume finds no more items than the backup's total and re-applies the first file with upserts, step 7) and at its second (`--second-reach`, the re-application of at most one batch after a recorded one). MongoDB records each batch's `insert_many` and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to ACTIVE rather than `target_not_empty`; only a crash before the first batch commits leaves an empty target, which resumes from the start. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after every file has verified and before the first load (`restore_after_download`; on an S3 store the spool holds every `.verified` marker, the manifest and the sidecar, and no item was written); after a file's transaction commits and before the next file starts (`restore_after_file_commit`); MongoDB after a batch's `insert_many` and its progress commit in one transaction (`restore_mongodb_after_insert_many`, the path a resume continues from the recorded `items_applied`); Cassandra between a batch's last write and its progress write (`restore_cassandra_before_progress`); SQLite mid-batch (`restore_sqlite_mid_batch`); PostgreSQL between the ACTIVE flip and the progress-row deletion (`restore_postgres_before_progress_delete`); MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip (`restore_mongodb_before_worker_flip`), where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7 (`restore_between_failed_and_removal`); a fenced-out restore job that resumes after a successor claimed the row (`restore_fenced_out`: its DONE write matches zero rows, it makes no table access and the successor's table survives with the exact count); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). - Distribution matrix for the filesystem default path: unprivileged local launch, the npm launcher, the container image with and without a volume, `sqlite-memory`. - Cross-backend matrix: nightly. - Live DynamoDB: Phase 2, measured. -- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table). Index contents are not stored in the backup; every index is rebuilt from the items on restore, so the vector cell measures the dominant restore cost. +- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table) in the restore's own terms: on SQLite a table with secondary indexes costs SQLite's page cache for the index b-trees above the no-index figure, an unordered backup costs its page cache across the base b-tree above the key-ordered figure, and the final `COUNT(*)` costs the counting connection's page cache, each bounded by `cache_size` (section 7, the record). The backup side's 1M figures are PR 8's record; the restore's are PR 9a's (`docs/design/15-backup-restore-bench-pr9.md`): 1M items of 200 bytes in key order in 9.5 s on PostgreSQL and 26.8 s on SQLite, the peak over idle 16 and 89 MiB, the growth from 100k to 1M 5.5 MiB on PostgreSQL and 51 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`, not the items; from 1M to 3M rows the SQLite peak moved 11 MiB), and the same 1M items as an unordered backup 16 MiB on PostgreSQL and 144 to 164 MiB on SQLite in 66 to 90 s (SQLite's base b-tree under scattered inserts, not the restore's buffers). The MongoDB and Cassandra 1M restore and the vector cell are still to be measured (PR 11). Index contents are not stored in the backup; every index is rebuilt from the items on restore, so the vector cell measures the dominant restore cost. ## Review checklist for implementation PRs - No data file is uploaded outside a snapshot; the snapshot is opened before the first read and closed after the last. - `extenddb-manifest.json` is written last and only after every other object succeeded. - Every failure path deletes the prefix or leaves it unreachable from List, and the row never says AVAILABLE for an incomplete prefix. -- Restore writes no item before every data file's checksum has verified; a verification failure deletes the CREATING target table, which never accepted a read or a write; the request refuses before creating the table when the target cannot represent the manifest or the key ranges overlap. -- A file's data and its progress record commit in one transaction on PostgreSQL and SQLite (per 10k-item batch on SQLite, with the items-applied count), so no item is applied twice; on MongoDB the progress record follows the insert, re-application of the one unrecorded file is idempotent by duplicate-key suppression (code 11000 only; any other write error fails the restore), and the post-resume document count is checked against the manifest; on Cassandra the progress record follows each batch under a lightweight transaction and at most one batch is re-applied. A key-ordered restore runs the per-file key-range check on every file, and an unordered one ends with a count of the target equal to the manifest's item count. On PostgreSQL the ACTIVE flip precedes the deletion of progress rows; on MongoDB the job's DONE write and `restore_backfill_pending` are one transaction, and the worker's ACTIVE flip is conditioned on that flag and CREATING, not on the job's generation. -- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched. +- Restore writes no item before every data file's checksum has verified once; on a filesystem store the load's second read is hashed again and a file replaced between the passes fails the file, its items removed with the target (5.3 step 3); a verification failure deletes the CREATING target table, which never accepted a read or a write; the request refuses before creating the table when the target cannot represent the manifest or the key ranges overlap. +- A file's data and its progress record commit in one transaction on PostgreSQL (the `restore_progress` row, per file) and SQLite (per batch of at most 10,000 items or 32 MiB of item text, with the items-applied count), so no item is applied twice; on MongoDB the `file_index` record follows the insert, re-application of the one unrecorded file (`BulkLoadFile::reapplying`) is idempotent by duplicate-key suppression (code 11000 only, on that file only; any other write error fails the restore); on Cassandra the progress record follows each batch under a lightweight transaction and at most one batch is re-applied. A key-ordered restore runs the per-file key-range check on every file, and every restore ends with a count of the target equal to the manifest's item count. On PostgreSQL the ACTIVE flip precedes the deletion of progress rows; on MongoDB the job's DONE write and `restore_backfill_pending` are one transaction, and the worker's ACTIVE flip is conditioned on that flag and CREATING, not on the job's generation. +- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched on a backup row and refreshes it on a restore record (4(e)). - The ARN is parsed by `BackupArn` once, in the engine, before any catalog or store access; no other code splits ARN strings. - Account scoping is checked on the ARN before any store access, every backup row is read by account and backup id and accepted only when its account, table and ARN equal the request's, and every restore-target method takes the target's account, name and table id. - The backup job validates every item with `validate_restorable_item` before encoding it, keeps validating after the first refusal, and the restore binds the same function. -- A restore's registration re-checks, under the backup row's lock, that the row is AVAILABLE and not in use, and DeleteBackup's in-use check and DELETING write run in one transaction under the same lock. -- A failed restore's target is removed through `remove_restore_target`, never DeleteTable; on PostgreSQL and SQLite `bulk_load` writes index rows in the batch's transaction, with the base rows, and the ACTIVE flip sets every index of the target ACTIVE. +- A restore's registration re-checks, under the `backup_index` row's lock (5.3 step 4 names it per backend), that the row is AVAILABLE and not in use, and DeleteBackup's in-use check and DELETING write run in one transaction under the same lock. +- A failed restore's target is removed through `remove_restore_target`, never DeleteTable, and only after the FAILED write matched one record (`fail_restore`); on PostgreSQL and SQLite `bulk_load` writes the secondary- and vector-index rows in the batch's transaction, with the base rows, and the ACTIVE flip sets every index of the target ACTIVE. - No API operation or job touches the store while it is unhealthy, the reconciler acts on no row of the store then (its legacy duty, which touches no store, still runs), and only sync with a resolving option and the health check itself read it. - No credential, bucket name with credentials, or presigned URL appears in a log line, and no API error names a store root. -- New workers stop on the shutdown signal within the drain window: a backup runs its failure path, which the server's shutdown awaits, and a restore stops with its row left RUNNING. +- New workers stop on the shutdown signal within the drain window: a backup runs its failure path, which the server's shutdown awaits, and a restore stops with its row left RUNNING and its spool kept. - Tests create composite-key tables, not only hash-only tables. ## Drawbacks @@ -557,3 +589,5 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 31. (2026-10-31, PR 8 as implemented.) The job and the reconciler live in the engine's backup module (`crates/engine/src/backup/{job,reconciler}.rs`), not in `crates/server/src/workers.rs`, and nothing crosses an in-process channel between the request and the job but the response (the pipeline's item and chunk channels are the job's own): the request's detached task opens the snapshot, claims the row, checks the maintenance claim and answers through a one-shot channel, then continues as the job; the server hands the engine the shutdown token, the metrics collector, the job settings and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, and `drain_jobs` (30 s) runs before the workers drain (PR 8); a stopped pipeline's sink answers the encoder's write with `BrokenPipe`, which nothing retries, where an `Interrupted` answer (std's "try again") kept a job cancelled in the middle of a 300k-row table spinning in the compressor's trailer write through the whole drain with its row left CREATING; the job ends within milliseconds of the signal on every store (5.1 step 5, PR 8). The reconciler's takeover is `BackupCatalog::claim_dead_backup(key, stale_after)`, not `claim_backup`, which inserts; `list_backup_rows_older_than(status, store, clock, age)` with `StaleClock::{Heartbeat, Completed}` carries the database's clock as a backend predicate for the FAILED-prefix delay and the tombstone retention, so the engine never holds a clock; `sweep_restore_registrations` is the Cassandra `active_restores` sweep with a default empty body; `StatusFields` gains `source_table_details_json`, which the job's and the reconciler's AVAILABLE writes fill with the manifest's `source_table` (4(e), PR 8). The PostgreSQL page is three layers, since a window function runs before `LIMIT`, and the stream ends when the inner query returned fewer rows than asked and none was cut (a page cut short of the row bound is not the end); the outer `ORDER BY` is a stated guarantee the executor gives anyway; SQLite's stream pulls no row past its cut (the driver's worker reads at most five rows ahead of it: the four `row_buffer_size` on the pool buffers and the one it holds while its `send` waits), so the rows-past-cut figure is PostgreSQL's alone (section 3, PR 8). `heartbeat_interval_secs` reaches the store-backed job as `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`; the in-catalog paths keep the constant until PR 14, which corrects item 27's "until the knob reaches the jobs" (5.1 step 4, section 7). The job's pre-put check is two periods, not one (a beat sent at the tick returns up to a period later), which stays inside the three-period floor; the check measures from the instant taken before the claim's insert, not from the heartbeat handle's start, so an unbounded maintenance check (a catalog query held behind the pool) cannot admit a first put on a row already past `stale_job_after`; when the pipeline returns no put is in flight, so the failure path's `delete_prefix` follows the FAILED write with nothing to await; a lost or stalled heartbeat ends the job with no catalog write, the claim connection being the one that failed; each unknown outcome of a status write is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row; within one cycle the duties run dead-owner first and FAILED-prefix after it, so a row the cycle just failed loses its prefix in that cycle once the dead owner's beat is stale (5.1 steps 4 to 6, 4(e)). SQLite's snapshot slot is kept as the backend's admission bound and the pool raised to `2 * (max_concurrent_jobs + 1)`, at most 34, which is what item 27's "folds it into `max_concurrent_jobs`" became (section 7, PR 6). On SQLite `claim_dead_backup` reads no window: a row whose slot no task holds is dead at once, and the engine calls it on every CREATING row each cycle (one set probe per live row); the dead owner's handle cannot observe `NotMatched` after a takeover on SQLite or PostgreSQL, whose slot and lock are the owner's life, and the engine reaches a taken-over row through `Unknown` and the re-read (4(e)). On MongoDB and Cassandra a successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row in that gap; the generation fence makes the loser's write `NotMatched` and one resolution lands; Cassandra's takeover condition carries the heartbeat it read (`IF owner_generation = AND backup_status = 'CREATING' AND heartbeat_at = `), which puts the liveness test inside the write; Cassandra's `list_backup_rows_older_than` reads the coordinator clock before its scan where `list_dead_restore_jobs` reads it after, both conservative; MongoDB's `Completed` listing requires the column present, since `age_is_stale` alone reads an absent column as the epoch (4(e), PR 8). MongoDB's stream applies no cut of its own (the server's 16 MiB reply bound is under the budget) and feeds no page-bytes counter, so `run_page_bounds` runs on PostgreSQL and SQLite only (section 3, PR 8); Cassandra's registrations sweep is one range read of `backups_by_arn` per cycle, enough while a catalog holds thousands of backups and not more, and the `backup_size_bytes` of `backups_by_account` and `backups_by_table` is written at the claim and not refreshed by `set_status`, which the index listings do not read (they re-read the row), as before PR 8 (PR 6). The spool sweep and the `restore_progress` sweep land with PR 9a, which adds the records they read, and the duty that removed a CREATING restore target with no job row is withdrawn: the job record is the registration on every backend (section 4, PR 8). The vector "round trip" PR 8 adds is its create half, the restore half being PR 9a's; the vector wire test asserts the store's ValidationException on the restore until then, and the fidelity suite's 24 restore cells skip on a server with a store (PR 8, PR 9a). `source_table.time_to_live` is read beside the open, outside the snapshot, and may trail it by the open's duration; DescribeBackup's `ItemCount` is the stats pass's figure while CREATING and the stream's count once AVAILABLE (5.1 step 2). Section 9 lists the metrics as `/metrics` emits them, the `MetricName` variants PR 8 adds (`BackupJobs`, `BackupJobDuration` in microseconds, `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors`, the outcome and the store call in the operation dimension) as JSON under those names, and says the snake-case spellings are a converter's names and not a rendering the server does (section 9). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the pytest jobs and the Rust integration jobs (`filesystem` on PostgreSQL, which appends a store after `init`, and on SQLite, the defaulted root; `s3` on the SQLite job against moto; `none` on MongoDB and Cassandra and on the PostgreSQL Rust integration jobs; `run-integration-dev-mode` runs `tests/test_dev_mode_authorization.py` alone, which creates no backup, without it) and `devtools/run-tests --pytest` refuses without it (section 10, PR 8). PostgreSQL's snapshot-slot refusal now goes through `limit_exceeded_message` naming `[storage.postgres] pool_size snapshot-slot` (item 26's one-implementation rule). The acceptance's "on each backend" for the 100k backup reads "on each backend with a `SnapshotEngine`", Cassandra keeping the legacy create, and for the #618 cells "on PostgreSQL and SQLite", the two deployments `tests/test_backup_job.py`'s fixture starts (the MongoDB deployment and its CI job came with the review, revision item 32); an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`, and UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone (PR 8). The 1M-item RSS runs and the kill points were measured on a 16-core, 30 GiB host (PostgreSQL 16.15 and MongoDB 7.0 in containers on loopback, the bundled SQLite 3.46) and the figures recorded with the PR in `docs/design/15-backup-restore-bench-pr8.md`; the memory bound holds (the 1M run's peak within 10 MiB of the 100k run's on PostgreSQL and SQLite, and the 500k run's within 5 MiB on MongoDB, where the 500k load alone took 27 minutes through the wire on that host; the bound is 112 MiB). Section 7's memory paragraph is rewritten to the terms the worst-expansion run measured: the earlier "the budget plus one row of text, plus its decoded form" left out the page's decoded rows waiting for the pipeline, the items in the feeder, the item channel and the encoder, one row's parse through the deserializer's intermediate value, and the line buffer, and it said nothing of the allocator; with 27.7 MB rows (three to a page) the terms sum to about 400 MB and the high-water mark over idle was 880 MiB on PostgreSQL and 580 to 760 MiB on SQLite across runs (507 MiB under `MALLOC_ARENA_MAX=1`), with MongoDB measured at 474 MiB on 13.5 MB rows, its 16 MiB document cap refusing the full shape; SQLite's driver run-ahead was bounded (sqlx's `row_buffer_size`, 50 rows by default, set to four on the pool: the 24-row run peaked at 844 MiB over idle with 50 and 696 MiB with four, back to back, and a 1M-row backup took 11.4 s with either), and whether the peak then grows with the number of worst rows is settled by the 12, 24 and 48-row series of the 2026-11-01 sentences below (it does with glibc's default arenas and does not under `MALLOC_ARENA_MAX=1`), and a one-item channel saved about 50 MiB of it for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders), so `ITEM_CHANNEL` stays at four. The PR 8 acceptance's "stays within section 7's formula" and "RSS must match the formula" read "under 1.2 times the sum of section 7's terms at the measured row with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one binary, the default-arena peak recorded beside it" (section 7, PR 8). The body sentences each of these corrects are edited with this item (4(e), 5.1 steps 2, 4, 5 and 6, section 7's knob comments, memory paragraph and pool paragraph, section 9, section 10's table, PR 6's slot sentence, PR 8's acceptance, interfaces, page paragraph and duty list, PR 9a's lead-in); earlier items keep their wording, as item 22 requires. (2026-11-01, PR 8 as measured and corrected.) The PostgreSQL page's row bound `n` is sized from the previous page's largest row, not its average, and the floor after the first page is 4 (`MIN_NEXT_PAGE_ROWS`; `MIN_PAGE_ROWS`, 16, is the first page's alone), because the server serializes and spills every one of the `n` rows the inner `LIMIT` reads, so a page costs the server `n` rows of text and not the rows it returns: after a page of 984 small rows and 3 large ones the average sized the next page at 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned, where the largest row sizes it at 4 (208 MiB of temp, 1.1 s), and a page of 27 MB rows now reads one row past its cut rather than thirteen (its steady state 1.2 s a page, 4.4 s before); the over-read that remains is the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, which the sentence of the same day below bounds (section 3, 7, PR 8). The `EXPLAIN` acceptance holds for a page with the planner's estimate above `n` and not everywhere: on a table of 1,060 rows a page near the end whose `n` is above the rows left gets a bitmap scan of the primary key index and a `Sort` of the rows after the seek under the `Limit` (PostgreSQL 16), while on the 100k-row table the index scan holds ten rows from the end; both are pinned by a test and stated in the module doc (PR 8). PostgreSQL's `fetch_page` pulls the page's rows from the connection one at a time and parses each as it arrives, so the page's text is no longer a term of section 7 (the record's 83 MB at the worst shape); the item channel and the decoded page are unchanged; with that change the PostgreSQL worst-shape run peaked at 790 MiB (`VmHWM`) in 16.5 s against the head's 905 MiB in 18.2 s, and its 1M run's peak was 4.0 MiB over the 100k run's (section 7, PR 8). `run_page_bounds` fails at once, naming the key yielded twice, when a stream has yielded more items than the table holds, so an inclusive seek fails in 3 s where it ran until another test's lock wait gave out (PR 8). SQLite's run-ahead is five rows, not four: sqlx's worker steps and copies the next row before its `send` on the four-row channel blocks, 160 MiB at the line cap (section 7, PR 8). The allocator: the peak over idle of the worst-expansion run grows with the number of large rows under glibc's default arenas and is flat under `MALLOC_ARENA_MAX=1`, which is documented for operators (the admin guide's `[backup]` rows, the deployment guide's service unit, section 7) rather than set from the binary; section 7's memory paragraph is the sizing rule (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite, times the allocator's factor, times `max_concurrent_jobs`), measured at 27.7 MB rows, 1,000 small rows first, 12, 24 and 48 such rows back to back, in two series of six runs each on one SQLite binary: 466, 588 and 760 MiB over idle with the default arenas and 481, 452 and 480 MiB under `MALLOC_ARENA_MAX=1` on a quiet host, then 707, 700 and 1,281 MiB and 452, 480 and 455 MiB on the final build with the kill-point runs beside it (the terms' sum at that row 545 MiB; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows), which replaces this item's earlier "the peak did not move between 12 and 24 worst rows" (section 7, PR 8). The #618 cells run on PostgreSQL and SQLite; the RSS acceptance states its 112 MiB bound and MongoDB's 500k run; the testing summary says the backup kill points are run by hand with the posted script, a `before_manifest` point added (the three documents and the data file under the prefix, no manifest, FAILED `owner dead` and the prefix deleted as at `after_data_file`) and the SIGTERM exit timed by polling (55 ms on SQLite, not the script's 5 s sleep); 5.1 step 1 names the store-health check in its order, before the permit; the record is committed as `docs/design/15-backup-restore-bench-pr8.md` and cited by that path, with its SQLite cache explanation corrected to `cache_size = -32000` (31 MiB per connection, `sqlite_util.rs`) and its two 12-row rows labelled by build; the differences document's backup-creation figures are the record's and its restore-interruption row says the prefix of a SQLite row goes once the dead owner's heartbeat is stale, not in the first cycle; the usage guide's backups row says RestoreTableFromBackup restores in-catalog backups only this release and the upgrade manual says a store-backed backup is listed, described and deleted but its restore answers the ValidationException until the restore job lands (section 5.1, 7, 10, testing summary, PR 8). The harness: `tests/test_abac.py`'s backup cell waits for AVAILABLE before DeleteBackup; the job-bound cell's two diagnostics say what each case is; the fidelity suite reads `/health` once and checks that the server's `backup_store` component is the kind `EXTENDDB_TEST_BACKUP_STORE` names and healthy (absent for `none`), the `none` branch of the vector cell captures the ARN of a create accepted against expectation, `_failed_rows` decodes Rust's `Debug` escapes, a cell pins the whole key of a refused item with a UUID hash key and a ten-digit `N` range key in the log line and in `backup list --failed`; the CI start steps of the PostgreSQL, SQLite and S3 jobs wait for the `backup_store` component to be healthy, the pytest lines carry `-rP` so a passed cell's printed branch reaches the log, and the S3 job dumps moto's log on failure and names what its by-name step covers (PR 8). Also: `validate_restorable_item` takes `&RestorableLimits` (the item size, hash key size, range key size and attribute name size of `LimitsConfig`), not `max_item_size_bytes` alone, so the restore's "every item against the target's limits" (5.3 step 5) is the same call; every untrusted text its error carries (a key value, an attribute name, a path segment, the number the wire's reason echoes) is bounded on its own at 4096 bytes of the text the message writes (a key value at 12,288, six times the hash key limit, so a key of control characters is whole too; 4096 is a round figure that keeps a path segment to four times the longest key attribute name), with a 1024-byte prefix and the length for a longer one, so a key of any legal size appears whole in the log line and the FAILED reason; every such text, the attribute path and the echoed number included, is written as a JSON string's content, with U+0085, U+2028 and U+2029 escaped as `\u0085`, `\u2028`, `\u2029` beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log, for a byte-oriented reader and for one that breaks lines the Unicode way (Python's `str.splitlines`) alike; an item whose data file line would exceed `MAX_LINE_BYTES` under a raised `max_item_size_bytes` is refused by the job's feeder through the same count-and-log path (`RestorableItemReason::LineTooLong`, whose reason names the cap, the line's own length being unknown once the count stops), the line measured on the blocking pool for items over the default item limit only, through a counting writer that holds no line, and such an item's validation (the number rule over every member, CPU work proportional to the item's size) run on the blocking pool with it, an item within the default validated on the feeder's own task, so the encoder's own refusal is never the first to see it; a put that fails while the snapshot stream is between pages ends the pipeline at once (the uploader cancels a token the feeder selects on), so the FAILED write does not wait for the stream; the completion log line carries the stream's item count; a job the shutdown cancelled records `BackupJobs{cancelled}` and neither `WorkerLastSuccess` nor `WorkerErrorCount`; `BackupJobDuration` is microseconds from the snapshot's open to the end of the job's last catalog or store call (the AVAILABLE write, or the FAILED write and the prefix's deletion); the reconciler's `CycleReport` counts a dead-CREATING resolution whose write matched no row as left to its owner (`creating_left_to_owner`), and its cycle line counts the dead legacy restores and the FAILED-record targets apart; `reconciler.rs`'s module doc states the FAILED-prefix duty's timing as the code has it, in this cycle on PostgreSQL, MongoDB and Cassandra and once the dead owner's last beat is stale on SQLite (5.1 steps 4 and 5, section 9, PR 8). (2026-11-01, PR 8 as measured and corrected, the row bound's memory.) The sentence above that the over-read happens "once" was false on a table whose large rows are scattered among small ones: with the bound sized from the previous page alone, a large row set the next bound to 4, four small rows set the one after back to 1000, and that page met large rows again, so the 1000-row page recurred every other page for the whole table (5,555 rows read for 1,000 kept, 13.4 GB of server temp files, on a 1,000-row table with 30 such rows; on the release binary the same bytes took 1.9 times as long and 3.2 times the temp files by key layout alone). The bound is now sized from the largest row of the last pages that together hold at least a row bound's worth of rows (`RowBoundMemory`), so it stays at the floor while large rows keep arriving within that span and returns to the row bound a row bound's worth of small rows after the last large one: 1,901 rows read for 1,000 kept and 3.9 GB on the probe (a memory of four pages read the same 5,555 as one page, since four pages of four rows are sixteen rows), and the interleaved table's backup 54.6 s and 3.2 GB of temp against 72.0 s and 9.2 GB, the grouped table unchanged. The bound that remains, stated as it is: the page that follows a row bound's worth of small rows reads up to `n` rows of up to the line cap, at most once per row bound's worth of small rows, and nothing in the window form stops the server at the budget. A page query that does stop there, a recursive walk of the primary key one index probe per row, was measured and not taken: it read exactly the rows it returned on every shape (the interleaved table in 38.8 s with 2.5 GB of temp) and cost one probe per row, 23.1 s against 11.9 s for the 1M-row backup; the record carries both. `the_row_bound_remembers_large_rows_among_small_ones` pins the memory's figures on a 300-row table with nine such rows (201 rows read past the cut over 53 round trips), `the_page_query_is_an_index_scan_on_every_key_shape` pins the index scan on the six key shapes (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation, each at `n` 16 and 1000, first page and a page after a key, and the stream over 20,000 rows), and `page_sql_for` takes the collation flag `open_stream` decides, so the legacy table's query is tested as the engine writes it (section 3, 7, PR 8). Section 7's rule is scoped: the terms are at the default `max_item_size_bytes`, where no stored item's line exceeds the cap, and on an instance that raises it they are at the largest line a stored item expands to; the feeder's measurement of an item over the default limit holds no line (a counting writer that stops at the cap), where the earlier text said its buffer was bounded by the cap and the code held the whole line (section 7, PR 8). The one-arena factor is measured on both backends: the PostgreSQL worst shape at 12 rows peaked at 340 MiB `VmHWM` under `MALLOC_ARENA_MAX=1` against 750 to 788 MiB under the default arenas, 0.82 of the terms at that row on PostgreSQL (413 MiB, no run-ahead), and the setting's cost on the request path is measured: 3.5 to 9.2% on SQLite (GetItem, Query, Scan, paged Scan) and about 2% on PostgreSQL (the scans, 2.1%), so the deployment guide's service unit carries the line commented out with the condition and the cost, and section 7 states both (section 7, PR 8). 4(e)'s unknown-outcome rule reads as `write_status` has it: each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read that shows the row in the target status counts as matched, anything else as matching no row; the review checklist's bullet says the same (4(e), PR 8). The PR 8 duty list marks the resume of a dead RUNNING restore as PR 9a's, this release's reconciler logging such a record once; section 10's table gains PR 8's rows (the admin guide's `[backup]` rows and `[backup.store]` section, the deployment guide's service unit, the upgrade manual's asynchronous-create and rollback paragraphs, the design README's row 15b); the PR 9a lead-in names `RestorableLimits`; the ratios are given to two places (0.86 to 2.35; 2.35 at 48 rows); the SIGTERM exit is 55 ms in every statement; the record attributes the PostgreSQL worst run's idle to the restarted process's startup work on a table of 27.7 MB rows, not to the 1M load (sections 4, 7, 10, PR 8, PR 9a). (2026-11-01, PR 8 as corrected.) Section 3's parenthesis on the PostgreSQL cut, the PR 8 bullet, its two paging acceptance cells and the Interfaces paragraph's `n` sentence described the recursive walk that was measured and not taken (one row per step, no row read past the cut, two index probes) and the rule before the memory (`n` from the previous page's largest row, the over-read "once"); each is corrected to the shipped form: the three-layer window page with up to `n - 1` rows read past the cut, `n` sized from the memory, the first paging cell naming the record's `EXPLAIN (ANALYZE, BUFFERS)` figure (4 read, 1 past the cut at the floor) and `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips), the `EXPLAIN` cell the plans the tests pin (`Limit` over one `Index Scan`; the bitmap scan with a `Sort` near the end of a small table) (section 3, PR 8). The over-read's statement gains the second page, which follows the first page's 16 rows and asks for the row bound: the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, each up to `n` rows of up to the line cap, said the same way in the PR 8 page paragraph, the module doc, the test's doc and the record (the earlier sentence of this item said "the page that follows a row bound's worth of small rows" alone); the floor's cost is stated beside it: after a page holding a row at the line cap the bound stays at the floor for a row bound's worth of rows, 250 pages of four, whatever those rows hold, so a table whose large rows are rarer than one per row bound pays 250 round trips per large row where one page of memory pages it at the row bound (2,585 page queries against 38 on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, 1.06 against 1.09 GB of temp; the record), and the record's "changes no page of a small-row table" reads "of a table with no large row"; the one-page-memory figure the test's doc, the record and the PR text cite is 306 rows past the cut over 7 pages (section 7, PR 8). 4(e) said the PostgreSQL reconciler "reads the heartbeat alone and never probes the lock" and named "the one liveness rule the three share": PostgreSQL's `claim_dead_backup` tries the advisory lock before it tests the heartbeat, so it takes over only a row whose heartbeat is stale and whose lock is free; a stopped owner (SIGSTOP, a paused VM) whose session lives keeps its row until the session ends or it finishes, and a vanished host's lock is released when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two; 4(e) and 5.1 step 4 now say so (the SIGSTOP sentence scoped to MongoDB and Cassandra, with the PostgreSQL outcome: no takeover while the session lives; on resume the job completes under its own generation or, when the stop outlasted two heartbeat periods, stops as a lost claim with no catalog write and the reconciler resolves the row), the trait doc of `claim_dead_backup`, the admin guide's `stale_job_after_secs` row and the differences document's dead-owner clause gain "on PostgreSQL, once the owner's claim session has ended as well", and item 27's "the heartbeat alone" stands for PR 6's `remove_dead_creating_backup_row`, which `backup purge` still uses (4(e), 5.1 step 4, PR 8). The one-arena cost on PostgreSQL reads "about 2% (the scans, 2.1%)" in section 7, this item, the admin guide, the deployment guide and the PR text, the record's scans being −2.1%; section 7 labels the base of each ratio (the peak over idle on SQLite, the `VmHWM` on PostgreSQL, about 0.47 over idle); the admin guide's `max_concurrent_jobs` row restates section 7's sizing rule (1.2 times the terms per job under `MALLOC_ARENA_MAX=1`, no bound measured under the default arenas, the PostgreSQL figure as `VmHWM`, 2.35) and its `stale_job_after_secs` row the dead-row outcomes (AVAILABLE with a complete manifest, FAILED without, a restore job's target removed); the upgrade manual's rollback paragraph names `backup sync --adopt-store`, as 4(c) requires (section 7, 10, PR 8). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the Rust integration jobs included: the SQLite Rust integration job's file-backed server has the defaulted root, so its CreateBackup is the asynchronous one, and `tests/rust`'s backup tests assumed the synchronous create and left their backups; the suite now waits for AVAILABLE, deletes what it makes and reads the variable as the Python suite does, and `run-tests`' usage, `CONTRIBUTING.md` and the developer test guide name the variable and the refusal (section 10, PR 8). The PR 8 kill-point list names `before_manifest`, and section 10's limits-doc row the rewritten "Backup store" row (PR 8). 5.1 step 6 says a job cancelled after its manifest put completes AVAILABLE, the manifest being the commit record, where it said a cancelled job can never complete (5.1 step 6, PR 8). On PostgreSQL a job stopped and resumed after more than two heartbeat periods stops as a lost claim with no catalog write, its session's end frees the advisory lock and the reconciler resolves the row, where 5.1 step 4 said it resumes and completes; the unknown-outcome rule's second write is stated as the rule of a backend whose claim connection survives the outcome (MongoDB, Cassandra), PostgreSQL's failing with the session; the reconciler's failed `get`, `list` and `delete_prefix` calls are `BackupStoreErrors` as the job's are; and `extenddb stop` waits for the sum of the server's shutdown bounds, the 30 s job drain included (4(e), 5.1 step 4, section 9, PR 8). After #335 and #681 merged, PR 8 is rebased onto PR 7's rebased head: the server's shutdown drains the jobs before the registration supervisor's return and the worker set's stop (#666), a job still running at the drain's bound keeps the serving registration as a worker that could not be stopped does, so a migration cannot start beside it, `extenddb stop` waits for the sum of the bounds that path has (76 s: 5 s of connections, 30 s of jobs, 10 s for the supervisor, 16 s for the workers' stop, 10 s for the release, 5 s for the runtime), and the reconciler's withdrawn duty is stated after migration 6 (PR 8). After #433 (attribute-name size on every write path) merged, the Drawbacks line names an attribute name over the limit that `UpdateItem` and a `TransactWriteItems` `Update` stored before it beside the #618 values (Drawbacks, PR 8). 32. (2026-11-04, after the maintainer's review of PR 8.) The heartbeat's one-period bound is placed by the backend (`JobClaim::heartbeat_within`, a provided method that bounds the whole call; SQLite's override takes the engine write lock first and bounds only the `UPDATE`, so a writer holding that lock past a period, a GSI backfill say, delays the beat and the claim re-check and ends no job; the re-check's confirmation counts as a beat for the pre-put check; after the maintainer's second round, the pre-put check also passes while a beat is in flight and has not failed, since a backup large enough to reach a put while the beat was parked on the lock past two periods stopped at that put, the maintainer's 180,000-item backup beside a GSI build twice at the third or fourth data file, and the reconciler then failed the row and deleted its objects; the safety argument is in 5.1 step 4 and `heartbeat.rs`'s module doc, and the tests are a SQLite cell whose second data file is put under a hold of the lock, `a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it`, and an engine cell with a claim whose beat parks, `a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost`), and `heartbeat.rs`'s SIGSTOP text says that a job with no beat in flight resumes under its own generation once its next beat succeeds. The store-backed create checks the maintenance claim before the permit, as the in-catalog paths do, and inserts no row for a refusal (5.1 step 2 and 4(f) are edited: a FAILED `maintenance` row carried the insert's heartbeat and `backup purge`'s wait counted it as a live job for the stale window). Before every put the job checks the store as the runtime holds it (healthy, and its identity the row's) and on a failure fails the row with the store's reason and makes no store call, the prefix's deletion included (4(c)'s swapped-root case). The open and the insert are selected against the shutdown token. The uploader cancels the feeder's token before it awaits an encoder, so a put that fails mid-file ends the job while the stream is held. A data file whose stored size differs from the encoder's compressed count fails the job (the dead-owner probe refuses that difference). The AVAILABLE write sets `TableSizeBytes` from the stream as it sets `ItemCount` (the differences document names both). `failed_row_retention_secs` is at least `stale_job_after_secs` (a FAILED row removed sooner left a stopped owner's late object with no row to delete it under). A registration lost while serving cancels the jobs' token, which the runtime replaces for the jobs that start under the new registration; the shutdown sequence after the listener is one function the server's tests drive with a job that outlives a short drain. The plain-HTTP listener's drain is bounded as the TLS listener's is. CreateBackup is bound to the in-catalog path on SQLite where the serve lock is not taken (not Unix), since the slot claim is sound only with one process per file. 4(c) states the gap of the different-roots guard before the first AVAILABLE row. The compose files' `stop_grace_period` is 80 s. The kill-point scripts are committed under `devtools/` and the clippy job lints `backup-test-hooks`; a MongoDB job runs the backup suites against a filesystem store. The tests added are the review's: the lock held past two periods on SQLite, the mid-file put failure with the stream held, the unhealthy and renamed store mid-job, the key-ordered stream out of order, `BackupStoreErrors` for the job's `delete_prefix` and the probe's `list`, the re-read under another generation, the two-period sample, the nesting rule through lists and mixed levels, the keepalive literals, the driver's five-row run-ahead, the beat between the takeover's read and its LWT on Cassandra, `pool_size_for_jobs(11, 4)`, and the retention floor. After the third round the clause is bounded where the beat does not park before its bound (`JobClaim::heartbeat_parks_before_its_bound`, `true` on SQLite only): a beat in flight for more than a period on PostgreSQL, MongoDB or Cassandra confirms nothing, since a process stopped mid-beat and resumed would otherwise pass the check until its timer raised the loss (`a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks`). + +33. (2026-11-01, PR 9 as implemented — the design's PR 9a and PR 9b as one PR, the maintainer's decision.) The restore job runs from the store on PostgreSQL, SQLite, MongoDB and Cassandra, same-backend and cross-backend, from a filesystem or an S3 store; the matrix and the nightly are PR 9b's, shipped here. `RestoreTableFromBackup` of a `backup_index` row no longer answers the "in a backup store; this release restores in-catalog backups only" refusal `check_catalog_row_for_restore` carried; a healthy configured store's AVAILABLE row is restored, and the refusal is gone from the engine and the differences, usage and upgrade docs (5.3, section 10, PR 9a). The interface: `RestoreEngine`, a new trait in `StorageEngine`'s bound with `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied`, `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }`, `count_restore_target(target) -> u64`, `read_restore_progress` (default `[]`) and `sweep_restore_progress(running) -> u64` (default 0); `StorageEngine` has no methods of its own, so `bulk_load` is not "a new `StorageEngine` method" (section 3's PR 9a interface list). `BackupCatalog` gains `claim_restore(account_id, input: CreateTableInput, &RestoreRegistrationRow) -> (TableDescription, Box)` (the registration creates the CREATING target and returns the description the request answers; the record is born owned under the backend's generation, `RestoreRegistrationRow.owner_generation` being `UNOWNED_GENERATION` and the backend minting its own), `claim_restore_dead(key, stale_after, spool_host) -> Option>` (the restore analogue of `claim_dead_backup`, writing the claimant's `spool_host`, a fresh generation and a fresh `heartbeat_at` on the database's clock in one conditional write, where the backup takeover leaves the heartbeat: nothing waits on a restore record's old heartbeat after a takeover, and a second taker must find the record live at once, 4(e)), `read_restore_job`, `list_restore_jobs(Option)` and `remove_restore_target`/`fail_restore_job` as the reconciler uses them; `RestoreJobRow` carries `store`, `spool_host`, `file_index`, `items_applied`, `failure_reason`, `completed_at`, so the FAILED-target duty reads a row's reason and the sweeps read its store and `spool_host` (section 3, 4, 5.3, PR 9a, PR 9b). The design's `claim_restore(&RestoreJobRow) -> Box` and `record_progress(&dyn JobClaim, file_index, items_applied)` are gone: there is no `record_progress`; every backend records progress inside `bulk_load`, in the write that applied the items (PostgreSQL a `restore_progress` row in the file's data transaction, SQLite and Cassandra a `tables`-row update per batch, MongoDB a `restore_job.file_index` update per file), and the design's "conditioned on the generation" holds on SQLite, MongoDB and Cassandra but on PostgreSQL the progress row lives in the data database, which holds no job record, so the fence there is the base table's primary key (a file applied twice fails on its first key, a recorded file on the progress row's own key) and the claim's liveness, which the engine confirms before each file and whose loss, the heartbeat's, aborts a file in flight; the generation fences only the record's own writes in the catalog (DONE, FAILED, the takeover) (5.3 step 5, PR 9a). `bulk_load` carries `reapplying` so MongoDB's duplicate suppression is explicit: on a resume the first file the engine meets is `reapplying = true` and every later file false; MongoDB suppresses an all-E11000 `insert_many` only when `reapplying`, else "the backup holds a key twice"; PostgreSQL and SQLite never suppress (their base rows are plain `INSERT`s, a duplicate fails the file whatever the flag, and SQLite has no unrecorded partial write, its progress being in the batch's transaction); Cassandra's `put_restored_item` is an upsert, so a duplicate collapses and `count_restore_target` one short of the manifest's total is the only duplicate check there (5.3 step 6, PR 9a). The restore ends by counting the target against the manifest's item total on every backend, so an unordered backup holding one key in two files and a key-range check that missed are both caught; a key-ordered file runs #394's per-file `KeyRange` check, which ends the file's stream at an item outside its range. The migration for PostgreSQL's `restore_progress` is id 6 (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL (`data_migrations/0006_restore_progress.sql`), `NotApplicable` on SQLite, and nothing on MongoDB and Cassandra, whose bootstrappers have no `MigrationId` table; this corrects "data migration 005 for `restore_progress`" in the PR 9a section (its acceptance line and its load paragraph) and the PR 6 line on reserving numbers; item 14's "data migration 005 reserved" stands as the history it is (4(i), PR 6, PR 9a). The manifest registration locks the `backup_index` row (the lock `start_backup_delete` takes: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), while the legacy registration keeps the `backups` row; a legacy and a store-backed restore into one name exclude each other (PostgreSQL the advisory `restore_lock_key`, which both paths take; on SQLite the store-backed path's `restore:/
    ` slot and, for a legacy restore, the `tables` row under the write transaction), so a `DeleteBackup` of a `backup_index` row meets only store-backed registrations and the two orderings resolve one before the other (DeleteBackup wins → the registration is `BackupNotFound` and no target; the registration wins → DeleteBackup is `BackupInUse` and the row stays AVAILABLE) (5.2, 5.3, PR 9a). The restore's ACTIVE flip sets the table and every secondary and vector index of the target in one catalog transaction on PostgreSQL and SQLite (an index of a restore target is created ACTIVE, and `activate_restore_target` sets ACTIVE any `indexes` or `vector_indexes` row that is not), on MongoDB the job writes DONE and `restore_backfill_pending` and the backfill worker flips the table ACTIVE once every index has left CREATING (`finish_restore` returns `Pending` there), and on Cassandra the one lightweight transaction flips the table alone, its index rows created ACTIVE (`Active`, never `Pending`), so a Query on each GSI and `SearchVectors` answer as soon as the target reports ACTIVE (5.3 step 6, PR 9a). The request bounds its wait for the detached registration at `RESTORE_REGISTRATION_TIMEOUT` (60 s) on every backend; past it it answers `InternalServerError` while the task goes on and the target appears through `DescribeTable`, so the differences document's MongoDB-only sentence becomes every backend's (5.3 step 4, PR 9a). The maintenance check runs after the registration on the manifest path (5.3 step 4 as written) and before the permit on the legacy path (item 29), both orders in `restore_table_from_backup` and `restore::start`; the restore's first confirmed beat stands at the instant `claim_restore` returned (the record's `heartbeat_at` was stamped inside the registration, before the return, so the instant is at or after the beat it stands for by the registration's remaining duration, which each backend bounds, 4(e)), where the backup job takes the instant before its claim's insert (5.1 step 4), so a registration that took longer than two heartbeat periods does not end the job at its first check; a held claim on the manifest path leaves a FAILED `maintenance` record whose target the job removes before it answers `LimitExceededException`. Only `StoreError::Transport` is retried on a data-file download, for up to `DOWNLOAD_RETRY_BOUND` (15 min, counted from the job's first download attempt, every file's retries against it) with `min(2^n s, 60 s)` plus up to a second of jitter, so the retry covers S3 stores (a filesystem store's `get` errors are `Io` and fail at once); `NotFound`, a denial, and a digest or size mismatch fail at once naming the file. On a filesystem store "verified in place" is two `get` passes, a hashing pass before any load and a decoding pass through `ExtenddbManifest::reader` that hashes again (so a file replaced between the passes fails in the load) with no spool; on an S3 store the files are spooled as `restore-
    /.json.gz` with `.partial` → `sync_all` → rename → `.verified` (the marker holding the file's sha256, compared with the manifest's before the file is trusted; an entry left by a run of this host for a record another host then owned is removed before use) and read through `reader_prehashed`, the spool at `spool_path` (the defaulted `backup-spool/` on file-backed SQLite). A restore the shutdown stops records `RestoreJobs{cancelled}` and no worker metric, keeps its spool and its progress and writes no status (the `SHUTDOWN` reason is a backup's alone); a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` with no catalog or table access; `fail_restore` writes FAILED then runs `remove_restore_target` only when that write matched. A job that learnt the record is another's (the lost claim, the DONE write's `NotMatched`, `fail_restore`'s no-table-access arm) leaves its spool entry to the sweep, since a same-host successor may be reading it (the entry is keyed by table id); every other end but the shutdown's removes it (4(e), 5.3 step 7, PR 9a). The load hands the backend one `bulk_load` per file over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB of data-file line text) through a channel of two chunks; a restore holds the parsed manifest, one decoded line and its parse, the decoded items of four chunk positions and the backend's batch (32 MiB of item text on SQLite and Cassandra, 4 MiB per INSERT on PostgreSQL, 1,000 documents on MongoDB), which is section 7's restore sentence, with the figures measured on this PR's final tree (`docs/design/15-backup-restore-bench-pr9.md`: 848 MiB over idle on SQLite and 778 MiB on PostgreSQL at 24 worst-shape items under the default arenas, 392 and 427 MiB under one arena; 89 and 16 MiB at 1M key-ordered items of 200 bytes, in 26.8 s and 9.5 s, 144 to 164 and 16 MiB for the same items unordered), which replace the earlier sentence, "A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table": the terms stand, the figures are measured, and a restore of large items is of the backup's order. The metrics are `MetricName::{RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded}` with the outcome in the operation dimension, `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job (the request's and a resume's two document reads and each download attempt the store failed, retried or not; a digest mismatch and the spool's own errors are not the store's); section 9's `restore_jobs_total{outcome}` and `restore_job_duration_seconds` are a converter's spellings, as item 31 said for the backup names (section 9, PR 9a). A backend's refusal of an item in `bulk_load` is not a wire exception: on MongoDB an `N` sort-key value past Decimal128's 34 significant digits is the write path's `ValidationException`, which becomes the restore's FAILED reason, and the client sees the target disappear; a 38-digit non-key number is stored as text and restored intact on every backend. The reconciler resumes a dead store-backed restore from its recorded progress (per file on PostgreSQL and MongoDB, per batch on SQLite and Cassandra) under `claim_restore_dead` once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; it fails a dead manifest record under a maintenance claim before it looks at the record's store (`OWNER_DEAD`, the target removed; `target_not_empty`, the target left for `remove-target`, when it recorded no progress and the target holds items), fails a record of a store other than the configured one or of no store with `OTHER_STORE` (naming `backup sync --move-store`, the target left to the FAILED-target duty), skips a resume without a permit, and FAILs a resume with no recorded progress into a non-empty target as `target_not_empty`, left for `extenddb backup remove-target --account
    ` (PR 10's command, shipped with PR 9, below); its two sweeps (`restore_progress` rows of tables that are not RUNNING restore targets; spool entries of restores not RUNNING with this host's `spool_host`) run only without the maintenance claim and read `list_restore_jobs(Some(RUNNING))`, a backend that cannot list them logging once and skipping both (section 4, 5.3 step 7, PR 9a). A store-backed backup of a multi-part-key table is written `unordered` and restores on every backend, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions), while a legacy backup of one keeps the reader's refusal; the #384 override tests run through both the legacy and the manifest paths for every case with a definition and assert the same input or the same refusal, the manifest path deriving the SSE specification from the manifest's `sse_description` where the legacy path keeps the specification as written. The cross-backend matrix (PR 9b): a cell copies the source's backup prefix into a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so the cell proves the restore and not the adoption; it runs from the committed `devtools/backup-restore-matrix`, and `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs the twelve plain cells and the duplicate-key and vector cells nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do; the seed stands in for `backup sync --adopt-store` until PR 10 replaces it. The restore kill points are run by hand with the `kill-points.py` posted with PR 9 (outside the repository, as PR 8's was), one mode per backend where it applies: `restore_after_download`, `restore_after_file_commit`, `restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, `restore_mongodb_before_worker_flip`, `restore_between_failed_and_removal`, `restore_fenced_out`, `pg_terminate_backend` and `sigterm`, each a SIGKILL or a signal at the named point with a restart and the outcome asserted (the exact count after a resume, or the target gone for a failing restore). Two points pause before the first progress record of a file or batch (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), so a kill at their first reach leaves written items and no progress and the resume answers `target_not_empty` (the rule of 5.3 step 7, asserted); the re-application path (MongoDB's E11000 suppression of the one unrecorded file, Cassandra's re-applied batch) is run by killing at the point's second reach, after an earlier file's or batch's progress is recorded (the script's `--second-reach`): on MongoDB a 24,000-item backup in three files resumed from file 1 with file 0 recorded and reached ACTIVE with the exact count (file 1's 8,227 items re-applied as duplicates, 7,544 applied by the resume); on Cassandra a one-file backup of 12,000 items killed before the second batch's progress write resumed from item 10,000 and reached ACTIVE with 12,000 (2,000 applied by the resume), and `restore_after_file_commit` resumed from file 1 to the exact count. A Cassandra server's CreateBackup is the in-catalog format (section 3), so a Cassandra kill point takes its backup on a SQLite source, copies the prefix under the Cassandra target's root and seeds the target's `backup_index` row as the matrix does; the single-node Cassandra of the harness needs `replication_factor = 1` in the server's configuration, as `init` writes it, since under the default of 3 every lightweight transaction and quorum read on a keyspace the server creates answers "No nodes available in query plan", which is what had blocked these points. The feature chain for the backends' pause points: `extenddb-app/backup-test-hooks` → `extenddb-server/backup-test-hooks` now pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks`, so a hooks build compiles `extenddb_storage::backup_catalog::restore_pause_point` and each backend's own point (`restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, each a `pub const` beside `restore_pause_point` in `extenddb_storage::backup_catalog`, the one spelling the backends and the kill-point script use) in without naming a backend crate (the server depends on none directly); the engine's five restore points are its `KillPoint` variants (`RestoreAfterDownload`, `RestoreAfterFileCommit`, `RestoreMongodbBeforeWorkerFlip`, `RestoreBetweenFailedAndRemoval`, `RestoreFencedOut`), and that enum is the engine's list, not the whole list. Corrections made before PR 9 merged: (1) the restore takeover writes `heartbeat_at = now()` in the one conditional write with the generation and `spool_host`, on every backend, so a second taker finds the record live at once; the backup takeover keeps leaving it (4(e) states why the rules differ); (2) `LOAD_CHUNK_BYTES` is summed from the decoded line's length (the decoder's `bytes_decoded` across one item), not `item_size_bytes`; section 7's terms are restated in that unit; (3) the download retry bound is per job: `download_all` takes the job's download start once and every file's retries count against it; the module doc, 5.3 step 3 and this item say so; (4) a job that learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the reconciler's sweep; (5) the `.verified` marker holds the file's sha256 and `is_verified` compares it with the manifest's; `Entry::open` removes an entry whose record's `spool_host` was another host's before use; the resume reads the manifest from the spool only when the record named this host before the takeover (it read the record after the takeover, which had just written this host's name); (6) the decoder checks the heartbeat's pre-write rule (`confirms_the_claim`) at every chunk boundary beside the shutdown token; MongoDB and Cassandra re-read the record per batch; (7) a restore's `claimed_at` is the instant `claim_restore` returned, the record's `heartbeat_at` having been stamped inside the registration. Cassandra's takeover re-applies the `active_restores` entry when absent; the SQLite and Cassandra batch is bounded by `RESTORE_BATCH_BYTES` (32 MiB of item text) beside `RESTORE_BATCH_ITEMS` (10,000), both in `extenddb_storage::restore_engine`; MongoDB and Cassandra read the record before each write batch; `sweep_restore_progress` deletes from a positive list; the backends' pause-point names are `pub const`s beside `restore_pause_point` in `extenddb_storage::backup_catalog`, one spelling each (`RESTORE_SQLITE_MID_BATCH`, `RESTORE_MONGODB_AFTER_INSERT_MANY`, `RESTORE_CASSANDRA_BEFORE_PROGRESS`, `RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE`), and the engine's `KillPoint` is its own list; Cassandra's `count_restore_target` reads the catalog first and answers `TableNotFound` for a key that is not a CREATING restore target with that id, as the other three do; the MongoDB registration stamps the record's heartbeat from the server's clock inside its transaction (the registering process's clock is replaced; the test registers from a process an hour behind); a duplicate primary key fails the file with `data file : holds a primary key the target already has (the backup holds a key twice): `, the engine naming the file by its key once and the backends naming no file; SQLite's startup line for an aged RUNNING record names the reconciler's duty per kind. The reconciler's store check runs first in the cycle, so the manifest arm reads this cycle's health, and the skip line names `not yet checked` for a store whose first check has not recorded; a maintenance-held dead manifest restore is counted once, in `manifest_restores_failed_and_removed`; `RestorableKeySchema::resolve` runs in the request with step 2's refusals; the request's second table-name check and second `check_supported` are gone: the parser's identity rule and version gate run first; the engine's kill points log `job reached kill point `; each backend's own point is named beside `restore_pause_point`; `finish_restore`'s `Pending`, the DONE write's `NotMatched`, the pre-write claim checks of `run`, the download's pre-attempt check and `WorkerErrorCount{RestoreJob}` on a FAILED job have engine tests; the #384 override tests run through both paths for every case with a definition. The wire test `test_create_backup_of_a_vector_table_is_refused` is `test_vector_table_backup_refused_in_catalog_and_restored_from_store`, since it asserts the round trip on a server with a store; the multi-part-key wire cell lives in `tests/test_backup_job.py` on a deployment that enables `allow_multipart_table_keys`, compares the restored item set with the source's `Scan` and makes no `GetItem` by the whole key (the preview's item paths address such a table by its first HASH attribute on every backend: a `GetItem` by the whole key finds nothing on PostgreSQL and the first HASH key's one item on SQLite, restored or not). Corrections made before PR 9 merged, second set: `extenddb backup remove-target --account
    ` ships with PR 9 rather than PR 10 (the PR 10 section's clause says so): the `target_not_empty` rule was written for a catalog restored from a dump that names a restore which finished and was written to since, but its common trigger on MongoDB and Cassandra, which record progress after the write, is an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra: the kill leaves written items and no progress), and without the command this release had no way to remove such a target (DeleteTable refuses a restore target, the FAILED-target duty skips the reason, DeleteBackup and a second restore of the backup are `BackupInUseException`); the first-reach runs of `restore_mongodb_after_insert_many` and `restore_cassandra_before_progress` are that state, so the PR 9a acceptance line, the PR 9a load paragraph, the testing summary and the differences doc's restore-interruption row name it as the third outcome beside "finishes with the right count" and "gone", where they had named two; the command is the one section 4's duty list describes (the maintenance claim, a FAILED record's target whatever the reason, a RUNNING record's dead target, a legacy record's target, the three refusals, no store needed, the audit line), its tests are the engine's maintenance tests and one wire cell on SQLite seeded into the state at the catalog, and the two first-reach kill points run it and assert the target gone and DeleteBackup allowed. The spool sweep lists the spool before the records (the design's rule, which the shipped order broke); the spooled manifest and sidecar are written as the data files are and a spooled pair that fails verification yields to the store's; the resume's first confirmed beat is the takeover's return, taken beside `claim_restore_dead`; a `finish_restore` error is an unknown outcome re-read through the claim; `BackupStoreErrors{get}` counts every `get` the store failed; the request's and the resume's document reads are bounded (4 KiB, 16 MiB); the restored table's size is the source table's; `RestorePostgresBeforeProgressDelete` is gone from the engine's `KillPoint` (one spelling each holds: the four backend names are the storage crate's constants and nothing else); a registration that fails after the request stopped waiting is logged. The restore's re-applied batch on Cassandra relies on the write path rewriting an unchanged item's GSI/LSI row instead of deleting it (#742, fixed on main by #424), and the `restore_cassandra_before_progress` second-reach run reaches ACTIVE with the GSI count exact where before #742 was corrected it was one batch short. PostgreSQL's `finish_restore` deletes the progress rows after the flip's commit and does not fail the restore if that delete errors (logged and swept). A file's PostgreSQL progress row is a plain INSERT from `start_at == 0` and an upsert from `start_at > 0`. Two concurrent registrations of one backup leave one target on Cassandra (the registration LWT's `active_restores = null` is the fence). What bounds a registration's duration per backend (the advisory/`FOR UPDATE` lock, the slot, `transactionLifetimeLimitSeconds`, the LWT timeouts), so the `claimed_at`-minus-`heartbeat_at` gap can never reach the stale window, replaces the earlier "conservative" in 4(e) and in this item, which had the direction wrong: an instant later than the stamp makes the pre-write check admit a beat older than two periods by the gap, so the gap needs a bound, not a name. The SQLite restore's published memory figures are the no-index shape; a table with indexes costs SQLite's page cache (`cache_size`) plus arena fragmentation, bounded, not the restore's buffers (the record's index table: 143 to 225 MiB at 100k and 165 to 172 MiB at 300k with two GSIs on SQLite, 23 to 34 MiB on PostgreSQL), and the 60,000-item shape of the differences doc's in-catalog timings was measured store-backed on PostgreSQL, SQLite and MongoDB (0.67, 1.85 and 3.74 s to ACTIVE), the row now labelling which path each figure is. The MongoDB CI jobs' `EXTENDDB_TEST_BACKUP_STORE` is `filesystem` from this PR (`devtools/run-mongodb-tests` serves a filesystem root), where it was `none`; the Cassandra job's stays `none`; the runner's `--backup-store none` is the one way to run the legacy LSI fallback cell, the in-catalog restore path's one wire cell on MongoDB, which no CI job runs. The PR 11 line owns the two unmeasured cells (the MongoDB and Cassandra 1M restores, the vector cell) and no longer the nightly schedule, shipped here. The kill-point cells as run on the pushed head's hooks binaries (4c353191), every ACTIVE outcome asserting the base table's and the GSI's `Scan` counts: `restore_after_download`, `restore_after_file_commit`, `restore_between_failed_and_removal` and `sigterm` on all four backends; `restore_sqlite_mid_batch`; `restore_postgres_before_progress_delete`, `restore_fenced_out` and `pg_terminate_backend` on PostgreSQL; `restore_mongodb_after_insert_many` at its first reach (`target_not_empty`, then `remove-target`) and its second (the re-application, 24,000 items in three files); `restore_mongodb_before_worker_flip`; `restore_cassandra_before_progress` at its first reach (`target_not_empty`, then `remove-target`) and its second (12,000 items, the GSI exact); the Cassandra points with a SQLite source. Rerun on the build after the last corrections (the first beat before the reply, the Cassandra re-read and fence, the PostgreSQL vector drop): `pg_terminate_backend` as reworked (the lost claim at the next check 20 s after the termination, the record RUNNING with its `(0, 8228)` progress row, the takeover 45.6 s after the termination and 46.7 s after the last beat, ACTIVE 24,000 on base and GSI), `restore_after_file_commit` and `restore_mongodb_before_worker_flip` on MongoDB (resumed from data file 1 after 0 of its items, 3,771 applied by the resume, DONE then the worker's flip, 12,000 exact; DONE before the kill and the restarted worker's flip after it), `restore_cassandra_before_progress` at its second reach (resumed after 10,000, 2,000 applied, 12,000 on base and GSI) and `restore_between_failed_and_removal` on PostgreSQL with a vector backup (the target gone after the stale window, its data table and `_ddb_vec_` relation gone with it, where the pushed head's build left the relation). The matrix's four `--duplicate-key` and eight `--vector` cells ran on the final binaries. Corrections made before PR 9 merged, third set. The job's first heartbeat is sent right after `claim_restore` returns and before the reply, and the pre-write check counts its two periods from that beat's return; a `Lost` or failed first beat fails the record ("the claim was lost before the job started") and answers `InternalServerError`. `extenddb backup remove-target` decides a RUNNING owner dead through `list_dead_restore_jobs(owner_window)` on the database's clock, and the purge's wait decides a backup row's owner live through `list_backup_rows_older_than` on the heartbeat clock; neither reads a host clock. The reconciler takes nothing over and writes no FAILED on its manifest arm once the server's shutdown has begun. Under a maintenance claim a dead record with no progress over a target that holds items is failed `target_not_empty` through `fail_restore_job`, not `OWNER_DEAD`. The reconciler's `target_not_empty` lines print `extenddb backup remove-target --account
    `. The restore job's stop line names the heartbeat timer's reason when the timer raised the loss, and the timer logs the reason at WARN. A DONE write whose answer was lost and whose record re-reads DONE under the job's generation ends with "complete (DONE)", not "ACTIVE". A spool document write that fails and a resume that fails before it runs remove this host's entry. The engine's tests of the 60 s bound, the slot held to the job's end and the re-reads' generation checks are cells of their own. Cassandra's progress lightweight transaction re-reads an unknown outcome (applied, one retry, or the lost claim; a second unknown outcome fails the restore), where the earlier text said the restore fails and the reconciler resumes it, which the FAILED write's removal made false. The Cassandra registration's refusal path conditions the target's removal on its own generation; a record a reconciler took over during the DDL keeps its target. PostgreSQL's `abort_restore` drops a vector index's relation with the data table. The PostgreSQL record's `heartbeat_at` is the registration transaction's start. Tests: `test_an_unknown_progress_write_outcome_is_decided_by_a_re_read`, `test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target`, `test_a_mid_file_resume_records_start_at_in_every_batch_progress`, `remove_restore_target_drops_the_vector_relation_with_the_data_table`. The engine's `KillPoint` has five restore variants, named above. SQLite's takeover clock is `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, the `NOW` constant of its catalog (4(e) names it). `extenddb backup remove-target` refuses a RUNNING record with a live owner by its own text ("stop it or wait for it"); the DONE and no-record refusals are the two "not a restore target" ones. The published 1M figures are the key-ordered shape's and the record says so beside them, with the same items measured as an unordered backup (SQLite 144 to 164 MiB over idle in 66 to 90 s against 87 to 98 MiB in 27 to 56 s key-ordered; PostgreSQL 16 MiB either way), and SQLite's final `COUNT(*)` is the counting connection's page cache, bounded by `cache_size` (32 MiB) and the same size at 1M and at 3M rows (22 to 28 MiB), where the record had called it the one term that grows with the table from a single measurement. The `pg_terminate_backend` cell of `kill-points.py` pauses the job at `restore_after_file_commit`, terminates the claim session while it is paused (psql must answer a row), waits past one heartbeat period, releases, and asserts the lost-claim stop line, the record RUNNING with its progress row, the takeover no earlier than `stale_job_after_secs` after the owner's last beat and the exact counts; the shipped cell had terminated the session after a fixed sleep and asserted the final count alone, which a 2.6 s restore satisfied without the case ever running. The script's `--vector` switch gives the source table a vector index, and the tamper cell on PostgreSQL then asserts the removed target's data table and `_ddb_vec_` relation gone. `.github/workflows/matrix-nightly.yml` installs `postgresql-client` and checks `psql --version` and the two drivers' import before the cells; it has had no run, since `workflow_dispatch` needs the file on the default branch, so its first run after the merge is the proof of the runner image. The upgrade manual counts migrations 1 to 6 as breaking and migration 7 as declared breaking on PostgreSQL for want of the additive-declaration job (on SQLite, MongoDB and Cassandra it does not apply). The body sentences each of these corrects are edited with this item (section 1's duplicate rule and Decimal128 bullets, section 3's PR 9a interface list, 4(b)'s progress members and metric name, 4(e) and 4(i), 5.2, 5.3's preamble and steps 1 and 3 to 7, section 7's restore memory sentence, section 9, section 10's table, the PR 6, PR 8, PR 9a and PR 9b sections, the PR 10 clause naming what the seed stands in for and the command shipped here, the PR 11 line, the testing summary and the review checklist); earlier items keep their wording, as item 22 requires. With the rebase onto #666's head (2026-11-03): the jobs and the reconciler are not gated by the PostgreSQL serving registration, and 4(e) says what they do in the registration's retry window, GitHub issue #755 being the gate; `backup purge`'s own cycle under the SQLite serve lock fails a dead store-backed restore and removes its target rather than resuming it (4(f)), since the command tells the cycle it holds the claim instead of the cycle reading it through a zero window; the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB, as 5.3 step 5 does; and MongoDB's snapshot open waits, at most 2 s, for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a store-backed CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a table that does not exist. With the rebase onto `main` (2026-11-04), after #335, #666 and #681 merged: `restore_progress` is migration 7 (4(i)), declared not applicable in the MongoDB and Cassandra registries #681 added, as on SQLite; 4(e) cites the module doc of `crates/server/src/registration.rs` for what a lost serving registration does, which replaces this item's 2026-11-03 clause on it: the reconciler stops with the server's workers and spawns no resume once its own token has fired, and the running jobs go on, #755 being the gate for them; a restore job joins PR 8's job drain as a backup job does; and `remove-target` releases its serving registration when it exits and is named on MongoDB and Cassandra as on PostgreSQL. With the move onto `main` at e8ca7df3 (2026-11-04), after #424 merged: the Cassandra write path's rule for an unchanged index row (rewritten by its INSERT alone, #742) and its version stamps on every index row are #424's, so this PR no longer changes `sync_indexes` or the GSI worker; the PR 9a load paragraph, 5.3 steps 5 and 6 and this item cite #424 for what a re-applied batch relies on, and a restored item's index rows carry its base row's `version` as #424's write path gives them (`put_restored_item` is that path). After the second review round (2026-11-05): the Cassandra load no longer goes through `put_restored_item`'s lightweight transaction but writes each item's base and index rows in one logged batch `USING TIMESTAMP` a reading taken before the batch's claim check (`put_restored_item_at`), the fence against a batch a stalled writer lands after a takeover (the PR 9a load paragraph); the index rows still carry the base row's `version`, which is now that timestamp. A resume with no recorded progress fails `target_not_empty` only when the target holds more items than the backup's row counts; at or below the total it re-applies the first file with upserts (step 7), so a Cassandra crash before the first progress record resumes to ACTIVE. One instance's sixth takeover of a record fails it (`RESTORE_TAKEOVERS`, step 7). `restore_target_ready` on Cassandra checks each GSI's and LSI's table as well as the data table. The nightly workflow runs every restore kill point of every backend in four jobs (the testing summary); the kill-point script's first-reach cells assert ACTIVE with the exact count and its `--remove-target` flag is gone (the `target_not_empty` state is seeded by `tests/test_backup_job.py`'s `remove-target` cell). diff --git a/docs/design/README.md b/docs/design/README.md index 4fb516ca3..f0ee934ed 100755 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -21,6 +21,7 @@ This directory contains the authoritative design documents for ExtendDB (extendd | 13 | [MongoDB Storage](13-storage-mongodb.md) | MongoDB storage backend: engine and catalog traits over the official driver, replica-set transactions | | 15 | [Backup and Restore](15-backup-restore.md) | Overview of the off-host backup series: the invariants, the per-operation contracts and a short section per later PR; the format, store and snapshot documentation lives with their crates (`crates/core/src/backup_format`, `crates/backup-store`, `crates/storage/src/snapshot.rs`) | | 15b | [Backup memory and duration record, PR 8](15-backup-restore-bench-pr8.md) | The measurements section 7 and revision item 31 of the design cite: peak RSS and duration of the 100k, 1M, worst-expansion and incompressible backups per backend, the allocator series, the PostgreSQL page's cost | +| 15c | [Restore memory and duration record, PR 9](15-backup-restore-bench-pr9.md) | The measurements section 7 and revision item 33 of the design cite for the restore: peak RSS and duration of the 100k, 1M, worst-expansion and large-item restores on SQLite and PostgreSQL, with and without one allocator arena, the terms the code bounds and the sizing sentence | ## How to Use These Docs diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index 4b9960450..5eea17fae 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -91,11 +91,11 @@ from a request sent to that service or backend. | Vector index update propagation | Eventually consistent, the same model as a GSI | Matches DynamoDB. Maintenance is queued on the same propagation queue as async GSIs, so a search immediately after a write may not see it. Governed by the same `index_propagation_delay_ms` setting; unlike a GSI there is no per-index override. A value of 0 applies maintenance inline in the write's own transaction, which is stricter than the service and exists so a test can assert steady state without waiting. Inline additionally requires the table's propagation queue to be empty: while rows queued during an index build are still draining, a new write queues behind them instead of overtaking them, so a brief search-after-write window exists even at 0 until the queue drains. That window is ordering, not data loss; the write is applied, in order, by the queue worker. That zero behaves differently while an index is still building, and the two backends differ: **PostgreSQL** applies inline only to an index that is already `ACTIVE`, and defers a write to a building index whatever the delay says, because the write must not reach the index ahead of the backfill's older snapshot of the same item; **SQLite** does not check the status on this path, so with a zero delay a write during a backfill is applied inline and bypasses the hold that keeps the two ordered. The SQLite behaviour is tracked as a defect (F-20), not intended, and it is reachable only with the zero delay, which is a test setting. | | `SearchVectors` score at extreme magnitudes (PostgreSQL backend only) | Returns the true distance as a number for any vector of finite components | The score is bounded to a finite value instead, and the bound is not a measured service answer. pgvector accumulates distances in single precision, so magnitudes far below `f32::MAX` overflow inside the extension: Euclidean above about 9.2e18, dot product above about 1.8e19, and cosine at both ends, above about 1.8e19 and below about 3.7e-23, which is where a component's single-precision square rounds to zero. A non-finite score cannot be serialised as JSON at all, so the result is bounded in SQL: `1e308` for an overflowed distance, `-1e308` for an overflowed negated inner product, which the score contract negates so a client sees `1e308` in `Score`, and 1.0 for a cosine that comes back NaN. Ranking is unaffected at the overflow end, because each bound sits at the end its metric overflows towards, so the farthest row stays farthest and the most similar stays most similar; two rows that both overflow tie, and the tie breaks on the base key. At the underflow end cosine loses resolution rather than being bounded, and which side is tiny decides how much. For a tiny **query** vector, its norm underflows while the inner product usually does not, so the quotient is an infinity that pgvector clamps and the reported distance collapses to one of 0, 1 or 2 following the sign of the inner product, with the 1.0 substitute firing only when the vectors are exactly orthogonal and the quotient is therefore 0/0 (measured). For a tiny **stored** vector it is worse: the stored norm is computed in single precision and reaches zero, so the zero-vector guard fires and that row reports 1.0 at every angle, parallel included. A corpus of tiny embeddings therefore loses ranking altogether rather than losing resolution. For a tiny query vector, by contrast, ranking still separates nearer-than-orthogonal from farther and loses resolution only within each half. The SQLite backend owns its own arithmetic, computes in double precision, and reports the true value, which is why this row is scoped to PostgreSQL. | | Vector index deletion window | `UpdateTable` Delete leaves the index in `DELETING` long enough to observe, then removes it | No observable `DELETING` window on either backend: the catalog row is removed inside the `UpdateTable` transaction, so a `DescribeTable` immediately afterwards already omits the index. The index's data table is dropped after that commit, in a separate transaction, and the two backends handle a failure there differently. **PostgreSQL** treats it as best effort, because the data table lives in a different database entirely: a failure is logged and skipped rather than failing the request. **SQLite** propagates it, so a failed drop returns an error from an `UpdateTable` whose catalog change has already committed, which is the more surprising outcome of the two: the index is gone from the catalog and the caller saw a failure. So an operator debugging a leftover `_ddb_vec_*` table should look for that warning rather than assume the delete was incomplete. | -| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Partial. With a configured `[backup.store]`, `CreateBackup` of such a table succeeds on PostgreSQL and SQLite, the two backends with vector indexes: the backup is written to the store and its manifest records the index specifications, and `DescribeBackup` and `ListBackups` report it as any other backup (wire test on both backends). Without a store (the section absent on PostgreSQL, or `kind = "disabled"`), where `CreateBackup` writes the in-catalog format, it is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend" and no backup is created, because that format cannot carry the index data across. `RestoreTableFromBackup` of any backup in the store answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" until a later release's restore job, which restores a vector table into a backend with vector support and refuses it into one without. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | +| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Partial. With a configured `[backup.store]`, `CreateBackup` of such a table succeeds on PostgreSQL and SQLite, the two backends with vector indexes: the backup is written to the store and its manifest records the index specifications, and `DescribeBackup` and `ListBackups` report it as any other backup (wire test on both backends). Without a store (the section absent on PostgreSQL, or `kind = "disabled"`), where `CreateBackup` writes the in-catalog format, it is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend" and no backup is created, because that format cannot carry the index data across. `RestoreTableFromBackup` of a store-backed vector backup restores the table with its vector indexes into PostgreSQL and SQLite, the two backends with vector support (`SearchVectors` answers as soon as the table is `ACTIVE`; matrix cells), and is refused into MongoDB or Cassandra with `ValidationException` "backup has vector index(es); restoring a table with vector indexes is not supported by this storage backend" before any target table is created. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | | `SearchVectors` endpoint | Served only on `search-dynamodb..amazonaws.com`. The standard `dynamodb..amazonaws.com` endpoint answers the same request with HTTP 400 `UnknownOperationException` ("This operation is not supported by this endpoint"); every control-plane and item operation stays on the standard endpoint. Signing is unchanged either way (service name `dynamodb`, target prefix `DynamoDB_20120810`) | Served on the same endpoint as every other operation, so a client is pointed at one ExtendDB endpoint for all of them. Two consequences worth knowing: an SDK that resolves a separate search hostname from its endpoint ruleset needs its endpoint overridden to reach ExtendDB, and ExtendDB does **not** reproduce the service's refusal, so a test asserting `UnknownOperationException` for vector search on the base endpoint passes against Amazon DynamoDB and fails here. | | `SearchVectors` result order for equal distances | Measured unstable: three identical searches returned tied rows in three different orders, and a top-k that truncates a tie group keeps an arbitrary subset of it | Deterministic on both backends, by different means. **PostgreSQL** sorts explicitly on the base table's full primary key after the score, so the order is a property of the query. **SQLite** issues no `ORDER BY` and resolves ties by scan order through a stable top-k, so its order is a property of the plan rather than something the query guarantees. Either way a client re-issuing an identical search sees the same order, which is stricter than the service rather than divergent in outcome. Do not rely on the two backends agreeing on which subset of a truncated tie group they keep. | | Vector indexes on the MongoDB backend | Vector indexes and `SearchVectors` are available | Not supported at all. That backend provides no vector search implementation, so the engine's capability gate refuses `CreateTable` and `UpdateTable` carrying `VectorIndexes` and every `SearchVectors` request, with the same capability message a PostgreSQL deployment without pgvector returns. Every other vector row in this document describes the PostgreSQL and SQLite backends. | -| Multi-part base table keys | Not supported | Preview extension (opt-in via `enable_multipart_keys` setting). Standard single/composite keys work identically. | +| Multi-part base table keys | Not supported | Preview extension (opt-in via the `limits.allow_multipart_table_keys` setting). Standard single/composite keys work identically. | | GSI throughput on a `PROVISIONED` table | Every GSI needs `ProvisionedThroughput` of at least 1 read and 1 write unit | A GSI `ProvisionedThroughput` below 1 (including 0/0) is `ValidationException` on `CreateTable` of a `PROVISIONED` table and on `UpdateTable`'s GSI `Create` ("ReadCapacityUnits and WriteCapacityUnits must both be greater than or equal to 1 for index: "; the service's text was not captured). A GSI with no `ProvisionedThroughput` is still accepted on a `PROVISIONED` table, and `UpdateTable` to `PROVISIONED` leaves an on-demand table's GSIs without one; see the `BillingModeOverride` row under Backup and Restore for how a backup of such a table restores | | Projected attributes across all indexes | At most 100 | Enforced on `CreateTable` (and so on `RestoreTableFromBackup`, which validates its target the same way) and on `UpdateTable`'s GSI `Create`: every `NonKeyAttributes` entry of every GSI and LSI is counted, so an attribute projected into two indexes counts twice. Over 100 is `ValidationException` ("Number of projected attributes in all indexes exceeds limit of 100, projected: "; the service's text was not captured) | | Index key validation on `UpdateItem` and transaction writes | A secondary-index key attribute must carry its declared scalar type and may not be empty. On `UpdateItem` a literal operand of the wrong type (`SET k = :v`, `ADD`, `DELETE`, `list_append` of literals) is refused on the statement, ahead of the `ConditionExpression`, naming the attribute and the index; a value computed from the item (an attribute reference, the fallback of `if_not_exists`) is refused after the condition passes with the message "The update expression attempted to update the secondary index key to unsupported type"; a key left empty is refused after the condition passes with the update-expression wording. In `TransactWriteItems` a wrong-typed key (table or index) on any member cancels the transaction with the first such member's `ValidationError` in position and `None` elsewhere; otherwise an empty index key on any member is a top-level `ValidationException`; otherwise members report their own conditions. On a `Put` member both checks come ahead of the member's condition. (Measured on the service.) | Matches on every backend. Two known shared differences: an empty index key supplied through the legacy `AttributeUpdates` parameter is reported with the update-expression wording where the service uses the `PutItem` wording naming the index; and a request whose own value exceeds 400 KB is refused for size by the service before any index-key check, where ExtendDB reports the index-key fault first. | @@ -107,8 +107,8 @@ Behaviour stated as measured, and every number below, was observed with release | Area | DynamoDB | ExtendDB | |------|----------|------| -| What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | -| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On MongoDB the request waits at most 60 s for its background task to register the `CREATING` target (a bound against a connection pool that has stopped handing out connections); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the copy ends; `DescribeTable` is where a client finds it *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB). A restore runs inside the server process; if the process stops, see the abandoned-restore row | +| What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand. `DescribeTable` on a freshly restored table reports `ItemCount` 0 and `TableSizeBytes` 0 (measured on 15 restored tables, one of them ten minutes after `ACTIVE`) | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. `DescribeTable` on a restored table reports `ItemCount` 0 and `TableSizeBytes` 0 when the table goes `ACTIVE`, as the service does (measured: 0 and 0 on 15 restored tables, one of them ten minutes after it went `ACTIVE`), until the statistics pass next rewrites both from the restored rows *(from the code)*. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | +| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured *(in-catalog restore, an earlier build)* with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. Measured *(store-backed restore, this release)* with the same shape (60,000 items of 200 bytes, composite key, no secondary index) from a filesystem store on loopback: response in 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB); `ACTIVE` after 0.67 s, 1.85 s and 3.74 s (on MongoDB the job's DONE at 1.8 s and the index backfill worker's flip after); 1M items of 200 bytes in key order (the server's own backups) `ACTIVE` after 9.5 s on PostgreSQL and 26.8 s on SQLite, the same items as an unordered backup (a MongoDB source, a multi-part-key table) after 17.5 s and 66 to 90 s (`docs/design/15-backup-restore-bench-pr9.md`, which has the host and its load); the store-backed response includes the sidecar and manifest reads and its `ACTIVE` time the download and the per-file verification, and Cassandra as a store-backed target was measured at 12,000 items only (the kill points: a resumed restore of 12,000 items of 4,000 bytes in two data files reached `ACTIVE` 8.3 s after its takeover on a single-node container). On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On every backend the request waits at most 60 s (`RESTORE_REGISTRATION_TIMEOUT`) for its background task to register the `CREATING` target of a store-backed restore (a bound against a connection pool that has stopped answering; the manifest and sidecar reads that precede the registration are not under it); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the restore ends; `DescribeTable` is where a client finds it *(from the code)*. Because that 60 s bound equals botocore's default read timeout, a client whose first attempt times out while the registration is still in flight (a backup's `backup_index` row held under a slow lock) retries: a retry that names the same backup and target waits for the in-flight registration and is answered its `CREATING` description rather than registering a second target, and a restore of a *different* backup into the same name while the first is in flight is `TableInUseException` *(from the code)*. The wait is per server process: each server keeps its own list of in-flight registrations, so a retry waits for the first attempt's registration only when it reaches the server that holds it; a retry that reaches another server behind a load balancer finds the name free until the registration commits and then `TableInUseException` (the registration took the name) or, once the target exists, the same `CREATING` description from the catalog *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB); on the store-backed path the permit is taken before the manifest is read, after the catalog row's state, the target name and the backup's in-use rule are checked, in that order (a restore of a backup into its own CREATING target is `TableInUseException`, measured on DynamoDB). A restore runs inside the server process; if the process stops, see the abandoned-restore row | | `RestoreSummary` | The restore response and `DescribeTable` while `CREATING` report `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` (the backup's `BackupCreationDateTime`, not the time the restore ran), and `RestoreInProgress=true`. Once the table is `ACTIVE`, `DescribeTable` omits `RestoreSummary` | The same on all four backends. No response that describes a table (`CreateTable`, `UpdateTable`, `DeleteTable`, `DescribeTable`, `RestoreTableFromBackup`) carries `RestoreSummary` unless the table is `CREATING`; what the service's `UpdateTable` returns for a restored table was not measured | | Reads on a restore target | A `GetItem` or `Scan` on the target while `CREATING` is `ResourceNotFoundException`; the first read after `DescribeTable` reports `ACTIVE` succeeds (measured) | The same on all four backends. The server's key-info cache stores a negative entry (5 s TTL by default) for a name the catalog does not have, and does not store one for a name the catalog has in a state other than `ACTIVE`: on a not-found answer it asks the catalog once more with `DescribeTable`, and authorization records the name as missing in the key-info snapshot it hands the engine, so the engine does not look it up again. A `GetItem`, `PutItem`, `UpdateItem`, `DeleteItem`, `Query`, or `Scan` on the target while it is `CREATING` therefore costs exactly one `table_key_info` and one `describe_table` catalog load, uncached, per request (on SQLite five SQL statements: one for the key info, four for `DescribeTable`; on PostgreSQL `describe_table` is two queries), plus one `describe_table` on the first such request for the throttle registry; the first read after `ACTIVE` loads once more and is then cached. Measured on SQLite with a `GetItem` loop on a `CREATING` restore target of 100,000 items: 40 requests refused, 40 key-info and 41 `DescribeTable` loads in the server's statement log; with the hand-off to the engine removed, 80 and 81. In an earlier run of the fidelity test the first `GetItem` after `ACTIVE` answered in 3 ms after 72 refused reads. | | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | See the overrides table below. A refused override is refused before the target table is created | @@ -119,10 +119,10 @@ Behaviour stated as measured, and every number below, was observed with release | Tag conditions on backups | The backup resource has no `aws:ResourceTag` condition key: a backup has no tags to match | `DescribeBackup`, `DeleteBackup`, and `RestoreTableFromBackup` are authorized on the backup ARN as a resource with no tags. An `aws:ResourceTag/` condition finds the key absent for a backup (and `dynamodb:ResourceTag/` is absent for every resource), so, for example, an Allow conditioned on `StringEquals` never grants on a backup and a Deny conditioned on `StringEquals` never denies one. This holds whatever an earlier build stored under the backup ARN (it accepted `TagResource` on one; see Tagging), and a backup does not take its table's tags. For `RestoreTableFromBackup`, the seven item actions evaluated on `table/` (see the authorization rows below) read that ARN's resource tags. A target that does not exist has no tags, so an `aws:ResourceTag/` condition on those actions finds none and does not grant the restore; a target that exists is evaluated with its tags, and a request that is otherwise valid then passes authorization and fails `TableAlreadyExistsException` (or `TableInUseException` for a `CREATING` restore target) when the condition is met, and fails `AccessDeniedException` naming the first item action when it is not (measured on SQLite). Whether the service reads a target's tags in these evaluations was not measured. Scope backup permissions by ARN (`table//backup/*`). Measured on PostgreSQL and SQLite for the Allow case, including tags written under a backup ARN by the earlier build and then served by this one; the Deny case and role sessions rest on unit tests of the authorization path, which is the same for every backend | | Authorization of `ListBackups` | `dynamodb:ListBackups` is evaluated on `arn:aws:dynamodb:::table/*/backup/*`, or on `table//backup/*` when `TableName` filters the list (measured on DynamoDB) | The same on all four backends (the check runs in the server before the backend is reached). A statement on `*` or `table/*` covers every list; one on `table//backup/*` (or `table//*`) covers only the list filtered to that table; one on the table's own ARN, `table/`, covers none. The unfiltered list is one resource, so a Deny on `table//backup/*` denies the list filtered to that table and not the unfiltered list that would include its backups, which follows from the resources the service evaluates. A `TableName` that is not a string is evaluated as no filter; the engine then refuses the request. These resources have been evaluated since v0.1.14; the admin guide's compatibility notes describe what earlier builds evaluated | | Authorization of `RestoreTableFromBackup` | `dynamodb:RestoreTableFromBackup` on the backup ARN, and `dynamodb:Scan`, `dynamodb:Query`, `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:UpdateItem`, `dynamodb:DeleteItem`, and `dynamodb:BatchWriteItem` on `table/`, each of which must be allowed; `dynamodb:RestoreTableFromBackup` is not evaluated on the target (measured on DynamoDB) | The same eight evaluations on all four backends, in that order: the backup first, then the target under each item action in the order listed, stopping at the first denial, which the `AccessDeniedException` names (`User: ... is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:::table/`); the service's order was not measured. A missing `TargetTableName` is evaluated as `table/` (an empty name, which no statement but `*`-style wildcards covers); the engine then refuses the request. The seven item-action evaluations read the target ARN's resource tags: a target that does not exist has none, one that exists is evaluated with its tags and the restore then fails `TableAlreadyExistsException` (see Tag conditions on backups). Every DynamoDB condition key (`dynamodb:LeadingKeys`, `dynamodb:Attributes`, `dynamodb:Select`, `dynamodb:ReturnValues`, `dynamodb:ReturnConsumedCapacity`, `dynamodb:FullTableScan`, `dynamodb:EnclosingOperation`) is absent in these seven evaluations, so `ForAllValues:*` and `Null: true` conditions pass and bare `StringEquals` and `Bool` conditions fail: an Allow of the item actions conditioned on `ForAllValues:StringEquals dynamodb:LeadingKeys` grants the restore of every item in the backup, an Allow of `dynamodb:Scan` conditioned on `Bool dynamodb:FullTableScan: false` blocks every restore, and a Deny of `dynamodb:Scan` conditioned on `Bool dynamodb:FullTableScan: true` blocks none; the keys the service sets in these evaluations were not measured. The backup ARN is evaluated as the backup it names in this deployment: the table and backup id from the request, in the server's region and the caller's account, whatever partition, region or account the caller wrote, with surrounding whitespace trimmed as the engine trims it. The engine resolves the same table and id (under the server's region first, then as written for a backup created under an earlier `server.region`), so it never acts on a backup other than the one evaluated. `DescribeBackup` and `DeleteBackup` are evaluated the same way. These resources have been evaluated since v0.1.14 (see the admin guide's compatibility notes). A Deny of `dynamodb:RestoreTableFromBackup` on a table does not block restores into it; deny one of the seven item actions on it instead | -| Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only *(from the code and storage-level tests; not checked on a running server)* | +| Multi-part base table keys (preview) | Not supported | A store-backed backup of a table created under `limits.allow_multipart_table_keys` is written `unordered`, and its restore preserves the item set: the restored table holds what the source's `Scan` held, with the source's key schema and attribute definitions, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions) *(measured on PostgreSQL and SQLite with the preview enabled, `tests/test_backup_job.py`; the engine test `a_multi_part_key_manifest_is_registered`)*. The preview's item paths address such a table by its first HASH attribute on every backend, restored or not: on PostgreSQL a `GetItem` by the whole key finds nothing on the source or on the restored table (a `Query` naming both key attributes does find the item), on SQLite the source itself keeps one item per first HASH key value and a `GetItem` returns that item whatever second key it was given, and a `Query` on the first HASH key alone is refused on both ("Query condition missed key schema element") *(measured on PostgreSQL and SQLite with the preview enabled)*. An in-catalog backup of such a table (no store, or Cassandra) is refused at restore with `ValidationException` *(from the code and storage-level tests)* | | `DeleteBackup` and `DeleteTable` during a restore | `DeleteTable` on the `CREATING` target is `ResourceInUseException`. `DeleteBackup` during a restore was not measured | While the target is `CREATING`, `DeleteTable` on it is `ResourceInUseException` on all four backends, as on the service. `DeleteBackup` on its source backup is `BackupInUseException` on all four: that refusal is ExtendDB's contract, not measured against the service. Both succeed once the target is `ACTIVE`. How the restore and `DeleteBackup` are ordered differs *(from the code)*. PostgreSQL and SQLite register the restore in the transaction that creates the target, holding the backup row, so a concurrent `DeleteBackup` is ordered strictly before (the restore then finds no backup) or after (it finds the restore). MongoDB runs the restore's registration and `DeleteBackup`'s claim each as one session transaction that first increments a lock field on the backup's catalog document while it is `AVAILABLE`, so the two conflict and one orders wholly before the other: a `DeleteBackup` that wins leaves the registration no `AVAILABLE` backup (`BackupNotFoundException`), a registration that wins makes the `DeleteBackup` `BackupInUseException`. Cassandra uses two lightweight transactions on different partitions: the target's row is inserted first (`IF NOT EXISTS`, which reserves the name), then the restore is registered on the backup's row (`IF backup_status = 'AVAILABLE'`), which `DeleteBackup` claims on the same partition; a concurrent `DescribeTable` can therefore see a `CREATING` target for a restore that then fails with `BackupNotFoundException`, after which the target is removed. On MongoDB `DeleteBackup` writes the backup `DELETING` before it drops the data and `DELETED` after; a `DeleteBackup` whose request fails or is cancelled after the drop started leaves the backup `DELETING`, which `DescribeBackup`, `ListBackups` and a restore treat as gone (`BackupNotFoundException`), and a retry of `DeleteBackup` finishes the drop and answers `DELETED` *(from the code and storage-level tests)*. MongoDB reads the catalog from the primary with read concern `local` regardless of the connection string, which the lock and the job records rely on; the connection string must use `readPreference=primary` (any other read preference is refused at startup). The write concern follows the connection string and must be `majority` (the server default): a connection string with `w=1` is refused at startup, because a `DELETING` write or a job record acknowledged under `w=1` just before a primary failover can be rolled back | -| Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | -| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | +| Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. For a store-backed restore the registration takes the same lock on the `backup_index` row instead of the legacy `backups` row (PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB a `$inc lock_seq` on the `backup_index` document, Cassandra the `active_restores` lightweight transaction on the backup's partition), the same ordering guarantee, so a legacy restore and a store-backed restore of two different backups never contend and a `DeleteBackup` of a store-backed backup meets only store-backed registrations. Two concurrent registrations of one backup leave one restore target on Cassandra as on MongoDB: the registration's lightweight transaction (`IF backup_status = 'AVAILABLE' AND active_restores = null`) is linearized on the backup's partition, so the second finds `active_restores` non-null and is `BackupInUseException`; the in-use pre-read is advisory *(storage-level tests on both)*. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | +| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. A store-backed restore (one run from a `[backup.store]`) is instead resumed, not discarded: its owner's death leaves a RUNNING restore job record with a stale heartbeat and the target `CREATING`, and the reconciler of any instance with the configured store takes the record over under a fresh owner generation once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), stamping a fresh heartbeat in the same write so no second instance takes it too, and resumes the load from its recorded progress (per data file on PostgreSQL, per batch on SQLite, MongoDB and Cassandra), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; the target reaches `ACTIVE` with the backup's full item count, its secondary indexes complete: a resumed Cassandra restore re-applies its last batch as idempotent upserts that rewrite the same base and index rows (the write path rewrites an unchanged item's GSI and LSI rows rather than deleting them, #424). A resume that finds no recorded progress and a target holding more items than the backup's total is `FAILED` with reason `target_not_empty` and the target is left in place: the rule protects a catalog restored from a dump that names a restore which finished and was written to since. A target holding no more than the total with no recorded progress is a first data file applied in part: on Cassandra, which records a batch's progress after its writes, an ordinary crash before the first progress record (the first 10,000 items or 32 MiB) leaves written items and no progress, and the resume re-applies the first file with idempotent upserts and reaches `ACTIVE` with the exact count *(from the code; the storage and engine cells)*. MongoDB records each batch's documents and its progress in one transaction, so a mid-file crash leaves no items the record does not account for and resumes to `ACTIVE`; only a crash before the first batch commits leaves an empty target, which resumes from the start. A record that one instance's reconciler has taken over five times without the restore ending (a resume whose final count or progress read fails every time, or that stops with the record RUNNING) is failed on the next takeover with reason `owner dead: taken over 5 times by one instance without the restore ending` and its target removed, so a record that can never finish does not hold its name and its backup for as long as the server runs; the count is per instance and a restart starts it over *(from the code)*. The target stays `CREATING` with a `FAILED` record: `DeleteTable` on it is `ResourceInUseException`, `DeleteBackup` of its backup and a second restore of that backup are `BackupInUseException`, and the reconciler does not remove it; `extenddb backup remove-target --account
    ` (the admin guide's `backup` section) removes the target and its record, after which `DeleteBackup` and a new restore into the name succeed. A legacy (in-catalog) restore is still failed and its target removed. Measured *(binary-level, this release)*: SIGTERM during a store-backed restore left the record RUNNING and the restarted server resumed it to `ACTIVE` with every item (on PostgreSQL after the 45 s stale window, on SQLite at once); a SIGKILL at each restore kill point left the target to finish with the exact count (the base table's and the GSI's) or, for a failing restore, gone, except at the first reach of the Cassandra point that pauses before a batch's first progress record: on Cassandra (killed after the first batch's 10,000 upserts, before their record) the resume of that build answered `target_not_empty` and left the target `CREATING`, and `extenddb backup remove-target` then removed it, after which `DescribeTable` was `ResourceNotFoundException` and `DeleteBackup` succeeded; this release resumes that case with upserts to `ACTIVE` *(from the code; the nightly kill-point job runs it)*. On MongoDB a SIGKILL after a file's `insert_many` leaves the batch's items and its progress committed together, so the resume continues to `ACTIVE` rather than `target_not_empty`. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | | Point-in-time recovery | 35-day window: `UpdateContinuousBackups` enables continuous backups, and `RestoreTableToPointInTime` restores to any second in the window | Not supported; use on-demand backups (`CreateBackup` and `RestoreTableFromBackup`) instead. `DescribeContinuousBackups` reports `ContinuousBackupsStatus: ENABLED` (the service reports this unconditionally) with `PointInTimeRecoveryStatus: DISABLED` and no restorable-time window. `UpdateContinuousBackups` with `PointInTimeRecoveryEnabled: true` returns `ContinuousBackupsUnavailableException`; with `false` it returns the same description. `RestoreTableToPointInTime` checks the source table, then returns `PointInTimeRecoveryUnavailableException` ("Point in time recovery is not enabled for table ''"), the exception the service models on that operation. A table that does not exist is `TableNotFoundException` ("Table not found: ") on all three operations, as on the service. Same on all four backends | | Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
    //`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` and `TableSizeBytes` in `SourceTableDetails` are the table's figures as last counted by the stats pass while `CREATING`, which is 0 and 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact figures once `AVAILABLE`: the items written and the sum of their DynamoDB sizes; the service reported 0 and 0 in every fresh-table backup measured). `CREATING` lasts the whole upload (the durations below), and while it lasts `DeleteBackup` and `RestoreTableFromBackup` of the backup answer `BackupInUseException` "Backup is being created: " on all three backends; the 16 service backups measured were `AVAILABLE` 12 to 90 ms after `CreateBackup` answered, so a client that deletes or restores a backup right after creating it polls `DescribeBackup` first here. The backup is the table as it stood when its snapshot opened, which is inside the request but after the request's checks (the table's state, the store's health, the maintenance claim) and after any wait for a job permit, up to 30 s (below): every write acknowledged before the open is in the backup and none acknowledged after it, so a `PutItem` acknowledged while the request waited for a permit is in the backup (measured: a write acknowledged 1.01 s after a `CreateBackup` that waited for a permit was in it, 2 of 2 runs), and `BackupCreationDateTime` is the open's millisecond. Measured on the service: a restored backup contained every write acknowledged before `CreateBackup` was sent and the 18 writes acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed, so the service takes the table at an instant inside the request as well. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). A `CreateBackup` still waiting in its snapshot open or row insert when the server stops writes no row; it is answered `ServiceUnavailable` "The server is shutting down. Retry the request." only after a registration loss, where the jobs' token is cancelled while the listener still serves, since on SIGTERM the listener's 5 s connection drain closes the connection before that token is cancelled (measured: the connection closed 8.2 s after the request was sent). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"), the account checked before the region, so another account's ARN in another region is `AccessDeniedException` (measured on DynamoDB for CreateBackup and ListBackups, twice each); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). Without a store, and on Cassandra, a table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s): the in-catalog format writes no store prefix. With a store, a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it with `ValidationException` before it opens a snapshot or writes a row ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"). Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | @@ -130,8 +130,9 @@ Behaviour stated as measured, and every number below, was observed with release | A backup maintenance command holding the catalog's claim | Not applicable: the service has no maintenance command | While `extenddb backup purge` holds the catalog-level maintenance claim, `CreateBackup` and `RestoreTableFromBackup` on every instance of the catalog answer HTTP 400 `LimitExceededException` "A backup maintenance command holds this catalog's maintenance claim; backup and restore jobs are refused until it finishes", after every other check of the request (a request refused on its own terms, a bad table name or override, gets its own error) and before the job permit. The refusal ends when the command releases the claim, or when a command that died goes stale after `[backup] stale_job_after_secs`. | | A store-backed backup while the configured store is unhealthy | Not applicable: managed storage | `CreateBackup` on PostgreSQL, SQLite and MongoDB with a configured `[backup.store]`, and `DeleteBackup` and `RestoreTableFromBackup` of a backup in that store, while the store is unhealthy or its first check is pending, answer HTTP 400 `ValidationException` "The filesystem backup store configured on this server is unhealthy (owned by another catalog); the operation is refused until the store's check passes", the kind (`filesystem`, `s3`) and the category in parentheses being the store's (`/health` reports the same category); before the first check of a store checked after the listener: "The s3 backup store configured on this server has not passed a check yet (its first check is pending); the operation is refused until it does". `DescribeBackup` and `ListBackups` still describe such a backup. Once `DeleteBackup`'s DELETING write has committed the request answers the description with `BackupStatus` `DELETED` whatever the object deletion does; a deletion the store does not answer within its bound is left DELETING for the reconciler, logged, and the request still answers DELETED. `CreateBackup`'s refusal comes after the table's ACTIVE probe and the name checks and before the maintenance check and the job permit, so it takes no slot and writes no row; the in-catalog create (no store, Cassandra) touches no store and is not refused. | | `DescribeBackup` when the definition cannot be read | Not applicable | A definition row that does not parse, a renamed column, or a revoked privilege (`Internal` from the backend) is an `InternalServerError`, as it is for `RestoreTableFromBackup` on the same read; only a definition version this build does not know is described without `SourceTableFeatureDetails`. | -| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | +| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. That snapshot is the majority-committed one, which can trail `DescribeTable`'s `local` read by the majority-commit lag, so a CreateBackup sent as soon as `DescribeTable` shows a new table `ACTIVE` could find it missing; the open then waits, at most 2 s, until the snapshot shows the table, and answers `TableNotFoundException` only for a table a `local` read does not show `ACTIVE` either. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | | `CreateBackup` of a table holding an `N` value the wire did not validate | Every stored value is a valid number, and a backup holds the table as it is | With a configured `[backup.store]` the backup fails. Four write paths of earlier builds stored an `N` value such as `1E126` or `abc` without validating it (#618: `BatchWriteItem`, `TransactWriteItems`, `UpdateItem` on an absent item in the key it creates, `ImportTable`), and `UpdateItem` and a `TransactWriteItems` `Update` of builds before #433 stored an attribute name over 65,535 bytes without checking it, and the store-backed job validates every item before encoding it with the rule the restore applies (`validate_restorable_item`: every `N` and `NS` member a valid number, the nesting depth, the key shape (each key attribute present, of its declared type, not empty, an `N` key normalized), the item size against the instance's `max_item_size_bytes`, the hash and range key sizes and the attribute name sizes against the instance's limits, and for an item over the default item limit the data file line against the format's 32 MiB cap), so no backup that no restore can load is reported `AVAILABLE`. After the first refusal the job reads and validates the rest of the snapshot, logs each refused item with its key, attribute path and reason beside the backup ARN (up to 1,000, then a count), and fails the backup with a reason giving the count and the first item, naming #618 only when that item's refusal is an `N` value ("3 item(s) cannot be restored; first: key {...}, attribute amount: Number overflow... (#618)"); an item over `max_item_size_bytes` (stored under a higher limit) fails it the same way with a reason naming the size and the limit and not #618. The client sees `CREATING` and then `BackupNotFound`; the reason reaches the operator (`extenddb backup list --failed`, and the log, which a tenant's support case can cite; every tenant-supplied text in the line is written as a JSON string's content, control characters and the Unicode line separators U+0085, U+2028 and U+2029 escaped, so a newline or a line separator in an attribute name or a number value cannot forge a second log record); the way out is to rewrite or delete each named item through the wire (a bad range key cannot be rewritten, since a PutItem with that key is refused; `DeleteItem` removes it). The in-catalog create (no store, Cassandra) backs such a table up and restores it as before (wire tests on PostgreSQL and SQLite for each path) | +| Restoring into MongoDB an `N` key value past Decimal128 | Not applicable | A store-backed restore into the MongoDB backend of a backup holding an `N` sort-key value of more than 34 significant digits (the sort-key column is BSON Decimal128) fails: the restore job applies the file through the write path, whose `ValidationException` ("Numeric sort key value '' exceeds supported precision (Decimal128, 34 significant digits)") becomes the restore's `FAILED` reason, so the client sees the target `CREATING` and then gone and the server log carries the text. PostgreSQL, SQLite and Cassandra restore the same value (their sort-key columns are numeric or decimal). A non-key `N` of up to 38 digits is stored as the item's text and restores intact on every backend, MongoDB included *(wire tests against the MongoDB runner)* | | `ListBackups` paging | At most 100 summaries per page, with `LastEvaluatedBackupArn`; `Limit` 1 to 100. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list. `TableName` is checked as the model says (3 to 255, `[a-zA-Z0-9_.-]+`, the "N validation error(s) detected" texts). An `ExclusiveStartBackupArn` of another account is `ValidationException` "Invalid Request: The supplied ExclusiveStartBackupArn does not match the account"; of another region, or of a table other than `TableName`'s, `ValidationException` ("does not match the region", "does not match the table name predicate"); a time bound before the epoch is `ValidationException`; a lower bound above the upper bound is `ValidationException` "Time range lower bound must be less than or equal to upper bound" (all measured) | The same paging on all four backends: `Limit` outside 1 to 100 is `ValidationException`, `ExclusiveStartBackupArn` resumes after the given backup, and summaries are ordered by creation time then ARN. Summaries carry `TableId`. `TimeRangeLowerBound` (inclusive) and `TimeRangeUpperBound` (exclusive) keep backups created in that range, and `BackupType` keeps backups of that type (`USER` when absent, every type for `ALL`); every ExtendDB backup is `USER`, so `SYSTEM` and `AWS_BACKUP` list nothing. The filters apply before paging, so `Limit` and the cursor count only backups that pass. Measured on SQLite and PostgreSQL with whole-second bounds between two backups taken 2 s apart; the millisecond edges are from the engine's unit tests. The request checks above are the service's, with its texts where captured: the account text in full; the region and table-name texts complete the captured fragments on the account text's pattern ("Invalid Request: The supplied ExclusiveStartBackupArn does not match the region" / "... the table name predicate"), checked in that order after the ARN's shape; a bound before the epoch is refused with ExtendDB's own text naming the member; equal bounds list nothing. An `ExclusiveStartBackupArn` that is not in the listing (excluded by the filters, or deleted since the previous page) is placed by the creation time in its id; one whose id carries no time (the 13-digit ids of earlier builds, with or without the hex suffix: only the 14-digit form carries one) is then a `ValidationException`. A cursor in another region is accepted only when it is the stored ARN of a backup of the account (or of `TableName`), which is what `LastEvaluatedBackupArn` hands out for a backup created under an earlier `server.region`, so paging passes such a backup; any other cursor in another region is refused ("does not match the region"). If that backup is deleted between two pages, its ARN is no longer a stored one and the next page is refused the same way (a same-region cursor would be placed by its time instead); start the listing again. Under a `server.region` outside `[a-z0-9-]` the cursor is still the stored ARN: a listing of backups created under a valid region pages, and a cursor that is the ARN of a backup an earlier release created under the invalid region is refused with "Invalid Request: BackupArn is not valid". `TableName` is a table name or a table ARN, as `CreateBackup` accepts it (the service accepts both, measured on DynamoDB): the ARN form lists what the name lists, an ARN whose region is not the server's (compared as written) is `ValidationException` "1 validation error detected: Invalid AWS region in ''" (the service's answer, measured on DynamoDB twice), another account's ARN is `AccessDeniedException`, and the cursor's table check runs against the resolved name. A numeric `TableName` is refused with the text the service uses for a map ("Start of structure or map found where not expected"), not measured for a number. Every `ListBackups` request reads all of the account's backups (or all of the table's, with `TableName`) from storage and pages them in memory, so walking N backups 100 at a time reads about N²/100 rows in total *(from the code)* | | Backup ARN and creation time | Backup id is 14-digit zero-padded epoch milliseconds, `-`, 8 hex digits; `BackupCreationDateTime` has sub-second precision | The same shape on all four backends; `BackupCreationDateTime` is the millisecond in the ARN | | Running two servers against one MongoDB catalog | Not applicable | MongoDB uses the fixed database names `extenddb_catalog` and `extenddb_data` whatever `--catalog-db` and `--data-db` say, and takes no serve lock: two servers on one replica set share one catalog and both start. Each server's reconciler acts only on a restore whose job record's heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, so one server does not remove another live server's restore; every heartbeat a restore writes, the registration's first one included, is stamped with the database's `$$NOW` inside the registration's transaction, so a host whose clock is behind the database's plays no part in that judgement *(from the code and storage-level tests)*. Running more than one server per catalog is otherwise unsupported (see the deployment guide) | @@ -209,7 +210,7 @@ ExtendDB exposes runtime settings that have no DynamoDB equivalent: | `control_plane_delay_seconds` | 5 | Simulated delay for table state transitions (CREATING → ACTIVE, DELETING → removed) | | `index_propagation_delay_ms` | 10 | System-wide default propagation delay for asynchronous secondary-index maintenance (milliseconds), covering GSIs and vector indexes alike. Per-GSI overrides stored in catalog; vector indexes have no per-index override. 0 = synchronous. Accepts the pre-rename name `gsi_propagation_delay_ms` as a deprecated alias, and a catalog created before the rename keeps honouring a value stored under it. | | `throttling_enabled` | `true` | Enable provisioned capacity throttling (token bucket per table/partition) | -| `enable_multipart_keys` | `false` | Enable multi-part base table key extension | +| `limits.allow_multipart_table_keys` | `false` | Enable multi-part base table key extension | | `log_level` | `info` | Runtime log level (trace, debug, info, warn, error) | | `sqlx_log_level` | `warn` | Separate log level for sqlx query traces | | `allow_credential_import` | `true` | Allow importing credentials via the management API | diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index 376072943..23649d7a1 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -128,13 +128,13 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | Limit | DynamoDB Value | Status | Notes | |-------|---------------|--------|-------| -| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a server raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34 (the slots are capped at 16), with a warning naming both numbers, so the slot bound refuses no job the `max_concurrent_jobs` permit admitted (no change at the defaults: a pool of 10 has 4 slots for 4 jobs); above 16 jobs a 17th concurrent job is refused by the slot. A `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); PostgreSQL bounds open table snapshots at half its (raised) pool the same way, naming `[storage.postgres] pool_size snapshot-slot`; a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | +| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request that finds every slot held waits up to 30 s for one (holding nothing while it waits: the permit is the last of its checks) and is then refused with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1` on an earlier build that did not wait: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a server raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34 (the slots are capped at 16), with a warning naming both numbers, so the slot bound refuses no job the `max_concurrent_jobs` permit admitted (no change at the defaults: a pool of 10 has 4 slots for 4 jobs); above 16 jobs a 17th concurrent job is refused by the slot. A `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); PostgreSQL bounds open table snapshots at half its (raised) pool the same way, naming `[storage.postgres] pool_size snapshot-slot`; a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A restore request on any backend that is not registered within 60 seconds (`RESTORE_REGISTRATION_TIMEOUT`) answers an internal error while the restore continues in the background. A restore from an S3 store holds the compressed backup size at `spool_path` while it runs; a restore's write batch on SQLite and Cassandra is bounded at 32 MiB of item text, measured on the serialized item (not on the service's `item_size_bytes`), and on Cassandra the item is serialized once more to measure it, a CPU cost per item the bound buys; a restore ends by counting the target against the manifest's item total, which on Cassandra is a full `COUNT(*)` of the restored table (one scan of a CREATING table, run once per restore and once more on a resume without recorded progress; no run has measured its cost past 12,000 rows, the largest Cassandra restore measured; a transient failure of this count is retried a few times, and a count that keeps failing leaves the record RUNNING for the reconciler to resume rather than failing the restore and removing its loaded target) | | `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed and backed up whatever its name, as on the service: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it before it opens a snapshot or writes a row, and the in-catalog create (no store, Cassandra) carries it (see the differences document's backup creation row). Both members may also be the table's ARN; an ARN whose region is not the server's is `ValidationException` "1 validation error detected: Invalid AWS region in ''", the service's text. The value is echoed in the message as sent | | `BackupArn` | 37 to 1024 characters | Maximum enforced; minimum implied | Surrounding ASCII whitespace is trimmed first, then over 1024 bytes, not of the backup ARN shape, a control or non-ASCII character, or a region outside `[a-z0-9-]`, an account that is not digits, or a table name outside `[a-zA-Z0-9_.-]` is `ValidationException` "Invalid Request: BackupArn is not valid"; the right shape with an id that is not a backup id is "Invalid Backup ARN"; a digit-only account that is not the caller's is `AccessDeniedException`. No minimum is checked: the shortest string of the right shape is longer than 37 characters. Another region of the caller's account resolves (the server's region is tried first, then the ARN as written). `ListBackups`'s `ExclusiveStartBackupArn` must also match the account, the region, and `TableName` when given | | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | | Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB (Cassandra, any backend without a store, and SQLite on a platform where the serve lock is not taken (not Unix), write the in-catalog format) and where restores and deletes of store-backed backups read: a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses CreateBackup where it writes to the store and the restore and delete of store-backed backups with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | | Backup data file | Not specified | N/A | A data file holds one item per line as DynamoDB JSON, gzip-compressed, in key order on a table whose key schema is orderable (`item_order` in the manifest); a line is at most 32 MiB (`MAX_LINE_BYTES`, which holds the largest line a 400 KB item can expand to, 27.7 MB); a file closes at the first item that takes its uncompressed bytes to or past `[backup] data_file_target_bytes` (default 64 MiB, minimum 32 MiB, the line cap), so it exceeds the knob by less than one line. The snapshot streams on PostgreSQL and SQLite page the table by key and cut each page at 64 MiB of item text (`SNAPSHOT_FETCH_BYTES`, a constant), so a page holds at most that plus one row; MongoDB's batches are bounded by the server's 16 MiB reply | -| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root (the differences document's "Where backups live" row); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | +| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root and restores into the backend that took it, and into another backend once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes (the differences document's "Where backups live" row). Vector indexes are restored where the target supports them, and a backup of a multi-part-key table (the `limits.allow_multipart_table_keys` preview) restores with its item set intact (the engine's `a_multi_part_key_manifest_is_registered` and `tests/test_backup_job.py`'s multi-part cell, which restores such a backup on a server with the preview enabled and compares the restored item set with the source's); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | | Point-in-time recovery | 35-day window | N/A | Not supported: `UpdateContinuousBackups` refuses to enable it (`ContinuousBackupsUnavailableException`) and `RestoreTableToPointInTime` refuses every restore (`PointInTimeRecoveryUnavailableException`); use on-demand backups (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/03-usage-guide.md b/docs/manuals/03-usage-guide.md index 63c45d01b..f2b4c96e7 100755 --- a/docs/manuals/03-usage-guide.md +++ b/docs/manuals/03-usage-guide.md @@ -571,7 +571,7 @@ extenddb reproduces DynamoDB error responses exactly. Common errors: | PartiQL | ✗ (not planned) | | DAX | ✗ (not applicable) | | Global Tables | ✗ (future) | -| On-demand backups (CreateBackup / RestoreTableFromBackup) | Partial: CreateBackup ✓ (to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB; in-catalog without one and on Cassandra); RestoreTableFromBackup ✓ for in-catalog backups only, store-backed backups from a later release (their restore answers a `ValidationException` naming the store this release; the upgrade manual) | +| On-demand backups (CreateBackup / RestoreTableFromBackup) | ✓: CreateBackup to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB (in-catalog without one and on Cassandra); RestoreTableFromBackup from the store into every backend (a filesystem or S3 store; a backup taken on one backend restores into another once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes), and of in-catalog backups | | Point-in-time recovery | ✗ (`UpdateContinuousBackups` refuses to enable it; see differences-from-dynamodb.md) | | Import/Export (local filesystem) | ✓ (FileSource instead of S3) | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 28ceef124..9e1b44f59 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -151,11 +151,11 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau |-----|---------|-------------| | `data_file_target_bytes` | `67108864` | Uncompressed bytes per backup data file (minimum 32 MiB, the cap on one line): a data file closes at the first item that takes it to or past this, so it exceeds it by less than one line | | `heartbeat_interval_secs` | `15` | The job heartbeat timer. The store-backed backup job heartbeats every `max(heartbeat_interval_secs, 15)` s and starts no store write unless a beat confirmed its claim within two periods or a beat is still in flight (on SQLite a beat waits behind any writer holding the engine write lock, a GSI backfill say, for as long as it holds it, and a backup goes on writing meanwhile; on the other backends a beat in flight counts for one period from its start), the claim's insert counting as the first beat; the in-catalog backup and restore paths heartbeat every 15 s whatever this says | -| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead restore job is FAILED and its target removed. Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | +| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead in-catalog restore job is FAILED and its target removed; a dead store-backed restore is resumed from its recorded progress (the differences document's restore-interruption row). Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | | `failed_row_retention_secs` | `604800` | How long FAILED and DELETED backup rows of the configured store are kept once their objects are gone (7 days); `extenddb backup list --failed` prints the FAILED ones with their reason. At least `stale_job_after_secs`: the reconciler deletes a stopped job's late object only under its row, so the row outlives the window in which that job can still write; a smaller value is refused at startup | | `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup. Each cycle checks the store, resolves backup and restore jobs whose owner is dead, finishes deletions, deletes the objects of FAILED backups once their heartbeat is older than `stale_job_after_secs`, and removes tombstones past `failed_row_retention_secs` (the differences document's restore-interruption row lists the duties) | -| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen) | -| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | +| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it waits up to 30 s for a slot, then `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen). A running restore holds the parsed manifest, one decoded data-file line and its parse, the decoded items of 32 MiB of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions between the decoder and the load (two in the channel, the one being filled, the one being drained; a chunk also closes at 1,000 items, and the chunks never hold more than one data file's items), and the backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text; 500 items (or 4 MiB) per INSERT statement inside one transaction per data file on PostgreSQL; 1,000 documents per `insert_many` on MongoDB (the driver splits the call); times this knob. For a table of large items that is the same order as the backup, not less: measured `VmHWM` over idle from a fresh start (`docs/design/15-backup-restore-bench-pr9.md`), 24 items of 27.7 MB restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), and 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`, the same setting as above; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`; the count's share was 22 to 28 MiB at 1M and at 3M rows) and 16 MiB on PostgreSQL, and the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s against 27 to 56 s key-ordered (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16 MiB on PostgreSQL; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. Size the host for about 450 MiB per running restore of large items under `MALLOC_ARENA_MAX=1`, times this knob, plus about 32 MiB of page cache per running restore of an indexed table on SQLite, and read the 1M figure for 200-byte items as the key-ordered shape's: an unordered backup of them restores in about 1.5 to 1.8 times that on SQLite | +| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. A store-backed restore from an S3 store downloads each data file here under `restore-
    /` (`.json.gz`, `` zero-padded to six digits, with a `.verified` marker holding the file's sha256, written only after the file's digests matched and trusted only while it names the digest the manifest records, plus the manifest and sidecar), reads it from there, and removes the entry when the restore ends, except when it ended because the record had become another instance's (its claim lost, a status write that matched no record), where the sweep removes it; an entry left by an earlier run of this host for a record another host then owned is removed before a new run of the same restore uses it; the reconciler sweeps the `restore-` entry of a restore no longer RUNNING with this host's `spool_host`. Its disk cost is the compressed backup of the restore in flight. A filesystem store needs no spool (files are hashed in place and read again). Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | #### [backup.store] @@ -167,13 +167,24 @@ The store CreateBackup writes to on PostgreSQL, SQLite and MongoDB, and the one | `filesystem.path` | `backups/` beside a SQLite database file | The root directory of a filesystem store. It must exist, be owned by the extenddb user with mode `0700`, and sit under directories owned by root or that user which others cannot write to; `validate` refuses anything else at startup. A relative path, like a relative `spool_path`, is resolved against the working directory the server was started from (the daemon's own working directory is `/`) | | `s3.bucket`, `s3.prefix`, `s3.region`, `s3.endpoint`, `s3.force_path_style`, `s3.allow_insecure_endpoint`, `s3.upload_part_size_bytes`, `s3.max_concurrent_uploads` | | The S3 store; see the crate's `config` module doc. Credentials come from the AWS SDK's chain and are never in the file | -When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call. With one configured, CreateBackup on PostgreSQL, SQLite and MongoDB writes the backup to the store: the request takes the table's snapshot (after its checks and after any wait for a job permit, up to 30 s, so the backup is the table at that open and a write acknowledged during the wait is in it), writes the catalog row and answers `CREATING`, and a job of the server uploads gzip data files and the manifest under `/
    //` below the root and marks the row `AVAILABLE` (the job's log lines carry the ARN: `backup : started, items in the table as last counted by the stats pass, key-ordered` (or `unordered`, for a table whose key shape or column collation has no format order) at the start, the stats pass's count, which is 0 on a table the pass has not reached, and `backup : AVAILABLE, items, bytes written in s` at the end, with the backup's own count); Cassandra, which has no table snapshot, keeps the in-catalog create with a store configured. Backups of either format stay listable, describable and deletable, and in-catalog backups restorable; the restore of a store-backed backup is a later release's and answers `ValidationException` until then. +When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call. With one configured, CreateBackup on PostgreSQL, SQLite and MongoDB writes the backup to the store: the request takes the table's snapshot (after its checks and after any wait for a job permit, up to 30 s, so the backup is the table at that open and a write acknowledged during the wait is in it), writes the catalog row and answers `CREATING`, and a job of the server uploads gzip data files and the manifest under `/
    //` below the root and marks the row `AVAILABLE` (the job's log lines carry the ARN: `backup : started, items in the table as last counted by the stats pass, key-ordered` (or `unordered`, for a table whose key shape or column collation has no format order) at the start, the stats pass's count, which is 0 on a table the pass has not reached, and `backup : AVAILABLE, items, bytes written in s` at the end, with the backup's own count); Cassandra, which has no table snapshot, keeps the in-catalog create with a store configured. Backups of either format stay listable, describable, deletable and restorable: a store-backed backup is restored by the restore job (the differences document's "How `RestoreTableFromBackup` runs" row; the `spool_path` row above and the `remove-target` command below are its operator surface), an in-catalog one by the previous path. Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), CreateBackup (where it writes to the store) and the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check's store calls are bounded (5 s for a filesystem root, 30 s for a bucket); the catalog's own reads (the catalog id, the rows' store ids, the newest AVAILABLE row) run beside the calls and not under the bound, so a slow catalog costs the check time and never marks the store unhealthy. A store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned check's call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), a root's calls run one at a time so that calls started before the first reached its bound (two `DeleteBackup` requests together, a `DeleteBackup` beside a check) wait for it under the bound, then have the bound for their own call (a slow call behind another finishes, where one bound over both would abandon it), and park no thread of their own, and `stop` or SIGTERM still ends the server, the runtime giving that thread 5 s before the process exits without it (after the job wait of up to 30 s when a backup job is running; see Stopping). The deletions `DeleteBackup` and the reconciler run against the store go through the same bound and rule: `DeleteBackup` answers the description with `BackupStatus` `DELETED` once its DELETING write has committed, and a deletion the store does not answer within the bound, or that leaves an object under the prefix, leaves the row DELETING for the reconciler and is logged, never an error to the client. A default SQLite root whose startup checks did not finish starts `checking` with its rule kept: a first check that fails while the catalog has no rows for it is the same warning and leaves the store unconfigured. CreateBackup on Cassandra is not refused by an unhealthy store: its in-catalog create touches no store. The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no AVAILABLE backup for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id, and a file copied from one directory to the other gives both one id), and the manifest probe has nothing to probe: a backup one instance completed under its own directory looks, to the other instance's reconciler, like a dead owner's row with no manifest, which it marks FAILED, and the first instance's restart then deletes the only copy. Point every instance at one shared root before the first backup; copying `extenddb-store.json` between directories defeats the check rather than satisfying it. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. -`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. Neither command runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database under the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse; the one exception is a RUNNING restore of kind `manifest`, which no release before PR 9a writes: the command leaves it for the reconciler that resumes it, waits on it as on any RUNNING restore and refuses naming the table; and a server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra both commands open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. None of the backup commands runs the startup recovery a server runs (on PostgreSQL the vector index rebuild, the control-plane recovery and the GSI queue workers; on MongoDB the key-repair pass; on Cassandra the control-plane transitions; on SQLite the interrupted-restore and index passes): that work belongs to the servers, and a command that ran it could exit with it half done. RUNNING restores are read by status, with no age, so a restore whose record was stamped this instant holds the wait too. On SQLite `purge --confirm` opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says, a heartbeat stamped ahead of the clock by a backward clock step included (the server that wrote it is stopped, so the zero window the command passes selects by status alone), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL, MongoDB and Cassandra the backup commands (`list`, `purge` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup remove-target`: on PostgreSQL the session's `application_name`, on MongoDB the `command` field of its `extenddb_servers` document, on Cassandra the `release` column of its `extenddb_servers` row, after the release and ` as `), and release it when they exit: `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release what it holds and exit: the serving registration after the open, if the signal came during it; the claim at once while it waits, and before its next backup while it deletes (the release itself is bounded by 10 s, past which the claim is left to the stale rule), so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. The dry run on an unhealthy store lists the rows and says that `--confirm` refuses until the store's check passes. + +`extenddb backup remove-target --account
    ` removes a restore target that `DeleteTable` refuses (a `CREATING` table) together with its restore record: the target of a `FAILED` restore whatever the reason (`target_not_empty` included: the state a resume leaves when it finds no recorded progress and more items in the target than the backup holds, which a catalog restored from a dump can produce when it names a restore that finished and was written to since; an ordinary crash before the first progress record on Cassandra leaves no more items than the backup holds and is resumed, not failed; until the command runs a `target_not_empty` target keeps its backup in use, so `DeleteBackup` and a second restore of the backup are `BackupInUseException` and the name is held; `owner dead: taken over 5 times by one instance without the restore ending` is the reason of a record one server's reconciler resumed five times without the restore ending, whose target the reconciler removes itself), the target of a `RUNNING` restore whose owner is dead (no heartbeat within `stale_job_after_secs` on the database's clock, the listing the reconciler uses; on SQLite under the serve lock every owner is dead), and a legacy (in-catalog) restore's target the same way. It refuses a `RUNNING` restore with a live owner ("stop it or wait for it"), a `DONE` restore and a table no restore record names, the latter two as "not a restore target", and exits 1. It needs no store (it runs whether or not `[backup.store]` is configured, and creates neither the default root nor its identity file), takes the catalog-level maintenance claim first as `purge` does and releases it before it returns, logs the removal under `extenddb::audit` with the account, the table and the backup's ARN, and prints what it did: + +``` +=== extenddb backup remove-target === +Taking the maintenance claim as "extenddb backup remove-target on (pid )". +removed the restore target
    (account , table id
    ) and its record: its store-backed restore from was FAILED (target_not_empty) +The maintenance claim is released. +``` + +The third line's tail names the record's state: `was FAILED ()` or `was RUNNING with a dead owner`, after `its store-backed restore from ` or `its in-catalog restore from ` for the record's kind. A refusal is one `Error:` line on stderr (`
    (account ) is not a restore target: no restore record names it`; `... its restore from is DONE (the table is ACTIVE, or awaits its index backfill)`; `the restore into
    (account ) from is RUNNING with a live owner (a heartbeat within stale_job_after_secs); stop it or wait for it`; on PostgreSQL, `the restore into
    (account ) from is a CREATING restore target (table id ) whose owner's lock is still held: its owner's session is alive though its heartbeat is stale; wait for the owner to stop, or stop it, and run the command again`, the owner's claim session holding the target's advisory lock, a frozen process for instance; `the maintenance claim is held by ""; one maintenance command runs at a time, and a claim whose holder died expires after stale_job_after_secs ( s)`). On SQLite it opens the database as `serve` does and takes the serve lock: stop the server first, as for `purge`; under that lock every owner is dead, so a `RUNNING` record's target is removed too. After it, `DescribeTable` of the name is `ResourceNotFoundException`, `DeleteBackup` of the backup succeeds and a new restore into the name runs. #### [auth] diff --git a/docs/manuals/06-developer-test-guide.md b/docs/manuals/06-developer-test-guide.md index 3eefa6ea2..6482a1a63 100755 --- a/docs/manuals/06-developer-test-guide.md +++ b/docs/manuals/06-developer-test-guide.md @@ -157,14 +157,71 @@ devtools/run-tests --extenddb --pytest --filter test_put_item server without the section has by default (`backups/` beside its database file); `s3` for an S3 store; `none` for a server without a store (`kind = "disabled"`, or PostgreSQL, MongoDB and Cassandra without the - section). With a configured store CreateBackup is asynchronous and this - release refuses to restore its backups, so the backup tests derive their - expected outcome from the variable and fail rather than guess when it is - unset; `run-tests --pytest` refuses before any test runs with + section). With a configured store CreateBackup is asynchronous and + RestoreTableFromBackup of its backups is the store-backed restore job + (the response is `CREATING` and the table is loaded in the background), + so the backup tests derive their expected outcome from the variable and + fail rather than guess when it is unset; `run-tests --pytest` refuses before any test runs with `error: --extenddb --pytest requires EXTENDDB_TEST_BACKUP_STORE to be set.` The suites check the value against the server's `/health` `backup_store` component, so a wrong value fails too. +**The backup store and job suites** (`tests/test_backup_store_config.py` +and `tests/test_backup_job.py`) start servers of their own, so `run-tests` +excludes them and they run by name: +`python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v`. +`EXTENDDB_BINARY` names the release build under test and decides which +deployment the cells use: a SQLite build (the default root beside the +database file) or a PostgreSQL build, which also needs +`EXTENDDB_TEST_PG_CONNECTION_STRING` and +`EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING`. The S3 cells (the store +configuration suite's S3 rows and the job suite's store-outage cell) need a +moto server at `EXTENDDB_TEST_S3_ENDPOINT` (`http://127.0.0.1:`); +without it they skip. They set the credentials moto accepts +(`testing`/`testing`, `us-east-1`) themselves, in the server's environment +and on their own client, so nothing else needs to be exported. The job suite's restore cells size their +backup from `EXTENDDB_TEST_RESTORE_ITEMS` (default 10,000; at least two data +files are needed), and the multi-part-key cell enables +`[limits] allow_multipart_table_keys` on its own deployment. CI runs both +suites on the PostgreSQL job and the two SQLite jobs; the PostgreSQL job +and the SQLite S3 job start moto, so the outage cell runs there and skips on +the plain SQLite job (`.github/workflows/integration.yml`). + +**The MongoDB backup cells** of `tests/test_mongodb_backup_restore.py` that +read the store's root and the server's log skip unless +`EXTENDDB_TEST_BACKUP_ROOT` (the server's filesystem store root) and +`EXTENDDB_TEST_SERVER_LOG` (the file the server logs to) are set; +`devtools/run-mongodb-tests` serves a filesystem store and exports both +(and `EXTENDDB_TEST_BACKUP_STORE=filesystem`), so run them through it: +`devtools/run-mongodb-tests -- --pytest --filter test_mongodb_backup_restore`. +The suite's legacy LSI fallback cell edits an in-catalog backup and runs +only on a server without a store: +`devtools/run-mongodb-tests --backup-store none -- --pytest --filter test_mongodb_backup_restore` +(the runner then writes no `[backup.store]` and exports +`EXTENDDB_TEST_BACKUP_STORE=none`; the store-backed cells skip under it). +The CI jobs run the suite with `filesystem`. + +**The cross-backend restore matrix** is `devtools/backup-restore-matrix`: +one cell per invocation (`--source --target +`, plus `--duplicate-key` or `--vector`), +each starting a source and a target server from the per-backend release +binaries (`--bin-`, defaulting to +`target/rel-/release/extenddb`) against PostgreSQL at +`127.0.0.1:55432`, MongoDB at `127.0.0.1:27117` and Cassandra at +`127.0.0.1:9042`; its MongoDB and Cassandra cells need `pymongo` and +`cassandra-driver` (`requirements-matrix.txt`). The nightly workflow +`.github/workflows/matrix-nightly.yml` builds the four binaries, starts the +three services and runs the twelve plain cells, the four duplicate-key +cells and the eight vector cells; `workflow_dispatch` runs it on demand. The +same workflow runs the restore kill points of every backend in four jobs +(`restore-kill-points-`), each building its backend's binary with +`extenddb-app/backup-test-hooks` and starting its backend as the matrix job +does, through `devtools/backup-kill-points.py restore +--bin --work ` (the Cassandra job builds a plain SQLite binary +too, the `--source-bin` whose backup the Cassandra target restores); every +point ends ACTIVE with the exact base and GSI counts, except the +tampered-file point, whose target is removed. + The `run-tests` script automatically: - Performs health check on the target endpoint - Provisions test credentials via `devtools/provision-test-credentials` diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 6c81ffbdc..103374a71 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -4,9 +4,9 @@ ## Current Status -This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 6, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), and migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)). +This release versions the PostgreSQL, SQLite, Cassandra and MongoDB catalogs by a migration ledger and carries catalog migrations 1 to 7, which `extenddb migrate --list` prints. **Every deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema, migration 5 drops `continuous_backups`, a table and on MongoDB a collection (see [Catalog 0.0.5](#catalog-005)), migration 6 adds the backup catalog index and the restore job records ([below](#migration-6-the-backup-catalog-index)), and migration 7 adds the `restore_progress` table to the PostgreSQL data database (it does not apply on the other backends). -Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described and deleted like any other, but its restore is a later release's: until the restore job lands, RestoreTableFromBackup of a backup in the configured store answers `ValidationException` ("is in a backup store; this release restores in-catalog backups only"), while in-catalog backups, those of the previous release included, restore as before; a deployment that needs to restore the backups it takes this release stays on the in-catalog format (`kind = "disabled"`, below) until then. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. +Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 7 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 7 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
    ` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. @@ -41,7 +41,7 @@ Exit status 6 means the migration lock, or a running server, was still held when `verify` also checks the connection to the data database. When that check fails, `verify` exits 1 on a Behind catalog instead of 3, because `migrate --yes` cannot apply the pending migrations while the data database cannot be reached. It still exits 4 or 5 for the statuses that need a person. -A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 6 are breaking: do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). +A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 6 are breaking. Migration 7 creates a table in the PostgreSQL data database that the previous release never reads; on PostgreSQL it is declared breaking only because the job that proves an additive declaration against the old binary does not yet exist (CONTRIBUTING, step 4 of adding a migration), and on SQLite, MongoDB and Cassandra it does not apply (`migrate --list` prints it as not applicable, and `migrate` counts it as additive). Do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). ### Migration 6: the backup catalog index @@ -258,7 +258,7 @@ On SQLite, `extenddb serve` reads the catalog's migration ledger before it opens The `extenddb migrate` command that `serve`, `migrate` and `verify` advise names the config file by its absolute path, quoted for a POSIX shell, so it can be pasted as printed. A relative `path` in `[storage.sqlite]`, such as the `extenddb.sqlite` that `init` writes, resolves against the directory a command runs in. So when the config file names the database by a relative `path`, or names none, the advised command starts with `EXTENDDB__STORAGE__SQLITE__PATH=`, and it reaches the same database from any directory. -On startup the SQLite server ages every RUNNING restore record to the epoch, which is only safe because one server at a time can hold the file, and its reconciler's first cycle then fails an in-catalog restore that a server stopped mid-copy and removes its `CREATING` target (the cycle runs beside the listener, so a request can still see the target `CREATING` until it ends). A backup keeps a WAL read snapshot open while it copies; checkpoints cannot reclaim pages needed by that snapshot, so allow disk headroom for WAL growth proportional to the backup duration and concurrent write volume. Set `pool_size` to 3 or more (see the Admin Guide); the server runs at most `pool_size / 2 - 1` backups at once (4 with the default of 10) and answers `LimitExceededException` at once to a `CreateBackup` that finds no slot free. Running backups hold one connection each for their read snapshot plus one shared writer (`n` backups hold `n + 1` connections), so from `pool_size` 4 up a flood of backups leaves at least half the pool to other requests; with a `pool_size` of 2 or 3 the one admitted backup holds two of them. A GSI added by `UpdateTable` with `ProvisionedThroughput` on a `PROVISIONED` table made every `DescribeTable` of that table fail on earlier builds (`Internal("index pt: missing field NumberOfDecreasesToday")`); this build writes the index's throughput in the shape `DescribeTable` reads and reads the shape earlier builds wrote (as the same units with `NumberOfDecreasesToday` 0), so such a table is describable again after the upgrade without any change to the database file. +On startup the SQLite server ages every RUNNING restore record to the epoch, which is only safe because one server at a time can hold the file, and its reconciler's first cycle then fails an in-catalog restore that a server stopped mid-copy and removes its `CREATING` target (the cycle runs beside the listener, so a request can still see the target `CREATING` until it ends); a restore from a backup store is resumed instead, from its recorded progress, and its target stays `CREATING` until it finishes. A backup keeps a WAL read snapshot open while it copies; checkpoints cannot reclaim pages needed by that snapshot, so allow disk headroom for WAL growth proportional to the backup duration and concurrent write volume. Set `pool_size` to 3 or more (see the Admin Guide); the server runs at most `pool_size / 2 - 1` backups at once (4 with the default of 10) and answers `LimitExceededException` at once to a `CreateBackup` that finds no slot free. Running backups hold one connection each for their read snapshot plus one shared writer (`n` backups hold `n + 1` connections), so from `pool_size` 4 up a flood of backups leaves at least half the pool to other requests; with a `pool_size` of 2 or 3 the one admitted backup holds two of them. A GSI added by `UpdateTable` with `ProvisionedThroughput` on a `PROVISIONED` table made every `DescribeTable` of that table fail on earlier builds (`Internal("index pt: missing field NumberOfDecreasesToday")`); this build writes the index's throughput in the shape `DescribeTable` reads and reads the shape earlier builds wrote (as the same units with `NumberOfDecreasesToday` 0), so such a table is describable again after the upgrade without any change to the database file. #### Cassandra diff --git a/docs/schema/postgres.txt b/docs/schema/postgres.txt index 144a6d8c4..f14e9a125 100644 --- a/docs/schema/postgres.txt +++ b/docs/schema/postgres.txt @@ -346,6 +346,7 @@ sequence stream_seq: bigint START 1 INCREMENT 1 table extenddb_migrations table gsi_pending table idempotency_tokens +table restore_progress table schema_history table stream_records table stream_shards @@ -369,6 +370,9 @@ column idempotency_tokens.account_id: text NOT NULL column idempotency_tokens.token: text NOT NULL column idempotency_tokens.fingerprint: text NOT NULL column idempotency_tokens.created_at: timestamp with time zone NOT NULL DEFAULT now() +column restore_progress.table_id: text NOT NULL +column restore_progress.file_index: bigint NOT NULL +column restore_progress.items_applied: bigint NOT NULL column schema_history.filename: text NOT NULL column schema_history.applied_at: timestamp with time zone NOT NULL DEFAULT now() column stream_records.shard_id: text NOT NULL @@ -394,6 +398,7 @@ index idx_idempotency_tokens_created: CREATE INDEX idx_idempotency_tokens_create index idx_stream_records_created: CREATE INDEX idx_stream_records_created ON public.stream_records USING btree (created_at) index idx_stream_shards_table: CREATE INDEX idx_stream_shards_table ON public.stream_shards USING btree (table_id) index idx_vector_index_holds_table: CREATE INDEX idx_vector_index_holds_table ON public.vector_index_holds USING btree (table_id) +index restore_progress_pkey: CREATE UNIQUE INDEX restore_progress_pkey ON public.restore_progress USING btree (table_id, file_index) index schema_history_pkey: CREATE UNIQUE INDEX schema_history_pkey ON public.schema_history USING btree (filename) index stream_records_pkey: CREATE UNIQUE INDEX stream_records_pkey ON public.stream_records USING btree (shard_id, sequence_number) index stream_shards_pkey: CREATE UNIQUE INDEX stream_shards_pkey ON public.stream_shards USING btree (shard_id) @@ -401,6 +406,7 @@ index vector_index_holds_pkey: CREATE UNIQUE INDEX vector_index_holds_pkey ON pu constraint extenddb_migrations.extenddb_migrations_pkey: PRIMARY KEY (id) constraint gsi_pending.gsi_pending_pkey: PRIMARY KEY (id) constraint idempotency_tokens.idempotency_tokens_pkey: PRIMARY KEY (account_id, token) +constraint restore_progress.restore_progress_pkey: PRIMARY KEY (table_id, file_index) constraint schema_history.schema_history_pkey: PRIMARY KEY (filename) constraint stream_records.stream_records_pkey: PRIMARY KEY (shard_id, sequence_number) constraint stream_records.stream_records_shard_id_fkey: FOREIGN KEY (shard_id) REFERENCES stream_shards(shard_id) ON DELETE CASCADE diff --git a/requirements-matrix.txt b/requirements-matrix.txt new file mode 100644 index 000000000..9489acf13 --- /dev/null +++ b/requirements-matrix.txt @@ -0,0 +1,9 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# The drivers devtools/backup-restore-matrix uses to seed and tear down its +# MongoDB and Cassandra deployments (.github/workflows/matrix-nightly.yml +# installs this file beside requirements.txt). Separate from requirements.txt +# because nothing else in the test suites talks to those services directly +# and cassandra-driver builds a C extension on install. +pymongo==4.7.3 +cassandra-driver==3.29.3 diff --git a/tests/rust/src/backup_restore.rs b/tests/rust/src/backup_restore.rs index 4c09373db..640b3482d 100755 --- a/tests/rust/src/backup_restore.rs +++ b/tests/rust/src/backup_restore.rs @@ -23,9 +23,9 @@ use std::time::{Duration, Instant}; /// Where the server under test writes its backups, from /// `EXTENDDB_TEST_BACKUP_STORE`: `filesystem` or `s3` (a configured -/// `[backup.store]`, under which a backup is in the store and this release -/// refuses to restore it, PR 9a adding the restore) or `none` (the -/// in-catalog create, which restores). Against the service the variable is +/// `[backup.store]`, under which a backup is in the store and the restore +/// job restores it) or `none` (the in-catalog create, which the legacy +/// reader restores). Against the service the variable is /// not read (the restore runs). Against ExtendDB a test whose outcome depends /// on it fails when it is unset, with the text the Python fixture prints, /// rather than accept either outcome. @@ -306,49 +306,39 @@ async fn restore_table_from_backup() { .backup_arn(&arn) .send() .await; - match backup_store() { - // A backup in the configured store is listed, described and deleted - // but not restored this release (the restore job is PR 9a's). - BackupStore::Filesystem | BackupStore::S3 => { - let err = restore.expect_err("a store-backed backup restored"); - assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); - let msg = err_msg(&err); - assert!( - msg.contains("is in a backup store"), - "the refusal does not name the store: {msg}" - ); - println!("restore of a store-backed backup refused: {msg}"); - assert!( - c.describe_table() - .table_name(&restored) - .send() - .await - .is_err(), - "the refused restore created {restored}" - ); - } - BackupStore::None => { - let resp = restore.unwrap(); - assert_eq!( - resp.table_description() - .unwrap() - .table_status() - .unwrap() - .as_str(), - "CREATING" - ); - - wait_for_active(c, &restored).await; - let scan = c.scan().table_name(&restored).send().await.unwrap(); - assert_eq!(scan.count(), 5); - println!( - "restore of an in-catalog backup completed with {} items", - scan.count() - ); - - c.delete_table().table_name(&restored).send().await.ok(); - } - } + // The round trip on either path: a backup in the configured store is + // restored by the restore job, an in-catalog one by the legacy reader. + let resp = restore.unwrap(); + let desc = resp.table_description().unwrap(); + assert_eq!(desc.table_status().unwrap().as_str(), "CREATING"); + let summary = desc + .restore_summary() + .expect("a CREATING restore target carries its RestoreSummary"); + assert_eq!(summary.source_backup_arn(), Some(arn.as_str())); + assert!(summary.restore_in_progress()); + + wait_for_active(c, &restored).await; + let scan = c.scan().table_name(&restored).send().await.unwrap(); + assert_eq!(scan.count(), 5); + let after = c + .describe_table() + .table_name(&restored) + .send() + .await + .unwrap(); + assert!( + after.table().unwrap().restore_summary().is_none(), + "RestoreSummary is reported only while CREATING" + ); + println!( + "restore of a {} backup completed with {} items", + match backup_store() { + BackupStore::Filesystem | BackupStore::S3 => "store-backed", + BackupStore::None => "in-catalog", + }, + scan.count() + ); + c.delete_table().table_name(&restored).send().await.ok(); delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&table).send().await.ok(); diff --git a/tests/rust/src/restore_active_completeness.rs b/tests/rust/src/restore_active_completeness.rs index 93ca7bcd5..bab83e8f1 100644 --- a/tests/rust/src/restore_active_completeness.rs +++ b/tests/rust/src/restore_active_completeness.rs @@ -25,9 +25,8 @@ //! window. //! //! On a server with a configured `[backup.store]` (`EXTENDDB_TEST_BACKUP_STORE` -//! at `filesystem` or `s3`) the restore of a backup this server created is -//! refused this release (the restore job is PR 9a's), so the race test pins -//! that refusal in place of the race. +//! at `filesystem` or `s3`) the restore is the restore job's, from the store, +//! and the race runs against it as against the in-catalog copy. use crate::backup_restore::{ backup_store, delete_backup_quietly, wait_for_backup_available, BackupStore, @@ -169,33 +168,13 @@ async fn restored_table_has_all_items_when_first_active() { wait_for_backup_available(c, &arn).await; let dst = format!("RestoreRaceDst_{}", ts()); - // A backup in the configured store is not restored this release (the - // restore job is PR 9a's): the race has nothing to observe, and the test - // pins the refusal instead. - if matches!(backup_store(), BackupStore::Filesystem | BackupStore::S3) { - let err = c - .restore_table_from_backup() - .target_table_name(&dst) - .backup_arn(&arn) - .send() - .await - .expect_err("a store-backed backup restored"); - assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); - let msg = err_msg(&err); - assert!( - msg.contains("is in a backup store"), - "the refusal does not name the store: {msg}" - ); - println!("restore of a store-backed backup refused: {msg}"); - assert!( - c.describe_table().table_name(&dst).send().await.is_err(), - "the refused restore created {dst}" - ); - delete_backup_quietly(c, &arn).await; - c.delete_table().table_name(&src).send().await.ok(); - return; - } - println!("restore of an in-catalog backup: racing the observer"); + println!( + "restore of {} backup: racing the observer", + match backup_store() { + BackupStore::Filesystem | BackupStore::S3 => "a store-backed", + BackupStore::None => "an in-catalog", + } + ); // Race an observer against the in-flight restore: the moment it sees // ACTIVE, it counts. Returns None if ACTIVE never arrives within the // bound, which means the restore itself failed. diff --git a/tests/test_auth_permissions.py b/tests/test_auth_permissions.py index 3c2e33465..6cf89d714 100755 --- a/tests/test_auth_permissions.py +++ b/tests/test_auth_permissions.py @@ -1356,6 +1356,14 @@ def source_backup(self): backup_arn = admin_client.create_backup( TableName=table, BackupName="authz", )["BackupDetails"]["BackupArn"] + # A store-backed backup is CREATING until its job ends; a restore + # of a CREATING backup is BackupInUse, so the cells wait here. + deadline = time.monotonic() + 120 + while admin_client.describe_backup( + BackupArn=backup_arn + )["BackupDescription"]["BackupDetails"]["BackupStatus"] == "CREATING": + assert time.monotonic() < deadline, f"backup {backup_arn} still CREATING" + time.sleep(0.2) yield admin_client, table, backup_arn finally: for t in self.created: @@ -1446,14 +1454,6 @@ def test_restore_needs_the_item_actions_on_the_target(self, source_backup): """RestoreTableFromBackup on the backup plus the seven item actions on the target, each required; RestoreTableFromBackup on the target is neither required nor evaluated.""" - if os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() in ("filesystem", "s3"): - # The denials hold on any server; the restores that must run are - # of a backup in the configured store, which RestoreTableFromBackup - # refuses until the restore job lands (PR 9a). - pytest.skip( - "RestoreTableFromBackup of a backup in the configured store is refused " - "until the restore job lands (PR 9a)" - ) admin_client, table, backup_arn = source_backup user = f"u-{uuid.uuid4().hex[:8]}" ak, sk = self._user_with_key(user) diff --git a/tests/test_backup_job.py b/tests/test_backup_job.py index 90be4e39b..0d761c05b 100644 --- a/tests/test_backup_job.py +++ b/tests/test_backup_job.py @@ -25,12 +25,35 @@ named by its whole key, and the wire's reason for its value, in the log line and in `backup list --failed`. +The store-backed RestoreTableFromBackup job, on both deployments: + +- a data file tampered with under the root fails the restore before any item + is loaded, the target gone and the log naming the file; +- SIGTERM mid-restore leaves the record RUNNING and the restarted server + resumes it (PostgreSQL after the stale window, SQLite at once); +- an S3 store (moto) cut mid-download leaves the record RUNNING and the + restore completes once the store answers; +- an unordered backup holding one key in two files, and a key-ordered file + leaving its `key_range` (both hand-built under the root), fail the restore + with the target gone; +- a table keyed on two HASH attributes (the `allow_multipart_table_keys` + preview, enabled on the cell's own deployment) backs up and restores with + the source's item set, key schema and attribute definitions; +- (SQLite) a RUNNING restore record with no recorded progress whose target + holds more items than the backup, the state a catalog restored from a dump + leaves when it names a restore that finished and was written to since, + seeded at the catalog, is FAILED `target_not_empty` + by the restarted server's reconciler, the target left CREATING and + `DeleteTable`, `DeleteBackup` and a second restore refused, and `extenddb + backup remove-target` removes the target and its record, after which + `DescribeTable` is `ResourceNotFoundException` and `DeleteBackup` succeeds. + The binary at `EXTENDDB_BINARY` decides which deployment runs: a SQLite build, a PostgreSQL build (with `EXTENDDB_TEST_PG_CONNECTION_STRING`) or a MongoDB build (with `EXTENDDB_TEST_MONGODB_URI`, a replica set of this suite's own: the catalog's database names are fixed, so each test destroys the previous test's -catalog and initializes its own). Excluded from the backend-agnostic suite; run -by name. +catalog and initializes its own); the restore cells run on the SQLite and +PostgreSQL deployments. Excluded from the backend-agnostic suite; run by name. """ from __future__ import annotations @@ -57,13 +80,18 @@ EXTENDDB_BINARY, _append, _client, + _Forwarder, + _free_port, + _moto_or_skip, _pg_build, _provision, + _s3_env, _Server, _sqlite_build, _SqliteDeployment, _wait_backup_available, _wait_health, + moto_bucket, # noqa: F401 - the fixture of the store-outage cell ) ACCOUNT = "123456789012" @@ -85,15 +113,19 @@ class _Deployment: - """A deployment with a filesystem store, its config and its root.""" + """A deployment with a filesystem store, its config, its root and its + catalog (`catalog`: the SQLite database file, or the PostgreSQL catalog + database's connection string), where a cell edits a `backup_index` row + it rewrote the prefix of.""" - def __init__(self, backend: str, config: str, port: int, root: str, cert: str, cwd: str): + def __init__(self, backend: str, config: str, port: int, root: str, cert: str, cwd: str, catalog: str = ""): self.backend = backend self.config = config self.port = port self.root = root self.cert = cert self.cwd = cwd + self.catalog = catalog def run(self, *args): import subprocess @@ -159,6 +191,7 @@ def deployment(request, tmp_path) -> _Deployment: root = os.path.join(base, "backups") store = "" backend = "sqlite" + catalog = d.db_path elif _pg_build(): cli_env = request.getfixturevalue("cli_env") result = lh._run_extenddb( @@ -173,6 +206,7 @@ def deployment(request, tmp_path) -> _Deployment: os.mkdir(root, 0o700) store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n' backend = "postgres" + catalog = cli_env["conn_string"] elif _mongodb_build(): uri = os.environ.get("EXTENDDB_TEST_MONGODB_URI") if not uri: @@ -183,6 +217,8 @@ def deployment(request, tmp_path) -> _Deployment: os.mkdir(root, 0o700) store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n' backend = "mongodb" + # The restore cells that edit the catalog run on SQLite and PostgreSQL. + catalog = "" else: pytest.skip("EXTENDDB_BINARY is neither a SQLite, a PostgreSQL nor a MongoDB build") os.mkdir(os.path.join(base, ACCOUNT)) @@ -192,7 +228,7 @@ def deployment(request, tmp_path) -> _Deployment: + store + f'[import]\npaths = ["{base}"]\n', ) - return _Deployment(backend, config, port, root, cert, base) + return _Deployment(backend, config, port, root, cert, base, catalog) def _started(d: _Deployment, log_path: str) -> tuple[_Server, object, dict]: @@ -660,3 +696,848 @@ def test_an_item_over_max_item_size_bytes_fails_the_backup_without_naming_618(de assert listed == ["under-default"], listed finally: server.stop() + + +# --- the restore job --------------------------------------------------------------- + +# Items of this count and padding make a backup of two data files (the knob's +# floor is 32 MiB of uncompressed lines) whose restore runs for long enough, +# on a PostgreSQL build of this host, that a SIGTERM sent right after the +# restore's response finds the job inside its download or its first file's +# load. The cells check the precondition they rely on (the target still +# CREATING when the signal is sent) rather than assume it. +RESTORE_ITEMS = int(os.environ.get("EXTENDDB_TEST_RESTORE_ITEMS", "10000")) +RESTORE_PAD = 4000 + + +# The `[backup]` keys the restore cells add to the deployment's: data files +# close at the knob's floor, so RESTORE_ITEMS padded items make two of them. +RESTORE_BACKUP_SETTINGS = "data_file_target_bytes = 33554432\n" + + +def _add_backup_settings(d: _Deployment, extra: str) -> None: + """Add `extra` to the deployment's `[backup]` table before the server + starts (TOML refuses a second `[backup]` header).""" + with open(d.config) as f: + text = f.read() + head = "[backup]\nreconcile_interval_secs = 1\n" + assert text.count(head) == 1, text + with open(d.config, "w") as f: + f.write(text.replace(head, head + extra)) + + +def _load_padded(d: _Deployment, creds: dict, name: str, n: int, pad: int) -> None: + """`_load` with items of `pad` bytes of padding and a `g` the GSI keys + on, from 8 threads with one client each.""" + local = threading.local() + + def write(start: int) -> None: + if not hasattr(local, "client"): + local.client = _client(d.port, creds) + requests_ = [ + { + "PutRequest": { + "Item": { + "pk": {"S": f"p{i % 500:04d}"}, + "sk": {"N": str(i)}, + "g": {"S": f"g{i % 50}"}, + "pad": {"S": "x" * pad}, + } + } + } + for i in range(start, min(start + 25, n)) + ] + pending = {name: requests_} + while pending: + pending = local.client.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + if pending: + time.sleep(0.05) + + with ThreadPoolExecutor(max_workers=8) as pool: + list(pool.map(write, range(0, n, 25))) + + +def _count(ddb, name: str) -> int: + total, start = 0, None + while True: + kwargs = {"TableName": name, "Select": "COUNT", "ConsistentRead": True} + if start: + kwargs["ExclusiveStartKey"] = start + resp = ddb.scan(**kwargs) + total += resp["Count"] + start = resp.get("LastEvaluatedKey") + if not start: + return total + + +def _wait_table_gone(ddb, name: str, timeout: float = 120.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + status = ddb.describe_table(TableName=name)["Table"]["TableStatus"] + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + return + assert status == "CREATING", f"{name} is {status}, it was never writable" + time.sleep(0.2) + raise AssertionError(f"{name} still exists after {timeout} s") + + +def _backed_up_table(d: _Deployment, ddb, creds: dict, prefix: str) -> tuple[str, str]: + """A composite-key table with a GSI holding RESTORE_ITEMS padded items, + backed up to AVAILABLE in two data files; the name and the ARN.""" + name = f"{prefix}-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load_padded(d, creds, name, RESTORE_ITEMS, RESTORE_PAD) + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=180) + files = sorted(os.listdir(os.path.join(_prefix(d, name, arn), "data"))) + assert len(files) >= 2, f"the backup has one data file ({files}); raise EXTENDDB_TEST_RESTORE_ITEMS" + return name, arn + + + + +def _restore_cell(d: _Deployment) -> None: + """The store-backed restore cells run on the SQLite and PostgreSQL + deployments; on MongoDB the store-backed restore is covered by + tests/test_mongodb_backup_restore.py and the crate's live tests.""" + if d.backend == "mongodb": + pytest.skip("the restore cells run on the SQLite and PostgreSQL deployments") + +def test_a_tampered_data_file_fails_the_restore_before_any_item(deployment, tmp_path): + """A byte of the backup's second data file flipped under the root: the + restore request is accepted (CREATING with a RestoreSummary), the target + is removed once the job fails, and the server log names the file. That + the job verified every file before it loaded any is pinned by the FAILED + reason's text: "does not match the manifest" is the hashing pass's, which + runs over every file before the first `bulk_load`; a mismatch found while + a file is loaded reads "data file : data file sha256 checksum + mismatch" or, when the flipped byte sits in the gzip trailer so the + decoder's own CRC fires first, "data file : data file decode error: + corrupt gzip stream does not have a matching checksum" instead. A + CREATING target answers no read whatever the job did, so the GetItem + probe below only confirms the target never became readable. The backup + itself stays AVAILABLE.""" + d = deployment + _restore_cell(d) + _add_backup_settings(d, RESTORE_BACKUP_SETTINGS) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _backed_up_table(d, ddb, creds, "tamper") + path = os.path.join(_prefix(d, name, arn), "data", "000002.json.gz") + with open(path, "r+b") as f: + f.seek(-1, os.SEEK_END) + last = f.read(1) + f.seek(-1, os.SEEK_END) + f.write(bytes([last[0] ^ 0xFF])) + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + readable = [] + deadline = time.monotonic() + 120 + while time.monotonic() < deadline: + try: + got = ddb.get_item(TableName=target, Key={"pk": {"S": "p0000"}, "sk": {"N": "0"}}, ConsistentRead=True) + if "Item" in got: + readable.append(got["Item"]) + except ClientError: + pass + try: + ddb.describe_table(TableName=target) + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + break + time.sleep(0.2) + else: + raise AssertionError(f"{target} was not removed within 120 s") + assert readable == [], f"an item of the failed restore was readable: {readable[:1]}" + log = server.log_text() + failed = [line for line in log.splitlines() if "FAILED (data file data/000002.json.gz does not match the manifest" in line] + assert len(failed) == 1, log[-4000:] + assert "the target was removed" in failed[0], failed[0] + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" + finally: + server.stop() + + +def test_sigterm_leaves_the_restore_running_and_it_finishes_after_restart(deployment, tmp_path): + """SIGTERM while the restore runs: the server drains, the job stops at + its next file or chunk with no status write, the record stays RUNNING + (the target CREATING with its RestoreSummary across the restart), and the + restarted server's reconciler resumes it: on PostgreSQL once the dead + owner's heartbeat is older than `stale_job_after_secs` (45 s, the floor, + on the database's clock); on SQLite at the first cycle, since the restart + frees the record's slot and ages every RUNNING record. The target reaches + ACTIVE with every item and the GSI answers.""" + d = deployment + _restore_cell(d) + _add_backup_settings(d, RESTORE_BACKUP_SETTINGS + "stale_job_after_secs = 45\n") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _backed_up_table(d, ddb, creds, "sigterm") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + status = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + assert status == "CREATING", ( + f"the restore finished ({status}) before the signal was sent; raise EXTENDDB_TEST_RESTORE_ITEMS" + ) + server.proc.terminate() + code = server.wait_exit(timeout=60) + assert code == 0, server.log_text()[-4000:] + log = server.log_text() + assert "stopped by the server's shutdown; the record stays RUNNING" in log, log[-4000:] + assert ": FAILED (" not in log, log[-4000:] + finally: + server.stop() + stopped_at = time.monotonic() + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + table = ddb.describe_table(TableName=target)["Table"] + if d.backend == "postgres": + # PostgreSQL resumes only once the dead owner's heartbeat is + # stale (45 s), so right after the restart the target is still + # CREATING, carries its RestoreSummary, and a retry into the name + # is refused as a restore in progress. On SQLite the restart frees + # the record's slot and the first reconciler cycle resumes at + # once, so the target may already be ACTIVE here; the + # resume-timing assertion below pins the at-once behaviour. + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["SourceBackupArn"] == arn, table + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "TableInUseException", refused.value.response + _wait_active(ddb, target, timeout=240) + resumed_after = time.monotonic() - stopped_at + if d.backend == "postgres": + assert resumed_after >= 45 - 5, f"the resume waited for the stale window ({resumed_after:.1f} s)" + else: + assert resumed_after < 45 - 5, f"the restart's slot rule resumes at once ({resumed_after:.1f} s)" + assert _count(ddb, target) == RESTORE_ITEMS + by_g = ddb.query( + TableName=target, + IndexName="by-g", + KeyConditionExpression="g = :g", + ExpressionAttributeValues={":g": {"S": "g7"}}, + Select="COUNT", + )["Count"] + assert by_g == RESTORE_ITEMS // 50, by_g + log = server.log_text() + assert "taken over by this instance and resumed" in log, log[-4000:] + assert ": ACTIVE, " in log, log[-4000:] + finally: + server.stop() + + +def test_a_store_outage_mid_download_leaves_the_restore_running_and_it_completes(request, tmp_path, moto_bucket): + """An S3 store (moto) reached through a TCP forwarder, with a spool: the + forwarder is cut right after the restore's response, so the job's data + file downloads fail as transport errors; the record stays RUNNING (the + target CREATING) while the job retries with backoff; once the forwarder + is back the download completes and the target reaches ACTIVE with every + item, the spool entry removed. On PostgreSQL the spool is `spool_path`; + on SQLite the defaulted `backup-spool/` beside the database file.""" + moto_host, moto_port = _moto_or_skip() + bucket = moto_bucket + base = os.path.realpath(str(tmp_path)) + port = _free_port() + store = ( + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "{bucket}"\nendpoint = "http://127.0.0.1:{port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n' + ) + if _sqlite_build(): + sd = _SqliteDeployment(str(tmp_path)) + spool = os.path.join(base, "backup-spool") + _append(sd.config, "[backup]\nreconcile_interval_secs = 1\nmax_concurrent_jobs = 1\n" + RESTORE_BACKUP_SETTINGS + store) + d = _Deployment("sqlite", sd.config, sd.port, "", sd.cert, base) + elif _pg_build(): + cli_env = request.getfixturevalue("cli_env") + result = lh._run_extenddb( + "init", *lh._init_args(cli_env), config=cli_env["config_path"], + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(cli_env["config_path"], cli_env["port"]) + spool = os.path.join(base, "spool") + _append( + cli_env["config_path"], + "[backup]\nreconcile_interval_secs = 1\nmax_concurrent_jobs = 1\n" + + RESTORE_BACKUP_SETTINGS + + f'spool_path = "{spool}"\n' + + store, + ) + d = _Deployment("postgres", cli_env["config_path"], cli_env["port"], "", os.path.join(cli_env["tls_dir"], "cert.pem"), base) + else: + pytest.skip("EXTENDDB_BINARY is neither a SQLite nor a PostgreSQL build") + forwarder = _Forwarder(port, (moto_host, moto_port)) + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + try: + _wait_health( + d.port, + lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy", + timeout=60, + server=server, + ) + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + name = f"outage-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load_padded(d, creds, name, RESTORE_ITEMS, RESTORE_PAD) + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=180) + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + # The cut: every data file get fails until the forwarder is back. + forwarder.stop() + cut_at = time.monotonic() + while time.monotonic() - cut_at < 8: + table = ddb.describe_table(TableName=target)["Table"] + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["RestoreInProgress"] is True, table + time.sleep(0.5) + log = server.log_text() + retried = [line for line in log.splitlines() if "retried in" in line and "data/" in line] + assert retried, "no download attempt was retried during the cut:\n" + log[-4000:] + forwarder = _Forwarder(port, (moto_host, moto_port)) + _wait_active(ddb, target, timeout=240) + assert _count(ddb, target) == RESTORE_ITEMS + entries = [e for e in os.listdir(spool) if e.startswith("restore-")] + assert entries == [], f"the spool entry stays: {entries}" + log = server.log_text() + assert ": ACTIVE, " in log and ": FAILED (" not in log, log[-4000:] + finally: + server.stop() + forwarder.stop() + + +# --- `backup remove-target`: the target_not_empty state ------------------------------ + + +def _seed_target_not_empty(d: _Deployment, target: str) -> str: + """Put a finished restore (table ACTIVE, record DONE) back into the state + a catalog restored from a dump leaves when it names a restore that + finished and was written to since: the target CREATING with its items + (one more than the backup holds, written after the restore) and a RUNNING + manifest record with no progress, which the restarted server's reconciler + ages, takes over, and fails `target_not_empty` since the target holds + more items than the backup's total (a count at or below the total is a + first data file applied in part and is resumed). No kill produces the + state on any backend; the catalog is edited with the server stopped. + Returns the target's table id.""" + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + row = conn.execute( + "SELECT table_id FROM tables WHERE account_id = ? AND table_name = ?", (ACCOUNT, target) + ).fetchone() + assert row, f"no catalog row for {target}" + table_id = row[0] + n = conn.execute( + "UPDATE tables SET table_status = 'CREATING' WHERE table_id = ?", (table_id,) + ).rowcount + assert n == 1, n + n = conn.execute( + "UPDATE table_restores SET status = 'RUNNING', file_index = NULL, items_applied = NULL, " + "failure_reason = NULL, completed_at = NULL, heartbeat_at = '1970-01-01T00:00:00.000Z' " + "WHERE table_id = ? AND kind = 'manifest' AND status = 'DONE'", + (table_id,), + ).rowcount + assert n == 1, n + conn.commit() + finally: + conn.close() + return table_id + + +def test_remove_target_removes_a_target_not_empty_restore(deployment, tmp_path): + """The operator's remedy for a resume that found no progress and more + items than the backup holds. A backup restored to ACTIVE, with one item + written to the target since, is seeded back into the `target_not_empty` + state (`_seed_target_not_empty`: the shape a catalog restored from a dump + leaves; a catalog edit is the one way to produce it). The restarted server's + reconciler fails the record `target_not_empty` and leaves the target + CREATING, which `DeleteTable` refuses (`ResourceInUseException`) and + which keeps the backup in use (`BackupInUseException` to `DeleteBackup` + and to a second restore). `extenddb backup remove-target`, run with the + server stopped (on SQLite it takes the serve lock), removes the target + and its record and prints what it did; the restarted server then answers + `ResourceNotFoundException` for the target, `DeleteBackup` succeeds, and + a new restore into the name runs to ACTIVE with every item.""" + d = deployment + _restore_cell(d) + if d.backend != "sqlite": + pytest.skip("the seeded target_not_empty state is built on the SQLite catalog (the PostgreSQL command runs beside a live server and is the engine tests')") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "rmt") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + # Written to since: one item the backup does not hold, so the seeded + # record's target holds more than the backup's total. + ddb.put_item(TableName=target, Item=_hand_item(HAND_BUILT_ITEMS)) + assert _count(ddb, target) == HAND_BUILT_ITEMS + 1 + finally: + server.stop() + table_id = _seed_target_not_empty(d, target) + + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + deadline = time.monotonic() + 60 + while "target_not_empty" not in server.log_text(): + assert time.monotonic() < deadline, server.log_text()[-4000:] + time.sleep(0.2) + log = server.log_text() + failed = [line for line in log.splitlines() if "target_not_empty" in line] + assert any(target in line for line in failed), failed + assert ddb.describe_table(TableName=target)["Table"]["TableStatus"] == "CREATING" + with pytest.raises(ClientError) as refused: + ddb.delete_table(TableName=target) + assert refused.value.response["Error"]["Code"] == "ResourceInUseException", refused.value.response + with pytest.raises(ClientError) as refused: + ddb.delete_backup(BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "BackupInUseException", refused.value.response + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName=f"{name}-again", BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "BackupInUseException", refused.value.response + # The reconciler's next cycles leave it: the FAILED-target duty skips + # this reason, and the target is still CREATING after two more cycles. + time.sleep(2.5) + assert ddb.describe_table(TableName=target)["Table"]["TableStatus"] == "CREATING" + finally: + server.stop() + + res = d.run("backup", "remove-target", "--account", ACCOUNT, target) + assert res.returncode == 0, res.stdout + res.stderr + assert "=== extenddb backup remove-target ===" in res.stdout, res.stdout + res.stderr + assert 'Taking the maintenance claim as "extenddb backup remove-target on ' in res.stdout, res.stdout + assert ( + f"removed the restore target {target} (account {ACCOUNT}, table id {table_id}) and its record: " + f"its store-backed restore from {arn} was FAILED (target_not_empty)" + ) in res.stdout, res.stdout + res.stderr + assert "The maintenance claim is released." in res.stdout, res.stdout + # A second run finds no record for the name and refuses, exit 1. + res = d.run("backup", "remove-target", "--account", ACCOUNT, target) + assert res.returncode == 1, res.stdout + res.stderr + assert f"{target} (account {ACCOUNT}) is not a restore target: no restore record names it" in res.stderr, ( + res.stdout + res.stderr + ) + + server, ddb, creds = _started(d, str(tmp_path / "serve3.log")) + try: + with pytest.raises(ClientError) as gone: + ddb.describe_table(TableName=target) + assert gone.value.response["Error"]["Code"] == "ResourceNotFoundException", gone.value.response + # The name is free and the backup no longer in use: a new restore runs. + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + ddb.delete_table(TableName=target) + _wait_table_gone_any_status(ddb, target) + ddb.delete_backup(BackupArn=arn) + _wait_backup_gone(ddb, arn) + log = server.log_text() + assert "target_not_empty" not in log, log[-4000:] + finally: + server.stop() + + +def _seed_running_with_progress(d: _Deployment, target: str) -> None: + """Put a finished restore (table ACTIVE, record DONE) back into the state + a server killed after the restore's first data file committed leaves + (the `restore_after_file_commit` kill point): the target CREATING with + its items and a RUNNING manifest record whose progress names the file + and its item count. Edited at the catalog with the server stopped, as + `_seed_target_not_empty` does.""" + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + row = conn.execute( + "SELECT table_id FROM tables WHERE account_id = ? AND table_name = ?", (ACCOUNT, target) + ).fetchone() + assert row, f"no catalog row for {target}" + table_id = row[0] + n = conn.execute( + "UPDATE tables SET table_status = 'CREATING' WHERE table_id = ?", (table_id,) + ).rowcount + assert n == 1, n + n = conn.execute( + "UPDATE table_restores SET status = 'RUNNING', file_index = 0, items_applied = ?, " + "failure_reason = NULL, completed_at = NULL " + "WHERE table_id = ? AND kind = 'manifest' AND status = 'DONE'", + (HAND_BUILT_ITEMS, table_id), + ).rowcount + assert n == 1, n + conn.commit() + finally: + conn.close() + + +def test_purge_fails_a_killed_servers_store_backed_restore_rather_than_resuming_it(deployment, tmp_path): + """`backup purge --confirm` under the SQLite serve lock, after a server + that was running a store-backed restore stopped (the record RUNNING with + recorded progress, `_seed_running_with_progress`): the command's own + reconciler cycle holds the maintenance claim, so it fails the record and + removes its target, as it does a legacy restore, and resumes nothing (no + `taken over by this instance` line, no restore job in the command). The + restarted server finds no target and no backup.""" + d = deployment + _restore_cell(d) + if d.backend != "sqlite": + pytest.skip("the purge's own reconciler cycle runs under the SQLite serve lock only") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "prg") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + _seed_running_with_progress(d, target) + + res = d.run("backup", "purge", "--confirm") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert "taken over by this instance" not in out, out + assert "resumed by this instance" not in out, out + assert "a maintenance claim is held; it is failed rather than resumed" in out, out + assert "Waiting for running jobs" not in out, out + + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + with pytest.raises(ClientError) as gone: + ddb.describe_table(TableName=target) + assert gone.value.response["Error"]["Code"] == "ResourceNotFoundException", gone.value.response + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=arn) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", gone.value.response + log = server.log_text() + assert "taken over by this instance" not in log, log[-4000:] + finally: + server.stop() + + +def _wait_table_gone_any_status(ddb, name: str, timeout: float = 60.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + ddb.describe_table(TableName=name) + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + return + time.sleep(0.2) + raise AssertionError(f"{name} still exists after {timeout} s") + + +# --- hand-built prefixes: the format's rules met at the server's boundary ---------- + +HAND_BUILT_ITEMS = 12 + + +def _hand_item(i: int) -> dict: + """Item `i` of a hand-built data file, in the shape `_create_composite_table` + keys on (`pk` S, `sk` N) with the GSI's `g`.""" + return {"pk": {"S": f"p{i % 4:04d}"}, "sk": {"N": str(i)}, "g": {"S": f"g{i % 3}"}, "v": {"S": f"value {i}"}} + + +def _gzip_lines(rows: list[dict]) -> tuple[bytes, int]: + """The data file of `rows`: one `{"Item": ..}` line each, gzipped; the + compressed bytes and the uncompressed length.""" + import gzip + import io + + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":")).encode() + b"\n" for r in rows) + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + return buffer.getvalue(), len(lines) + + +def _write_manifest(prefix: str, manifest: dict) -> None: + """Write the manifest and its sha256 sidecar under `prefix`, the files + private as the store writes them, and the two DynamoDB export manifests + for the manifest's data files, so the prefix stays a whole backup in the + format's sense (one `md5Checksum` and one `itemCount` per file, agreeing + with the extenddb manifest by construction).""" + import hashlib + + text = json.dumps(manifest, separators=(",", ":")).encode() + path = os.path.join(prefix, "extenddb-manifest.json") + with open(path, "wb") as f: + f.write(text) + os.chmod(path, 0o600) + sidecar = os.path.join(prefix, "extenddb-manifest.sha256") + with open(sidecar, "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(sidecar, 0o600) + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + files_path = os.path.join(prefix, "manifest-files.json") + with open(files_path, "w") as f: + for entry in manifest["data_files"]: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(files_path, 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in manifest["data_files"]) + summary["billedSizeBytes"] = manifest["backup_size_bytes"] + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def _write_data_files(prefix: str, files: list[list[dict]]) -> list[dict]: + """Replace the data files under `prefix` with one file per list of rows + (unordered: no `key_range`); the manifest entries.""" + import base64 + import hashlib + import shutil + + data_dir = os.path.join(prefix, "data") + shutil.rmtree(data_dir) + os.mkdir(data_dir, 0o700) + entries = [] + for index, rows in enumerate(files, start=1): + compressed, uncompressed = _gzip_lines(rows) + key = f"data/{index:06d}.json.gz" + path = os.path.join(prefix, key) + with open(path, "wb") as f: + f.write(compressed) + os.chmod(path, 0o600) + entries.append({ + "key": key, "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": uncompressed, "key_range": None, + }) + return entries + + +def _update_row(d: _Deployment, arn: str, size_bytes: int, item_count: int) -> None: + """Set the `backup_index` row's `size_bytes` and `item_count` to the + rewritten prefix's, as `backup sync` would record them.""" + backup_id = arn.rsplit("/", 1)[1] + if d.backend == "sqlite": + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + n = conn.execute( + "UPDATE backup_index SET size_bytes = ?, item_count = ? WHERE account_id = ? AND backup_id = ?", + (size_bytes, item_count, ACCOUNT, backup_id), + ).rowcount + conn.commit() + finally: + conn.close() + else: + import psycopg2 + + conn = psycopg2.connect(d.catalog) + try: + with conn.cursor() as cur: + cur.execute( + "UPDATE backup_index SET size_bytes = %s, item_count = %s WHERE account_id = %s AND backup_id = %s", + (size_bytes, item_count, ACCOUNT, backup_id), + ) + n = cur.rowcount + conn.commit() + finally: + conn.close() + assert n == 1, f"{n} backup_index rows match {arn}" + + +def _small_backed_up_table(ddb, prefix: str) -> tuple[str, str]: + """A composite-key table with a GSI holding HAND_BUILT_ITEMS items, + backed up to AVAILABLE; the name and the ARN.""" + name = f"{prefix}-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + pending = {name: [{"PutRequest": {"Item": _hand_item(i)}} for i in range(HAND_BUILT_ITEMS)]} + while pending: + pending = ddb.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=60) + return name, arn + + +def _restore_fails(server: _Server, ddb, target: str, arn: str, *needles: str) -> str: + """The restore of `arn` into `target` is accepted CREATING and the target + is gone once the job fails, no item of it ever readable; the server log's + FAILED line, which must carry every needle. The job logs the line right + after it removes the target, so the log is polled for a moment.""" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + _wait_table_gone(ddb, target) + deadline = time.monotonic() + 15 + while True: + failed = [line for line in server.log_text().splitlines() if ": FAILED (" in line and f" into {target} " in line] + if failed or time.monotonic() > deadline: + break + time.sleep(0.25) + assert len(failed) == 1, server.log_text()[-4000:] + for needle in needles: + assert needle in failed[0], failed[0] + assert "the target was removed" in failed[0], failed[0] + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" + return failed[0] + + +DUPLICATE_KEY_TEXT = "holds a primary key the target already has (the backup holds a key twice)" + + +def test_an_unordered_backup_holding_one_key_in_two_files_fails_the_restore(deployment, tmp_path): + """The backup's prefix rewritten under the root as an unordered backup of + two data files that share item 0's key (the manifest and its sidecar + rewritten, the row's size and count updated): the restore is accepted, + the job fails the second file on the duplicate, the target is gone and + the log names the reason. On PostgreSQL and SQLite, the two deployments + here, the detector is the base row's plain INSERT and the reason is its + text (`DUPLICATE_KEY_TEXT`); the engine's final count against the + manifest's total ("the manifest lists") is the detector on a backend + whose write is an upsert (Cassandra, through the matrix). A backend with + neither would restore 12 of 13 items.""" + d = deployment + _restore_cell(d) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "dup") + prefix = _prefix(d, name, arn) + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + rows = [_hand_item(i) for i in range(HAND_BUILT_ITEMS)] + entries = _write_data_files(prefix, [rows[:6], rows[6:] + [rows[0]]]) + manifest["item_order"] = "unordered" + manifest["data_files"] = entries + manifest["backup_size_bytes"] = sum(e["bytes"] for e in entries) + manifest["source_table"]["item_count"] = HAND_BUILT_ITEMS + 1 + _write_manifest(prefix, manifest) + _update_row(d, arn, manifest["backup_size_bytes"], HAND_BUILT_ITEMS + 1) + line = _restore_fails(server, ddb, f"{name}-r", arn, "data/000002.json.gz", DUPLICATE_KEY_TEXT) + assert "the manifest lists" not in line, line + finally: + server.stop() + + +def test_a_key_ordered_file_leaving_its_key_range_fails_the_restore(deployment, tmp_path): + """The backup's manifest rewritten so its first data file's `key_range` + ends at its first item: the reader's key-range check ends the file's + stream at the second item, the restore is FAILED naming the file and the + rule, and the target is gone.""" + d = deployment + _restore_cell(d) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "range") + prefix = _prefix(d, name, arn) + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["item_order"] == "key_ordered", manifest["item_order"] + first = manifest["data_files"][0] + assert first["item_count"] >= 2, first + first["key_range"]["last"] = first["key_range"]["first"] + _write_manifest(prefix, manifest) + _restore_fails(server, ddb, f"{name}-r", arn, "data/000001.json.gz", "key_range") + finally: + server.stop() + + +# --- the multi-part-key preview ------------------------------------------------------ + +MULTIPART_ITEMS = 60 + + +def _scan_items(ddb, name: str) -> list[dict]: + items: list[dict] = [] + kwargs: dict = {"TableName": name, "ConsistentRead": True} + while True: + resp = ddb.scan(**kwargs) + items += resp["Items"] + if "LastEvaluatedKey" not in resp: + return items + kwargs["ExclusiveStartKey"] = resp["LastEvaluatedKey"] + + +def _canonical(item: dict) -> str: + return json.dumps(item, sort_keys=True, separators=(",", ":")) + + +def test_a_multi_part_key_table_restores_with_the_sources_item_set(deployment, tmp_path): + """A table keyed on two HASH attributes (`[limits] + allow_multipart_table_keys = true` on this cell's deployment, so a + server that refuses the second HASH key fails the cell rather than + skipping it) backs up to the store as an `unordered` backup and + restores: the restored table's `Scan` holds what the source's held, + with the source's key schema and attribute definitions. The comparison + is with the source's `Scan`, not with the items written: the preview's + item paths address such a table by its first HASH attribute on every + backend (on SQLite the source keeps one item per `h1` value; on + PostgreSQL a GetItem by the whole key finds nothing on the source + either), and that is the preview's behaviour, not the restore's; for the + same reason the cell makes no GetItem by the whole key.""" + d = deployment + _restore_cell(d) + d.set_limit("allow_multipart_table_keys", "true") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name = f"mpk-{uuid.uuid4().hex[:8]}" + key_schema = [ + {"AttributeName": "h1", "KeyType": "HASH"}, + {"AttributeName": "h2", "KeyType": "HASH"}, + ] + attr_defs = [ + {"AttributeName": "h1", "AttributeType": "S"}, + {"AttributeName": "h2", "AttributeType": "N"}, + ] + ddb.create_table(TableName=name, KeySchema=key_schema, AttributeDefinitions=attr_defs, BillingMode="PAY_PER_REQUEST") + _wait_active(ddb, name) + for i in range(MULTIPART_ITEMS): + ddb.put_item( + TableName=name, + Item={"h1": {"S": f"tenant-{i % 5}"}, "h2": {"N": str(i)}, "v": {"S": f"value-{i}"}, "n": {"N": str(i * 1.5)}}, + ) + expected = sorted(_canonical(i) for i in _scan_items(ddb, name)) + assert expected, "the source holds no item" + print(f"branch: the source holds {len(expected)} of {MULTIPART_ITEMS} items written ({d.backend})") + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=60) + with open(os.path.join(_prefix(d, name, arn), "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["item_order"] == "unordered", manifest["item_order"] + assert manifest["source_table"]["key_schema"] == key_schema, manifest["source_table"]["key_schema"] + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + _wait_active(ddb, target, timeout=120) + table = ddb.describe_table(TableName=target)["Table"] + assert table["KeySchema"] == key_schema, table["KeySchema"] + assert sorted(table["AttributeDefinitions"], key=lambda a: a["AttributeName"]) == attr_defs, table["AttributeDefinitions"] + assert "RestoreSummary" not in table, table + got = sorted(_canonical(i) for i in _scan_items(ddb, target)) + assert got == expected, f"{len(got)} restored items against {len(expected)} in the source" + log = server.log_text() + assert f" into {target} " in log and ": ACTIVE, " in log and ": FAILED (" not in log, log[-4000:] + finally: + server.stop() diff --git a/tests/test_backup_restore_fidelity.py b/tests/test_backup_restore_fidelity.py index 8d152b607..12da741c7 100644 --- a/tests/test_backup_restore_fidelity.py +++ b/tests/test_backup_restore_fidelity.py @@ -6,8 +6,9 @@ Every backend; hash-only keys and composite keys with S, N, and B sort keys, plus secondary indexes and provisioned throughput. A restore must reach ACTIVE with the source's key schema, attribute definitions, and exactly the source's -items, attribute for attribute. (Multi-part base keys are a preview that -restore refuses; that refusal is tested at the storage level.) Comparing whole items rather than counts is what catches a +items, attribute for attribute. (Multi-part base keys are a preview: a +store-backed backup of such a table restores through the store, an in-catalog +one is refused, and the refusal is tested at the storage level.) Comparing whole items rather than counts is what catches a backend that restores the right number of rows under the wrong keys. Readiness assessment P0-2: on PostgreSQL every table with a sort key backed up @@ -44,29 +45,15 @@ # Where the server under test writes its backups: `filesystem` or `s3` (a # configured `[backup.store]`, under which CreateBackup is asynchronous and -# backs up vector tables) or `none` (the in-catalog create). The CI jobs set -# it beside EXTENDDB_TEST_ENDPOINT; the one test whose expected outcome -# depends on it fails when it is unset rather than accept either outcome. +# backs up vector tables, and RestoreTableFromBackup runs the restore job +# from the store) or `none` (the in-catalog create). The CI jobs set it +# beside EXTENDDB_TEST_ENDPOINT; the one test whose expected outcome depends +# on it fails when it is unset rather than accept either outcome. Every cell +# that restores a backup it created waits for AVAILABLE first +# (`_create_backup`), on either path. BACKUP_STORE = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() or None BACKUP_STORE_VALUES = ("filesystem", "s3", "none") -# RestoreTableFromBackup of a backup written to a configured store answers -# ValidationException ("is in a backup store; this release restores -# in-catalog backups only") until the restore job of PR 9a lands. The cells -# that restore a backup this server created skip on such a server and come -# back with PR 9a; the in-catalog create's backups (`none`, and the service) -# restore as before. The marker is gated on the target as well as the -# variable, so a stale `EXTENDDB_TEST_BACKUP_STORE=filesystem` in the -# environment of a real-DynamoDB run (which does not require the variable) -# cannot skip the 24 cells against the service. -restores_a_backup_of_this_server = pytest.mark.skipif( - BACKUP_STORE in ("filesystem", "s3") and not _is_real_dynamodb(), - reason=( - f"EXTENDDB_TEST_BACKUP_STORE={BACKUP_STORE}: RestoreTableFromBackup of a backup " - "in the configured store is refused until the restore job lands (PR 9a)" - ), -) - @pytest.fixture(scope="session") def backup_store_component(endpoint_url): @@ -428,13 +415,11 @@ def _round_trip(client, sort_kind: str | None, hash_kind: str = "S") -> None: _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server @pytest.mark.parametrize("hash_kind", ["S", "N", "B"]) def test_restore_hash_only_table(dynamodb_client, hash_kind): _round_trip(dynamodb_client, None, hash_kind) -@restores_a_backup_of_this_server @pytest.mark.parametrize("sort_kind", ["S", "N", "B"]) def test_restore_composite_key_table(dynamodb_client, sort_kind): _round_trip(dynamodb_client, sort_kind) @@ -451,7 +436,6 @@ def _index_by_name(indexes: list[dict]) -> dict: return {i["IndexName"]: i for i in indexes} -@restores_a_backup_of_this_server def test_restore_preserves_indexes_and_provisioned_throughput(dynamodb_client): """GSIs and LSIs, their key schemas, projections, and throughput, the table's provisioned throughput, and every index's contents come back.""" @@ -639,7 +623,6 @@ def ranked(table: str) -> list[str]: _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_client): """A backup taken after an on-demand switch does not restore stale capacity.""" client = dynamodb_client @@ -714,7 +697,6 @@ def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_cli _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): client = dynamodb_client source = f"restore-override-ppr-{uuid.uuid4().hex[:10]}" @@ -753,7 +735,6 @@ def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): client = dynamodb_client source = f"restore-override-prov-{uuid.uuid4().hex[:10]}" @@ -788,7 +769,6 @@ def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_provisioned_restore_override_requiring_gsi_override_leaves_no_target( dynamodb_client, ): @@ -1031,7 +1011,6 @@ def large_backup(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_restore_response_arrives_before_its_copy_finishes(dynamodb_client, large_backup): """The CREATING response reaches a client with a 10 s read timeout and no retries before the copy finishes, and the table then reaches ACTIVE with @@ -1094,7 +1073,6 @@ def _table_appears(client, name: str, timeout: float) -> bool: time.sleep(_poll_interval() * 5) -@restores_a_backup_of_this_server def test_restore_survives_a_client_that_disconnects_before_the_response( dynamodb_client, large_backup ): @@ -1150,7 +1128,6 @@ def test_restore_survives_a_client_that_disconnects_before_the_response( _cleanup(dynamodb_client, None, target) -@restores_a_backup_of_this_server def test_first_read_after_active_succeeds(dynamodb_client, large_backup): """A read on the restore target while it is CREATING is ResourceNotFoundException, and the first read after DescribeTable reports @@ -1207,7 +1184,6 @@ def test_first_read_after_active_succeeds(dynamodb_client, large_backup): _cleanup(client, None, target) -@restores_a_backup_of_this_server def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, large_backup): """CreateBackup on a table that is still CREATING is TableNotFoundException "Table not found: " (measured on DynamoDB), @@ -1237,7 +1213,6 @@ def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, l _cleanup(client, None, target) -@restores_a_backup_of_this_server def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused( dynamodb_client, large_backup ): @@ -1280,7 +1255,6 @@ def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_r _cleanup(client, None, target, other) -@restores_a_backup_of_this_server def test_target_and_backup_are_held_while_the_copy_runs(dynamodb_client, large_backup): """DeleteTable of a CREATING restore target is ResourceInUseException, and (ExtendDB) DeleteBackup of its source is BackupInUseException. @@ -1731,7 +1705,6 @@ def test_backup_arn_shape_and_account_are_checked(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dynamodb_client): client = dynamodb_client source = f"restore-noop-{uuid.uuid4().hex[:10]}" @@ -1782,7 +1755,6 @@ def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dy _cleanup(client, backup_arn, plain, restored, source) -@restores_a_backup_of_this_server @pytest.mark.parametrize("billing", ["PROVISIONED", "PAY_PER_REQUEST"]) def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_client, billing): """GlobalSecondaryIndexOverride and LocalSecondaryIndexOverride built from @@ -1953,7 +1925,6 @@ def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_clien _cleanup(client, backup_arn, changed, restored, source) -@restores_a_backup_of_this_server def test_on_demand_throughput_override_is_applied(dynamodb_client): """OnDemandThroughputOverride is the restored table's on-demand throughput, equal to the backup's or changed (measured on DynamoDB). @@ -2009,7 +1980,7 @@ def test_on_demand_throughput_override_is_applied(dynamodb_client): @extenddb_only -def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_store_component): +def test_vector_table_backup_refused_in_catalog_and_restored_from_store(dynamodb_client, backup_store_component): """CreateBackup of a table with vector indexes: the outcome of the server's configured path, read from EXTENDDB_TEST_BACKUP_STORE and checked against the server's `/health` (`backup_store_component`). @@ -2019,17 +1990,18 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_stor created. `filesystem` or `s3`: the store-backed create backs the table up (the manifest records its vector indexes): CreateBackup answers CREATING, DescribeBackup reaches AVAILABLE with the table's features, - ListBackups lists it, and RestoreTableFromBackup of it answers the - ValidationException naming the store, which PR 9a's restore job - replaces with the restore (its refusal then narrows to a target without - vector support). The test fails rather than accept either outcome when - the variable is unset (the fixture fails first), and prints which branch - ran. + ListBackups lists it, and RestoreTableFromBackup of it restores the + table with its vector index: the response is CREATING with a + RestoreSummary, the target reaches ACTIVE, DescribeTable shows the + vector index ACTIVE, and SearchVectors on it answers the item. The test + fails rather than accept either outcome when the variable is unset (the + fixture fails first), and prints which branch ran. """ from test_vector_index_query_scan import _error_message, _signed_post client = dynamodb_client name = f"restore-vec-{uuid.uuid4().hex[:10]}" + restored = f"{name}-r" resp = _signed_post( "CreateTable", { @@ -2106,21 +2078,39 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_stor assert [g["IndexName"] for g in gsis] == ["by-g"], desc["SourceTableFeatureDetails"] listed = client.list_backups(TableName=name)["BackupSummaries"] assert [b["BackupArn"] for b in listed] == [backup_arn], listed - # Until PR 9a's restore job, the restore of a backup in the store is - # refused; PR 9a replaces this assertion with the round trip. - with pytest.raises(ClientError) as exc_info: - client.restore_table_from_backup(TargetTableName=f"{name}-r", BackupArn=backup_arn) - error = exc_info.value.response["Error"] - assert error["Code"] == "ValidationException", error - assert error["Message"] == ( - f"Backup {backup_arn} is in a backup store; this release restores in-catalog " - "backups only" - ), error + # The round trip: the restore runs from the store and the target + # comes back with its vector index. + resp = client.restore_table_from_backup(TargetTableName=restored, BackupArn=backup_arn) + table = resp["TableDescription"] + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["SourceBackupArn"] == backup_arn, table + assert table["RestoreSummary"]["RestoreInProgress"] is True, table + wait_for_active(client, restored, timeout=RESTORE_TIMEOUT_S) + desc = _signed_post("DescribeTable", {"TableName": restored}).json()["Table"] + assert "RestoreSummary" not in desc, desc + indexes = desc.get("VectorIndexes") or [] + assert [v["IndexName"] for v in indexes] == ["vidx"], desc + assert indexes[0]["IndexStatus"] == "ACTIVE", indexes + assert indexes[0]["Dimensions"] == 4, indexes + assert [g["IndexName"] for g in desc["GlobalSecondaryIndexes"]] == ["by-g"], desc + search = _signed_post( + "SearchVectors", + { + "TableName": restored, + "IndexName": "vidx", + "SearchVector": [{"N": "0.1"}] * 4, + "TopK": 1, + }, + ) + assert search.status_code == 200, search.text + results = search.json()["SearchResults"] + assert [r["Item"]["pk"] for r in results] == [{"S": "a"}], results + got = client.get_item(TableName=restored, Key={"pk": {"S": "a"}}, ConsistentRead=True) + assert got["Item"]["g"] == {"S": "g1"}, got finally: - _cleanup(client, backup_arn, name) + _cleanup(client, backup_arn, name, restored) -@restores_a_backup_of_this_server def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_client): """Once a restored table is ACTIVE, no description-returning response carries RestoreSummary: UpdateTable's no more than DescribeTable's. @@ -2153,7 +2143,6 @@ def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_cl _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server @extenddb_only def test_a_provisioned_gsi_without_throughput_restores_with_the_table_throughput( dynamodb_client, @@ -2522,7 +2511,6 @@ def test_create_backup_accepts_the_table_arn(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_table_class_is_not_restored(dynamodb_client): """A STANDARD_INFREQUENT_ACCESS source restores as STANDARD (measured on DynamoDB); DescribeBackup still reports nothing of the diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py index 4494144ce..74085bb0a 100644 --- a/tests/test_backup_store_config.py +++ b/tests/test_backup_store_config.py @@ -512,8 +512,8 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql default of that build) on the SQLite build: with `[backup.store.filesystem] path` set the store is configured and healthy, its identity file is written, and a backup created there is written to the store (CREATING, - then AVAILABLE with its manifest under the root; its restore is - refused naming the store until the restore job lands); without + then AVAILABLE with its manifest under the root) and restores from it + with its item; without `[backup.store]` there is no default root (no database file to put it beside), no component, and CreateBackup writes a legacy backup with the warning naming `[backup.store]`.""" @@ -543,10 +543,15 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql _wait_backup_available(ddb, arn) prefix = os.path.join(root, "123456789012", "Music", arn.rsplit("/", 1)[1]) assert os.path.isfile(os.path.join(prefix, "extenddb-manifest.json")), os.listdir(root) - with pytest.raises(ClientError) as refused: - ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) - assert refused.value.response["Error"]["Code"] == "ValidationException", refused.value.response - assert "is in a backup store" in refused.value.response["Error"]["Message"], refused.value.response + restored = ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + assert restored["TableDescription"]["TableStatus"] == "CREATING", restored + deadline = time.monotonic() + 60 + while ddb.describe_table(TableName="Restored")["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the restore did not finish" + time.sleep(0.1) + item = ddb.get_item(TableName="Restored", Key={"pk": {"S": "a"}})["Item"] + assert item == {"pk": {"S": "a"}, "v": {"N": "1"}}, item + assert "restore of " + arn in server.log_text(), "the restore ran as a store-backed job" assert "configure [backup.store]" not in server.log_text(), server.log_text() finally: server.stop() @@ -1373,6 +1378,22 @@ def claim_rows(): finally: server.stop() + def test_remove_target_creates_neither_the_default_root_nor_its_identity_file(self, sqlite_deployment): + # `backup remove-target` needs no store: on a fresh deployment it + # refuses a table no restore record names and leaves the default root + # as it found it, missing, then present and empty. + d = sqlite_deployment + root = os.path.join(d.dir, "backups") + res = d.run("backup", "remove-target", "--account", "000000000000", "nosuch") + assert res.returncode == 1, res.stdout + res.stderr + assert "nosuch (account 000000000000) is not a restore target" in res.stderr, res.stderr + assert not os.path.exists(root), "remove-target created the default root" + os.mkdir(root, 0o700) + res = d.run("backup", "remove-target", "--account", "000000000000", "nosuch") + assert res.returncode == 1, res.stdout + res.stderr + assert "is not a restore target" in res.stderr, res.stderr + assert os.listdir(root) == [], "remove-target wrote into the default root" + def test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock(self, sqlite_deployment, tmp_path): d = sqlite_deployment # Create the default root with a server once, then stop it. @@ -1552,7 +1573,12 @@ def command(*args): with conn.cursor() as cur: cur.execute("SET application_name = 'holds the serving key'") cur.execute("SELECT pg_advisory_lock(%s, 2)", (0x00454442,)) - for args in (("backup", "list"), ("backup", "purge"), ("backup", "purge", "--confirm")): + for args in ( + ("backup", "list"), + ("backup", "purge"), + ("backup", "purge", "--confirm"), + ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ): res = command(*args) out = res.stdout + res.stderr assert res.returncode == 6, f"{args}: {out}" @@ -1565,7 +1591,11 @@ def command(*args): res = command("migrate", "--forget", "6", *lh._pg_args()) assert res.returncode == 0, res.stdout + res.stderr - for args in (("backup", "list"), ("backup", "purge", "--confirm")): + for args in ( + ("backup", "list"), + ("backup", "purge", "--confirm"), + ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ): res = command(*args) out = res.stdout + res.stderr assert res.returncode == 3, f"{args}: {out}" diff --git a/tests/test_mongodb_backup_restore.py b/tests/test_mongodb_backup_restore.py index adee9b5ac..e3209023c 100644 --- a/tests/test_mongodb_backup_restore.py +++ b/tests/test_mongodb_backup_restore.py @@ -5,7 +5,10 @@ The table operations are exercised through the DynamoDB API. These tests are gated by the MongoDB runner because the backup metadata being checked is part -of the MongoDB implementation; other backends retain the shared API tests. +of the MongoDB implementation; other backends retain the shared API tests. The +runner's server has a filesystem backup store, so CreateBackup is the +store-backed job and RestoreTableFromBackup the store-backed restore; the +cells at the end meet the MongoDB target's own rules through that path. """ from __future__ import annotations @@ -40,7 +43,13 @@ def mongodb_container() -> str: return container +BACKUP_STORE = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() or None + + def _create_backup(client, table_name: str, backup_name: str) -> str: + """CreateBackup, waited for AVAILABLE: with the runner's filesystem store + the create is the asynchronous job, and a restore or a delete of a + CREATING backup is BackupInUseException.""" deadline = time.monotonic() + 30 while True: try: @@ -48,7 +57,8 @@ def _create_backup(client, table_name: str, backup_name: str) -> str: TableName=table_name, BackupName=backup_name, ) - return response["BackupDetails"]["BackupArn"] + arn = response["BackupDetails"]["BackupArn"] + break except ClientError as exc: if ( exc.response["Error"]["Code"] @@ -57,6 +67,8 @@ def _create_backup(client, table_name: str, backup_name: str) -> str: ): raise time.sleep(0.2) + _wait_backup_available(client, arn) + return arn def _remove_legacy_capacity_metadata(container: str, backup_arn: str) -> None: @@ -250,7 +262,16 @@ def test_restore_preserves_provisioned_throughput_and_secondary_indexes( def test_restore_legacy_backup_falls_back_and_populates_lsi( dynamodb_client, unique_table_name, mongodb_container ): - """A pre-throughput backup still restores and its LSI remains queryable.""" + """A pre-throughput backup still restores and its LSI remains queryable. + The backup edited is an in-catalog one (the legacy `backups` document), + so the cell runs only on a server without a backup store: + `devtools/run-mongodb-tests --backup-store none -- --pytest --filter + test_mongodb_backup_restore` (the CI jobs run the suite with a filesystem + store, where this cell skips). With a store, CreateBackup writes the + store and the fallback is pinned by the crate's + `restore_uses_legacy_fallback_when_capacity_metadata_is_missing`.""" + if BACKUP_STORE != "none": + pytest.skip("the legacy throughput fallback is a rule of in-catalog backups (EXTENDDB_TEST_BACKUP_STORE is not none)") source = unique_table_name restored = f"{source}-restored" dynamodb_client.create_table( @@ -321,3 +342,250 @@ def test_restore_legacy_backup_falls_back_and_populates_lsi( _cleanup(dynamodb_client, restored, source) if backup_arn: dynamodb_client.delete_backup(BackupArn=backup_arn) + + +# --- the store-backed restore's MongoDB rules --------------------------------------- +# +# These cells rewrite a backup's prefix under the runner's filesystem store +# (EXTENDDB_TEST_BACKUP_ROOT) and read the restore job's log lines from the +# server's log (EXTENDDB_TEST_SERVER_LOG); devtools/run-mongodb-tests exports +# both. A rule the backend's write path enforces (an `N` sort key beyond +# Decimal128) is met inside the job, so the restore is FAILED with that text +# as its reason and the client sees the target disappear; a rule the request +# checks (vector indexes into a backend without vector support) is a +# synchronous ValidationException before any table exists. + +ACCOUNT = "123456789012" + + +@pytest.fixture() +def backup_root(mongodb_container) -> str: + root = os.environ.get("EXTENDDB_TEST_BACKUP_ROOT", "").strip() + log = os.environ.get("EXTENDDB_TEST_SERVER_LOG", "").strip() + if not root or not log: + pytest.skip("requires devtools/run-mongodb-tests (EXTENDDB_TEST_BACKUP_ROOT, EXTENDDB_TEST_SERVER_LOG)") + return root + + +def _wait_backup_available(client, arn: str, timeout: float = 120.0) -> None: + deadline = time.monotonic() + timeout + while True: + status = client.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] + if status == "AVAILABLE": + return + assert status == "CREATING", status + assert time.monotonic() < deadline, f"backup {arn} still CREATING after {timeout} s" + time.sleep(0.2) + + +def _prefix(root: str, table: str, arn: str) -> str: + return os.path.join(root, ACCOUNT, table, arn.rsplit("/", 1)[1]) + + +def _read_manifest(prefix: str) -> dict: + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + return json.load(f) + + +def _write_manifest(prefix: str, manifest: dict) -> None: + """The manifest and its sha256 sidecar rewritten under `prefix`, private + as the store writes them, and the two DynamoDB export manifests for the + manifest's data files (one `md5Checksum` and one `itemCount` per file, + agreeing with the extenddb manifest by construction), so the prefix + stays a whole backup in the format's sense, as `tests/test_backup_job.py` + and `devtools/backup-restore-matrix` rewrite theirs.""" + import hashlib + + text = json.dumps(manifest, separators=(",", ":")).encode() + path = os.path.join(prefix, "extenddb-manifest.json") + with open(path, "wb") as f: + f.write(text) + os.chmod(path, 0o600) + sidecar = os.path.join(prefix, "extenddb-manifest.sha256") + with open(sidecar, "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(sidecar, 0o600) + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + files_path = os.path.join(prefix, "manifest-files.json") + with open(files_path, "w") as f: + for entry in manifest["data_files"]: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(files_path, 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in manifest["data_files"]) + summary["billedSizeBytes"] = manifest["backup_size_bytes"] + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def _write_unordered_file(prefix: str, rows: list[dict]) -> dict: + """The one data file of an unordered backup holding `rows`; its manifest + entry.""" + import base64 + import gzip + import hashlib + import io + + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":")).encode() + b"\n" for r in rows) + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + compressed = buffer.getvalue() + path = os.path.join(prefix, "data", "000001.json.gz") + for name in os.listdir(os.path.join(prefix, "data")): + os.remove(os.path.join(prefix, "data", name)) + with open(path, "wb") as f: + f.write(compressed) + os.chmod(path, 0o600) + return { + "key": "data/000001.json.gz", "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": len(lines), "key_range": None, + } + + +def _composite_table(client, name: str) -> None: + client.create_table( + TableName=name, + AttributeDefinitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + ], + KeySchema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + wait_for_active(client, name) + + +def _delete_backup_once_available(client, arn: str | None) -> None: + if arn: + _wait_backup_available(client, arn) + client.delete_backup(BackupArn=arn) + + +def test_a_vector_manifest_is_refused_before_any_table(dynamodb_client, unique_table_name, backup_root): + """A backup whose manifest records a vector index (the prefix of a real + backup rewritten under the root) is refused by RestoreTableFromBackup + with the vector ValidationException, and no table of the target's name + exists afterwards: the request checks the manifest against the backend + before it registers anything.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + try: + _composite_table(client, source) + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": "1"}, "emb": {"L": [{"N": "0.1"}]}}) + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + prefix = _prefix(backup_root, source, backup_arn) + manifest = _read_manifest(prefix) + manifest["source_table"]["vector_indexes"] = [{ + "IndexName": "by-emb", "Dimensions": 3, "DistanceFunction": "COSINE", + "VectorAttribute": {"AttributeName": "emb"}, + }] + _write_manifest(prefix, manifest) + with pytest.raises(ClientError) as refused: + client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn) + err = refused.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"] == ( + f"backup {backup_arn} has 1 vector index(es); restoring a table with " + "vector indexes is not supported by this storage backend" + ), err + with pytest.raises(client.exceptions.ResourceNotFoundException): + client.describe_table(TableName=target) + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn) + + +def test_a_sort_key_beyond_decimal128_fails_the_restore(dynamodb_client, unique_table_name, backup_root): + """A data file holding an `N` sort key of 35 significant digits (more + than Decimal128's 34, which the sort-key column is): the restore is + accepted CREATING, the job fails the file with the write path's + ValidationException text, the target is gone, and the server log carries + the Decimal128 text as the FAILED reason.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + try: + _composite_table(client, source) + for i in range(3): + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": str(i)}}) + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + prefix = _prefix(backup_root, source, backup_arn) + manifest = _read_manifest(prefix) + long_key = "12345678901234567890123456789012345" + rows = [{"pk": {"S": "a"}, "sk": {"N": str(i)}} for i in range(3)] + rows.append({"pk": {"S": "a"}, "sk": {"N": long_key}}) + entry = _write_unordered_file(prefix, rows) + manifest["item_order"] = "unordered" + manifest["data_files"] = [entry] + manifest["backup_size_bytes"] = entry["bytes"] + manifest["source_table"]["item_count"] = len(rows) + _write_manifest(prefix, manifest) + resp = client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + deadline = time.monotonic() + 120 + while True: + try: + status = client.describe_table(TableName=target)["Table"]["TableStatus"] + except client.exceptions.ResourceNotFoundException: + break + assert status in ("CREATING", "DELETING"), f"{target} reached {status}" + assert time.monotonic() < deadline, f"{target} still exists after 120 s" + time.sleep(0.2) + deadline = time.monotonic() + 15 + while True: + with open(os.environ["EXTENDDB_TEST_SERVER_LOG"], errors="replace") as f: + failed = [line for line in f if ": FAILED (" in line and f" into {target} " in line] + if failed or time.monotonic() > deadline: + break + time.sleep(0.25) + assert len(failed) == 1, failed + assert "Decimal128" in failed[0] and long_key in failed[0], failed[0] + assert "the target was removed" in failed[0], failed[0] + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn) + + +def test_a_non_key_number_of_38_digits_restores_intact(dynamodb_client, unique_table_name, backup_root): + """A non-key `N` of 38 significant digits (the service's precision, past + Decimal128's) is stored as the item's text and comes back from a restore + equal to what the source serves.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + big = "12345678901234567890123456789012345678" + try: + _composite_table(client, source) + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": "1"}, "big": {"N": big}}) + key = {"pk": {"S": "a"}, "sk": {"N": "1"}} + served = client.get_item(TableName=source, Key=key, ConsistentRead=True)["Item"] + assert served["big"] == {"N": big}, served + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn) + wait_for_active(client, target) + restored = client.get_item(TableName=target, Key=key, ConsistentRead=True)["Item"] + assert restored == served, (restored, served) + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn)