Repository navigation
feat(migrations): catalog migrations with a ledger in every store, for Cassandra and MongoDB - #681
Merged
Merged
Conversation
Cassandra and MongoDB now use the numbered catalog migrations and the runner that SQLite and PostgreSQL use. Each keyspace or database keeps a ledger, the status is read from the ledgers, serve refuses a catalog that does not match the binary, and every change runs under the migration lock. Cassandra: migration 1 is a CQL file for the catalog keyspace and one for each account keyspace, with every statement qualified by its keyspace. Migration 2 is the shared policy rewrite; 3 and 4 do not apply. Cassandra has no transactions: every statement of a body can be sent again without harm, and a body's ledger row is written at QUORUM after it. Migration 1's row and the catalog_version row go in one logged batch; for a later id the version row is written first, so a run stopped between the two leaves that id pending. The lock is a settings row written by lightweight transactions at SERIAL, with a 60 s TTL renewed every 15 s while the run makes progress; a holder whose renewals fail stops writing 25 s before the row can expire. Each request under the lock is bounded by 10 s and each schema statement by 20 s. After every schema statement the runner waits until every live node reports the same schema version, and a node that does not answer is not counted as agreeing (#637). init creates the catalog keyspace and its settings table, takes the lock, and runs everything else under it, the default account's migrations included. Under an authorizer other than AllowAllAuthorizer it grants the application role its permissions on the catalog keyspace and the default account's keyspace. CreateAccount runs the account keyspace's migrations through the runner. MongoDB: the bodies are code for the catalog and data databases, and a body's ledger row and the catalog_version row commit in one transaction. The lock and the server registrations are documents written with majority write concern. Commands connect with 10 s connect and server selection timeouts, and each request under the lock is bounded by 10 s. Both backends read the status under the lock as its holder (status_under_lock), so migration 2's scan over every principal keeps the lock. Migration 2's error names its first failed lookup whatever its kind, and the status names a migration pending in several stores once, with the number of stores. With every backend on the runner, the bootstrapper's migration methods, the old branches of init, migrate and verify, serve's resource-tag marker check and OperationsEngine::catalog_version are removed. Fixes #637.
docs/schema/cassandra.txt and docs/schema/mongodb.txt hold the schema a fresh install creates. The catalog check jobs dump a fresh install's schema (devtools/schema-dump on Cassandra, devtools/check-mongodb-schema on MongoDB) and fail when it differs from the committed file. Cassandra: the storage job runs tests/migrations.rs and tests/migrations_stopped.rs. New jobs run the migration tests on a two-node cluster, check the permissions init grants under CassandraAuthorizer, and run the catalog checks: the schema, the lock and the serving registration, and a catalog the last 0.1.x release wrote. MongoDB: devtools/run-mongodb-tests runs the catalog migration tests against its replica set, and the catalog job checks the schema, the lock, a user with readWrite on the two databases and no other role, the released catalogs and the last 0.1.x image. The new test steps set EXTENDDB_TEST_CASSANDRA=required or EXTENDDB_TEST_MONGODB_REQUIRED=1, so a test without its server fails rather than skips.
ADR-0023 describes every backend on the runner and gives the Cassandra and MongoDB lock and registration values. The upgrade manual gives this release's Cassandra and MongoDB procedures, the GRANT statements a Cassandra deployment created before 1.0 needs under CassandraAuthorizer, and how a stopped Cassandra run is finished. docs/cassandra-setup.md says what init grants. The design docs, the architecture, admin and storage extension manuals and CONTRIBUTING.md's procedure for adding a migration cover all four backends, and docs/build-docs.py says extenddb version prints the catalog migration on every backend.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 30, 2026 19:56
The fixture in tests/migrations_stopped.rs created its users with
create_user, 32 at a time. create_user inserts the user row with a
lightweight transaction, and every fixture user is in the same account's
partition, so the 32 transactions contend for that partition. On the
cassandra-storage runner one of them failed, and
the_status_read_under_the_lock_keeps_it_through_a_long_scan stopped in
its fixture with user: Internal("Database error"). On a node limited to
one CPU the old fixture failed in both runs. The fixture now inserts the
user row with the columns create_user writes, without the transaction,
and adds the self-service policy create_user adds, so each user holds
the same rows as before.
a_bodys_request_to_a_node_that_stops_answering_fails_at_the_request_timeout
sends its requests through a proxy that stops passing bytes. On the
two-node runner the driver also read the second node from system.peers
and connected to it directly, and the second node answered the request
sent after the proxy went silent, 7 ms later. The test now uses a
session that leaves every node but the proxy out of its query plans. On
a local two-node cluster the old session's plan held both nodes and the
second node answered the first request after the proxy went silent; the
new session's plan holds the proxy only, and both of its requests waited
for the 10 s request timeout.
… two-node test two_migrators_started_together_apply_a_legacy_catalog_once starts two migrators through sessions that reach the cluster at different nodes. The driver reads the other nodes from system.peers_v2 and sends requests to every node in turn, so each migrator sent its requests to both nodes, and the two migrators' requests did not always go through different coordinators. The test now gives each migrator a session whose query plans hold its own node only. On a local two-node cluster the plan of each such session held one node, a different one for each, while a session built as before held both. The schema agreement wait still asks every node for its version, on a connection to each, and it agreed through both sessions. pinned_session in tests/migrations_support builds that session. Like CassandraEngine::create_session, it logs in with the test credentials when both are set and gives up on the connect after 10 s, so a node that cannot be reached fails the test after 10 s instead of leaving it waiting until the job's timeout. The proxy test's session now comes from it. The proxy test now checks that the request sent before the proxy goes silent is answered by the node, which reports that the keyspace does not exist. Before, any error passed, so a proxy that passed nothing would not have been caught. With the proxy silenced before that request, the test now fails. The fixture in tests/migrations_stopped.rs binds password_hash to null, as create_user does, so each user row holds the same cells as before. Corrections to the previous commit's message, which says system.peers: on Cassandra 4.0 and later the driver reads system.peers_v2. The CI log does not say which of create_user's two statements failed, because create_user reports any driver error as Database error; the lightweight transaction is the statement the new fixture no longer sends.
The proxy test's first request must now be answered through the proxy. The error a node returns names the server the driver read it from, and the check requires the proxy's address there as well as "does not exist". A session that also sent requests straight to the second node passed the old check whenever its next request, the one after the proxy went silent, fell to the proxy. On a local two-node cluster, a session built as before the previous commit failed the new check in three runs out of three, answered by the second node, and the pinned session passed it in three out of three. pinned_session resolves its contact point first, so a host name works as it does for CassandraEngine::create_session, and the session is given the address it resolved to. The doc comments on the pinned sessions now say what they hold: query plans with one node. The schema agreement wait still reads every node. A correction to the previous commit's message: create_user's account_exists read reports the driver's text, and only its two writes, the lightweight transaction and the self-service policy insert, report Database error. The CI log ruled out the read and could not tell the two writes apart.
This was referenced Nov 1, 2026
robinnsc
reviewed
Nov 1, 2026
robinnsc
left a comment
robinnsc
left a comment
Collaborator
There was a problem hiding this comment.
This holds up well on the upgrade paths. We used three separate reviewers: Cassandra, MongoDB, and the shared code, CLI and CI, each against real Cassandra 4.1 and MongoDB 7 nodes with the head binaries and #666's head as the control.
What was checked:
- Base-binary and real 0.1.14 Cassandra catalogs with data (accounts, ResourceTag policies, GSI/LSI, streams, TTL, backups and restores), and 0.1.12, 0.1.13 and 0.1.14 MongoDB image catalogs (including the #643 default account), migrate to Current with the policy rewrites named, and the head server serves them.
- #726 is fixed: with the Cassandra superuser password rotated,
migrate,--yes,--list,--forget,verifyandsettingsconnect as the application role, a wrong password fails in a bounded 10 s, and no password shows up in output, logs, the ledger or a dump. The Cassandra container image in #725, stacked on this PR, now passes its smoke test end to end, includingmigrate --yeswith a rotated superuser password. - Under
CassandraAuthorizer,initgrants exactly the ten permissions,serveworks, and the upgrade manual's GRANT list matches the code. A MongoDB user with onlyreadWriteon the two databases runs every command. - Lock and registration: kill -9 of a migrator, a paused node, two migrators, two
inits,rs.stepDownand a killed primary mid-migration all stayed bounded and never let a migration run beside a server. - SQLite and PostgreSQL give identical output and exit codes to #666's head across init, every
migrateform, verify and serve, including upgrades from the 0.1.14 dev image and the 0.1.10 PostgreSQL image. The PR's MongoDB migration tests pass on a 3-member set, anddocs/schema/*.txtmatch a fresh install.
The ones I'd hold this for are the first four inline comments: init writing into a foreign Cassandra keyspace whose tables share catalog names, the registration rule taking servers down on a short Cassandra outage or a hung MongoDB primary, and a server crash during CreateAccount stopping every later server start. Each is a regression against #666's head and was reproduced with a control run.
A few things outside the diff:
docker-compose.mongodb.ymlrunsmigrate --yesin its one-shot bootstrap service on everyup. On an up-to-date MongoDB catalog with the server running, that now waits the 60 s lock timeout and exits 6, where #666's head exited 0 at once, so a seconddocker compose upon a running stack fails. The PostgreSQL stack has behaved this way since #666. One option would be for the bootstrap to run the preview and only use--yeswhen it exits 3.- On Cassandra and MongoDB,
init,--yesand--reconvergeprint "Applied migration 3 ..." and "Ran again 3 ..." for the migrations--listcalls not applicable, anddevtools/check-cassandra-catalog-lock:267pins that text after--forget 3, so that cycle runs no body on Cassandra. Migration 2 might be the better one for that cycle there. - "Fixes #637" won't close #637 on merge, because the PR's base is
feat/catalog-migrationsrather than main.
| let tables = connect::tables_of(session, keyspace).await?; | ||
| let mut foreign: Vec<String> = tables | ||
| .iter() | ||
| .filter(|t| *t != "extenddb_migrations" && !ours.contains(t.as_str())) |
Collaborator
There was a problem hiding this comment.
I think this lets init write into another program's keyspace when its tables happen to use catalog table names. The check counts a table as ours by name alone, and the names are generic (metrics, tags, tables, accounts, settings, schema_history). With a keyspace r1f_catalog holding another program's metrics (id int PRIMARY KEY, reading double) and tags (name text PRIMARY KEY, color text) with a row each, head init --keyspace-prefix r1f created the keyspace's 23 catalog tables (including settings and extenddb_servers), took and released the lock, ran migration 1, and only then failed the post-check ("column metrics.bucket is missing ..."). The role r1f_app was left behind, and after that run the keyspace holds only catalog-named tables, so a re-run is no longer refused. With a foreign tables and schema_history, init created accounts and settings and stopped at CREATE INDEX ... ON tables (table_id). Had that table had a table_id column, the body would have gone on to the ALTER TABLE tables ADD ... and the INSERT INTO schema_history rows. #666's head refused any existing keyspace ("Catalog keyspace 'r1h_catalog' already exists"), and the description says init writes nothing to a store that is not a catalog.
Could consider comparing each catalog-named table's columns with the expected definitions (postcheck::Definitions is already loaded here), and treating rows in any table other than settings as foreign.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 5513001 and a16bfdd. A table with a catalog table's name now belongs to another program when it has a column that migration 1 does not give it, when it has a column that migration 1 defines otherwise (kind, position, clustering order or type), or when it lacks a column that migration 1 creates it with. A column that migration 1 adds with ALTER TABLE ... ADD may be missing, since a table created before that statement lacks it, so a keyspace that an init left before migration 1 recorded anything is still taken as a catalog to finish. A row in any catalog-named table other than settings, and a schema_history row that migration 1 does not insert, also belong to another program. init runs the check in its first step, before it creates the application role.
The test tables_with_catalog_names_and_other_columns_or_rows_are_refused refuses a keyspace with your metrics and tags tables and a row in each, one with empty accounts and tables tables that have only a part of the catalog's columns, and one whose schema_history has the catalog's columns and a row that migration 1 does not insert, and it still takes a keyspace that an init left before migration 1 recorded anything. init_refuses_a_foreign_keyspace_before_it_creates_its_role runs init's first step on your keyspace and finds no row for the application role in system_auth.roles afterwards.
| let mut next = last_ok + REGISTRATION_RENEW; | ||
| loop { | ||
| tokio::time::sleep_until(next.min(last_ok + REGISTRATION_LOST_AFTER)).await; | ||
| if last_ok.elapsed() >= REGISTRATION_LOST_AFTER { |
Collaborator
There was a problem hiding this comment.
This check runs before the renewal that would be due at the 30 s mark, and after a failure the retries are 5 s apart, so how long an outage a server survives depends on where in the 15 s cycle it starts. An outage that begins just before a scheduled renewal has to end, and the driver has to reconnect, within about 10 s. With docker restart of a single node (RF 1, which init writes by default) the outage was 10 s: the node logged "Starting listening for CQL clients" at 08:24:41.5, the server's renewal at 08:24:41.8 still got "No nodes available in query plan!", and at 08:24:46.8 it exited 1 ("the serving registration was not renewed for 30 s"). It does not come back on its own. #666's head server on a base catalog, same restart, stayed up and was healthy once the node returned. docker pause of the node stopped the foreground server after 28 s and a daemon-mode server after 18 s.
So any Cassandra restart of an RF 1 deployment (an upgrade, a JVM or container restart), or a network blip of that length, now takes the server down until something restarts it, while the docs describe a 30 s tolerance. One option would be one last renewal attempt at the deadline, renewing every 5 s (the 45 s TTL allows it) with 1 s retries after a failure, so 30 s means 30 s. It might also be worth saying in docs/cassandra-setup.md what an RF 1 deployment does when its node restarts.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 1c3b2bf, with the cadence you suggested, and in ca13267, 6282493, 21ff73c and d8fc8d7. The registration is renewed every 5 s, and every second after a renewal failed, and the renewals are tried for 30 s after the last one that succeeded. Each renewal is bounded by 5 s, and one sent before that deadline also by the time left to it (6282493). A server therefore keeps serving through an outage of 25 to 30 s, depending on when the outage starts, and reports the loss 30 to 35 s after its last successful renewal was sent, at least 10 s before the row can expire. Since 21ff73c the first deadline after a registration counts from the send of the INSERT that registered, not from its answer.
A loss no longer ends the process. As on PostgreSQL since #666, the server answers new requests with ServiceUnavailable (HTTP 503). Since ca13267 it also stops its background workers, aborting the ones still running after 5 s, and since d8fc8d7 that includes the GSI backfills the recovery worker starts, which went on writing before. Work that requests started in the background is not stopped: a RestoreTableFromBackup's item copy and an UpdateTimeToLive backfill run until they end or the process exits, and e8f507f says so in registration.rs and the upgrade manual. The server then registers again every second for up to 30 s, with a new row only while the lock row shows no migrator, then a read of the status. When the status is the one it started with, it starts its workers again and serves; otherwise it stops through the shutdown path with exit 1.
That comparison does not detect a run that ends at the status the server started with, such as migrate --yes --reconverge, or --forget 2 and then --yes, while the server was unregistered. The server then serves again with what it had cached, and its caches replace those entries once their [auth.cache] times pass, as they do after another server's writes. A run id that migrate --yes writes beside the lock would detect it; this PR does not add one.
Measured at 1c3b2bf with the release binary on a single node (replication factor 1), polling /health every 100 ms:
docker pausefor 20 s:/healthstayed 200 throughout.docker pausefor 40 s: 503 from 37 s into the pause until 0.1 s after the unpause (3.3 s), then 200.docker restart: on this host the node took 28 to 35 s to answer CQL again, where yours took 10 s. In the run where it answered after 28 s, the registration was lost at 30 s, and the server registered again 5.4 s after the node answered, after 7.1 s of 503. In three runs where the node answered after 34 s, the server stopped with exit 1 at 60 s, 26 to 29 s after the node answered: until then, every request of its sessions still failed with "No nodes available in query plan!", and the driver logged "No nodes found for control connection!" while it tried to reconnect. In the run that recovered, the engine's own session still got the same error for 66 s after the node answered. A build with a constant 1 s reconnection policy stopped at 60 s the same way. So after a restart the driver's reconnection, which this PR does not change, decides whether the server comes back within the window, anddocs/cassandra-setup.mdnow says so and that the server should run under a supervisor that starts it again (9c6c248). I could not make a restart as short as yours on this host.
With ca13267 and 6282493, devtools/check-cassandra-catalog-lock measured the first 503 35.2 s after the node was paused, against 39 s before 6282493, and the exit 1 at 80 s.
| doc! { "_id": &token }, | ||
| vec![doc! { "$set": { "expires_at": { "$add": ["$$NOW", millis(timing.registration)] } } }], | ||
| ); | ||
| let answer = tokio::time::timeout(REQUEST_TIMEOUT.min(left(renewed)), renewal).await; |
Collaborator
There was a problem hiding this comment.
I think MongoDB has the same tight budget as the Cassandra renewal. A renewal sent to a primary that has stopped answering spends the full 10 s, and the retry gets what is left of the 30 s, which an election (10 s electionTimeoutMillis) plus the driver finding the new primary often outlasts. Freezing the primary of a 3-member set with docker pause for 40 s, the head server exited 1 in 3 of 14 runs ("could not renew its registration for 30 s (no answer)"), 15 to 19 s after the freeze, when the freeze came 10.9, 12 and 14 s after its last renewal. #666's head server on the same set survived three freezes at 14 s, and the head server survived every docker kill and rs.stepDown of the primary. With several servers their phases differ, so a hung primary would take some of them down on each such failover. Renewing more often than one request timeout (say every 5 s, with a shorter per-renewal timeout), or retrying until close to the 45 s expiry minus a margin, would ride through it; a test that freezes the primary just before a renewal would pin it.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 1c3b2bf, with the same cadence as the Cassandra registration (D2), and in ca13267. The registration is renewed every 5 s, and every second after a renewal failed, with each renewal bounded by 5 s and by the time left to the deadline. It counts as lost only after 30 s without a successful renewal. A loss no longer ends the process: the server answers new requests with ServiceUnavailable, stops its background workers (ca13267), and registers again for up to 30 s, as a PostgreSQL server does since #666, and once it holds a new registration and the catalog's status is the one it started with, it starts its workers again and serves.
Measured at 1c3b2bf with release binaries on a 3-member replica set of mongo:7.0 containers, with a server in a container on the same network and the primary frozen with docker pause for 40 s, 14 times, each freeze starting at a different offset from the last:
- b8584d5: the server stopped with exit 1 after 2 of the 14 freezes ("could not renew its registration for 30 s (no answer)").
- this branch: the server kept running through all 14.
a0a902b adds the test you suggested, on the single-member set the MongoDB CI jobs run, and 448c36f and aeeeaca make it tell the old bound from the new one. A proxy in front of the primary passes nothing for 1.2 s, starting 50 ms before a renewal is due, inside the 4 s deadline the test gives the registration, and a connection that carried bytes during that silence never answers after it, as a connection to a primary that hung does. The registration is not lost: the renewal sent into the silence times out after the 400 ms renewal interval, and a later attempt on another connection gets through. With each renewal bounded only by the time left to the deadline, as at b8584d5, the test fails, because that renewal waits on its silent connection until the deadline. No automated test covers a failover of a replica set; the freezes above are the measurement for that.
| Ok(()) | ||
| } | ||
| CatalogStatus::Current { .. } => Ok(()), | ||
| status => Err(MigrationError::Status(status)), |
Collaborator
There was a problem hiding this comment.
CreateAccount writes the accounts row first and then creates and migrates the keyspace (catalog_store.rs:188). If the server dies between the two (kill -9, OOM, host loss, or the 30 s registration exit), the catalog's status counts that account keyspace as a store behind migration 1, and this refuses to serve. Reproduced by killing the server 1.5 s into POST /management/accounts: the accounts row is there, 5 tables and no extenddb_migrations in the keyspace, and the restarted server exits 3 with "the catalog is at migration 0, and ExtendDB 0.1.14 needs migration 4. Pending (all breaking): 1 baseline (account)". migrate --yes then repaired it. On #666's head, the same kill at the same point left a server that restarted healthy, and the retried CreateAccount answered "Account already exists" and completed the keyspace.
So a crash of the one server becomes an outage until an operator runs migrate --yes. It might be worth letting serve finish an account store that has an accounts row and no complete ledger, through the account-migration path CreateAccount already uses (the server is registered, so no migration can run meanwhile), or at least serving and refusing only that account. The message also says "the catalog is at migration 0" when the catalog keyspace's ledger holds 1 to 4; naming the store would help.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 4dae6cf by changing the order, so that serve writes nothing to finish an account. CreateAccount now creates the account's keyspace and runs the account migrations on it before it writes the accounts row. The status reads the account keyspaces from accounts, so a call stopped between the two leaves a keyspace that no status counts, the catalog stays Current, and servers start. A CreateAccount retried with the same account id completes the account. A retry without an account id gets a new id, and the first call's keyspace stays without a row. init's default account step works the same way (D9).
When a store lacks a migration that the catalog store has, the message now names that store: "account store is at migration 0, behind the catalog store at migration 4, and ExtendDB needs migration 4". serve and verify print it, and since 206b090 so does the migrate preview. With more than five stores behind, the line names the first five and counts the others (935c422).
Tests: a CreateAccount stopped by the test hook between its keyspace and its row leaves the keyspace migrated, no accounts row and a Current catalog, and the retried call writes the row; with the old order the test fails, since the stopped call leaves the row and no keyspace. An account keyspace whose ledger lacks migration 1 is named in the message.
| return Err(MigrationError::Status(CatalogStatus::Uninitialized)); | ||
| } | ||
| let lock = MigrationLock::acquire(session, &catalog, LeaseTimes::DEFAULT, timeout).await?; | ||
| if let Err(e) = wait_for_no_servers(&lock, timeout).await { |
Collaborator
There was a problem hiding this comment.
--lock-timeout bounds the lock wait and this server wait separately, each with its own start, so migrate --yes can wait up to twice the timeout and then report only the second. With a server serving, migrator A at --lock-timeout 20 and migrator B at 25 started 2 s later: A exited 6 at 20.2 s, B exited 6 at 44.3 s with "after waiting 25 s". During the server wait the migrator also holds and renews the lock, so every server that starts or restarts in that window exits 6, while the ones already running keep serving and the migrator ends with exit 6 anyway. A --lock-timeout 3600 run against a live deployment would block crash restarts for an hour. Could consider one deadline for both waits, and a sentence in the manual that servers cannot start while a migrator waits.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 69a1bbc and fa4f16a. The wait for the lock and the wait for the servers now end at one deadline, --lock-timeout counted from the start, and the refusal gives the time it waited for both. The upgrade manual says that one --lock-timeout bounds both waits, and that a server started while a migrator waits for the servers exits 6. fa4f16a removes the read before the lock that the timeout did not bound: whether the keyspace is a catalog is now read from its ledgers and settings alone, as MongoDB does since 4dfbea8 (D6), so migration 2's read of every principal's policies runs under the lock.
The test registers a server, lets migrator A take the lock and wait for that server, and starts migrator B 1 s later with a 6 s timeout: B gives up within 8 s of its start. With the server wait counted from when B took the lock, B waited 10.2 s. A second test, with migration 2 pending and the iam_policies table dropped, takes the lock and gets the failed read from the status under the lock; with the old check before the lock, taking the lock fails on that read.
| println!("--- Default account created"); | ||
|
|
||
| // Create account-specific keyspace | ||
| // The keyspace, then every account migration through the runner, |
Collaborator
There was a problem hiding this comment.
A similar case on Cassandra: init killed right after it printed "--- Default account created" can't be resumed. The re-run says "the catalog is already initialized (the catalog is at migration 0; pending: 1 baseline (account, breaking)); use 'extenddb migrate' for pending migrations", but no extenddb.toml was written yet, and migrate and destroy both need it, so the way out is a DROP KEYSPACE in cqlsh. #666's head was stuck at the same point too ("already exists. Run 'destroy' first"), so the outcome isn't new, only the advice. Stops earlier in init (a paused node during migration 1) did resume. When the only pending work is the default account's store and admin_users is empty, could init finish the run instead of refusing?
Collaborator
Author
There was a problem hiding this comment.
Fixed with D4 and D8 (4dae6cf and 835f9ce). init's default account step now creates and migrates the account's keyspace and grants the application role its permissions on it before it writes the accounts row. When the catalog already has an account, the step completes that account's keyspace instead of creating a second account. So an init killed anywhere in the default account step or after it leaves a catalog whose status is Current, and the next init finishes it as in D8.
One case leaves something behind: an init stopped after the default account's keyspace and before its accounts row leaves that keyspace with no row. The next init finds no account, creates another default account with a keyspace of its own, and the first keyspace stays until it is dropped; docs/cassandra-setup.md now says so and how to drop it (9c6c248).
The test init_finishes_a_catalog_stopped_before_its_admin_user stops an init after its default account and then runs init's steps again through a new bootstrapper, which takes the existing catalog keyspace as init does. The finish keeps the key and the one account and writes the admin user, and a third init is refused. The test the_default_account_step_run_again_keeps_one_account_and_completes_its_keyspace runs the step twice and keeps one account, and after the account's keyspace is dropped the step creates and migrates it again.
| let mut lost = registration.lost(); | ||
| loop { | ||
| tokio::select! { | ||
| () = shutdown.cancelled() => { |
Collaborator
There was a problem hiding this comment.
From reading the code: hold_registration is one of the handles the server drains, and drain_workers cancels one token for all of them at once and waits up to 5 s. So the registration is deleted at the moment the control-plane, GSI, TTL and restore-sweep workers are told to stop, not after they have stopped, and a migrate --yes started in that window reads no server at ALL and can start while the old process still writes. Every body is idempotent and the window is a few seconds, so the effect is small, but releasing the row after the other workers have joined would keep the "no migration beside a server" property through shutdown. I didn't construct an overlapping write.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 1c3b2bf, ca13267 and d8fc8d7. ServingRegistrar gains release(), which serve calls once every background worker has ended, on Cassandra and MongoDB, with the release bounded by 10 s. PostgreSQL keeps the default, which does nothing, since its key ends with the session. A registration dropped without a release, such as an attempt to register again that the server drops at its bound after its write returned, deletes its row or document from a task of its own. An attempt dropped while that write is in flight has no registration yet, so a row or document it writes stays until it expires, and e8f507f says so in both registrations' docs.
1c3b2bf released the registration after the drain, but the drain gave up after 5 s, and MongoDB's four workers and five of Cassandra's (GSI propagation, control plane, GSI delay, transaction recovery and GSI backfill recovery) took no shutdown token, so on those backends the release came while they still ran. ca13267 gives each of them the token and drains the GSI propagation workers with the others. A worker still running 5 s after its token fired is aborted, and the registration is released only when every worker has ended; when one could not be stopped, the row or document is left to expire. Cassandra's GSI backfill recovery worker ran each backfill in a task of its own that took no token, so a backfill could still write at the release; since d8fc8d7 each backfill stops when the worker's token fires, and the worker returns only after its backfills have ended. Work that requests started in the background, such as a restore's item copy, is not a worker and can still be running at the release; e8f507f documents that.
Unit tests of the stop: the registration is released after a worker that writes once more after its token fired; a worker that ignores its token is aborted before the release and runs no more after it; a worker that blocks its thread keeps the registration. Each fails with its part of the change reverted. a_backfill_the_recovery_worker_started_ends_before_the_worker_returns fires the recovery worker's token during a backfill of 20,000 items and requires the index to stay CREATING with no more rows written; with the backfill given no token, as before, it fails.
| let kinds: Vec<Kind> = pending.iter().map(|p| p.kind).collect(); | ||
| let same = kinds.windows(2).all(|w| w[0] == w[1]); | ||
| let items: Vec<String> = pending | ||
| // One item per id and kind of store: a body that runs in several stores |
Collaborator
There was a problem hiding this comment.
The grouping here reaches verify and serve ("1 baseline (account in 4 stores)"), but the extenddb migrate preview, which is what an operator runs first, prints its own list in cmd_migrate_runner.rs preview(), one line per store with no store name. With 4 account keyspaces after --forget 1, the preview printed the 1 baseline (account, breaking) line four times. On a 0.1.x Cassandra deployment with a thousand accounts, the upgrade preview would be a thousand identical lines per migration. Grouping in preview() the same way, or printing the keyspace, would fix it.
Collaborator
Author
There was a problem hiding this comment.
Fixed in fe3b4f8. The preview groups its pending lines by id, kind of store and kind, as the advice does, so four account keyspaces after --forget 1 print "1 baseline (account in 4 stores, breaking)" once. The preview and the advice share one function for it.
| /// Nothing: `init` takes the migration lock before its first write, and | ||
| /// the lock's claim creates the catalog database. A second `init` is | ||
| /// refused under the lock, because the catalog is initialized. | ||
| async fn create_catalog_db(&self) -> OpResult<()> { |
Collaborator
There was a problem hiding this comment.
The description says init refuses a store that is not a catalog and writes nothing to it. That holds on Cassandra (a keyspace with another program's orders table and a theme settings row was refused, naming both), but on MongoDB init adopts the database: with extenddb_catalog.orders holding a document and settings holding {_id: "theme"}, head init exited 0, created the catalog collections beside orders, and wrote its rows into that settings. migrate --yes refuses the same database as not initialized. #666's head and 0.1.x did the same, and the database names are fixed, so this isn't a regression. Either scoping that sentence to Cassandra, or porting the Cassandra check to MongoDB init, would make them agree.
Collaborator
Author
There was a problem hiding this comment.
Fixed in 6dc1648 by porting the check. MongoDB init now refuses a catalog database that is not a catalog and holds another program's contents, before it writes the lock document, naming each item and exiting 1. Another program's contents are collections that migration 1 does not create, a catalog-named collection of another type, with other options or with an index that migration 1 does not create, settings documents other than the lock's, schema_history documents other than 001_initial, and documents in any other catalog collection. "Not a catalog" is decided from the catalog ledger and the catalog_version document, as the status decides Uninitialized.
The test refuses your orders collection holding a document and your theme settings document with "the database is not an ExtendDB catalog: it holds the collection orders, the settings document 'theme'. Nothing was written to it.", and leaves both as they were, with no lock document and an empty data database. It also refuses a capped tags and a metrics holding a document, and takes a database with migration 1's collections, 001_initial and an expired lock document. The data database is not checked.
| for line in output.strip().splitlines()[-3:]: | ||
| print(f" | > {line}") | ||
| if command[0] == "serve": | ||
| report(code not in (0, -1), f"exit {code}: {label}: the 0.1.14 serve refuses a v1.0 catalog", output) |
Collaborator
There was a problem hiding this comment.
This passes when the 0.1.14 server exits non-zero for any reason (port in use, a config it can't parse, a refused connection). The MongoDB check requires "resource-tag" in the output, and the older Cassandra server prints the same refusal ("this catalog has not had the resource-tag condition-key migration applied", checked with #666's head binary on a head-initialised catalog), so asserting that text here would keep the check from passing on an unrelated failure.
Collaborator
Author
There was a problem hiding this comment.
Fixed in d123441. The check now also requires the 0.1.14 server's refusal text, "this catalog has not had the resource-tag condition-key migration applied", as the MongoDB check does.
The preview printed one line per store that a migration was pending in, without the store's name, so on Cassandra a deployment with 4 account keyspaces after --forget 1 printed "1 baseline (account, breaking)" four times, and one with a thousand accounts would print a thousand. The preview now groups the pending lines by id, kind of store and kind, as the advice that verify and serve print already does: "1 baseline (account in 4 stores, breaking)". The grouping moves to a function the two share.
…server
check-cassandra-older-binary passed when the 0.1.14 server exited
non-zero for any reason, such as a port in use or a config it could not
parse. It now also requires the server's refusal of the catalog for its
missing marker ("this catalog has not had the resource-tag
condition-key migration applied"), as the MongoDB check does.
…alog names init and the migration lock took a table for the catalog's by its name alone, and the names are common ones (metrics, tags, tables, accounts, settings, schema_history). A keyspace holding another program's metrics (id int PRIMARY KEY, reading double) and tags (name text PRIMARY KEY, color text), with a row each, received the catalog's other tables, the lock row and migration 1 before the post-check refused it, and the application role init created was left behind. A table of a catalog table's name now belongs to another program when it has a column migration 1 does not give it, or defines otherwise (kind, position, clustering order or type), and so does a row in any such table other than settings and the schema_history rows migration 1 inserts. A column that migration 1 adds with ALTER TABLE ... ADD may be missing, so a keyspace an init left before migration 1 recorded anything is still taken as a catalog to finish. init runs the check before it creates the application role, so a refused init creates nothing.
…assandra-mongodb Brings in the answers to the review of #666. Conflicts: cmd_migrate.rs and cmd_verify.rs keep this branch's paths, which every backend now takes through the ledger, with verify's --all-notes and its exit order from #666. Migration 2's refusal takes #666's list of failed lookups, each with its own text; this branch's test of a lapsed lease's message now checks that the text is in that list. The ADR and the upgrade manual keep both sides.
Migration 2 is now declared not convergent on Cassandra and MongoDB, as #666 declares it on SQLite and PostgreSQL. A dynamodb:ResourceTag key written through the API after the rewrite has no effect, as on the service, and running the rewrite again would give it one. A reconverge now leaves such a policy as it was written, also one that names both forms of a key, which the rewrite refuses, and says that it skipped migration 2. Tests on both backends: after a released catalog is migrated, policies with the dynamodb: form and one with both forms are written, the marker a 0.1.x release leaves is put back, and the reconverge leaves the text of every policy as it was and names migration 2 as skipped. Each test fails with migration 2 declared convergent.
…ead the others Migration 2 reads each principal's policies and boundary through list_policy_documents and boundary_document, which #666 overrides on SQLite only. On Cassandra the trait's default went through list_policies, which fails the whole list when one stored document does not decode, so the status, verify and migrate --yes stopped with exit 1 and "Database error". On MongoDB list_policies and the boundary getters skip a document they cannot read as JSON, so migration 2 left such a policy as it was and reported nothing. Both stores now override the two methods and decode each document on its own, and their leased stores, which a migration reads through, pass the calls on. A document that does not decode is an item that needs attention, named with its account, principal, policy and error, as on SQLite. Tests on both backends: a principal with one undecodable policy, one tagged policy and a tagged boundary. The status under the lock is NeedsAttention and names the undecodable policy, and the rewrite rewrites the other two. Both tests fail without the overrides.
… registration last A Cassandra or MongoDB server whose registration could not be renewed for 30 s ended the process with std::process::exit(1), skipping the serve shutdown path, as a PostgreSQL server did before #666. Both backends now implement ServingRegistrar. On a loss the server answers new requests with ServiceUnavailable and registers again: a new row or document, then a read of the migration lock and of the catalog's status. It serves again when the status is the one it started with. A migration that holds the lock, a changed status, or 30 s without a registration stop it through the shutdown path with exit 1. The registration is now renewed every 5 s instead of every 15 s, and every second after a renewal failed, each renewal bounded by the interval. It counts as lost after 30 s without a successful renewal, so an outage shorter than that leaves it in place, and the server can register again during the 30 s after a loss. The registration was released while the other workers were told to stop, so a migration that waited for the servers could start while a worker still wrote to the catalog. ServingRegistrar gains release(), which serve calls after the workers have drained. PostgreSQL keeps the default, which does nothing, since its key ends with the session. A registration dropped without a release, as an attempt the server drops at its bound, deletes its row or document from a task of its own. Tests: on both backends, an attempt to register again replaces the row or document when the status is unchanged, and is refused, leaving none, while a migration holds the lock or once the status changed; a dropped registration's row or document is deleted. On Cassandra, a registration whose node stops answering is lost within the deadline plus one renewal's bound. The drain in serve releases the registration after the workers have stopped. Each of these fails with its change undone.
…by one lock timeout migrate --yes on Cassandra gave the wait for the lock and the wait for the servers a --lock-timeout each, the second counted from when it took the lock, so a run could wait twice the timeout and then report only the second wait. Both waits now end at one deadline, the lock timeout counted from the start, as on PostgreSQL and MongoDB, and the refusal says how long the run waited for the lock and the servers. The upgrade manual says that one --lock-timeout bounds both waits, and that a server started while a migrator waits for the servers exits 6. Test: with a server registered, migrator A takes the lock and waits for the server; migrator B, started 1 s later with a 6 s timeout, gives up within 8 s of its start. With the server wait counted from when B took the lock, B waited 10.2 s.
…'s ledger row Migration 1's catalog body deleted the resource_tag_migration settings document, which the 0.1.14 server needs, before the post-check and the recording of the step. When the post-check refused the catalog, the run recorded nothing and the document was gone, so neither 0.1.14 nor this release served the catalog. The backend now deletes the document in the transaction that records the catalog step, after the body and the post-check, and on every later run of the step, as a reconverge does. The upgrade manual's MongoDB paragraph now says that 0.1.12 and 0.1.13 servers read neither the document nor the catalog version, start on a migrated catalog and must not be run against it; that going back means restoring the backup taken before migrate --yes; and what a 0.1.x migrate does on a migrated catalog. Test: a 0.1.14 catalog whose post-check fails keeps the document and records nothing, and the run after the cause is removed deletes it. With the deletion back in the body, the document was gone after the failed run.
…ock without reading every policy Before it took the lock, migrate --yes on MongoDB read the full catalog status to refuse a store that is not a catalog, and that status ran migration 2's detect, which reads the policies of every principal. On a catalog with 6,000 tagged users that read took 42 s on the test server, outside the lock timeout: against another holder of the lock, a run with --lock-timeout 2 gave up after 41 s. The check now reads the ledgers and the settings alone, through a new Runner::initialized, so the lock timeout bounds the wait, and the run reads the policies one time fewer: 24,030 finds instead of 36,081 on that catalog. The status under the lock and the runner's detect still read them. Test: with 6,000 tagged users and another holder of the lock, a run with a 2 s lock timeout gives up within 4 s. Without the change it waited 41.2 s.
8 tasks done
…migrations are pending The bootstrap service of both compose stacks ran `migrate --yes` on every `up`. `migrate --yes` takes the migration lock and waits for every running server of the catalog to stop, so on a running stack with nothing to apply it waited the 60 s lock timeout and exited 6, and `docker compose up` failed. The PostgreSQL stack has done this since #666, and the MongoDB stack since its server registers. The bootstrap now runs the preview, which takes no lock, and runs `migrate --yes` only when the preview exits 3. Any other exit status ends the bootstrap with that status. The entrypoint in the deployment guide ran `migrate --yes` on every start and now runs it the same way, and the container README says what the bootstrap runs on later starts.
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 6, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 6, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 6, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#666 put SQLite and PostgreSQL on one numbered list of catalog migrations with a ledger in every store. This PR moves Cassandra and MongoDB onto the same list and runner. All four backends read their catalog status from their ledgers,
serverefuses a catalog that does not match the binary, and every change runs under the migration lock. With every backend on the runner, the code that served a backend without catalog migrations is removed: the bootstrapper's migration methods, the old branches ofinit,migrateandverify,serve's resource-tag marker check andOperationsEngine::catalog_version.Stacked on #666; the diff is against
feat/catalog-migrations. The design is ADR-0023, which now has Cassandra and MongoDB rows. Refs #637, to close once the stack reaches main, and #636, #346, #470 (permissions underCassandraAuthorizer).The migrations on each backend
Migration 1 deletes the marker row the 0.1.14 server requires, and writes the legacy
schema_historyrows (Cassandra) or001_initialdocument (MongoDB), so a 0.1.xmigratefinds nothing to do. 0.1.12 and 0.1.13 MongoDB servers read neither, so they must not be run against a migrated catalog.What changes for operators
migrate(with--yes,--list,--forget,--reconverge,--lock-timeout) andverifywork on Cassandra and MongoDB as on SQLite and PostgreSQL, with the same exit codes.serverefuses a catalog whose status is notCurrentorAheadCompatibleand prints the command that resolves it.initandmigrate --yesrefuse a keyspace or database that is not a catalog, or that holds another program's tables, collections or rows, and write nothing to it.initfinishes a catalog that an earlierinitleft before its admin user.migrate --yesonly when the preview exits 3, so anupon a running stack succeeds.SERIAL) or a document (MongoDB, majority writes) renewed while the run makes progress. Each server keeps a registration, andmigrate --yeswaits for the servers to stop, then exits 6 naming them. One--lock-timeoutbounds both waits.ServiceUnavailable, stops its background workers, and tries to register again for 30 s. It serves again when the catalog's status is unchanged, and otherwise exits 1. Requests already running, and work they left running in the background (a restore's item copy, a TTL backfill, an index build on PostgreSQL), go on until they end or the process exits.What changes for contributors
docs/schema/cassandra.txtanddocs/schema/mongodb.txthold the current schema, and CI fails when a fresh install differs.CONTRIBUTING.md"Adding a catalog migration" covers all four backends.tests/migrations.rsandtests/migrations_stopped.rs, a two-node job, aCassandraAuthorizerjob and the catalog checks, and MongoDB's catalog migration tests and catalog checks.Changes since the review of 1 November
Each reply on a review comment names its fix commits. In short:
feat/catalog-migrationswith feat(migrations): one list of catalog migrations with a ledger in every store, for SQLite and PostgreSQL #666's answers to its own review.init: 5513001, a16bfdd, 6dc1648, 4dfbea8, fa4f16a, 835f9ce, fd19d0b, 4dae6cf.Testing
GitHub CI runs on this head. Before the push, every job of the test, clippy, fmt, licenses and integration workflows ran on one host at e8f507f, with
--locked, andcargo checkpassed at each of the 36 commits. Everything passed except these, each of which also fails at b8584d5 or is not run by a pull request workflow:gsi_can_be_added_to_a_table_with_no_sort_key: the GSI recovery worker drops and recreates a new index's table, and a query in that window gets "Unknown CF".test_ttl_enable_rejects_asynchronous_gsi(Cassandra ttl_integration: test_ttl_enable_rejects_asynchronous_gsi asserts a refusal wording #384 replaced; masked by continue-on-error #566), in a step CI runs withcontinue-on-error./tmpis owned by another user), andcheck-released-throughandcheck-licenses.The MongoDB proxy test added in 448c36f failed when the rest of its binary ran beside it; aeeeaca fixes its deadline, and it passed 3 of 3 runs of the whole binary.
Known issues
#683, #684, #685, #686 and #689, filed from this work, and the GSI recovery race above. Work a request leaves running in the background is not stopped when the registration is lost.