Repository navigation
fix(mongodb): store the key when UpdateItem creates an item - #473
Merged
Merged
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 23, 2026 20:05
This was referenced Oct 24, 2026
robinnsc
reviewed
Oct 28, 2026
robinnsc
left a comment
robinnsc
left a comment
Collaborator
There was a problem hiding this comment.
The fix itself holds up. A creating UpdateItem now stores the same document PutItem does for every key shape we tried, and an upgrade through the API with 39 broken documents across 10 tables came out clean: the startup log counted exactly 39, and every key then read back with its key and took a PutItem. A few things are outside the diff or are pre-existing, so I'd treat them as follow-ups:
- A backup taken by 0.1.14 or earlier, restored on this build, brings the broken documents back, because restore copies the collection with
$outand the repair only runs at startup. GetItem returns items without their key and PutItem answers 500 until the next restart. Runningrepair_tableon the restore target before it goes ACTIVE would close it. - On a table that gained a GSI after the broken documents were written, the repaired items stay missing from the GSI. On base the backfill's resume cursor (
$gton a string_id,data_engine.rs:3044) skips ObjectId documents once a table has more than 500, so the index went ACTIVE without them. After the repair we saw 640 items in the table and 600 in the GSI. This was already wrong on base, but it might be worth qualifying "behaved as for any item" in the description for tables with a GSI. - A conditional create with
attribute_not_exists(pk)on the pushdown path (data_engine.rs:2912-2958) checks the condition against an empty item and then does an unconditional upsert, so concurrent creators can both get 200. 1 or 2 of 40 keys had 2 to 3 winners per run, at the same rate on base. It isn't part of #436, so it probably wants its own issue. - The description's base wire-test counts (16 failed, 10 passed, 2 xfailed) add up to 28, while the file has 32 cases. Against the base binary we got 20 failed, 10 passed, 2 xfailed.
| // yet answers as it did before the repair existed, and each batch is | ||
| // its own transaction, so running this while another process serves | ||
| // the same data is safe. | ||
| data_engine::key_repair::spawn_key_repair(engine.clone()); |
Collaborator
There was a problem hiding this comment.
The handle from spawn_key_repair is dropped here, so the task isn't one of the handles drain_workers waits for, and the repair loop never reads the shutdown token. A normal SIGTERM during a repair therefore leaves the batch transaction open on the server, the same as a kill. On the next start, writes to that batch's keys wait for the server to expire it. With 300,000 planted documents and SIGTERM 1.5 to 4.3 s after start, 2 of 4 runs had an UpdateItem on a held key block for 66 s and 86 s, which is past botocore's default 60 s read timeout. Keys outside the batch answered at once, and a SIGTERM after the repair finished left nothing open. A startup failure after this line (the catalog client and encryption key are set up below it) should behave the same way, though we only read that from the code. Returning the handle with the other workers, or checking the shutdown token between batches so the drain waits for the open batch, would avoid it. Either way the description and the module comment, which describe this as a killed-process case lasting 60 s, could be updated.
| //! start. The scan projects `_id` only, so the `_id` index answers it without | ||
| //! reading a document. | ||
| //! | ||
| //! A client write to a key whose document a repair transaction holds waits for |
Collaborator
There was a problem hiding this comment.
This holds for sessionless writes, but a write inside a transaction that hits a held document gets a WriteConflict at once instead of waiting. In the window after a stale repair transaction, a TransactWriteItems Delete on a held key answered TransactionCanceledException, and a DeleteItem on a cold cache (which takes the session path) answered TransactionConflictException. Also, the measured wait was 66 to 86 s, not 60, since the server's reaper runs periodically after the lifetime limit. Small thing nearby: the "rewritten after waiting N s" line counts only the 2 s sleeps, so 86.7 s of wall time was logged as 70 s.
| Item::new() | ||
| }; | ||
|
|
||
| let pk_name = key_schema |
Collaborator
There was a problem hiding this comment.
On a table with more than one HASH attribute (allow_multipart_table_keys), pk holds the netstring composite (2:x1,2:y1,), and this puts the whole text into the first attribute and never recovers the others. Two broken documents on such a table logged missing partition key attribute b, the summary said 0 repaired, 2 failed, the documents stayed broken, and the WARN comes back on every start. It's a preview flag that's off by default, so this is minor. The creation fix itself works on those tables: a new creating UpdateItem stored both key attributes and a PutItem then succeeded.
| let mut rebuilt: Vec<Document> = Vec::with_capacity(ids.len()); | ||
| let mut cursor = coll | ||
| .find(doc! { "_id": { "$in": ids.to_vec() } }) | ||
| .session(&mut *session) |
Collaborator
There was a problem hiding this comment.
Reading the batch inside the transaction is what keeps a concurrent client write from being lost, but no test pins it. Dropping this .session(&mut *session) so the read happens outside the transaction still passes all 11 unit and 16 live tests. With that change, a write that lands between the read and the delete_many is overwritten by the stale rebuilt document. A test that runs a client $set between the read and the delete (through a hook or a two-phase helper) and checks that the attribute survives would cover it. Of the 19 mutants we ran, these others also survived: the upsert retry budget cut from 50 to 2, CONFLICT_WAITS set to 0, the delete/read count check disabled, and the progress logging removed.
LeeroyHannigan
force-pushed
the
fix/mongodb-update-creates-key
branch
from
October 29, 2026 23:16
23feb86 to
58fd344
Compare
An unconditional UpdateItem on a table without an index runs as one findAndModify with upsert on the MongoDB backend. When the item did not exist, MongoDB built the inserted document from the filter fields and the update operators, so the document got a generated ObjectId as `_id` and an `item_data` without the key attributes. GetItem then returned the item without its key, and every PutItem on that key failed with code 66 (ImmutableField) and HTTP 500, because PutItem writes `_id` as the key text. A REMOVE-only update on a missing item produced a document with no `item_data` at all and answered 500 itself. The upsert now carries a `$setOnInsert` with `_id` as the key text and the key attributes under `item_data`, so a created document is the one PutItem writes, with `_v` 1 as on the transaction path. Two writers creating the same item now collide on `_id` instead of the key index, which the server does not retry for this filter, so the engine retries the upsert within the transient retry budget. The UpdateItem transaction path treats a duplicate key on its own insert as a transient conflict and runs again. Documents written without their key by earlier builds are repaired at startup: each table's data collection is scanned for an ObjectId `_id`, and each such document is rebuilt from its `pk` and `sk_*` fields and `item_data` and replaced in one transaction. The server logs a warning with the counts when it changed anything. New modules: data/key_fields.rs (insert-only key fields, repair of a document), data_engine_update_create.rs (the upsert and its retry), data_engine_key_repair.rs (startup repair), data_engine_live_tests.rs (live MongoDB tests, ignored without EXTENDDB_TEST_MONGODB_URI). The wire test tests/test_update_item_creates_key.py covers creation through SET, ADD on a number and on a set, REMOVE only, a condition, every ReturnValues, the later GetItem, PutItem, UpdateItem, Query, Scan and DeleteItem on the key, and the issue's race of UpdateItem and PutItem writers creating the same keys. Fixes #436
A REMOVE-only UpdateItem that created an item on the native fast path before the fix for issue #436 upserted a document with no item_data at all: the filter supplied pk and sk_*, the $unset contributed nothing and the $inc gave _v. The startup repair could not rebuild it, because repaired_document went through document_to_item, which fails with "Document missing item_data field". The document was counted failed and left in place, so its key answered 500 on GetItem, PutItem and UpdateItem on that start and every later one. Treat a missing item_data as an empty item, which rebuilds the document into the key-only item the fixed fast path writes for the same request. Two unit tests cover the shape for a hash key and a range key, and a live test plants it, reads the key as an error before the repair and as the key-only item after it. The live tests' store_without_key helper takes Some(attribute) for the SET form and None for the REMOVE-only form.
The comment described a check on both the _id and the item_data key attributes; the function tests only that _id is an ObjectId. State that, and why it is the whole test for a base-table document.
The repair held the listener closed: serve awaited the whole pass in server_components_factory, and the pass was one transaction per document. Measured on this host, 10,000 documents cost 25.64 s before anything answered on the port, and nothing reported progress while it ran. Run the pass as a background task, so the listener opens whatever the number of affected documents. A key the pass has not reached yet answers as it did before the repair existed, so a long pass delays the repair of that key rather than the start of the server. Rebuild up to 500 documents per transaction. A commit is one majority-acknowledged round trip and dominates the cost of a rebuild, so one commit per batch is what sets the rate: each batch is one read, one delete and one insert. The batch is atomic, so a process killed mid-pass still leaves every document either as it was or rebuilt. When a batch transaction cannot run, its documents are rebuilt one transaction at a time, so one document that cannot be written leaves the rest repaired. The batch reads its documents inside the transaction instead of taking them from the scan. A client write that lands between the scan and the rebuild is then either carried into the rebuilt document or turns the delete into a write conflict, which runs the attempt again. Progress is logged at INFO every 10,000 documents and once at the end with the elapsed time and the rate. A live test repairs 1,100 documents, which crosses two batch boundaries and ends on a partial batch.
The repair driver had no automated coverage. The scan, the batch transaction and the behaviour of two processes starting together are tested only by the live tests in crates/storage-mongodb, which are EXTENDDB_TEST_MONGODB_URI or passed --ignored. devtools/run-mongodb-tests already starts the replica set those tests need, so it now exports its URI and takes a --rust-live flag that runs them after the delegated suite. The MongoDB rust CI job passes that flag. Each live test works in its own pair of databases and drops them at the end, so it does not touch the served instance's catalog or data.
Measuring the batched repair after a SIGKILL exposed a regression against the one-transaction-per-document form. The killed process leaves its last transaction holding that batch's documents until the server expires it, which takes up to transactionLifetimeLimitSeconds, 60 s by default. The next start found 500 locked documents, spent the transient retry budget of about 0.3 s on them, and reported 114 of 200,000 as failed, so those keys stayed broken until a third start. Measured before this change: "152386 documents stored without their key repaired, 114 failed". A batch that cannot run is already rebuilt one document per transaction. Each of those now waits for a conflicting writer, 40 waits of 2 s, which covers the lock expiry with margin. Nothing waits on the repair, so waiting costs the start nothing and finishes the pass instead of leaving documents for the next one. A document rewritten after a wait is logged with how long it waited. Measured after this change on the same scenario: "129000 documents stored without their key repaired, 0 failed", one document logged as rewritten after waiting 62 s. The single-document rebuild now goes through the batch path with one id rather than its own delete-and-insert, so it reads the document inside its transaction and cannot overwrite a client's write with a stale copy. That makes the old replace_document redundant.
rustfmt 1.97.0 splits three calls in data_engine_key_repair.rs; no other change.
The native UpdateItem upsert writes each key attribute with a
$setOnInsert on the path item_data.<name>. MongoDB reads a dot in that
path as a step into a sub-document, so on a table with a key attribute
named a.b the creating call stored item_data { a: { b: ... } } where
PutItem stores the field "a.b". The call answered 500 and GetItem could
not read the document back. The same path also conflicts with an update
of an attribute named a, so SET a on an existing item of such a table
answered 500.
A table with a key attribute name that contains a dot or starts with $
now takes the session path, which writes the document item_to_document
builds. That is the test native_attribute_name already applies to the
updated attribute, and both gates now call one predicate.
The live tests create an item on a table with hash key a.b and on one
with range key s.k and compare the stored document with the one
item_to_document builds. They then read it with GetItem, Query and Scan,
set an attribute named a (or s) on it, and replace and delete it. A
third live test sets that attribute on an item PutItem stored. The wire
tests run the create test on tables with those key names and add a Query
and a Scan over them. A unit test pins the predicate for the dot and the
leading $.
The startup repair took a numeric sort key from its Decimal128 and used the display form as the number. Decimal128 displays a value below 1e-6 in exponent form (0.0000001 is 1E-7), while a request carries, and PutItem builds the _id from, the plain decimal that validate_and_normalize_number produces. A document repaired from 0.0000001 got the _id 2:u1,4:1E-7, where PutItem builds 2:u1,9:0.0000001,. PutItem on that key then matched the document by sk_n, tried to replace the _id and failed on the immutable field. The repair had counted the document as repaired. The repair now passes the display text through validate_and_normalize_number. A value at or above 1e-6 displays as the plain decimal already and is unchanged. The unit test adds 0.0000001, -0.00000012345, 1E-30 and 1E-7 (the last two spelled as a request spells them) to the numeric cases, with 0.000001, which Decimal128 displays as written, and compares the whole repaired document including the _id. The live test plants the pre-fix document for four numeric sort keys, runs the repair, and runs a PutItem on each key.
The UpdateItem transaction path ran its body again when the replace_one upsert that creates an item failed with a duplicate-key error, on the reasoning that another writer had created the item after the find_one. On MongoDB 7.0 that race does not surface as a duplicate-key error. An insert that loses to one committed after the transaction's snapshot, or still uncommitted in another transaction, fails with WriteConflict (code 112, TransientTransactionError), and the transient retry that was already there reruns the body and reads the winner's item. The duplicate-key error appears only when the colliding _id is visible in the snapshot, and every rerun meets it again, so the branch used all 50 attempts and ended in TransactionConflictException. The statement is back to the form main has: the error maps to Internal at once, with the server's duplicate-key text. A live test creates an item with eight concurrent numeric ADD calls, which take the transaction path, and checks that all apply: c is the sum and _v counts every call. It passes with and without the branch, because the write conflict is retried either way. A second live test plants a document that holds the key text as its _id under a different pk and checks that a creating ADD answers Internal with the duplicate-key error; with the branch it answers TransactionConflict after the 50 attempts.
The startup repair ended the whole pass at the first table-level error. A cursor error in repair_table, for instance from a table dropped while the pass ran, came back through the ? in repair_items_stored_without_key, so every table after it kept its keyless documents until the next start, and one error line was the only sign. The pass now logs the table, counts it in the summary as stopped by an error, and goes on to the next table. The final line is logged at warn when a table or a document failed. The scan returned whole documents although only the ObjectId _id is used. It projects _id now, so the _id index answers it. In a profiled run on MongoDB 7.0 the same find over five keyless documents examines five documents without the projection and none with it. The module comment states how a client write behaves while a repair transaction holds its document. The write waits for the transaction and sets no maxTimeMS. Normally that is one batch commit. After a process is killed mid-batch it is the time the server takes to expire the transaction, transactionLifetimeLimitSeconds, which is 60 s by default and was 60 on the test server, for up to 500 keys. A live test puts a catalog row whose table id MongoDB refuses as a collection name (it contains $) ahead of a readable table, and checks that the pass returns Ok, counts one stopped table and repairs the second table's document. A second live test reads the profiler entry of the scan and checks that it returned all 20 documents and examined none.
The live tests called repair_table and repair_items_stored_without_key directly, so spawn_key_repair could stop running the pass and every test would still pass. The new live test registers a catalog table, plants a document in the shape the earlier upsert stored (an ObjectId _id and no key attributes), awaits the JoinHandle that spawn_key_repair returns, and checks that the document is the one item_to_document builds with its _v kept. The call to spawn_key_repair in lib.rs is still not under a test. The MongoDB pytest job starts one server before any test runs and runs the files in parallel, so a pytest cannot plant a document ahead of the startup pass.
key_fields.rs said item_to_document writes _v, and update_create.rs said the created document is the one PutItem would have written with _v 1. Neither item_to_document nor PutItem writes _v; the update's $inc adds it. Both module comments now say so, and so do the doc comments of two live tests that described the stored document the same way. The live test of the created document asserted, for a REMOVE-only update, that ALL_NEW and the old image of the PutItem that follows both carry the key-only item. The service creates no item for that request and its ALL_NEW answer for one is not measured, so the REMOVE case now asserts the stored document and that the PutItem succeeds, and no returned item. repaired_document decodes a partition key from the pk text through pk_value, and only string partition keys were under test. The new unit test repairs a numeric partition key, a numeric partition key below 1e-6 (whose pk text keeps its plain form) and a binary partition key, and compares the whole rebuilt document with the one item_to_document builds.
data_engine_live_tests.rs had grown to 711 lines. The three tests of the document an UpdateItem writes when it creates an item move to data_engine_live_tests_create.rs, which is 227 lines. The shared helpers and the repair tests stay in the original file, which is now 501 lines. The moved block is byte-identical, and only the imports and the module declaration change. The test names gain the module name create::, so the filter data_engine::live_tests:: that devtools/run-mongodb-tests uses still selects all 16 live tests.
LeeroyHannigan
force-pushed
the
fix/mongodb-update-creates-key
branch
from
October 30, 2026 09:28
58fd344 to
9607fd7
Compare
robinnsc
self-requested a review
October 31, 2026 05:03
robinnsc
approved these changes
Oct 31, 2026
This was referenced Oct 31, 2026
LeeroyHannigan
added a commit
that referenced
this pull request
Nov 2, 2026
…assandra-mongodb Brings in main through #666's branch (4fa6769). The one conflict is in crates/storage-mongodb/src/lib.rs: #473's pass that rebuilds documents stored without their key now starts in build_components, after the engine is created and before the catalog client, where it started in the factory.
LeeroyHannigan
added a commit
that referenced
this pull request
Nov 2, 2026
…istered The pass #473 added runs from the server's start in a task of its own, outside the worker set, so it goes on while the server holds no registration. It writes only the data database.
robinnsc
added a commit
that referenced
this pull request
Nov 3, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 5` is the migration. `util::require_served_catalog_version` is the rule `serve` applies on a backend whose catalog carries a version string. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 5 (`0005_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 5 as a code body with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections, a `restore_job` subdocument on a restore target's `tables` document, and the versioned step `002_backup_catalog_index` recorded in `schema_history`, which backfills the registrations, writes the catalog id and moves `catalog_version` to `0.0.5` (one constant, `CATALOG_VERSION`, where three copies of `0.0.2` were). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores are removed. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, naming `extenddb migrate --yes`, and checks it before the factory builds the data engine or starts the startup key repair of #473 (`open_data_engine`), so a server that refuses has written nothing. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. `V006__backup_catalog_index.cql`, after which the runner writes `0.0.5` (once the backfill step and the file's `schema_history` row are in, so a run that fails in the step leaves 0.0.4 and `migrate --yes` runs the file again): the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, `backup_maintenance`, and a Rust backfill step keyed on the file. The file's statements are `ALTER TABLE ... ADD IF NOT EXISTS` and `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace (#638); the step and the version write name the keyspace in their own statements, and no code or test issues `USE`. An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, as on MongoDB. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration and versions. One change, one number: migration id 5 on PostgreSQL and SQLite, `0.0.5` on MongoDB and Cassandra. The migration is breaking (stop pre-upgrade servers first); a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Docs. The upgrade manual (migration 5 and its rollback, the Cassandra and MongoDB `0.0.5` section with the Cassandra resume sentence, the serve-time refusal, the removed 15 s sweep line, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the architecture guide, the MongoDB design document (the version check, run before the data engine is built), the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 6, the sample config, `docs/schema/postgres.txt` (regenerated for the collation) and `sqlite.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration numbers as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. extenddb-storage 204 (14 new; 205 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 new), extenddb-engine 208 (25 new: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (28 kept, 1 rewritten, 2 removed, 9 new), every other binary green (fresh 5, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 241 (3 rewritten, 12 new, among them a target removed only under its full key; 1 ignored benchmark). extenddb-storage-mongodb: lib 152 plus 45 live tests against a replica set (22 backup, 6 stream, 17 key repair; 11 new or rewritten, among them a catalog at the previous version refused with no repair started, 5 removed with the slots and the sweep), interpreter_selftests 18, pushdown_parity 20. extenddb-storage-cassandra: lib 78 (4 new), direct_integration 192 (12 new, 1 rewritten, 1 removed, 1 renamed; three of the new tests build with `--features test-hooks`), ttl_integration 26 passed and the one pre-existing failure below. tests/test_resource_tag_migration.py's released-catalog fixture strips migration 5's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Wire, on the tree before its rebase onto 4fa6769: tests/test_backup_restore_fidelity.py (37) and tests/test_table_operations.py (27) against a `serve` of this tree: 64 passed on PostgreSQL with `EXTENDDB_TEST_BACKEND=postgres EXTENDDB_CONFIG=<config>` (the DELETING test reads the catalog), 63 and 1 skipped without them; on SQLite 63 and the DELETING test skipped without them (with them it fails on the host's sqlite3 3.7.17 module, which cannot read a partial index); on Cassandra 62 passed and 2 skipped (the vector test and the DELETING test). On 4fa6769 the MongoDB workflow's commands pass against a release build of this tree: `run-mongodb-tests --rust-live -- --rust-integration` (tests/rust 512, key repair live 17, records live 6) and the pytest job's `--pytest --comprehensive --parallel` (1551 passed, 50 skipped; 340). `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88, `cargo doc` with no new warnings. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure predates this change (it fails the same way without it, checked on a clean bffa088: the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
8 tasks done
robinnsc
added a commit
that referenced
this pull request
Nov 3, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: on PostgreSQL its serving registration's session carries the name as `application_name`, so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL both commands hold a serving registration named after the command for their run, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the PostgreSQL registration, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This was referenced Nov 3, 2026
LeeroyHannigan
added a commit
that referenced
this pull request
Nov 3, 2026
Main added a live test module for the data engine in #473: its tests are #[ignore]d, read EXTENDDB_TEST_MONGODB_URI through one uri() helper, fail rather than skip when a run passes --ignored without it, and run in the mongodb-rust job through devtools/run-mongodb-tests --rust-live. Its scratch_engine helper builds the engine with MongoEngine::new and struct update syntax, so a field added to MongoEngine needs no change there. The two create-race storage tests move into that module as data_engine_live_tests_put_race.rs. They used their own URI gate, which printed SKIP without the variable, and a with_scratch_engine helper that built MongoEngine from a struct literal. Against main that literal missed the four fields #384 added, and the test build failed with E0063. Both helpers are gone, and the tests check what they checked before. devtools/run-mongodb-tests exported EXTENDDB_TEST_MONGODB_URI a second time for these tests. Main exports it since #473, so the line is removed.
LeeroyHannigan
added a commit
that referenced
this pull request
Nov 3, 2026
…EATE_RACE_ATTEMPTS Main's #473 added upsert_native_update, whose create race loop is sized by TRANSIENT_RETRY_ATTEMPTS. That is 50 on main and 1,000 on this branch, and the merge ac1f3b4 gave the loop this branch's value with no conflict, so a lost create race on _id retried up to 1,000 times with backoff, about 6.4 s of backoff before it answered TransactionConflictException, where main stops after 50. The loop now counts against CREATE_RACE_ATTEMPTS, the 50-attempt budget a lost create race has on the PutItem paths. Also from the review of this branch before the push: the BatchWriteItem doc comment says which writes answer TransactionConflict; the differences row says which error a batch answers when none of its requests was applied, and what the service answered; and a wire test checks that a batch whose only request is held past the deadline answers 503 and applies nothing.
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is #335's `drop_continuous_backups`). It is breaking (stop pre-upgrade servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
On the MongoDB backend, an unconditional UpdateItem on a table without an index runs as one
findAndModifywithupsert: true, filtered on the key fields. When the item did not exist, MongoDB built the inserted document from the filter fields and the update operators: a generated ObjectId as_idand anitem_datawithout the key attributes. GetItem returned the item without its key, and every PutItem on that key answered HTTP 500 withcode 66 (ImmutableField)in the server log, because PutItem writes_idas the key text. A REMOVE-only update on a missing item stored a document with noitem_dataand answered 500 itself.Fixes #436.
Change
data_engine_update_create.rs(new): the upsert carries a$setOnInsertwith_idas the key text and the key attributes underitem_data, built bydata/key_fields.rs::insert_only_key_fields. A created document is now the oneitem_to_documentbuilds, plus_v1 from the update's$inc, as on the transaction path.$. The native upsert writes each key attribute with a$setOnInserton the pathitem_data.<name>, and MongoDB reads a dot in that path as a step into a sub-document. On a table whose key attribute is nameda.b, the creating UpdateItem storeditem_data: {a: {b: ...}}where PutItem stores the field"a.b". The call answered 500 and GetItem could not read the document back. The same path also conflicts with an update of an attribute nameda, soSET aon an existing item of such a table answered 500. CreateTable accepts these names, so any table with a hash or range key attribute named like that was affected, for the UpdateItem calls that take the native upsert (an unconditional UpdateItem that does not return the old item, on a table with no index and no stream). A key attribute name that starts with$is excluded the same way. On MongoDB 7.0.45 a$setOnInsertonitem_data.$kstored the field$kliterally (one probe), and other server versions were not tried. A table with such a key attribute name now takes the session path, which writes the documentitem_to_documentbuilds. The fast-path gate uses the same test thatnative_attribute_namealready applied to the updated attribute, now one function (update_create::is_plain_path_name). Query and Scan over a table with hash keya.band over one with range keys.kreturn the created item (live and wire tests)._idset, two writers creating the same item collide on_id_, which the filter does not name, so the engine retries the upsert within the transient retry budget (findAndModifyreports E11000 as a command error). The transaction path of UpdateItem is unchanged. A creator that loses the race for a key there gets a write conflict from MongoDB (code 112, TransientTransactionError), which the existing retry handles. A duplicate-key error appears there only when the colliding_idis already visible in the transaction's snapshot, and it is returned at once as an internal error.data_engine_key_repair.rs(new) runs as a background task once the engine is built, so the listener opens without waiting for it. Each table's data collection is scanned with{_id: {$type: "objectId"}}, projecting_idonly, so the_idindex answers the scan and it examines no documents whether or not any match (MongoDB 7.0 profiler, five matching documents: 0 examined with the projection, 5 without). Up to 500 such documents are rebuilt per transaction from theirpkandsk_*fields and theiritem_data, keeping_v: the batch is read inside the transaction, deleted by ObjectId and inserted. A document stored with noitem_data(main's REMOVE-only create) is rebuilt as the key-only item the fixed upsert writes for that request. If a batch cannot commit, its documents are rebuilt one per transaction, each sleeping up to 80 s in total between attempts while a writer holds it. A document whose key fields cannot be read is logged and left in place. Progress is logged at INFO every 10,000 documents, and once at the end with the counts, the elapsed time and the rate. The final line is logged at INFO, and at WARN when a document failed or a table's scan stopped on an error. A scan that stops on an error, as when a table is dropped while the pass runs, is logged and counted in the summary, and the pass goes on to the next table; the documents that scan had not reached wait for the next start._idcannot be changed in place, and repairing on the read and write paths would add a second write to PutItem, GetItem, Query and Scan on every request.Decimal128and passes the display text throughvalidate_and_normalize_number.Decimal128displays a value below 1e-6 in exponent form (0.0000001is1E-7), while a request carries the plain decimal and PutItem builds the_idfrom that text. Without the normalisation, a document repaired from sort key0.0000001gets the_id2:u1,4:1E-7,where PutItem builds2:u1,9:0.0000001,, a PutItem on that key answers 500 on the immutable_id, and the repair counts the document as repaired. Partition key text is stored as written and needs no normalisation. The values0.000001,12.5and1000000000000000000000000display as written and are unchanged by it. The repair is new in this PR, so no released build has produced documents of that shape.maxTimeMS. Normally the wait is one batch commit. After a process is killed mid-batch it is the time the server takes to expire the transaction (transactionLifetimeLimitSeconds, 60 s by default and 60 on the MongoDB 7.0.45 server used here), for up to 500 keys.data_engine.rs: 22 insertions, 11 deletions; theNativeUpdate::Docarm calls the new module, and the fast-path gate adds the key attribute name test.E11000 duplicate key error ... index: _id_fromfindAndModify): that is the PutItem path fix(mongodb): retry an unconditional PutItem that loses the create race #431 changes. Before this change the same window ended in the ImmutableField 500 for that put and every later one.Tests
key_fields(9: two for the document main's REMOVE-only create stored, on a hash and a range table; one repairing numeric and binary partition keys; the sort key cases include 0.0000001, -0.00000012345, 1E-30, 1E-7 and 0.000001),update_create(2; one pins the key attribute name test).data_engine::live_testsfilter thatdevtools/run-mongodb-tests --rust-liveruns (ignored withoutEXTENDDB_TEST_MONGODB_URI): the created document equals the PutItem shape for SET, REMOVE and numeric ADD on hash and range tables, and a PutItem then replaces it; 64 UpdateItem calls creating one item from 8 threads all apply and_vis 64; an upsert whose insert always collides on_idanswers TransactionConflict, not an internal error; creation, GetItem, Query, Scan,SET a, PutItem and DeleteItem on a table with hash keya.band on one with range keys.k, andSET aon an item PutItem stored; eight concurrent numeric ADD calls creating one item all apply (cis 8 and_vis 8); a creating ADD whose_idis held by a document with a differentpkanswers the duplicate-key error at once; the repair rewrites documents stored the way the old upsert stored them (hash, and S, N and B sort keys, including 0.0000001, followed by a PutItem on each key), leaves healthy documents alone, and finds nothing on a second run; the pass goes on to the next table when one scan fails; the scan is answered from the_idindex (0 documents examined for 20 returned); the taskservestarts,spawn_key_repair, repairs a planted document; the whole-catalog entry point reads the key schema from the catalog row. The REMOVE case of the creation test asserts the stored document and that a PutItem succeeds, and no returned item, because the service creates no item for a REMOVE-only update and its ALL_NEW answer for one is not measured.tests/test_update_item_creates_key.py, 32 cases: creation through SET, ADD on a number and on a set, REMOVE only,attribute_not_exists(pk)once and refused the second time, ReturnValues ALL_NEW, UPDATED_NEW, ALL_OLD, UPDATED_OLD and NONE on creation, then GetItem, PutItem ALL_OLD, a second UpdateItem, Query, Scan, DeleteItem ALL_OLD and PutItem after the delete on the key; the creation test also runs on tables with hash keya.band with range keys.k, and a Query and a Scan run over both; and the issue's race, 8 UpdateItem and 8 PutItem writers on 30 keys at once. On main (MongoDB): 16 failed, 10 passed, 2 xfailed; the race left 16 of 30 (hash) and 8 of 30 (range) items without a key, and PutItem on them answered 500. On the final tree against MongoDB 7.0.45: 30 passed, 2 xpassed. The in-race PutItem assertion carries the same non-strict xfail marker on MongoDB astests/test_put_item_create_race.py, for the lost-create-race 500 that PR fix(mongodb): retry an unconditional PutItem that loses the create race #431 fixes.spawn_key_repairinlib.rshas no automated test, because the MongoDB pytest job starts one server before any test runs. It was run by hand on the final tree: a document planted in the pre-fix shape in a served table was repaired at start (startup key repair finished: 1 tables scanned, 0 stopped by an error, 1 documents stored without their key repaired, 0 failed), and GetItem, PutItem, UpdateItem, Query and Scan on its key then answered as for any item. Deleting the call fails the clippy gate (dead code); moving it or making it conditional would not fail any check.1 tables scanned, 0 stopped by an error, 2 documents stored without their key repaired, 0 failed, and GetItem, PutItem, UpdateItem, Query, Scan and DeleteItem on those keys then behaved as for any item.item_data: the port answered 0.2 s after start and the repair finished 0.83 s later. GetItem, PutItem and UpdateItem on the no-item_datakey answered 200. Before this round the same plant held the port closed for 25.64 s and left that document failed, its key answering 500.transactionLifetimeLimitSecondsis 60 on that server).devtools/run-mongodb-tests --rust-liveruns the data engine's live tests against the runner's replica set, and themongodb-rustjob in.github/workflows/integration-mongodb.ymlpasses the flag.Service measurements
Live DynamoDB, us-east-1, 2026-10-23 14:11 UTC, hash table
bb-w-M3-1(pk S) and range tablebb-arn-main(pk S, sk S), client retries off. Each case was run on both tables with the same result, the range table addingsk.SET w = :w, key absent, no ReturnValues{}; GetItem{pk, w: upd}ADD c :one(N 1), key absent{}; GetItem{pk, c: N 1}ADD tags :t(SS a, b), key absent{}; GetItem{pk, tags: SS [a, b]}REMOVE w, key absent{}; GetItem returns no ItemConditionExpression attribute_not_exists(pk), key absent{}; GetItem{pk, w: upd}; the same request again:ConditionalCheckFailedException, "The conditional request failed"{"Attributes": {pk, w: upd}}{"Attributes": {w: upd}}{}{pk, w: put}ALL_OLD on the created key{"Attributes": {pk, w: upd}}; GetItem{pk, w: put}SET x = :xALL_NEW on that key{"Attributes": {pk, w: put, x: N 1}}pk = :pbegins_with(pk, :prefix){"Attributes": {pk, w: put, x: N 1}}; GetItem returns no Item{}The REMOVE-only row is the one place the engine differs from the service on every backend measured (SQLite, MongoDB): ExtendDB creates an item holding the key. That is older than this change and the wire test does not assert it either way; it asserts the call succeeds and the key stays writable, which on MongoDB it did not before (500 on the call, since the created document had no
item_data). Key attribute names containing a dot or starting with$, the form in which the service returns a number such as 0.0000001, and the service's ALL_NEW answer for a REMOVE-only update on a missing item were not measured on the service.Relation to #431
is_duplicate_keyis unchanged. The command-error form of E11000 is matched in the new module, sogit merge-treeagainst the #431 head reports no conflict (git merge-tree --write-tree HEAD origin/pr431exits 0 against c3f8bc8).