Repository navigation
Conversation
23c1b71 to
38c7b00
Compare
4c35319 to
d27c9bb
Compare
09c881f to
ec4eb8d
Compare
d27c9bb to
a3e6c12
Compare
8d44bd8 to
22bfb5b
Compare
210165a to
19e4459
Compare
22bfb5b to
6bdd117
Compare
There was a problem hiding this comment.
I built the server at this head for SQLite, PostgreSQL, MongoDB and Cassandra, and at the base for SQLite. Each ran in a container with a filesystem store. I sent the same RestoreTableFromBackup requests, and the calls around a running restore, to each backend and to Amazon DynamoDB in us-east-1. I sent each error case twice. I killed, froze and stopped servers (SIGKILL, SIGSTOP, SIGTERM) at the pause points in this PR and at two I added. The two are right after the registration's commit and, on MongoDB, between the claim check and the insert_many. For the takeover cases a second server ran on the same catalog. I ran the engine's backup:: module (236 tests), the SQLite and PostgreSQL restore job tests (11 each), and the MongoDB and Cassandra restore cells. I ran 51 mutants of the code those tests guard, and for 45 a test failed on an assertion about the change. Clippy 1.97.0 with the test-hook features, which CI does not lint, passes.
The restore response and DescribeTable matched the service on RestoreSummary, on what a restore carries and drops, and on the tables restored with accepted overrides. They also matched on the errors for a missing backup, another account's backup, an ACTIVE target name and data-plane calls on a restoring table. On SQLite, PostgreSQL and MongoDB a restore of a backup in use answers BackupInUseException before any override error, as on the service. The fenced DONE write, the shutdown path, and remove-target on a target_not_empty record behaved as described. So did SQLite's takeover at restart and the job's first beat after a 75 s registration. Main has no store-backed restore, and the base answers ValidationException once a restore reads a store-backed backup, so each store-backed answer here is new. Two Blocking items, twelve Major and nine Minor ones are inline, and the Minor ones are marked. Smaller items:
- From #671:
backup_in_use_messagegivesBackup is being used to restore table <name>: <arn>, where Amazon DynamoDB givesBackup is being used to restore another table: <arn>(eight requests). This PR's registrations raise that text on all four backends. a_feeder_whose_timer_stopped_ends_the_stream_with_the_claim_lost_reasonfailed once in 40 runs of thebackup::module on a loaded host, at the 30 s deadline inwait_for_items_seen. It passed in the other 39 runs and in three reruns. Is 30 s enough on a loaded CI machine?
Once the Blocking and Major items are fixed, I will approve.
| /// the CREATING target with its provenance and its RUNNING `kind = | ||
| /// manifest` record under a fresh generation, the store and the | ||
| /// claimant's spool, with the server's clock on its heartbeat. The | ||
| /// transaction commits right after the insert, before the target's |
There was a problem hiding this comment.
Blocking: this commits the RUNNING record before the target's data collection, its unique key index and its GSI and LSI metadata exist. I killed the server right after this commit and restarted it, and the reconciler resumed the record into a collection indexed on _id only. The table went ACTIVE with no GSIs or LSIs, and Query on by-g answered ValidationException: The table does not have the specified index: by-g where the source returned 15 items. Please make the record resumable only after the last index metadata is written. Also have the resume check the target's indexes against the manifest.
| reapplying: bool, | ||
| batch: &mut Vec<Document>, | ||
| ) -> Result<u64, StorageError> { | ||
| self.confirm_restore_claim(target, generation).await?; |
There was a problem hiding this comment.
Blocking: this read checks the claim, and the insert_many that follows carries no condition, so a frozen writer can write into a finished restore. With two servers on one catalog, I froze the owner with SIGSTOP after this call, and the other server took the record over and finished. I removed 10 items through it and resumed the owner, and 20 s later all 10 were readable again with ConsistentRead and missing from by-g. Please put the fence in the write here, as the PostgreSQL and SQLite loads do. The Cassandra batch at crates/storage-cassandra/src/restore_engine.rs line 182 needs the fence too.
There was a problem hiding this comment.
On MongoDB I froze the owner inside its batch transaction, the other server finished, and the 10 items I deleted stayed deleted. On Cassandra the batch writes still follow a separate check, and the module doc says a stalled writer can write one batch after a takeover. Please fence those writes too, for example by writing each batch at a timestamp read before the check, so that later writes and deletes win.
There was a problem hiding this comment.
Fenced. The Cassandra batch now writes each item's base row and its GSI and LSI rows in one logged batch USING TIMESTAMP a reading of the coordinator's clock taken before the batch's claim check (put_restored_item_at, replacing the put path's lightweight transaction for the store-backed load; a CREATING target has no other writer). A writer that stalls between the check and its writes and lands them after a takeover lands them below every write and delete the successor made, so Cassandra keeps the successor's cells and tombstones and a key deleted through the ACTIVE table stays deleted; the base row's version is the timestamp, so a later ordinary write's version is above it. test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes (direct_integration, test-hooks) pauses the owner after its check, takes the record over, has the successor load the batch, flip ACTIVE and delete two items, then releases the owner: the two items stay deleted, their GSI and LSI rows stay gone, the survivors carry the successor's versions, and the owner's progress write reports the lost claim. With the timestamp read after the pause instead (when the writes land) the cell fails with 4 items in place of 2; it passes after the revert. The module doc states the rule and that it needs the hosts' clocks to agree, as the version scheme already does.
| ) | ||
| .await?; | ||
| } | ||
| restore_pause_point(RESTORE_MONGODB_AFTER_INSERT_MANY).await; |
There was a problem hiding this comment.
Progress is recorded once per data file, so a one-file restore records none until its load ends, and data files are 64 MiB by default. A SIGTERM in the middle of a 5,000-item one-file restore left the record RUNNING, and 39 s after the restart it was FAILED target_not_empty. The table then stays CREATING, with DeleteTable ResourceInUseException and DeleteBackup BackupInUseException, until an operator runs remove-target. But the differences row says a SIGTERM resumes to ACTIVE. Please mark the record before any insert_many and resume a marked record with reapplying (upserts on Cassandra). Keep target_not_empty for a count above the total.
There was a problem hiding this comment.
On MongoDB a SIGKILL after the first batch and a SIGTERM inside it both resumed to ACTIVE with every item and the full GSI. On Cassandra a crash before the first progress record still ends target_not_empty, as the differences row now says. Please resume that case with upserts, as my comment asked.
There was a problem hiding this comment.
Done. The reconciler's no-progress rule now compares the target's count with the backup's total (the backup row's item_count, exceeds_backup_total): above it is target_not_empty (the dump-restored catalog); at or below it the record is resumed and the first file re-applied with reapplying set, which on Cassandra is the same idempotent upserts. The engine cell a_half_applied_first_file_with_no_progress_is_resumed_with_upserts resumes one landed item of two to DONE with 2; the storage cell test_a_first_batch_applied_without_progress_resumes_to_the_exact_count lands 3 of 7 with no progress (two lost progress writes) and resumes to 7 with both index-row counts exact. The differences row, 5.3 step 7, the PR 9a load paragraph, the admin guide and the kill-point script's first-reach assertion (now ACTIVE with the exact count) are updated; the Cassandra point's first reach runs in the nightly job.
| Ok(count) => count, | ||
| Err(e) => { | ||
| tracing::error!( | ||
| "{what} was taken over but its target could not be counted; the next cycle \ |
There was a problem hiding this comment.
When the target's tables do not exist, this returns after claim_restore_dead refreshed the heartbeat, so the record is retaken every stale window and never fails. I killed SQLite and PostgreSQL servers after the catalog commit, before the data DDL, and each record was retaken every 45 s for three minutes. The table stayed CREATING, and DeleteTable, DeleteBackup and a restore into the name answered ResourceInUseException, BackupInUseException and TableInUseException. Please fail the record and call remove_restore_target when the target's tables are missing, and bound the takeovers of one record.
There was a problem hiding this comment.
A target without data tables now fails in one takeover on SQLite and PostgreSQL. Takeovers stay unbounded otherwise: when a resume fails on a read error or stops with LeftRunning, the reconciler claims it again every stale window. Please count the takeovers of a record and fail it after a small number.
There was a problem hiding this comment.
Bounded. ReconcilerState counts this instance's takeovers of each record by table id (RESTORE_TAKEOVERS = 5); the takeover after the fifth fails the record owner dead: taken over 5 times by one instance without the restore ending and removes its target as for OWNER_DEAD. The count is in process rather than a column: a takeovers column would reach every backend's record and PostgreSQL's catalog table_restores, which this PR's migration does not touch, so the count is per instance and a restart starts it over (stated in the module doc, the design and the differences row). the_takeover_after_the_fifth_fails_a_record_that_never_ends: the count fails on every attempt, five cycles take the record over and leave it RUNNING, the sixth fails and removes it. With the bound check disabled the sixth cycle resumes again and the cell fails; it passes after the revert.
| "{what} has no live owner and reads a store other than the configured one; the \ | ||
| record is FAILED and its target removed once the record is stale: {OTHER_STORE}" | ||
| ); | ||
| return match storage.fail_restore_job(target, OTHER_STORE).await { |
There was a problem hiding this comment.
fail_restore_job matches any RUNNING record, and an instance without [backup.store] takes this arm for every dead store-backed record it lists. So this write can fail a record another instance took over after the listing, and on PostgreSQL I queued such a takeover ahead of it. The record ended FAILED under the new generation, the new owner's DONE write for its 50 items matched no record, and the target was removed. Please condition this write and those at lines 689 and 719 on the listed owner_generation and a stale heartbeat. Also leave other stores' records alone.
| let index = | ||
| u32::try_from(index).map_err(|_| Stop::Fail("too many data files".to_owned()))?; | ||
| if let Dest::Spool(spool) = dest | ||
| && spool.is_verified(index, &entry.sha256) |
There was a problem hiding this comment.
Minor: a spooled file with a .verified marker is skipped here and later read as prehashed (load.rs line 118). So the restore loads a file changed in the spool between two runs without checking its digest again. Please hash spooled files at load, as the in-place path does.
| dest: &Dest<'_>, | ||
| ) -> Result<u64, Stop> { | ||
| // The retry bound's clock: one for the job, started here. | ||
| let started = Instant::now(); |
There was a problem hiding this comment.
Minor: download_all fetches every data file before the progress skip in restore.rs (line 674). So a resume on a host without the spool entry also downloads the files its progress records as applied. Please download only the files the progress leaves to load.
| }; | ||
| println!( | ||
| "removed the restore target {table_name} (account {account_id}, table id {}) and its \ | ||
| record: its {} restore from {} was {state}", |
There was a problem hiding this comment.
Minor: this prints its manifest restore from ... was FAILED (target_not_empty), the internal record kind, where the docs call these store-backed and in-catalog restores. Please print the documented terms.
| }) | ||
| } | ||
|
|
||
| fn claim_restore<'a>( |
There was a problem hiding this comment.
Minor: on Cassandra the engine's in-use pre-check, backup_in_use at line 1571, reads only the legacy backups row, so it misses the restores this function registers. While one restore of a backup ran, a second with a GlobalSecondaryIndexOverride naming an unknown index answered ValidationException: One or more parameter values were invalid: global secondary index nope is not in the backup on Cassandra. SQLite, PostgreSQL and MongoDB answered BackupInUseException, as Amazon DynamoDB did (Backup is being used to restore another table: <arn>) to six such requests. Please have backup_in_use read the store-backed registration as well, and add a Cassandra cell to the in-use precedence test.
There was a problem hiding this comment.
With a restore of the backup running, Cassandra answers BackupInUseException to the unknown GSI override, the throughput override and a plain restore, twice each. No test covers this on Cassandra yet. Please add the cell.
There was a problem hiding this comment.
Added test_a_running_store_backed_restore_keeps_its_backup_in_use_until_active to the Cassandra direct_integration suite: with a store-backed restore RUNNING, backup_in_use names the target and a second claim_restore is BackupInUse with the service's text, twice; with the backup's active_restores entry deleted (the registration's window before it lands) the target rows alone keep the backup in use; after finish_restore it is free. The order of the engine's checks (the in-use rule before the override validation) is pinned by the engine's cells; this is the Cassandra input to it.
| // The restored table's size is the source table's as the manifest | ||
| // records it (the statistics pass rewrites it once ACTIVE), not the | ||
| // compressed size of the backup's files. | ||
| let table_size_bytes = u64::try_from(manifest.source_table.table_size_bytes).unwrap_or(0); |
There was a problem hiding this comment.
Minor: this writes the manifest's item total and the source's size at ACTIVE, so a restored 20-item table showed ItemCount 20 on every backend. TableSizeBytes was 0 on SQLite and MongoDB, 49152 on PostgreSQL and 1385 on Cassandra, and 1385 on the in-catalog path of the same SQLite build. Amazon DynamoDB showed ItemCount 0 and TableSizeBytes 0 on all 15 tables I restored there, one of them 10 minutes after it went ACTIVE. Please write 0 and 0 here. Otherwise, add the service's 0 and 0 to the DynamoDB column of the "What a restore carries across" row.
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I checked every item from my last review at 2979565. I ran the round-one reproductions again on SQLite, PostgreSQL, MongoDB and Cassandra, including the SIGKILL, SIGTERM and SIGSTOP cases on the backends each item named. I applied the round-one mutants and ran the tests your reply names.
16 of the 23 inline items are fixed, and so is the first item in the review body. Each reproduction now gives the right answer, and each named test fails on its mutant and passes after the revert. The MongoDB halves of the claim fence and the one-file resume hold under a frozen owner, a SIGKILL and a SIGTERM.
I agree with your reply on the Cassandra job in the nightly matrix. Six items remain, with a reply on each thread. They are the Cassandra batch fence, the Cassandra first-batch resume, the takeover bound, the kill-point jobs, the claim_restore doc and the Cassandra in-use cell. The deadline in wait_for_items_seen is still 30 s. Please raise it in this PR, as your reply offers.
The new code brings one Blocking item, two Major items and four Minor items, all inline.
Once you fix the Blocking and Major items, including the ones on the threads, I will approve.
| for point in restore_after_download restore_after_file_commit \ | ||
| restore_sqlite_mid_batch restore_between_failed_and_removal; do | ||
| echo "=== SQLite restore kill point: $point ===" | ||
| python3 devtools/backup-kill-points.py --backend sqlite --point "$point" || rc=1 |
There was a problem hiding this comment.
Blocking: this call fails for every point, so the job runs none of the four. The script reads restore <backend> <point> --bin <path> --work <dir>, and these arguments exit 2 with "invalid choice: 'sqlite'". Please call restore sqlite "$point" --bin target/release/extenddb --work <dir>.
There was a problem hiding this comment.
Fixed: restore sqlite "$point" --bin target/release/extenddb --work "kill-point-logs/sqlite-$point". The job's exact call was parsed locally (--help, and a run with a bogus --bin that gets past the parser into init), and restore_after_file_commit and restore_sqlite_mid_batch ran end to end with it on a backup-test-hooks release build: both resumed and ACTIVE with 12,000 items, base and GSI exact. The MongoDB and Cassandra jobs' calls, with --second-reach and --source-bin, parse the same way.
| } | ||
| match storage.count_restore_target(target).await { | ||
| Ok(count) => return Ok(count), | ||
| Err(e) if attempt == COUNT_ATTEMPTS => { |
There was a problem hiding this comment.
Major: no test reaches the count retry or this stop. With a count error failing the restore at once, as before, all 261 backup:: tests pass. Please add a cell whose count fails three times, and assert that the record stays RUNNING with no FAILED write.
There was a problem hiding this comment.
Added a_count_that_fails_three_times_leaves_the_record_running: the fake's count fails three times (a new count_errors hook), the job stops with the record RUNNING at its last file's progress, no FAILED or DONE write, the loaded target kept. With the first count error failing the restore at once the cell fails (one count call, not three); it passes after the revert.
| // resume is left for the maintenance arm of a later cycle rather than | ||
| // run beside a command, and the record keeps this takeover's fresh | ||
| // heartbeat. | ||
| match storage.check_maintenance_claim(stale_job_after).await { |
There was a problem hiding this comment.
Major: with this second read of the maintenance claim skipped, all 261 backup:: tests still pass, so no test reaches it. Please add a cell where a command takes the claim during the cycle and the record is not resumed.
There was a problem hiding this comment.
Added a_claim_taken_during_the_cycle_leaves_the_taken_over_record_unresumed: a hook inside the fake's claim_restore_dead sets the maintenance claim held after the cycle's first read; the record is taken over (its generation moves) but not resumed, no bulk_load runs, and it stays RUNNING. With the second read replaced by a constant "not held" the cell fails (manifest_restores_resumed: ["MusicRestored"]); it passes after the revert.
| let store = runtime.configured_store_arc(); | ||
| // An instance with no configured store does not own any store-backed | ||
| // record: it leaves every one alone for the instance that holds its | ||
| // store, rather than failing records it cannot read (Lee's 4468802503, |
There was a problem hiding this comment.
Minor: this comment cites a GitHub comment id, as do nine other comments in this diff. Please describe each case in the code's own terms.
There was a problem hiding this comment.
Replaced all ten with descriptions in the code's terms (the engine's tests and reconciler, the Cassandra backup_catalog.rs and backup_engine.rs, the PostgreSQL backup_catalog.rs and tests/restore_job.rs); a grep for the ids over the tree finds none.
| .await | ||
| .map_err(|e| format!("cannot remove the restore target {what}: {e}"))?; | ||
| if !removed { | ||
| return Err(format!( |
There was a problem hiding this comment.
Minor: with the owner's lock held, this says the table "is not a restore target", but it is a CREATING restore target. Please report the held lock as its own case, so that the operator knows to wait.
There was a problem hiding this comment.
Reported as its own case. After the backend refuses the removal the command re-reads the record: unchanged, the table is still its CREATING target and the refusal was the lock, and the error is the restore into <table> (account <id>) from <arn> is a CREATING restore target (table id <id>) whose owner's lock is still held: its owner's session is alive though its heartbeat is stale; wait for the owner to stop, or stop it, and run the command again; changed or gone, it is not a restore target: it is no longer a CREATING restore target with table id <id>. remove_target_reports_a_held_owner_lock_as_its_own_case covers the held lock and the removal once it is released; the admin guide lists the line.
| let data_table = data_table_name(&key.table_id).to_lowercase(); | ||
| let rows = crate::cassandra_util::query_rows::<StorageError>( | ||
| &self.session, | ||
| "SELECT table_name FROM system_schema.tables WHERE keyspace_name = ? AND \ |
There was a problem hiding this comment.
Minor: this reads only the base data table, and the registration creates each GSI and LSI table after it, in separate statements. By the code, a crash between them passes this check. Please check the index tables too.
There was a problem hiding this comment.
Done: restore_target_ready reads each GSI's and LSI's table in system_schema.tables beside the data table, from the indexes the catalog names. test_restore_target_ready_requires_every_index_table drops the GSI's table and sees false.
| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra <!-- version-literal-ok: names the catalog that began recording backup definitions --> | | ||
| | How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On MongoDB the request waits at most 60 s for its background task to register the `CREATING` target (a bound against a connection pool that has stopped handing out connections); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the copy ends; `DescribeTable` is where a client finds it *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB). A restore runs inside the server process; if the process stops, see the abandoned-restore row | | ||
| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand. `DescribeTable` on a freshly restored table reports `ItemCount` 0 and `TableSizeBytes` 0 (measured on 15 restored tables, one of them ten minutes after `ACTIVE`) | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. `DescribeTable` on a restored table reports `ItemCount` 0 and `TableSizeBytes` 0 when the table goes `ACTIVE`, as the service does (measured: 0 and 0 on 15 restored tables, one of them ten minutes after it went `ACTIVE`), until the statistics pass next rewrites both from the restored rows *(from the code)*. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra <!-- version-literal-ok: names the catalog that began recording backup definitions --> | | ||
| | How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured *(in-catalog restore, an earlier build)* with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. Measured *(store-backed restore, this release)* with the same shape (60,000 items of 200 bytes, composite key, no secondary index) from a filesystem store on loopback: response in 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB); `ACTIVE` after 0.67 s, 1.85 s and 3.74 s (on MongoDB the job's DONE at 1.8 s and the index backfill worker's flip after); 1M items of 200 bytes in key order (the server's own backups) `ACTIVE` after 9.5 s on PostgreSQL and 26.8 s on SQLite, the same items as an unordered backup (a MongoDB source, a multi-part-key table) after 17.5 s and 66 to 90 s (`docs/design/15-backup-restore-bench-pr9.md`, which has the host and its load); the store-backed response includes the sidecar and manifest reads and its `ACTIVE` time the download and the per-file verification, and Cassandra as a store-backed target was measured at 12,000 items only (the kill points: a resumed restore of 12,000 items of 4,000 bytes in two data files reached `ACTIVE` 8.3 s after its takeover on a single-node container). On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On every backend the request waits at most 60 s (`RESTORE_REGISTRATION_TIMEOUT`) for its background task to register the `CREATING` target of a store-backed restore (a bound against a connection pool that has stopped answering; the manifest and sidecar reads that precede the registration are not under it); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the restore ends; `DescribeTable` is where a client finds it *(from the code)*. Because that 60 s bound equals botocore's default read timeout, a client whose first attempt times out while the registration is still in flight (a backup's `backup_index` row held under a slow lock) retries: a retry that names the same backup and target waits for the in-flight registration and is answered its `CREATING` description rather than registering a second target, and a restore of a *different* backup into the same name while the first is in flight is `TableInUseException` *(from the code)*. The job permit is the last of the request's checks: a restore that finds every `[backup] max_concurrent_jobs` slot held (default 4; backups and restores together, per server process) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), creating nothing, where the service accepted five restores sent together (measured on DynamoDB); on the store-backed path the permit is taken before the manifest is read, after the catalog row and the target name are checked. A restore runs inside the server process; if the process stops, see the abandoned-restore row | |
There was a problem hiding this comment.
Minor: a retry waits only when it reaches the server that holds the registration, since each server keeps its own in-flight list. Please say so in this row.
There was a problem hiding this comment.
Said in the row: each server keeps its own in-flight list, so a retry waits for the first attempt's registration only when it reaches the server that holds it; a retry on another server finds the name free until the registration commits and then TableInUseException, or the CREATING description once the target exists.
b6d1d34 to
41bf0db
Compare
2979565 to
07afb06
Compare
|
Pushed |
41bf0db to
0339924
Compare
07afb06 to
8c9b3bc
Compare
|
The six thread items are done (the Cassandra batch fence, the Cassandra first-batch resume, the takeover bound, the kill-point jobs, the |
0339924 to
0450e01
Compare
8c9b3bc to
d9c09bf
Compare
0450e01 to
141f911
Compare
d9c09bf to
ad78bbd
Compare
|
The replies above describe the head now pushed as |
141f911 to
b20e19e
Compare
ad78bbd to
94cd324
Compare
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I checked every item from my last review at ad78bbd. I also read the restack to 94cd324, which changes only #700's files under this commit. You fixed every item except the Cassandra job in the nightly matrix, where I agreed with your reply last time.
I ran each restore-kill-points-* job's commands as the workflow writes them. Every point passed, and every ACTIVE outcome had exact counts:
- SQLite: 5 of 5
- PostgreSQL: 8 of 8, with the
--vectorrun - MongoDB: 7 of 7
- Cassandra: 6 of 6
At the Cassandra job's restore_cassandra_before_progress point, the job killed the server with 10,000 items written and no progress recorded. The resume reached ACTIVE with 12,000 items, and the base and GSI counts were exact.
For each code item, the cell your reply names fails on its mutant and passes after the revert. That covers mutants R17 and R18. It also covers the Cassandra batch fence, the first-batch resume, the takeover bound, the held lock, the index tables and the in-use cell. wait_for_items_seen now waits 120 s.
The new code brings three Minor items, inline.
The restore handle in job/heartbeat.rs carries the in-flight clause from #700. When you bound that clause there, please make the same change here.
| } | ||
| if self | ||
| .count_errors | ||
| .fetch_update( |
There was a problem hiding this comment.
Minor: current stable deprecates fetch_update, and rustc 1.99.0 warns on this call. crates/storage-cassandra/src/migrations/backend.rs line 214 uses a compare-exchange loop instead, because try_update is not on 1.88. Please do the same here.
There was a problem hiding this comment.
Fixed. count_errors takes one error from its budget through a compare-exchange loop (load, checked_sub, compare_exchange until it lands or the budget is empty), as migrations/backend.rs does, with the reason in its comment.
| //! serialized item text (a batch closes at the first item that takes its | ||
| //! `item_data` text past the bound, so the batch term of a restore's memory | ||
| //! is bounded in bytes, not only in items). Before each batch's writes it | ||
| //! reads the coordinator's clock (the batch's write timestamp) and then |
There was a problem hiding this comment.
Minor: the stamp comes from the ExtendDB host's clock (data::version::now_micros). The deletes it must lose to are lightweight transactions, and the Cassandra coordinator sets their write time. Please name the host's clock here. Also say that the ExtendDB hosts and the Cassandra nodes need clocks that agree.
There was a problem hiding this comment.
Fixed. The module doc now says the stamp is read from the ExtendDB host's clock (data::version::now_micros, the clock the item versions already use), that the writes it must lose to are lightweight transactions and the deletes through the ACTIVE table, whose write times the Cassandra coordinator sets from its own clock, and so that the rule needs the ExtendDB hosts' clocks and the Cassandra nodes' clocks to agree, as the version scheme already needs the hosts' to.
There was a problem hiding this comment.
The new sentences name the host's clock and the clocks that must agree. The sentence at line 16 still says that the batch reads the coordinator's clock for its write timestamp. The code reads data::version::now_micros() before the claim check, so please change that sentence too.
| assert_eq!( | ||
| report.failed_prefixes_deleted, | ||
| std::slice::from_ref(&stale_failed) | ||
| // --- review round two: the count retry, the second maintenance read, the |
There was a problem hiding this comment.
Minor: this header, the one at line 18219 and the one at line 7840 of crates/storage-cassandra/tests/direct/backup_engine.rs say where the cells came from. Please name the cases they cover, in the code's own terms.
There was a problem hiding this comment.
Fixed. The three headers name their cases in the code's terms: in the engine tests, "the manifest total, remove-target's scope and lost claim, a target without tables, the registration retry" and "the count retry, the maintenance claim taken mid-cycle, the takeover bound, the first file resumed with upserts"; in direct/backup_engine.rs, "the batch's timestamp fence, the first batch resumed without progress, the readiness check's index tables, the in-use rule of a running restore".
b20e19e to
c461442
Compare
94cd324 to
9803ab7
Compare
|
The three Minors are fixed as above (head |
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I checked the three items from my last review at 9803ab7.
You fixed the compare-exchange loop and the three headers. The engine tests build with no warning on rustc 1.99.0, and backup:: passes 272 tests. The restore check carries parks, and the parks || true mutant fails #700's cell, because the backup and restore checks are one function. Both 55-minute jobs passed on this head, and one ran past the old 40-minute bound.
Two Minor items remain: one sentence of the clock doc, on its thread, and the new CI comment, inline. The stop-between-beats item I raised on #700 applies here too, through the shared function.
| timeout-minutes: 40 | ||
| # The job's own suites run close to 40 minutes; the backup store and job | ||
| # suites it ends with (their own servers, the restore cells among them) | ||
| # add about ten, so the bound leaves room for a slow runner. |
There was a problem hiding this comment.
Minor: this comment says that the backup store and job suites add about ten minutes. On this head they took 3 min 3 s in this job and 4 min 16 s in run-integration-sqlite. The two jobs took 39 min 49 s and 40 min 31 s.
Please use the measured figure, or remove the figure.
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
c461442 to
33a8e9c
Compare
9803ab7 to
0773f03
Compare
33a8e9c to
1c7cc29
Compare
Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and
backup purge· #700 the CreateBackup job · this PR: the restore job from the store, every backend, the cross-backend matrix andextenddb backup remove-target(the design's PR 9a and PR 9b as one PR, with PR 10'sremove-targetpulled forward) · next: PR 10backup sync.Closes #739 (part of #624).
Stacked on #700 (
feat/backup-job, head33a8e9c7), itself on #672'sa317b99conmainat6ab74d57, which carries #671 (merged asff6f74eb). The rebase after #671's merge conflicted only incrates/storage-sqlite/src/lib.rs's module list, where main's #704 addedmod password_hash_store;beside this PR'smod restore_engine;(both kept). (Rebased for this review round onto #700's current head;crates/storage/src/lib.rstook main's #776pub mod secret_cipher;beside this PR'spub mod restore_engine;.) This PR addsrestore_progressto the PostgreSQL data database (migration 7, not applicable on SQLite, MongoDB and Cassandra);docs/schema/postgres.txtcarries the table and the other three schema files are unchanged, each checked against a freshinit. It binds PR 7's store, spool and health and PR 6's claim, row and reconciler to the restore:check_catalog_row_for_restoreno longer refuses a store-backed backup; the request takesrequire_healthy_storeand the job permit and registers the target under the backup row's lock; the reconciler's cycle gains the dead-restore resume and therestore_progressand spool sweeps beside #700's backup duties; the backup commands' catalog refusals and named registrations are PR 7's, whichremove-targetuses. Since the last push main merged #335, #666 and #681 under the stack:restore_progressis migration 7 (#335 took 5, PR 6 took 6) and is declared not applicable in #681's MongoDB and Cassandra registries, and main's migration tests that count the ids now count it; the reconciler runs in #666's restartable worker set, which a lost serving registration stops while the jobs go on, so a reconciler whose own token has fired spawns no resume (a resume is a job); a restore job meets #700's drain in #666's shutdown path as a backup job does; andremove-targetreleases its serving registration on exit and carries its name on MongoDB and Cassandra as well as PostgreSQL. The rebase onto #700's head conflicted in twelve files:cmd_backup.rs(PR 7's registration release on this PR's split of the catalog opening from the store's), the Cassandra crate'sCargo.tomland backup-engine imports, the MongoDB backup live tests' imports and the GSI backfill worker's doc, the PostgreSQL migration table (migration 7 beside main's 5), and the design, differences, limits, usage, admin and upgrade documents, each resolved by keeping both sides' intent, main's text where #335 removed point-in-time recovery; Cassandra'sclaim_restoreno longer calls the catalog check #681 removed. The rebase onto the review rounds of #671, #672 and #700 conflicted in 23 files, resolved so that the three lower PRs' review changes stand beside this PR's: the threefail_restore_jobwrites of the resume arm (OWNER_DEAD,TARGET_NOT_EMPTYunder the claim,OTHER_STORE) now pass theowner_generationandheartbeat_atthe listing read, #671's fence, so a heartbeat that lands after the listing keeps the record its owner's (the engine'sFakeapplies the fence to its restore records); the restore request takestake_permit().await, #671's 30 s wait, before the manifest read; the target name is checked after the catalog row's state (and, on the legacy path, after the source resolves), #671's measured order, and before the backup's in-use rule and the manifest read (check_catalog_row_status, thencheck_restore_target_name, thencheck_catalog_row_for_restore;the_manifest_request_refuses_in_order_before_the_registrationreads one row read and no store read before a taken name's refusal), so a restore of a backup into its own CREATING target isTableInUseException, as the fidelity suite measured on DynamoDB, and the same backup into another nameBackupInUseException(a_backup_being_restored_is_in_use_until_its_target_is_activecarries the three cases, with another backup into the name being restored into;a_store_backed_restore_checks_the_target_name_after_the_row_and_before_the_storeis #700's cell adapted: the free name goes on to the manifest read). The first CI run of this head had the in-use rule before the name on the store-backed path and failedtest_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refusedon the three store-backed jobs; the in-use text is #671's service text naming the backup (backup_in_use_message), which the SQLite, PostgreSQL and Cassandra registration cells assert;HeartbeatHandlekeeps #700'sheartbeat_withinandrecheckwith this PR'sHeartbeatMetricandClaimCheck; the reconciler's state is theArc<Mutex<..>>#671'sserve.rsshares, its store check carries #672's defaulted-root rule, and the resume arm takes the configured store as theArc#700'sJobEnv::newneeds (BackupRuntime::configured_store_arc); SQLite's in-catalog restore slot (#671, by table id) islegacy_restore_slot_keybeside this PR'srestore_slot_keyon the target name, and the dead listing skips a record whose slot of either kind a task holds; every backend'slist_running_restore_jobs(#672) reads the restore record through this PR's wider row reader, andlist_dead_restore_jobskeeps #671's skip-and-log of a record this build cannot read;cmd_backup.rsopens the catalog forremove-targetahead of #672's store-requiring open, under the same signal listener;devtools/run-mongodb-testskeeps one store mechanism, this PR's--backup-store KIND(filesystem by default), with #700's--backup-store-root DIRnaming the root of that store (a named root is created and made owner-only, never cleared);tests/test_backup_job.pykeeps #700's MongoDB deployment, on which this PR's seven restore cells skip (_restore_cell), and the cell that seeds atarget_not_emptyrecord at the catalog edits SQLite's and PostgreSQL's as before;MAINTENANCEis back injob.rsfor the restore job's reason (the backup job, since #700's round, checks the claim before its row exists). #671's review item took revision item 28, #672's are 29 and 30 and #700's 31 and 32, so this PR's is 33 (every reference in the tree renumbered).What
Core (
crates/core). No new rule;validate_restorable_item,RestorableLimitsand the manifest reader (ExtenddbManifest::reader,reader_prehashed, the per-fileKeyRangecheck) are PR 4's and PR 8's, bound by the restore.DataFileDecoderandDataFileReaderexposebytes_decoded(), the uncompressed bytes decoded so far, which the load's chunk bound counts.Storage trait (
crates/storage). A newRestoreEnginetrait inStorageEngine's bound:bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied,finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched },count_restore_target,read_restore_progress(default[]),sweep_restore_progress(default 0);RESTORE_BATCH_ITEMS(10,000) andRESTORE_BATCH_BYTES(32 MiB), the SQLite and Cassandra batch bounds, live beside it.BackupCataloggainsclaim_restore(account_id, input, &RestoreRegistrationRow) -> (TableDescription, Box<dyn JobClaim>)(the registration creates the CREATING target and the RUNNING record under the backend's own generation),claim_restore_dead(key, stale_after, spool_host)(the restore analogue ofclaim_dead_backup; it writes a freshheartbeat_aton the database's clock in the same conditional write as the generation andspool_host, so a second taker finds the record live at once, where the backup takeover leaves the heartbeat for its FAILED-prefix clock),read_restore_jobandlist_restore_jobs;RestoreJobRowcarries the store,spool_host,file_index,items_applied,failure_reasonandcompleted_at.restore_input_from_manifestbuilds aCreateTableInputfrom the manifest'ssource_tableand applies the overrides, one implementation for both restore paths.restore_pause_point(name)is apub async fnundertest-hooks, a no-op without it, and the four backend pause-point names arepub consts beside it (RESTORE_SQLITE_MID_BATCH,RESTORE_MONGODB_AFTER_INSERT_MANY,RESTORE_CASSANDRA_BEFORE_PROGRESS,RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE), one spelling each: the engine'sKillPointholds the engine's five points alone.JOB_HEARTBEAT_INTERVAL's doc says the store-backed jobs run atmax(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)and the in-catalog paths and the maintenance claim at the constant.PostgreSQL (
crates/storage-postgres).restore_engine.rs:bulk_loadopens one data transaction per file, multi-row INSERTs of 500 base and index rows (or 4 MiB of item text) throughinsert_restore_rows, writes therestore_progressrow in the same transaction (a plain INSERT for a file applied from its start, an upsert on(table_id, file_index)for a file resumed past it, so a row a decoder panic left short is completed by the resume rather than failing on the row's key), and fails a duplicate on the base table; nothing in it conditions on the generation (the base table's key and the claim's liveness are its fence).finish_restoreflips the table and every index ACTIVE in one catalog transaction and then deletes the progress rows on the data pool (the pause pointrestore_postgres_before_progress_deletebetween the two); that delete's error is logged and swept by the reconciler, never the restore's failure, since the table is ACTIVE and the record DONE once the flip commits.claim_restorelocksbackup_index FOR UPDATE(the lockstart_backup_deletetakes; its comment says so);claim_restore_deadtakes the name's advisory lock and a stale RUNNING manifest record, refreshingheartbeat_atwith the generation andspool_host.sweep_restore_progressdeletes from a positive list (the table ids the progress rows name, read first, minus the RUNNING records read after and the caller's list), so a registration that commits between the two reads is never a candidate. Migration 7 addsrestore_progress(table_id, file_index, items_applied).remove_restore_target'sto_regclass('restore_progress')check and itsDELETEnow run on the data pool; on 09c881f both ran on the catalog pool, so with the table in a separate data database the PR 6 code found no table and deleted nothing.remove_restore_targetdrops the vector index's_ddb_vec_relation with the data and index tables (table_storage_relations); the record'sheartbeat_atis the registration transaction'snow(), refreshed by the engine's first beat before the reply.SQLite (
crates/storage-sqlite).bulk_loadwrites batches of at most 10,000 items or 32 MiB of serialized item text, each aBEGIN IMMEDIATEholding the base, index and vector rows and the record's(file_index, items_applied), conditioned on the generation (the pause pointrestore_sqlite_mid_batchbetween batches);claim_restoretakes the name's slot;claim_restore_deadtakes over a record whose slot no task holds, whatever its heartbeat (a restart frees every slot and ages every RUNNING record), refreshingheartbeat_at. The startup line for an aged RUNNING record names the reconciler's duty per kind (a store-backed restore is resumed; an in-catalog one is removed).remove_restore_targetno longer probes for arestore_progresstable (SQLite never has one; a store-backed restore's progress is on itstable_restoresrow,file_indexanditems_applied). The flip'sbackfilling = NULLon the vector rows is now observed by a fixture that sets it before the flip. A table with secondary indexes restores in more resident memory than the no-index figures the docs quote, and the excess is SQLite's own per-connection page cache for the index b-trees plus the allocator's arenas, not the restore's buffers, which stay bounded by the batch's 32 MiB (the module doc, the admin guide and the record state the shape beside the figures; a smaller batch for an indexed table was built, measured ineffective and not kept). A job's heartbeat queues behind the engine write lock like any writer, so the restore engine's and the heartbeat's module docs state what holds a beat, the batch times the bench record measured, and what a beat late by a period costs (the job stops as a lost claim and the reconciler resumes it from its last batch).MongoDB (
crates/storage-mongodb).bulk_loadwrites each batch of up to 1,000 base documents in one session transaction with the record's progress write: the transaction firstfind_one_and_updates the record (RUNNING under the claim's generation, settingrestore_job.file_index,restore_job.items_appliedto the file's running total andheartbeat_atto$$NOW) and ends the file as a lost claim when that write matches nothing, theninsert_manys the batch and commits, so a frozen owner whose record a takeover moved can write no batch after the takeover (the Blocking fence, [Lee 4468802456]: the fence is in the write). Because the progress commits atomically with each batch, a crash mid-file leaves no documents the record does not account for and a resume continues fromitems_appliedwith nothing re-applied;reapplyingis therefore vestigial on MongoDB (a transaction cannot doordered=falseduplicate suppression), andtarget_not_emptyarises only from the engine's count-above-total check, not from a mid-file crash. A duplicate key (E11000, the unique_id) is the backup's and fails the file. The pause pointrestore_mongodb_after_insert_manynow fires after each batch's transaction commits.finish_restorewrites DONE andrestore_backfill_pendingand returnsPending, the backfill worker flipping the table ACTIVE. A record carriesrestore_job.ddl_done, setfalseat the registration's insert andtrueonce the target's collections and index metadata exist; the reconciler'srestore_target_readyrefuses a record left unmarked (its owner died between the commit and the data DDL) and fails it rather than resuming into a target missing its indexes (the Blocking registration-order fix, [Lee 4468802443]).claim_restore_deadrefreshesheartbeat_atwith$$NOW, so a second taker right after the first is refused; the registration stamps the record's heartbeat from the server's clock inside its transaction. AnNsort key past Decimal128 fails the file with the write path's text.bulk_loadholds the GSI backfill test gate (test-hooksbuilds only) before the first insert of the restore's first file, as the in-catalog copy does before its$out, sorestore_worker_waits_for_base_copy_before_backfilling_indexesruns on the store path. The snapshot open that a store-backed CreateBackup takes reads the majority-committed snapshot, which can trail DescribeTable'slocalread, and a CreateBackup sent as soon as DescribeTable said ACTIVE was refusedTableNotFoundException(5 of 80 in a probe beside a write load; 0 of 80 on the in-catalog path); the open now waits, at most 2 s, while alocalread shows the table ACTIVE (0 of 150 after), pinned byan_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active, which writes the flip unjournaled so the snapshot trails it.Cassandra (
crates/storage-cassandra).bulk_loadwrites each item's base row and every GSI and LSI row in one logged batchUSING TIMESTAMPa reading of the coordinator's clock taken before the batch's claim check (put_restored_item_at, no lightweight transaction, eight in flight), in batches of at most 10,000 items or 32 MiB of serialized item text; the claim check before each batch is a lightweight transaction on the record (confirm_restore_claim), and the timestamp is the fence against a batch a stalled writer lands after a takeover: it carries a timestamp below every write and delete the successor made, so Cassandra keeps the successor's cells and tombstones and a key a client deleted stays deleted (the second round's thread item;test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletespauses a writer after its check, lets the successor write, flip ACTIVE and delete two items, and releases it); it records progress with a lightweight transaction per batch (the pause pointrestore_cassandra_before_progress);finish_restoreis one LWT (Active, neverPending; index rows created ACTIVE);count_restore_targetchecks the catalog first, as the other three do. The count is the only duplicate check. An unknown outcome of the progress lightweight transaction is re-read atLOCAL_SERIAL(applied, one retry, or the lost claim; a second unknown outcome fails the restore); a registration whose record was taken over during its DDL removes nothing and answers the lost claim; thetest-hooksfeature gainsfail_progress_writes(Landed,Lost).claim_restore_deadrefreshesrestore_heartbeat_atin its LWT and re-applies the backup'sactive_restoresentry for the target when a crash between the target's DDL and the registration left none, so the in-use rule holds through the resume.bulk_load's index rows carry the batch's timestamp, which is also the base row'sversion, so ADR-0027's rule holds (a later ordinary write's version is above it) and a re-applied restore batch keeps every GSI and LSI row under a later stamp (#424's in-place rewrite of an unchanged index row is what the legacy copy'sput_restored_itemrelies on; the store-backed load no longer goes through it).restore_target_readyreads each GSI's and LSI's table insystem_schema.tablesas well as the data table, since the registration creates them in separate statements after the data table (test_restore_target_ready_requires_every_index_tabledrops the GSI's table). A crash before the first batch's progress record leaves the batch's items with no progress; the engine resumes that record with the first file re-applied (upserts) when the target holds no more than the backup's total (test_a_first_batch_applied_without_progress_resumes_to_the_exact_count: the progress write's outcome unknown twice, the load failed with 3 of 7 items landed and no progress, the resume to 7 with both index-row counts exact). Two concurrent registrations of one backup leave one target: the registration's LWT (IF backup_status = 'AVAILABLE' AND active_restores = null) is the fence, linearized on the backup's partition; the in-use pre-read is advisory. New index-row tests: a re-appliedbulk_loadbatch, whose GSI and LSI rows keep their count and carry their base rows' versions as write times, and the two-registration race.Engine (
crates/engine/src/backup/).restore_table_from_backupand the reconciler take&Arc<S>(the manifest branch and the resume spawn tasks).restore.rsandrestore/{spool,download,load}.rs: the request verifies the sidecar and manifest (each read bounded, 4 KiB and 16 MiB; an object over its bound is aValidationExceptionnaming a corrupt prefix), the vector gate, the overrides, the key schema and the target, registers undertimeout(RESTORE_REGISTRATION_TIMEOUT)(60 s, every backend), and runs the job (download with a 15-minute transport retry counted from the job's first attempt, per-filebulk_load, the count,finish_restore). The load's chunk bound (LOAD_CHUNK_BYTES, 32 MiB) counts the decoded line's bytes, and the feeder checks the claim at every chunk boundary beside the shutdown token; a decoder that dies fails the file..verifiedmarkers hold the file's sha256 and a stale entry (a record that was another host's) is removed before use; a job whose record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the sweep. The spool sweep lists the spool before the records and keeps the entry of a restore registered between the two reads; the spooled manifest and sidecar are written.partial→ sync → rename and a spooled pair that fails verification yields to the store's; afinish_restoreerror is re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); the resumed job's first beat is the takeover's return, taken by the reconciler besideclaim_restore_dead;BackupStoreErrors{get}on everygetthe store failed; the restored table'sItemCountandTableSizeBytesare 0 at ACTIVE, as the service reports them (measured: 0 and 0 on 15 restored tables, one of them ten minutes after ACTIVE; [Lee 4468802685]), the statistics pass rewriting them later where the backend has one; a registration that fails after the request's 60 s bound is logged with the ARN, the target and the account. The job sends its first heartbeat right afterclaim_restorereturns and before the reply, andclaimed_atis that beat's return, so a slow registration does not end the job at its first check and the record'sheartbeat_atis fresh at the reply (on PostgreSQLnow()is the transaction's start, so a registration stalled on thebackup_indexlock or the DDL would otherwise commit a stale stamp); a first beat that finds the claim lost fails the record ("the claim was lost before the job started") and answersInternalServerError; the maintenance refusal recordsRestoreJobs{failed}.remove-targetrefuses or removes a RUNNING record's target bylist_dead_restore_jobson the database's clock (no host clock), and the purge's wait reads the backend's heartbeat listing the same way; the reconciler takes nothing over once the shutdown began, spawns no resume once its own token has fired while the jobs' token has not (the server stopping its workers while it holds no serving registration;a_stopping_reconciler_takes_nothing_over), and applies thetarget_not_emptyguard under a maintenance claim (a dead record with no progress over a target that holds more items than the backup's row counts is failedtarget_not_emptyand left forremove-target, notOWNER_DEADand removed); the lost-claim stop line carries the heartbeat timer's reason, which the timer also logs at WARN; the spool entry is removed on a failed spool document write and on a resume that fails before it runs; a DONE write whose answer was lost and whose record re-reads DONE ends withcomplete (DONE); the twotarget_not_emptylines printextenddb backup remove-target --account <account> <table>. The reconciler checks the store first in its cycle (a store whose first check has not recorded isnot yet checked, not unhealthy), resumes a dead store-backed restore from its progress underclaim_restore_dead, fails one of another store (OTHER_STORE) or no progress into a target holding more items than the backup's total (target_not_empty; at or below the total the first file is re-applied withreapplying,exceeds_backup_totalreading the backup row'sitem_count), re-reads the maintenance claim after the takeover and leaves the record unresumed when a command took the claim during the cycle, bounds its own takeovers of one record (RESTORE_TAKEOVERS, 5, counted per table id inReconcilerState; the sixth takeover fails the recordTAKEOVERS_EXHAUSTEDand removes its target asOWNER_DEADdoes, so a record whose every resume ends RUNNING again is not retaken every stale window for as long as the server runs; per instance, reset by a restart), counts a maintenance-held dead record in a field of its own, and sweepsrestore_progressrows and spool entries. The final count is attempted three times with a wait between; three failures stop the job with the record RUNNING and no FAILED write (a_count_that_fails_three_times_leaves_the_record_running). MetricsRestoreJobs,RestoreJobDuration,RestoreBytesDownloaded(each data-file download attempt's bytes);KillPointgains five restore variants (RestoreAfterDownload,RestoreAfterFileCommit,RestoreMongodbBeforeWorkerFlip,RestoreBetweenFailedAndRemoval,RestoreFencedOut; the engine's list; the engine's points logjob reached kill point <name>and the backends'restore job reached kill point <name>).crates/server/Cargo.toml'sbackup-test-hooksaddsextenddb-storage/test-hooksso the backends' pause points compile in.App (
crates/app).extenddb backup remove-target --account <id> <table>(PR 10's command, pulled forward): under the maintenance claim it removes a FAILED restore's CREATING target and record (target_not_emptyincluded, the state a catalog restored from a dump leaves when it names a restore that finished and was written to since), a RUNNING restore's target whose owner is dead, or a legacy record's target, throughremove_restore_target; it refuses a live RUNNING restore, a DONE restore, a table no record names, and, on PostgreSQL, a CREATING restore target whose owner's session still holds the target's advisory lock, reported as its own case (the record is unchanged after the backend refused the removal: "its owner's lock is still held ... wait for the owner to stop, or stop it, and run the command again";remove_target_reports_a_held_owner_lock_as_its_own_case). It needs no store and opens the catalog without creating the default root or an identity file, prints the claim it took and one line naming what it removed, and logs the removal underextenddb::audit; on PostgreSQL, MongoDB and Cassandra its serving registration is namedextenddb backup remove-target(PR 7's per-backend name) and released when the command exits, whatever the outcome, and a catalog under migration or behind the binary ends it with exit 6 or 3, aslistandpurgedo.backup purgeon SQLite fails a dead store-backed restore and removes its target rather than resuming it inside the command, since its cycle runs as the claim's holder. The command's logic and its tests are the engine's (maintenance::remove_target).Harness and CI. The 24 restore cells of
tests/test_backup_restore_fidelity.pyrun on a server with a store, andtest_create_backup_of_a_vector_table_is_refusedistest_vector_table_backup_refused_in_catalog_and_restored_from_store, since it asserts the round trip withSearchVectorson a server with a store;tests/test_auth_permissions.py's restore cell andtests/rust's backup suite run against the store;tests/test_backup_job.pygains the tamper, SIGTERM, store-outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments (the multi-part cell enablesallow_multipart_table_keyson its own deployment and compares the restored item set with the source'sScan; the hand-built prefixes rewrite the two DynamoDB export manifests too, so they are whole backups) and, on SQLite, thebackup remove-targetcell: a finished restore, with one item written to the target since, is seeded at the catalog back into thetarget_not_emptystate (a RUNNING manifest record with no progress over a CREATING target that holds more items than the backup, the shape a catalog restored from a dump leaves), the restarted server's reconciler fails ittarget_not_emptyandDeleteTable,DeleteBackupand a second restore are refused, the command removes the target and its record with the documented output, and the restarted server answersResourceNotFoundExceptionfor the name,DeleteBackupsucceeds and a new restore runs;tests/test_mongodb_backup_restore.pygains the vector-manifest refusal, the 35-digit Decimal128 failure and the 38-digit intact cells, and its hand-built prefixes rewrite the two export manifests as the job suite's do;devtools/run-mongodb-testsserves a filesystem store and exports its root, its log andEXTENDDB_TEST_BACKUP_STORE, and its--backup-store noneserves no store for the legacy LSI fallback cell (the one wire cell of the in-catalog restore path on MongoDB, which no CI job runs; the test guide and the cell's docstring give the command); the runner starts from an empty store root, since a reused--outputdirectory'sextenddb-store.jsonnames the previous run's catalog. The PostgreSQL CI job starts moto, so the store-outage cell runs there as on the SQLite S3 job, whose workflow comments now say that cell is the one S3 restore the job runs. The matrix isdevtools/backup-restore-matrixand.github/workflows/matrix-nightly.yml, which also runs every restore kill point of every backend in fourrestore-kill-points-<backend>jobs (the round-two section): one target directory and four sequential builds, the artifact action pinned, one run at a time,requirements-matrix.txtforpymongoandcassandra-driver, anapt-get install postgresql-clientstep for the hostpsqlthe script's PostgreSQL seed and teardown call, and a step that checkspsql --versionand the two drivers' import before the cells; the SQLite Rust-integration job's comment inintegration.ymlsays the backup tests restore the store-backed backups (it had said they expect the restore's refusal); its duplicate-key prefix writer chmods its files0600ascopy_prefixdoes, and its docstring's exit codes are the script's (1 on an assertion failure or a setup error); the nightly'srust-cachestep saves only on main, as #680 made every other step, and its four actions are pinned to the commits main pins (#752). The job suite gains a SQLite purge cell (a stopped server's RUNNING store-backed restore record with progress:purge --confirmfails it and removes the target, and notaken over by this instanceline is logged), and the store-config suite aremove-targetcell that leaves no default root and no identity file; the PostgreSQL exit-code cell coversremove-target. The engine'sa_file_replaced_between_the_two_passes_fails_in_the_loadruns at a 1 s heartbeat: its hashing pass over both files runs on the test's one runtime thread before the first load's claim check, and under load it outlasted two 50 ms periods.Docs. The differences document (vector, multi-part-key, how-restore-runs with its timings labelled as the in-catalog restore's beside the store-backed figures, concurrent-restore lock with Cassandra's two-registration guarantee, restore-interruption resume with the
target_not_emptystate, its remedy and the measured first-reach outcomes, a restored table'sTableSizeBytes, a new Decimal128 row), the limits document (concurrent-restore spool and batch bound and Cassandra COUNT, backup contents), the usage guide and upgrade manual (the restore lands this release; migrations 1 to 6 breaking and 7 declared so on PostgreSQL for want of the additive-declaration job, not applicable on the other backends; the rollback paragraph;backup remove-targetfor thetarget_not_emptystate), the admin guide (the spool's contents and sweep, the restore's memory terms and measured figures with the shape and the input order they were measured on, the two-GSI and the unordered figures beside them,backup listlists no restore, thebackup remove-targetcommand with its output and refusals, the owner's liveness on the database's clock, and the[backup.store]paragraph no longer saying a store-backed restore is a later release's), the developer test guide and CONTRIBUTING (the job suite, the MongoDB cells' variables and the runner's--backup-store none, the moto cells' credentials, the matrix and the nightly),docs/design/15-backup-restore-bench-pr9.md(the restore memory and duration record, with the two-GSI and the store-backed 60,000-item figures) and design revision item 33 with the body sentences it corrects. With the rebase: the admin guide'sstale_job_after_secsrow (a dead store-backed restore is resumed) and purge paragraph (under the serve lock such a restore is failed and its target removed), the upgrade manual's SQLite startup sentence (a store-backed restore is resumed) and itsrestore_progresssentence (migration 7, declared breaking on PostgreSQL, not applicable on SQLite, MongoDB and Cassandra), design 4(e) and the reconciler's module doc (what a lost serving registration does, citingregistration.rs's module doc: the reconciler stops with the server's workers and spawns no resume once stopping, the running jobs go on until they end or the server stops; #755), CONTRIBUTING's worked example (migration 8, 7 beingrestore_progress), the PR 9a load paragraph (500 rows or 4 MiB), and the differences document's consistency row (the MongoDB snapshot open's wait).Why
#396 section 5.3 and the PR 9a and 9b sections. PR 9a and 9b ship as one PR by the maintainer's decision (item 30). Decisions, each in the module doc of the code it concerns and in item 30:
bulk_loadis aRestoreEnginemethod, not aStorageEngineone; progress is recorded insidebulk_loadin the write that applied the items, so there is norecord_progressand PostgreSQL's progress row (in the data database, no job record) is fenced by the base table's key and the claim's liveness rather than the generation;reapplyingmakes MongoDB's duplicate suppression explicit; the manifest registration locksbackup_index(the lockstart_backup_deletetakes) so a store-backed restore and aDeleteBackupof its backup order wholly;RESTORE_REGISTRATION_TIMEOUTis 60 s on every backend; a filesystem store is verified in place (twogetpasses), an S3 store spooled; the restore takeover refreshes the heartbeat where the backup takeover leaves it; the memory bounds are in bytes of text (the chunk in line bytes, the SQLite and Cassandra batch in serialized item bytes); the download retry bound is the job's; the matrix seed stands in forbackup sync --adopt-storeuntil PR 10.extenddb backup remove-targetships here rather than in PR 10 because thetarget_not_emptyrule, written for a catalog restored from a dump, has as its common trigger on MongoDB and Cassandra an ordinary crash before the first progress record (both record progress after the write), and without the command this release had no way to remove such a target. The spool sweep lists the spool before the records, as the progress sweep lists its rows first, so neither removes what a registration between its two reads created. Afinish_restoreerror is an unknown outcome re-read through the claim, as the status writes are, since on PostgreSQL the reachable case (the progress delete after the flip's commit) is a finished restore.Testing done
After the rebase onto the review rounds of #671, #672 and #700, on this head:
cargo clippy --all-targetswithextenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks,extenddb-engine/backup-test-hooksunder-D warnings, clean;cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup::262 passed (the cells adjusted to #671's order and text named above); theextenddb-core(797),extenddb-storage(233),extenddb-storage-sqlite(276, therestore_job_testsamong them),extenddb-server(307),extenddb-app(47) andextenddb-config(24) lib suites;backup_restore(56) andrestore_job(12) on PostgreSQL at127.0.0.1:55432;backup_engine::live_tests::withtest-hookson the MongoDB replica set at127.0.0.1:27117, 42 passed;direct_integration -- backup_engine::withtest-hookson a fresh Cassandra 4.1 node, 60 passed (test_the_catalog_listings_are_scoped_and_orderedfailed once on a keyspace the node had not agreed on yet, "Keyspace ... doesn't exist" inside the migration runner, and passed when rerun). The wire suites (tests/test_backup_job.py's restore cells among them), the kill points and the memory runs below are from the previous heads and were not repeated; CI runs the wire suites.The first bullets, down to the restore kill points' SIGTERM cells, are measured on this tree after the move onto
mainat e8ca7df, except the parts a bullet scopes to03baa321(the tree before the move: #700's head42c4f6d8plus this commit); the bullets after them are scoped "Before the rebase" and are the tree on #700's head before the stack's two rebases (09c881f4), where "new" is against that head.cargo fmt --all -- --checkclean;cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings, the same--features extenddb-engine/backup-test-hooks, and-p extenddb-storage-cassandraand-p extenddb-storage-mongodbwith--all-targets --features test-hooks, all clean;cargo +1.88.0 check --locked --workspace --all-targetsclean with no warning; on03baa321,cargo doc --no-depsof the engine, app and storage crates warned in none of this PR's files and the licence notices (default, MongoDB, Cassandra, dev) were current (the move changes noCargo.lockorCargo.toml: main changed neither between03baa321ande8ca7df3).cargo test --locked --workspace --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb --exclude extenddb-backup-storewith PostgreSQL required: bin 3, app 47, auth 274, cache 27, config 23, core 796 (1 ignored), engine 389, server 305, storage 220, PostgreSQL lib 78 and every test binary green (backup_restore50,restore_job11, themigrations_*binaries,table_snapshot89 with 2 ignored among them), SQLite 267 (1 ignored) withrestore_job_resume1 andbackup_job_panic1.-p extenddb-backup-storewith moto: lib 37, conformance 123. MongoDB: lib 169 (62 ignored),interpreter_selftests18,migrations_fresh8,migrations_guards7,migrations_holders10,migrations_locks11,migrations_released5,pushdown_parity20,--lib -- --ignored --test-threads=1 live_62, and thetest-hooksfence test 1. Cassandra under the shared container's lock: lib 115,migrations22,migrations_stopped8,--features test-hooks --test direct_integration219 (three parts of 72, 74 and 73 tests named--exact; in the first part's first run two cells timed out on the node, one with "no answer within 60 s" on a keyspace's baseline file and one with "Connection timeout after 10s", and passed when rerun),ttl_integration26 passed, 1 ignored andtest_ttl_enable_rejects_asynchronous_gsifailing as it does onmain.not_applicable, its "later binary" rows move to id 8, its lock test counts eight steps and its released-catalog test eight pending; MongoDB's and Cassandra's registry tests list 7 as not applicable; Cassandra's backfill cell allows 7 pending beside 6.03baa321: the engine'sbackup::tests20 times undertaskset -c 0-1 nice -n 5beside one busy loop: 20 of 20 green (214 tests, 33 to 34 s each). Mutant:a_stopping_reconciler_takes_nothing_over(new) fails withmanifest_restores_resumed: ["MusicRestored"]when the guard on the reconciler's own token is disabled.main's head (release builds ofe8ca7df3) on SQLite and PostgreSQL:migrate --listshows 1 to 7 (7not applicableon SQLite,catalog, dataon PostgreSQL);serve,backup listandbackup remove-targeton the catalog behind exit 3 naming 6 and 7;migrate --yestakes it from 5 to 7 (PostgreSQL:restore_progressin the data database and 7 in both ledgers; SQLite: 7 recordednot_applicable);verifyHEALTHY;remove-targetof an unknown table exits 1, leaves nobackups/on SQLite and noextenddb backup%session on PostgreSQL;servestarts,extenddb stopreturns in 0.26 s, andmain'sserverefuses the migrated catalog with exit 4. On03baa321, on MongoDB and Cassandra (a replica set of its own; the shared node),list, the dry run andremove-targetleave noextenddb_serversentry,migrate --yesright after them does not wait, under a planted migration lock each exits 6 naming the holder, and on a catalogmigrate --forget 6put behind each exits 3 with themigrate --yesadvice.d27c9bb9first.test.yml:cargo test --locked --workspace --exclude extenddb-backup-storewith no service environment, as on its runner, 59 binaries and 3,264 tests passed (the Cassandra contact point pointed at a closed port, so the Cassandra suites found no node and returned as on the runner), the 1.88 row'scheckabove, and the store crate with moto.integration-mongodb.yml:mongodb-rust(devtools/run-mongodb-tests --rust-live -- --rust-integrationwithfilesystemon themongodb-test-hooksbuild): records 6, the migration binaries,tests/rust514 withrestore_worker_waits_for_base_copy_before_backfilling_indexesamong them on the store path, live 19;mongodb-pytest(-- --pytest --comprehensive --parallel): 1638 passed, 51 skipped, 8 xfailed, 4 xpassed, comprehensive 502,test_abac.pyandtest_rfc0003_concurrency.pyamong the passes;mongodb-catalog: the schema check,check-mongodb-catalog-lock29,-auth16,-released53,-older-binary28, each "all checks passed".integration.yml:run-integration'scheck-postgres-catalog-lock16 andcheck-postgres-older-binary26, and its backup suites step with moto (17 passed, 15 skipped);run-integration-sqlite: both lock checks, the older-binary check, the schema check,run-tests --pytest --comprehensive --parallel --filter "not import_export"(1666 passed, 24 skipped, 1 xfailed; comprehensive 502) and the by-name suites (26 passed, 6 skipped);run-rust-integration-sqlite'stests/rust514;run-integration-sqlite-s3: the S3 store on moto,--filter "backup"65 passed, 6 skipped, the by-name suites 28 passed, 4 skipped.integration-cassandra.yml:cassandra-storage(the Cassandra rows above),cassandra-rusttests/rust514,cassandra-catalog-checks(the schema check;check-cassandra-catalog-lock42 andcheck-cassandra-older-binary33 against v0.1.14 built from its tag, each on a node of its own),cassandra-two-nodesmigrations22.docs/schema/*.txtagainst a freshinit: SQLite 48 lines, PostgreSQL 416, Cassandra 320, identical; MongoDB's check passed. Not run:cassandra-pytestandcassandra-authorizer(this rebase changes no Cassandra request or auth path; the pytest leg is host-limited on this machine), the PostgreSQL pytest and Rust-integration jobs (their defaultextenddb_catalogexists on the shared PostgreSQL), and the nightly matrix.03baa321: the restore kill points' SIGTERM cell on PostgreSQL and SQLite (24,000 items,backup-test-hooksbuilds): the restore stopped by the server's shutdown inside feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's job drain, no FAILED written, 16 workers drained, the record RUNNING across the restart and the restore resumed to ACTIVE with the base table's and the GSI's counts exact.cargo fmt --all -- --checkclean;cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings, the same--features extenddb-engine/backup-test-hooks, and-p extenddb-storage-cassandra --features test-hooks --all-targetsall clean;cargo check --locked -p extenddb-storage --libwith and withouttest-hooksclean;cargo check --locked --workspace --all-targetsclean.extenddb-engine336 passed (62 new test attributes against 09c881f, one removed, 61 net; 7 of them added last: the maintenance-heldtarget_not_emptyguard, no takeover once the shutdown began, the loss raised inside a step named by the next check, the first beat refreshing the registration's heartbeat before the reply, the request stopping at the 60 s bound with the task going on, a resume that fails before it runs removing its spool entry, a spool document write failure removing the entry; the 28 before them: the slow registration, the stopped timer before each write and before a download attempt, the DONE write'sNotMatched, the lost-claim entry left,Pending, the job-wide retry bound, a decoder that dies, the same-host spool resume, a marker with another digest, a crash's spool states, a file replaced between the passes (the replacement matching the original in compressed length, uncompressed length and item count and failing on the sha256 alone), a file over one chunk by bytes, the feeder's claim check, an unhealthy or unchecked store with a known identity, SSE on both paths, a restore registered between the sweep's two reads, a failed DONE write re-read, the resume's first beat, a truncated spooled manifest, a resume whose store documents fail verification,Progress::from_recordson each backend's shape, a record gone after the takeover, a registration that fails after the request stopped waiting, the bounded document reads,remove_targetover a FAILED, a dead RUNNING, a live RUNNING, a DONE and a legacy record);extenddb-core728,extenddb-server246,extenddb-app44.extenddb-storage200.extenddb-storage-postgreswith the PG env: 353 passed over every binary before the last two cells were added, and thenrestore_job.rs10 passed (the takeover test asserting the refreshed heartbeat, the progress delete failing after the flip, the resumed file's upsert) with every other binary clean up to the slowtable_snapshot.rs, which the run's timeout cut short without a failure.extenddb-storage-sqlite: lib and tests 245 passed (restore_job_tests10,a_batch_closes_at_the_byte_bound_before_the_count_capnew, the flip test settingbackfillingbefore the flip).extenddb-storage-mongodb: unit 182 (the lib binary's 144 rerun after the last change; no MongoDB code changed in it),--lib -- --ignored live_43 in the earlier run (37 backuplive_cells,live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_callsnew), and thetest-hooksfence testa_takeover_during_bulk_load_fences_the_file_index_write(--features test-hooks -- --ignored --test-threads=1) 1; the 43live_cells pass under the shared replica set with--test-threads=1: run in parallel, three legacy-path copy cells can time out waiting for a GSI-backfill worker within the 90 s deadline (an environment limit, not a logic defect).extenddb-storage-cassandra: lib 78,--features test-hooks --test direct_integration213 passed in the full run before the last three cells were added (tests/direct/backup_engine.rs48 tests, five new: theactive_restoresre-apply, the delete between the registration's pre-read and its fence, the composite-key resume with index rows and an empty file, the byte bound, the lost claim between two batches), and the three new cells and two changed ones ran twice and passed (tests/direct/backup_engine.rs: two concurrent registrations leaving one target, the re-applied batch keeping every index row over a composite GSI+LSI target;tests/direct/index.rs: an unchanged item's re-put keeping its GSI and LSI rows, the GSI worker keeping the row), the full binary not rerun end to end on the shared container;--test ttl_integration25 passed, 1 ignored,test_ttl_enable_rejects_asynchronous_gsithe known pre-existing failure andtest_backfill_cursor_resumepassing alone.finish_restore's error as a failure, the resume's instant at its entry, the spooled documents written in place, the fallback to the store removed, no metric on a refusedget, the unlogged registration failure, theCancelledarm recording a worker error, a zero duration, the read bound removed, the in-place pass not hashing again,from_recordsignoring the MongoDB and PostgreSQL shapes, the sidecar and ARN re-checks dropped, the gone record as an error, the compressed size as the table's, and four onremove_target: the DONE refusal skipped, every owner live, the claim dropped unreleased, a zero window not read as every owner dead), 21 across the storage trait and the four backends (five added last: the Cassandra index-row skip removed, the GSI worker bypassing the shared writer, PostgreSQL's progress delete propagated, the plain INSERT for a resumed file, the SQLite flip'sbackfillingclause dropped).--features postgres|sqlite|mongodb|cassandra,-p extenddb), the four hooks binaries (<backend>,extenddb-app/backup-test-hooks) and themongodb-test-hooksbuild built on the final tree;CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targetsclean.docs/schema/postgres.txtandsqlite.txtmatch a freshinitof the release builds byte for byte (no schema changed since);cargo test -p extenddb-storage-postgres --test doc_version_literalspasses with the record in the tree.tests/test_backup_restore_fidelity.py test_auth_permissions.py(the 24 restore cells and the vector round trip withSearchVectors) — SQLite 100 passed, PostgreSQL 100 passed, no skip (the multi-part-key cell moved to the job suite); andtests/test_backup_store_config.py test_backup_job.pyby name withEXTENDDB_BINARYand the moto variables on both builds — SQLite 25 passed, 3 skipped (the PostgreSQL-only store cells skipped;test_backup_job.py14 of 14, theremove-targetcell included), PostgreSQL 16 passed, 12 skipped (the SQLite-only store cells and the SQLite-onlyremove-targetcell skipped;test_backup_job.py13 of 14), the outage cell against moto on both. The multi-part-key cell runs on both (the restored item set equals the source'sScan; on SQLite the preview's item paths keep one item per first HASH key value in the source itself, so the set is 5 of the 60 items written). The MongoDB cells throughdevtools/run-mongodb-tests: the CI way (filesystem)tests/test_mongodb_backup_restore.py5 passed, 1 skipped (the legacy LSI fallback cell skipped), and with--backup-store none3 passed, 3 skipped (the legacy LSI fallback cell and the two throughput cells passed; the three store-backed cells skipped).docs/design/15-backup-restore-bench-pr9.md; the admin guide'smax_concurrent_jobsrow and the design's section 7 quote it): release binaries, filesystem store,VmHWMover idle from a fresh start, 24 items of 27.7 MB (the worst shape) restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite) and 392 and 427 MiB underMALLOC_ARENA_MAX=1; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key and no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16.4 MiB on PostgreSQL in 17.5 s; at 3M rows SQLite 109 to 116 MiB key-ordered and 171 to 174 MiB unordered, the finalCOUNT(*)'s share 22 to 28 MiB at both sizes, bounded by the counting connection'scache_size(32 MiB) rather than growing with the table, as the record had said from one measurement. The same 200-byte items with two full-projection GSIs, measured on the pushed head's and on the final binaries: SQLite 143, 225 and 143 MiB over idle at 100k in three runs (10.2 to 13.9 s to ACTIVE) and 165 and 172 MiB at 300k (51.7 and 51.9 s), PostgreSQL 29.3 and 22.9 MiB at 100k (9.9 and 6.2 s) and 30.8 and 33.5 MiB at 300k (19.7 and 21.4 s); the excess on SQLite is the backend's per-connection page cache for the index b-trees (cache_size, 32 MiB) plus the allocator's arenas, not the restore's buffers, and a smaller batch for indexed tables was measured ineffective (91.7 against 92.7 MiB) and not kept; every GSI'sScancount equalled the item count after ACTIVE. The 60,000-item shape the differences document's in-catalog timings were measured on, restored from the store on the final binaries: response 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB), ACTIVE after 0.67 s, 1.85 s and 3.74 s (the MongoDB job's DONE at 1.8 s, the backfill worker's flip after), 9.3, 38.3 and 30.1 MiB over idle; the differences document's in-catalog timings for that shape now carry their label. The record carries the terms the code bounds and the sizing sentence (about 450 MiB per running restore of large items under one arena, timesmax_concurrent_jobs).devtools/backup-restore-matrix, one invocation per cell, on the final binaries: the twelve plain cells (PostgreSQL, SQLite, MongoDB sources into all four targets) PASSED (items, description, GSI and LSI queries match the source; 10 to 44 s each, the Cassandra targets the slowest), with the seed's timestamps as the job writes them; the four--duplicate-keycells and the eight--vectorcells PASSED on the final binaries too (the--duplicate-keycells into Cassandra by the count alone,restored 12 items, the manifest lists 13; the--vectorcells into PostgreSQL and SQLite withSearchVectorsanswering, into MongoDB and Cassandra refused before any table). The nightly workflow has had no run:workflow_dispatchneeds the workflow file on the default branch, so it cannot be dispatched until this PR merges, and its first run (the schedule's, or a dispatch after the merge) is the proof of the runner image. The workflow installspostgresql-clientand checkspsql --versionandpython3 -c "import pymongo, cassandra"before the cells, since the matrix script seeds and tears down its PostgreSQL deployments through the host'spsql, which no other workflow uses.kill-points.py, posted with this PR; every ACTIVE outcome asserts the base table'sScancount and the GSIby-g'sScancount both equal the item count), each killed or signalled at the point, restarted and asserted, on the final hooks binaries where the row says so and on the pushed head's otherwise (nothing on those paths changed between the two):restore_after_downloadrestore_after_file_commitrestore_sqlite_mid_batch(24,000 items, one file)restore_postgres_before_progress_deleterestore_mongodb_after_insert_many, first reachtarget_not_empty, the target left CREATING (DeleteTableResourceInUseException);extenddb backup remove-target --account 123456789012 <table>printed the four lines and removed it; DescribeTableResourceNotFoundException, DeleteBackup allowedrestore_mongodb_after_insert_many, second reach (--second-reach, 24,000 items in three files)insert_many; resumed from file 1 underreapplying, ACTIVE 24,000, GSI 24,000restore_mongodb_before_worker_fliprestore_cassandra_before_progress, first reach (12,000 items, one file)target_not_empty, the target left CREATING;remove-targetremoved it, DescribeTableResourceNotFoundException, DeleteBackup allowedrestore_cassandra_before_progress, second reachrestore_between_failed_and_removal(a tampered file)--vector(a vector index on the source) gone with its data table and_ddb_vec_relation (rerun on the rebuilt binary; the pushed head's build left the relation)restore_fenced_outpg_terminate_backend(the job paused atrestore_after_file_commit)t, the last beat 1.1 s old); released 20 s later, the job stopped at its next check as a lost claim naming the terminated session, the record RUNNING with its(0, 8228)progress row; the same server's reconciler took it over 45.6 s after the termination (46.7 s after the last beat) and finished it: 24,000, GSI 24,000 (rerun on the rebuilt binary)sigtermEvery cell above ran on the pushed head's hooks binaries (4c35319); the cells marked "rerun on the rebuilt binary" ran again on the binaries built after the last corrections (the first beat before the reply, the Cassandra progress re-read and registration fence, the PostgreSQL vector-relation drop), the
pg_terminate_backendcell as reworked (the shipped cell slept 2 s, terminated and asserted the final count alone, which a 2.6 s restore passed without the case running). The Cassandra points ran against a freshly recreated single-node container withreplication_factor = 1(noGCInspector,StatusLoggeror "No nodes available" line in its log after the runs). Two MongoDB cells (restore_between_failed_and_removal,sigterm) failed once at setup withResourceNotFoundExceptionon the fixture's first write when a second deployment on the same replica set dropped the fixed databases under them, and passed when rerun alone.Review round (2026-11-04): changes addressing Lee's review
Measured on this stack's head after rebase onto #700's current head. Counts here are re-measured on this head; the counts above are carried from the pre-rebase runs unless noted.
Code changes:
insert_manyand its(file_index, items_applied)record now commit in one session transaction whosefind_one_and_updatefence requires the record RUNNING under the claim's generation, so a frozen owner writes nothing after a takeover, and a mid-file crash resumes fromitems_appliedto ACTIVE rather thantarget_not_empty.reapplyingis vestigial on MongoDB (a transaction cannot doordered=falseduplicate suppression).restore_job.ddl_done, set true only after the target's collections and index metadata exist; the reconciler's newrestore_target_readyrefuses and fails an unmarked record instead of resuming into a target missing its GSIs/LSIs.restore_heartbeat_atrefreshedIFRUNNING under the generation) before its upserts. Round two completes the fence: the batch's writes carry a timestamp read before that check (the round-two section).restore_target_ready, not retaken every stale window; the reconciler re-checks the maintenance claim afterclaim_restore_dead(Minor, 4468802609).OTHER_STORE. The threefail_restore_jobwrites are conditioned on theowner_generationandheartbeat_atthe listing read (feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671's signature), so a record whose owner heartbeated after the listing keeps its record.CREATINGdescription; a different backup into the same name isTableInUseException.ExtenddbManifest::reader, notreader_prehashed), so a file changed in the spool between runs is caught.remove-target(Minors): the command prints the documented terms "store-backed"/"in-catalog" (4468802662); on PostgreSQL it takes the target name's advisory lock for the removal via a command-onlyremove_restore_target_exclusive, so a frozen owner keeps its target (4468802600); the account-scopedfind, the maintenance-claim-lost pre-removal check, and the count-above-total path are pinned by new engine cells (4468802541, 4468802553, 4468802532).ItemCount/TableSizeBytes0 at ACTIVE (Minor, 4468802685).Tests run this round (re-measured on this head):
cargo test -p extenddb-engine backup::withextenddb-engine/backup-test-hooks: 241 passed (236 carried plus five new: the count-above-total cell, the account-scopedremove-targetcell, the claim-lostremove-targetcell, the dead-restore-with-no-tables cell, and the registration-retry cell).cargo test -p extenddb-storage-postgres --test restore_jobagainst PostgreSQL: 12 passed (11 carried plus the new sweep-reads-order cell).cargo test -p extenddb-storage-sqlite --test restore_job_resumeand--test backup_job_panic: 1 + 1 passed; SQLite--lib restore57 passed.--features test-hooks, replica set atmongodb://127.0.0.1:27117): the rewritten fence, lost-claim and resume cells pluslive_finish_restore_hands_over_and_the_worker_flips_active,live_bulk_load_refuses_a_long_sort_key_and_keeps_a_long_number,live_restore_returns_creating_and_the_detached_copy_finishes— all passed.!=→<(4468802532), theremove-targetaccount comparison (4468802541), theclaim.lost()pre-removal check (4468802553), therestore_target_readyguard (4468802481), the in-flight registration dedup (4468802521), PostgreSQL'ssweep_restore_progresstwo-read order (4468802562), and the reconciler's spool/record sweep order (4468802548).cargo fmt --allclean;cargo clippywith-D warningson the engine, storage, PostgreSQL, SQLite, MongoDB (test-hooks) and Cassandra (test-hooks) crates clean;cargo check --workspace --testsclean;shellcheckondevtools/run-mongodb-testsreports only the pre-existing SC2174 the MongoDB runner already carried.Not run this round (no environment / CI only): the Cassandra live suite (shared container; the fence is compile-checked with
test-hooks), the nightly matrix, and the newmatrix-nightly.ymlkill-points job (itsdevtools/backup-kill-points.pyis committed by #700 and arrives by rebase).CI wiring:
devtools/run-mongodb-testsgained atest-hooksstep running the restore fence cells..github/workflows/matrix-nightly.ymlgained arestore-kill-pointsjob that builds the SQLite binary withbackup-test-hooksand runs the four SQLite restore points throughdevtools/backup-kill-points.py(its call did not match the script's CLI; round two corrects it and adds the other backends' jobs).legacy_create_only(it cannot take a store-backed backup), so serving a filesystem store in its Rust job would not give store-backed-restore coverage; the nightly matrix (SQLite/PostgreSQL/MongoDB source → Cassandra target via adopt-store) is that coverage. See the reply.Review round two (2026-11-05): Lee's second review
Measured on this head (the round-one head plus these changes). Counts marked re-measured were counted on this head; the rest are carried from the sections above.
Code changes:
devtools/backup-kill-points.py --backend sqlite --point <p>, which the script's parser refuses (exit 2). The call is nowrestore <backend> <point> --bin target/release/extenddb --work kill-point-logs/<backend>-<point>; the jobs' exact calls were parsed locally (--help, and runs with a bogus--binthat get past the parser intoinit), and two SQLite points (restore_after_file_commit,restore_sqlite_mid_batch) ran end to end with the job's exact call on abackup-test-hooksrelease build: both resumed and ACTIVE with 12,000 items, base and GSI exact.restore-kill-points-{sqlite,postgres,mongodb,cassandra}, each building its backend's binary withextenddb-app/backup-test-hooksand starting its backend as the matrix job does (the PostgreSQL job installspostgresql-clientfor the script'spsql; the MongoDB job starts the replica set and installspymongo; the Cassandra job configuresPasswordAuthenticator, installscassandra-driver, and builds a plain SQLite binary as the--source-binwhose backup the Cassandra target restores). Each runs every point of the script'sRESTORE_POINTSfor its backend, plusrestore_mongodb_after_insert_many --second-reach(24,000 items),restore_cassandra_before_progress --second-reachandrestore_between_failed_and_removal --vectoron PostgreSQL; logs are uploaded on failure. The script's first-reach assertion for the two no-progress points changed fromtarget_not_emptyto ACTIVE with the exact count (MongoDB since round one, Cassandra since this round), and its--remove-targetflag is gone (thetarget_not_emptystate is seeded bytests/test_backup_job.py's cell).a_count_that_fails_three_times_leaves_the_record_running: the fake's count fails three times, the job stops with the record RUNNING at(file 1, 2 items), no FAILED or DONE write, the loaded target kept,RestoreJobs{failed}recorded. Mutant: with the first count error failing the restore at once, the cell fails (count_calls1, not 3); passes after the revert.a_claim_taken_during_the_cycle_leaves_the_taken_over_record_unresumed: the fake's newtakeover_hooksets the claim held insideclaim_restore_dead, after the cycle's first read; the record is taken over (new generation) but not resumed (nobulk_load, RUNNING). Mutant: with the second read replaced by a constant "not held", the cell fails (manifest_restores_resumed: ["MusicRestored"]); passes after the revert.put_restored_item_atwrites each item's base row and index rows in one logged batchUSING TIMESTAMPa reading taken before the batch'sconfirm_restore_claim; a stalled writer's batch lands below the successor's writes and deletes.test_a_batch_landed_after_a_takeover_loses_to_the_successors_writes_and_deletes(direct_integration,test-hooks, the newpause_after_restore_claim_checkhook): the owner paused after its check, the record taken over, the successor loads, flips ACTIVE and deletes two items (GSI and LSI rows gone), the owner released: its progress write reports the lost claim, the two items stay deleted with their index rows gone, the survivors carry the successor's versions. Mutant: with the timestamp read after the pause (when the writes land), the cell fails (4 items, not 2: the deleted items resurrected); passes after the revert.item_count(exceeds_backup_total): above ittarget_not_empty(the dump-restored catalog), at or below it the resume re-applies the first file withreapplyingset (the resume already set it for the first file it loads). Engine cella_half_applied_first_file_with_no_progress_is_resumed_with_upserts(one of two items landed, no progress: resumed,reapplyingtrue, DONE with 2); storage celltest_a_first_batch_applied_without_progress_resumes_to_the_exact_count(3 of 7 landed with no progress through two lost progress writes, resumed to 7 with 4 GSI and 3 LSI rows). Thetarget_not_emptycells now seed three items against a backup of two. The differences row, design 5.3 step 7 and the PR 9a load paragraph, the admin guide and the kill-point description are updated.ReconcilerState.restore_takeoverscounts this instance's takeovers per table id; the sixth (RESTORE_TAKEOVERS= 5) fails the recordowner dead: taken over 5 times by one instance without the restore endingand removes the target as forOWNER_DEAD. In-process rather than a column: atakeoverscolumn would touch every backend's record and PostgreSQL's catalogtable_restores(not this PR's migration 7, a data-database table), and the count per instance is what the thread allowed; stated in the module doc, the design and the differences row (a restart starts the count over). Cellthe_takeover_after_the_fifth_fails_a_record_that_never_ends: the count fails on every attempt, five cycles take the record over and leave it RUNNING, the sixth fails and removes it with the reason in the FAILED write. Mutant: with the bound check disabled, the sixth cycle resumes again and the cell fails; passes after the revert.claim_restoredoc (thread): no longer "in one transaction": the catalog rows commit first, the data tables (MongoDB: collections and index metadata) after, in the orderrestore_target_ready's doc gives, and the resume checksrestore_target_ready.test_a_running_store_backed_restore_keeps_its_backup_in_use_until_active(direct_integration): with a store-backed restore RUNNING,backup_in_usenames the target and a secondclaim_restoreisBackupInUsewith the service's text, twice; with theactive_restoresmap entry deleted (the registration's window before the entry lands) the target rows alone keep it in use; afterfinish_restorethe backup is free. The engine's order (the in-use rule before the override validation) is the engine's cells'; this cell is the Cassandra input to it.wait_for_items_seendeadline: 120 s, with the reason in the comment (the decoder's hashing pass and first chunk on the test's one runtime thread under a loaded CI machine).(Lee 4468...)citations in the diff replaced with descriptions in the code's terms (engine tests and reconciler, Cassandrabackup_catalog.rsandbackup_engine.rs, PostgreSQLbackup_catalog.rsandtests/restore_job.rs);grep 4468over the tree finds none.remove-targetre-reads the record after the backend refuses the removal; an unchanged record means the table is still its CREATING target and the refusal was the lock, reported as "the restore into T (account A) from is a CREATING restore target (table id ) whose owner's lock is still held: its owner's session is alive though its heartbeat is stale; wait for the owner to stop, or stop it, and run the command again"; a changed or gone record is "not a restore target: it is no longer a CREATING restore target with table id ".remove_target_reports_a_held_owner_lock_as_its_own_case(the fake'sowner_lock_held), the legacy cell's text adjusted; the admin guide lists the line.restore_target_readychecks each GSI's and LSI's table insystem_schema.tablesbeside the data table;test_restore_target_ready_requires_every_index_tabledrops the GSI's table and seesfalse.RestoreTableFromBackupruns" row says the in-flight list is per server process, so a retry waits only when it reaches the server that holds the registration, and what a retry on another server sees.tests/test_backup_job.py'sremove-targetcell expected "its manifest restore from", where the command prints "its store-backed restore from" since round one; the cell and the admin guide's sample output now carry the printed text (the cell ran green on the hooks build).Tests run this round (re-measured on this head):
cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup::267 passed (--listcount 267; the five new cells above plus the held-lock cell);cargo test -p extenddb-storage-cassandra --features test-hooks --test direct_integration -- backup_engine::on the shared Cassandra 4.1 node 64 passed (60 carried plus the four new cells; the fence cell run alone three more times, once with the mutant);cargo test -p extenddb-storage-postgres --test restore_jobagainst PostgreSQL at127.0.0.1:5543212 passed;tests/test_backup_job.py -k remove_target_removes_a_target_not_emptyon the SQLitebackup-test-hooksrelease build 1 passed;devtools/backup-kill-points.py restore sqlite restore_after_file_commitandrestore_sqlite_mid_batchwith the job's exact call on that build, both ACTIVE with 12,000 items, base and GSI exact;cargo check --workspace --testsclean;cargo fmt --all -- --checkclean;cargo +1.97.0 clippy --all-targets -- -D warningson the engine, storage, Cassandra (test-hooks) and PostgreSQL crates withextenddb-engine/backup-test-hooks, and on the Cassandra crate withouttest-hooks, clean;shellcheckon the four jobs' run steps clean. Mutants verified (mutate, run the one cell, revert): the count error failing at once, the second maintenance read skipped, the takeover bound disabled, the Cassandra batch timestamp read after the check. Not run: the MongoDB live suite (no MongoDB code changed), the nightly workflow (it needs the workflow file on the default branch), and the PostgreSQL, MongoDB and Cassandra kill-point jobs' points (their calls were parsed locally; the SQLite points ran).Review round three (2026-11-05): Lee's third review
Three inline Minors, the review body's request to bound the restore handle's in-flight clause with #700's, and one CI change.
ClaimCheckcarriesparks(JobClaim::heartbeat_parks_before_its_bound, feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's new provided method) and passes it toin_flight_confirms, so the restore handle's check is feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's: a beat in flight confirms the claim for as long as it waits on SQLite, where the beat parks on this process's lock, and within one period of its start on PostgreSQL, MongoDB and Cassandra. feat(backup): CreateBackup job, worker and reconciler duties [backup/restore 8] #700's cella_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parksruns on this head too.fetch_update(Minor):count_errorsin the engine tests' fake is a compare-exchange loop, ascrates/storage-cassandra/src/migrations/backend.rsdoes, with the reason (try_updateis not on 1.88) in its comment.restore_engine.rs's module doc names the stamp's clock (the ExtendDB host's,data::version::now_micros) and the write times it must lose to (lightweight transactions and the deletes through the ACTIVE table, stamped by the Cassandra coordinator), so the rule needs the ExtendDB hosts' and the Cassandra nodes' clocks to agree.crates/storage-cassandra/tests/direct/backup_engine.rsname the cases their cells cover (the manifest total,remove-target's scope and lost claim, a target without tables, the registration retry; the count retry, the maintenance claim taken mid-cycle, the takeover bound, the first file resumed with upserts; the batch's timestamp fence, the first batch resumed without progress, the readiness check's index tables, the in-use rule of a running restore) instead of the review round they came from.run-integrationandrun-integration-sqlitewere cancelled at their 40-minutetimeout-minutesafter the backup store and job suites step, which this stack adds to the end of each job; both are now 55 minutes, with the reason in a comment. The other jobs keep 40.Tests run this round on this head:
cargo test -p extenddb-engine --lib --features backup-test-hooks -- backup::272 passed;extenddb-storage-sqlitewrite_lock_tests15;cargo +1.97.0 clippy --all-targetswith the Cassandra and MongoDBtest-hooksand the engine'sbackup-test-hooksfeatures,-D warnings, clean;cargo check --workspace --testsclean.Not changed here
PR 10's
backup sync --adopt-store,--move-store,--forget-backupandmigrate-legacy; PR 11's CI finalization and documentation, with the MongoDB and Cassandra 1M restores and the vector cell still to be measured; PR 13; PR 14's drop of the in-catalog format. The matrix seeds the target'sbackup_indexrow by hand in place ofbackup sync --adopt-store. MongoDB'sinsert_manyunit stays 1,000 documents with no byte bound of its own (the data file bounds it:bulk_loadis per file). SQLite's restore memory for a table with indexes stays SQLite's page cache (cache_size) above the no-index figure; a smaller batch for indexed tables was measured ineffective and not kept.Breaking changes
kind = "disabled"is no longer needed to keep a deployment able to restore the backups it takes. Migration 7 addsrestore_progressto the PostgreSQL data database: runextenddb migrate --yesbefore starting the new binary. An S3 store needsspool_pathon PostgreSQL, MongoDB and Cassandra for restores. The restore of a store-backed backup is asynchronous on every backend with a 60 s registration bound, as on the service.DCO
Signed-off-by: Scott Robinson robinnsc@amazon.com