Repository navigation
Conversation
8f310f1 to
09c881f
Compare
62d32f0 to
1234179
Compare
09c881f to
ec4eb8d
Compare
1234179 to
b6f683d
Compare
210165a to
19e4459
Compare
There was a problem hiding this comment.
I built this head with extenddb-app/backup-test-hooks for SQLite, PostgreSQL and MongoDB and ran it in containers with a filesystem store. The runs included SIGKILL after a job's first data file and after its manifest, SIGTERM during jobs and a 160 s freeze. Two servers also raced for the same dead rows, backup purge ran beside clients creating backups, and I swapped the store root during one job. I sent the backup requests to DynamoDB in us-east-1, to this head on the three backends and to #672's head. Mutants of the job, the reconciler, the item validator and the backends ran against the PR's tests. CI passes on this head (37 checks in 10 workflow runs), and every new Rust test runs there except the MongoDB takeover test.
The CreateBackup response, and DescribeBackup and ListBackups of a CREATING backup, match the service key for key on all three backends. A SIGTERM during a job failed its row and deleted its objects within 10 ms, and the server exited 0.15 s after the signal. Two servers racing for a killed server's 12 rows resolved each row once, on MongoDB and on PostgreSQL. On PostgreSQL, client timeouts of 1 to 43 ms on 1,200 creates left no CREATING row and no FAILED row with objects.
A PostgreSQL backup whose job ran while four writers put and deleted keys contained exactly the 2,500 items the table held before the request. A backup #672's head made on a SQLite file still restores after the upgrade to this head. DeleteTable and UpdateTable during a job, the item validator's limits and log, the SQLite pool raise and the job bound behave as the description says.
Two Blocking items and 28 more are inline. One item comes from #672:
- The store check's 5 s bound covers its catalog reads, so a slow catalog marks the store unhealthy for one cycle (60 s by default). With this PR the server also refuses CreateBackup while the store is unhealthy. I saw the store unhealthy for 2.4 s on MongoDB at a host load of 45.
Once you fix the inline items, I will approve.
| let store_backed = runtime.configured_store().is_some() | ||
| && !runtime.legacy_create_only() | ||
| && STORE_BACKED_CREATE; | ||
| let store_backed = runtime.configured_store().is_some() && !runtime.legacy_create_only(); |
There was a problem hiding this comment.
Blocking: this makes CreateBackup store-backed whenever a store is configured, and file-backed SQLite has one by default, but RestoreTableFromBackup refuses those backups until #740. On SQLite by default, and on PostgreSQL and MongoDB with a store, a restore right after CreateBackup answered ValidationException Backup <arn> is in a backup store; this release restores in-catalog backups only in all 8 tries. DynamoDB accepted that restore 0.6 s after CreateBackup, and #672's head, like main, restored all 100 items. Breaking changes lists this. Please merge this PR with #740, with no main commit between them, or disable the store-backed create until #740.
| ticks.tick().await; | ||
| loop { | ||
| ticks.tick().await; | ||
| let why = match tokio::time::timeout(period, claim.heartbeat()).await { |
There was a problem hiding this comment.
Blocking: on SQLite this beat is an UPDATE under the engine write lock, so a write holding that lock past one period ends a running backup. Beside UpdateTable building a GSI on a 400,000-item table, one backup failed here (the heartbeat did not return within 15 s). Another backup failed at the claim re-check in job.rs line 515 (the claim re-check did not return within one heartbeat period), after its uploads, and the reconciler then deleted its 5 objects. The in-catalog create that main runs, here with kind = "disabled", finished AVAILABLE in 98.2 s beside the GSI build. Please bound the beat and the re-check only once they hold the lock. Add a test whose writer holds the lock past two periods.
There was a problem hiding this comment.
The 60,000-item race from my comment now ends AVAILABLE. With 180,000 items the backup stopped twice at the check before data file 3 or 4. Both times a GSI build on the 400,000-item table held the write lock. The reconciler then failed each row and deleted its objects, and the in-catalog create of the same table ended AVAILABLE. Please let confirms_the_claim pass while the beat waits for the write lock, and add a test that puts a data file during the hold.
There was a problem hiding this comment.
Fixed. HeartbeatHandle::confirms_the_claim has a third clause: a beat that is in flight and has not failed confirms the claim, however old the last confirmed beat is. The timer and recheck set an in-flight flag around every heartbeat_within call and clear it through a drop guard on the way out (a beat that returns Lost, fails or outlives its bound raises the loss first, and the check reads the loss before the flight; a timer that stops mid-beat clears the flag, so the two-period clause remains for it). The module doc states why it is safe: on SQLite the beat parks only on this process's write lock, one process serves the file (backup purge refuses while a server runs on it) and the reconciler's liveness test is the in-process slot, so a parked beat cannot coincide with a takeover; on PostgreSQL, MongoDB and Cassandra no process-local lock sits in front of the beat and the bound covers the whole call, so a beat is in flight for at most one period after its tick and the clause admits no put the two-period rule refuses. Tests: write_lock_tests::a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it (storage-sqlite) backs up three items at a data_file_target_bytes of one byte through a store whose first put takes four periods, takes the write lock as the row appears and holds it until the second data file's put has started, so that put's check runs with the first beat parked and no beat confirmed since the insert (asserted: heartbeat_at unchanged under the hold); after the release the backup is AVAILABLE with its three data files and the manifest, and heartbeat_at has moved. review_round_3::a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lost (engine) drives the handle with a fake claim whose heartbeat_within parks outside the bound: the check stays true at two and a half and at twelve and a half periods, turns false the moment the released beat returns Lost, and after a parked beat released with a confirmation and the timer stopped, turns false two and a half periods later. Mutant: with the in-flight clause removed, the SQLite cell fails (one put started under the hold, the row left CREATING) and the engine cell fails at its first assertion; both pass after the revert.
| Ok(meta) => meta, | ||
| Err(why) => { | ||
| stop.store(true, Ordering::Release); | ||
| let _ = encoder.await; |
There was a problem hiding this comment.
This awaits the encoder before line 562 cancels failed, and an encoder in items.blocking_recv() returns only when the feeder reads the stream's next item. A test that holds the stream between items and fails a mid-file put found the row CREATING and the job running 3 s later. With failed.cancel() before this await and the one at line 495, as the module doc describes, the row was FAILED and the job ended. With that order, 156 backup::tests passed. Please make that change. Add the mid-file case to a_put_failure_ends_the_job_without_waiting_for_the_streams_next_item, which passes either way because MemStore::put fails after reading the whole body.
| } | ||
| kill_point(KillPoint::BeforeManifest, p.shutdown).await; | ||
| let (name, bytes) = docs.manifest_document(); | ||
| bytes_uploaded += put_document(p, name, bytes).await?; |
There was a problem hiding this comment.
I paused a job after its first data file and swapped the root for one whose extenddb-store.json names another catalog. The reconciler marked the store unhealthy, yet the job wrote its documents and this manifest into that root, and the row went AVAILABLE. With the original root back, the store stayed unhealthy after a restart, and CreateBackup and DeleteBackup answered ValidationException The filesystem backup store configured on this server is unhealthy (manifest probe failed); the operation is refused until the store's check passes, so no request clears it. Please check is_healthy() and that the identity equals env.identity before each put. When either check fails, fail the row with the store's reason and no store call.
| { | ||
| Ok(false) => Ok(Started::Job(Box::new(job))), | ||
| Ok(true) => { | ||
| fail_before_start(&*job.claim, &job.arn, MAINTENANCE).await; |
There was a problem hiding this comment.
A create refused here leaves a FAILED maintenance row with the insert's heartbeat, and backup purge counts such a row as a live job for stale_job_after_secs. With four clients creating backups and stale_job_after_secs = 45, all 8 purge attempts exited 1 after 90 to 93 s. Each attempt named such rows or rows inserted seconds before. Those runs left 16,054 FAILED maintenance rows, kept 7 days by default, and with them a restarted server's first reconciler cycle lasted over 4 minutes. Please refuse before the insert as the in-catalog path does, or write the refused row so the wait ignores it.
| /// The in-memory pool is not sized by this: it has one connection by nature. | ||
| pub(crate) fn pool_size_for_jobs(pool_size: u32, max_concurrent_jobs: usize) -> u32 { | ||
| let wanted = max_concurrent_jobs.min(BACKUP_SERVER_SLOTS_MAX); | ||
| if backup_server_slot_count(pool_size) >= wanted { |
There was a problem hiding this comment.
With >= changed to >, both pool tests pass, because the pools they size exactly are even. pool_size_for_jobs(11, 4) would then return 10, a pool lowered by one. Please add that case.
| [features] | ||
| # Kill-point pauses in the backup job (`EXTENDDB_BACKUP_PAUSE`), for the | ||
| # binary-level crash tests; never enabled in a release build. | ||
| backup-test-hooks = [] |
There was a problem hiding this comment.
No CI job builds or lints this feature, and kill-points.sh is outside the repository (design line 467). The pause code in job.rs lines 607 to 640 and the description's kill-point results depend on hand runs that CI cannot repeat. Clippy on 1.97.0 and a 1.88.0 check with the feature pass on this head today. Please add the feature to the clippy job. Commit the script under devtools/.
| | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace. Take database-level backups for off-host copies until the backup store program ships | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE` on all four backends. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw it `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB reports `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: another region's resolves (not measured on the service), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment could create but whose name no backup ARN can carry is refused with `ValidationException` where the service backs up any table it can create: a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"), and every table under a `server.region` outside `[a-z0-9-]` (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot | | ||
| | Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `<account id>/<table name>/<backup id>/`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` is the table's count as last counted while `CREATING`, which is 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact count once `AVAILABLE`). The backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: another region's resolves (not measured on the service), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment could create but whose name no backup ARN can carry is refused with `ValidationException` where the service backs up any table it can create: a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"), and every table under a `server.region` outside `[a-z0-9-]` (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot | |
There was a problem hiding this comment.
This says the backup is stricter than the service's one-minute window. In one DynamoDB run, the restored backup contained every write acknowledged before I sent CreateBackup. It also contained the 18 writes acknowledged in the 0.1 s up to BackupCreationDateTime. It contained none of the 11,754 acknowledged in the next 75 s, so the service took the table at an instant inside the request. Please replace the clause with that measurement, or cite the source of the one-minute window.
| let end = match writer::write_data_files(&pipeline, items, key_schema, limits, tracker).await { | ||
| Ok(files) => { | ||
| item_count = files.item_count; | ||
| source_table.item_count = i64::try_from(files.item_count).unwrap_or(i64::MAX); |
There was a problem hiding this comment.
The AVAILABLE write sets ItemCount from the stream and leaves TableSizeBytes at the statistics figure. On SQLite the backup of a fresh 1,000-item table reports 1000 and 0, as on MongoDB, and on PostgreSQL it reports 1000 and 1359872. DynamoDB reported 0 and 0 in every fresh-table backup I measured, and #672's head on SQLite, like main, reported 100 and 5989 for 100 items. Please set TableSizeBytes from the stream as well, or keep both at the snapshot's figures. Also name TableSizeBytes in the differences row.
| | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace. Take database-level backups for off-host copies until the backup store program ships | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE` on all four backends. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw it `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB reports `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: another region's resolves (not measured on the service), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment could create but whose name no backup ARN can carry is refused with `ValidationException` where the service backs up any table it can create: a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"), and every table under a `server.region` outside `[a-z0-9-]` (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot | | ||
| | Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `<account id>/<table name>/<backup id>/`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` is the table's count as last counted while `CREATING`, which is 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact count once `AVAILABLE`). The backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: another region's resolves (not measured on the service), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment could create but whose name no backup ARN can carry is refused with `ValidationException` where the service backs up any table it can create: a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"), and every table under a `server.region` outside `[a-z0-9-]` (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot | |
There was a problem hiding this comment.
While a backup is CREATING, which now lasts the whole upload, DeleteBackup and RestoreTableFromBackup of it answer BackupInUseException Backup is being created: <arn> on all three backends. All 16 DynamoDB backups I measured were AVAILABLE 12 to 90 ms after CreateBackup answered, and #672's head, like main, answered both requests at once. Please add both refusals and how long CREATING lasts to this row.
19e4459 to
b6d1d34
Compare
|
Thanks for the runs; every item is addressed on the new head, each test you asked for was run against the mutation you named and fails there.
|
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I checked every item from my last review at b6d1d34. I ran the round-one reproductions again on SQLite, PostgreSQL, MongoDB and Cassandra, with the store in a container where an item needed one. I applied the round-one mutants and ran the tests your reply names.
29 of the 30 inline items are fixed. Each reproduction now gives the right answer, and each named test fails on its mutant and passes after the revert.
The engine write lock item is partly fixed, and my reply is on its thread. The round-one race now ends AVAILABLE. A larger backup still stops at the check before a data file while a GSI build holds the lock. The reconciler then deletes its objects.
The new text and code bring two Minor items, both inline.
Once you fix the write lock case, I will approve.
| | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace. Take database-level backups for off-host copies until the backup store program ships | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE` on all four backends. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw it `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB reports `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in '<arn>'" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s); a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed writer of a later release refuses it before it writes, and the in-catalog format of this release carries it as earlier releases did. Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the check runs after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | | ||
| | Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `<account id>/<table name>/<backup id>/`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | ||
| | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '<v>' at 'backupName' failed to satisfy constraint: <c>" texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: <name>" (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` and `TableSizeBytes` in `SourceTableDetails` are the table's figures as last counted by the stats pass while `CREATING`, which is 0 and 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact figures once `AVAILABLE`: the items written and the sum of their DynamoDB sizes; the service reported 0 and 0 in every fresh-table backup measured). `CREATING` lasts the whole upload (the durations below), and while it lasts `DeleteBackup` and `RestoreTableFromBackup` of the backup answer `BackupInUseException` "Backup is being created: <arn>" on all three backends; the 16 service backups measured were `AVAILABLE` 12 to 90 ms after `CreateBackup` answered, so a client that deletes or restores a backup right after creating it polls `DescribeBackup` first here. The backup holds every write acknowledged before the request was received and none acknowledged after. Measured on the service: a restored backup contained every write acknowledged before `CreateBackup` was sent and the 18 writes acknowledged in the 0.1 s up to `BackupCreationDateTime`, and none of the 11,754 acknowledged in the 75 s that followed, so the service takes the table at an instant inside the request as well. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:<region>:<account>:table/<name>`) is accepted on all four backends: an ARN whose region is not the server's (`server.region`, compared as written, so `US-EAST-1` differs from `us-east-1`) is `ValidationException` "1 validation error detected: Invalid AWS region in '<arn>'" (the service's answer, measured on DynamoDB twice; an earlier build resolved it to the server region's table), another account's is refused with the service's answers (see "A resource ARN in another account"); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: <name>" for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). Without a store, and on Cassandra, a table the deployment can create is backed up whatever its name, as the service does (measured on DynamoDB: a table named `...` answered `ContinuousBackupsUnavailableException` "Backups are being enabled for the table: .... Please retry later" 13.2 s after `CreateTable` and was backed up at 33.7 s): the in-catalog format writes no store prefix. With a store, a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots fits no store prefix, so the store-backed create refuses it with `ValidationException` before it opens a snapshot or writes a row ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"). Every table under a `server.region` outside `[a-z0-9-]` is refused (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot. The permit is the last check: a request that finds every `[backup] max_concurrent_jobs` slot held (default 4) waits up to 30 s for one and is then `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later"), where the service accepted six `CreateBackup` calls sent together (measured on DynamoDB, two runs) | |
There was a problem hiding this comment.
Minor: this says the backup holds no write acknowledged after the server received the request. A PutItem acknowledged 1.01 s after a CreateBackup that waited for a job permit was in that backup, in 2 of 2 runs. The snapshot opens after the permit, so please describe the backup as the table at the snapshot's open.
There was a problem hiding this comment.
Fixed. The row now describes the backup as the table at its snapshot's open, which is inside the request but after the request's checks and after any wait for a job permit (up to 30 s), so a write acknowledged during that wait is in the backup, with your measurement (1.01 s, 2 of 2 runs) beside the service's; BackupCreationDateTime is the open's millisecond. Design 5.1 step 2 said the same as the row and is edited the same way; the admin guide's [backup.store] sentence names the open's place after the permit wait.
| // and no row is inserted for a server that is stopping. | ||
| let snapshot = tokio::select! { | ||
| biased; | ||
| () = env.shutdown.cancelled() => return Err(shutting_down(table_name)), |
There was a problem hiding this comment.
Minor: on SIGTERM, a CreateBackup that waited in this open got a closed connection 8.2 s after the client sent it. The listener's 5 s drain closes the connection before drain_then_release cancels the job token, so this ServiceUnavailable never reaches that client. Please say in the description that a client gets this answer only after a registration loss.
There was a problem hiding this comment.
Fixed. The job's module doc, the comment at the select and the shutting_down doc say that the ServiceUnavailable reaches a client only when the jobs' token is cancelled while the listener still serves, which is a registration loss (BackupRuntime::cancel_jobs from the restartable worker set's stop), and that on SIGTERM the listener's 5 s drain closes the connection before drain_then_release cancels the token, with your 8.2 s; the refusal's effect there is the row not written. The same statement is in the differences document's backup-creation row, the admin guide's Stopping paragraph and design 5.1 step 2; the PR body's What and Breaking-changes entries are corrected.
There was a problem hiding this comment.
The What and Breaking changes entries now say this. The Engine paragraph still says the request answers ServiceUnavailable when an open waits behind an ACCESS EXCLUSIVE lock during the drain. That is the SIGTERM case, where the client saw the connection close after 8.2 s, so please qualify that sentence too.
There was a problem hiding this comment.
Fixed. The Engine paragraph now says the ServiceUnavailable reaches a client only when the token is cancelled while the listener still serves, a registration loss, and that on SIGTERM the listener's 5 s drain closes the connection first.
b6d1d34 to
41bf0db
Compare
|
Pushed |
9765cdd to
fb12f6b
Compare
41bf0db to
0339924
Compare
fb12f6b to
23411fa
Compare
0339924 to
0450e01
Compare
|
The write-lock case is fixed as above, with the data-file-under-the-hold test and the engine test; the in-flight clause is the only code change this round, beside the two tests and the texts of the two Minor items. Engine |
23411fa to
eb15f90
Compare
0450e01 to
141f911
Compare
|
Pushed |
141f911 to
b20e19e
Compare
|
Pushed |
There was a problem hiding this comment.
I checked the items from my last review at 141f911. I also read the diff to b20e19e, which changes only the job-bound cell and one design line.
You fixed the write-lock case. I ran three SQLite races, each a 180,000-item backup beside a GSI build on the 400,000-item table. All three ended AVAILABLE, and the reconciler kept every object. Without the in-flight clause, both new cells fail.
You also fixed the snapshot row. The code docs, the differences row and the admin guide now state the ServiceUnavailable case. My reply on its thread covers the one sentence left in the description.
The in-flight clause brings one Major item, inline. On PostgreSQL, MongoDB and Cassandra it admits puts for a short time after a process stop, and the module doc says it never does. The other new items are Minor.
At 141f911 the job-bound cell failed in three CI jobs, because the second request waits for the slot. Your rewrite passed in mongodb-backup-store and run-integration-sqlite-s3. The other jobs that run it were still running when I checked.
Once you bound the clause on those three backends, I will approve.
| pub(crate) fn confirms_the_claim(&self) -> bool { | ||
| !self.lost.is_cancelled() | ||
| && (self.last_confirmed.borrow().elapsed() <= 2 * self.period | ||
| || self.in_flight.load(Ordering::Acquire)) |
There was a problem hiding this comment.
Major: the module doc says that on PostgreSQL, MongoDB and Cassandra this clause admits no put that the two-period rule refuses. The clause reads only the flag, and a beat in flight when the process stops keeps it set after the resume.
I stopped a test process for 1.5 s with kill -STOP while a bounded beat was in flight. After kill -CONT, checks passed for 64 µs and 85 µs in two runs, until the timer raised the loss. Without the clause, no check passed.
On MongoDB and Cassandra another owner can claim the row during the stop. Please apply the clause there only while the beat is inside its bound.
There was a problem hiding this comment.
Fixed: the in-flight clause is bounded where the beat does not park. JobClaim gains a provided heartbeat_parks_before_its_bound (default false; SQLite's claim answers true, its beat waiting behind the engine write lock of its own process, outside heartbeat_within's bound), the handle records the beat's start (in_flight: Mutex<Option<Instant>> instead of the flag), and in_flight_confirms is parks || since.elapsed() <= period: a beat in flight confirms the claim for as long as it waits on SQLite, and within one period of its start on PostgreSQL, MongoDB and Cassandra, so a process stopped mid-beat and resumed there passes no check in the window before its timer raises the loss (your 64 and 85 µs). review_round_3::a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks runs the same parked beat under both declarations: within the bound both confirm; past it only the parking one; and when the beat returns Lost the loss is read first. With the clause unbounded again (parks || true) its parks=false half fails. The module doc now states the rule per backend, and design 5.1 step 4, item 32 and the admin guide's heartbeat_interval_secs row say the same. #740's ClaimCheck takes the same parks so the restore handle's clause is bounded there too.
There was a problem hiding this comment.
The Engine paragraph of the PR description still says that on the other backends the clause admits no put that the two-period rule refuses. The module doc no longer says this, so please update that sentence too.
| /// ends AVAILABLE with its three data files and its manifest. With the | ||
| /// in-flight clause removed the second data file's check refuses the | ||
| /// put ("no heartbeat confirmed the claim recently enough for data file | ||
| /// 2"), the row ends FAILED and the prefix is deleted, which is what the |
There was a problem hiding this comment.
Minor: without the in-flight clause, this cell fails with one put started and the row left CREATING, because no reconciler runs here. The doc says the row ends FAILED and its prefix is empty, so please describe what the cell sees.
There was a problem hiding this comment.
Fixed. The cell's doc now says what the cell sees without the clause: one put started and the row left CREATING, since no reconciler runs there (on a server the reconciler fails the row and deletes its prefix).
| ) | ||
|
|
||
| ACCOUNT = "123456789012" | ||
| # The job's bound in the LimitExceeded cell: items of this count and padding |
There was a problem hiding this comment.
Minor: this comment still describes the LimitExceeded cell. In the new cell, a second request admitted with the first backup AVAILABLE is the passing branch, and a refusal is the slow-host branch. Please update it to match.
There was a problem hiding this comment.
Fixed. The comment now describes the slot-wait cell: the second request is sent while the first job holds the one slot and waits for it; the passing branch is the second admitted once the first job ended (the first AVAILABLE at its answer and seen CREATING while it waited); a second answered while the first is still CREATING is a broken bound; a refusal at the wait's bound is the slow-host branch; a first backup already AVAILABLE before the second is sent is a probe that came too late, where the cell asks for more items.
| assert waited >= 25, f"a refusal before the wait's bound: {waited:.1f} s" | ||
| print(f"branch: the second request waited {waited:.1f} s and was refused at the bound ({d.backend})") | ||
| else: | ||
| assert status_at_answer == "AVAILABLE", ( |
There was a problem hiding this comment.
Minor: the cell reads the first backup's status before the thread builds the second client and connects. If the first job ends in that gap, this branch passes and the second request never waited. Please assert that seen_creating is above zero here, so a late probe fails as it did before.
There was a problem hiding this comment.
Fixed. The admitted branch asserts seen_creating > 0, with the message naming EXTENDDB_TEST_BACKUP_BUSY_ITEMS, so a first job that ended before the thread connected fails the cell as the late probe did before.
eb15f90 to
ba6b82d
Compare
b20e19e to
c461442
Compare
There was a problem hiding this comment.
I checked the items from my last review at c461442. You fixed all of them.
I reran the stop probe with a beat in flight at the stop, and no check passed after the resume in three runs. With the clause unbounded, 15 and 114 checks passed. With SQLite's declaration, the check still passes while a beat waits, and the SQLite cell passes in a container. The parks || true mutant fails a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parks. The three Minor texts read as asked, you qualified the Engine paragraph, and the job-bound cell passed in all four CI jobs that run it.
One Major item is inline. A stop that falls between beats still lets checks pass after the resume, for the round trip of the beat that the missed tick starts. That path was already there at b20e19e, and the bound I asked for does not cover it. So it is not a condition for this approval. The simpler rule is to use the clause only where the beat parks. Please fix it in this PR, or open an issue for it before the stack merges.
My reply on the clause's thread covers one sentence left in the description.
| /// parks before its bound, and within one `period` of its start otherwise. | ||
| fn in_flight_confirms(in_flight: &Mutex<Option<Instant>>, parks: bool, period: Duration) -> bool { | ||
| match *in_flight.lock().unwrap_or_else(PoisonError::into_inner) { | ||
| Some(since) => parks || since.elapsed() <= period, |
There was a problem hiding this comment.
Major: after a stop that falls between beats, this clause still passes checks that the two-period rule refuses. On resume, the missed tick starts a beat at once, and every check passes during its round trip.
I stopped a test process for 1.5 s between two 5 ms beats, and the first beat after the resume answered Lost. Checks then passed for 5.2 to 6.2 ms after the resume, though the previous beat returned eight periods before them. Without the clause, no check passed.
The module doc still says that a row that passes the check "is never one a reconciler may take over". On PostgreSQL, MongoDB and Cassandra the two-period rule already covers a beat in flight, so only a claim whose beat parks needs the clause.
There was a problem hiding this comment.
My last review approved this PR with a Major item open. That item blocks approval, so I am changing my review to request changes.
After a stop that falls between beats, the clause at heartbeat.rs:129 passes checks for 5 to 6 ms after the resume. Another owner may hold the row by then. On PostgreSQL, MongoDB and Cassandra the two-period rule already covers a beat in flight, so only a claim whose beat parks needs the clause. Please apply the clause only there. Then add a test that stops the process between beats and fails without that change.
The restore checks in #740 call the same function, so this fix covers them too. Once it is in, I will approve.
ba6b82d to
a317b99
Compare
c461442 to
33a8e9c
Compare
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
33a8e9c to
1c7cc29
Compare
Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and
backup purge· this PR: the CreateBackup job · next: the restore job (PR 9a).Closes #699 (part of #624).
On
mainat81c7c340, which carries #671 (merged asff6f74eb) and #672 (merged as4a42c5e0); rebased onto it after each merge, with no conflicts (the last rebase adds main's #466, #754, #756 and #819). The text below still says "PR 6" and "PR 7" for #671's and #672's code: this PR adds nothing to the catalog schema (every column the job writes exists from PR 6's migration 6;docs/schema/*.txtare unchanged, checked against a freshiniton PostgreSQL and SQLite); it binds PR 7's store, identity file and health to the job, and PR 6's claim, row and reconciler to the duties. Where the PRs meet: the job takes PR 7'srequire_healthy_storeand the maintenance check before the permit (as the in-catalog paths do; the review moved the check in front of the row, so a refusal writes none) and PR 6'sclaim_backupafter the snapshot; the reconciler's cycle carries PR 7's store check and DELETING duty and PR 6's dead-restore duties beside this PR's dead-CREATING, FAILED-prefix, DELETED-prefix and tombstone duties, with the maintenance flag gating the last three;backup purge's zero-window pass on SQLite runs as the claim's holder (PR 7'sReconcilerState::for_maintenance_holder), so on this tree its cycle skips the three deleting duties and the registrations sweep, which the held-branch test now pins; the reconciler lists its FAILED-prefix and tombstone rows through PR 7'slist_backup_rows_older_than, which PR 7 added for the purge's live-owner test; the jobs' maintenance refusal is PR 7's text ("A backup maintenance command holds this catalog's maintenance claim; ..."); the PostgreSQL snapshot slot's refusal now goes through PR 6'slimit_exceeded_message;StatusFields(PR 6) gains one member every backend'sset_statuswrites;BackupCataloggains two methods beside PR 7's. Since the last push main merged #335, #666 and #681 under the stack: PR 6's migration is 6 (#335 took 5 for dropping continuous backups), Cassandra and MongoDB run it through #681's numbered runner, and the job drain now runs in #666's shutdown path, before the registration supervisor's return and the stop of the restartable worker set the reconciler belongs to, so this PR's own worker drain is gone, a job still running at the drain's bound keeps the serving registration as a worker that could not be stopped does, andextenddb stopwaits for the sum of that path's bounds (76 s). The earlier rebase onto #672's head conflicted in nine files:serve.rs(that order), the Cassandra workflow (this PR'snoneon #681'scassandra-rustjob), the engine tests' imports, and the differences, limits, usage and upgrade documents and the design, where #335's PITR rows and PR 6's migration 6 text meet this PR's text (main's rows kept beside this PR's). The rebase onto the review rounds of #671 and #672 conflicted in fifteen files, resolved so that both PRs' review changes stand beside this PR's:JobPermit::acquire_within(the 30 s wait beforeLimitExceededException) is what the store-backed path'stake_permit().awaittakes;fail_restore_jobcarries the owner generation and heartbeat #671's reconciler passes; the reconciler's state is theArc<Mutex<..>>#671'sserve.rsshares; the in-catalog create keeps #671's name rule (a dot-only or over-long table name is backed up in-catalog, as the service backs up any table it can create), and the store-prefix rules on the name that #671 removed from that path move to the store-backed path ascheck_store_prefix_fits(region, account_id, table_name), run before the store's health check, the maintenance check and the permit, so a name no prefix can carry is refused with the sameValidationExceptiontexts and writes no row (a_store_backed_create_refuses_a_name_no_store_prefix_can_carry, new); #672's unhealthy-store row in the differences document now names the store-backed create beside the delete and the restore; the clippy job keeps #671's run with the Cassandra and MongoDBtest-hooksfeatures and adds this PR's run withextenddb-engine/backup-test-hooks; the TableName ARN region refusal, the 30 s permit wait, the catalog reads beside the store check's bound, the DELETED answer and the defaulted-root rule from the two review rounds are re-applied to this PR's rows of the differences document and the admin guide. #671's review item took revision item 28 and #672's are 29 and 30, so this PR's are 31 and 32 (every reference in the tree renumbered). On the store-backed row's path the target name is checked once the row is known to exist and AVAILABLE (check_catalog_row_status), before the backup's in-use rule and before this release's refusal of a store-backed restore (check_catalog_row_for_restore): the service's order, measured (a restore of a backup into its own CREATING target isTableInUseException; a missing backup into any targetBackupNotFoundException), which the first CI run of this head caught on the S3 and MongoDB store jobs, wheretest_restore_into_an_existing_name_is_table_already_existsand the auth suite'stest_an_existing_target_is_evaluated_with_its_tagsmet the store refusal beforeTableAlreadyExistsException(a_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusal, new, pins the three answers and that a row that is not AVAILABLE answers before the name is read); design 5.3 step 1 states the order.Merge order. This PR and #740 (the restore job) merge together, with no
maincommit between them: this PR makes CreateBackup store-backed wherever a store is configured, which on file-backed SQLite is the default, and RestoreTableFromBackup of such a backup is refused until #740. Amainthat carried this PR alone could not restore the backups it takes (the maintainer measured 8 of 8 restores refused where #672's head and the service restored). The store-backed create is not disabled here; the two PRs are one release.What
With a configured
[backup.store]on PostgreSQL, SQLite and MongoDB,CreateBackupwrites the backup to the store and returns itCREATING; a job of the server uploads it and marks the rowAVAILABLE. Without a store, and on Cassandra always, the in-catalog create of PR 7 runs as before. The reconciler runs its full duty list. No request or response shape changes; the API-visible changes are the asynchronous create (clients pollDescribeBackup, as against the service),CreateBackupof a vector table succeeding on the store-backed path,CreateBackuprefused withValidationExceptionwhile a configured store is unhealthy or its first check pending,LimitExceededExceptionfor a secondCreateBackupovermax_concurrent_jobswhile one job runs and for aCreateBackupwhilebackup purgeholds the maintenance claim (checked before the permit, as on the in-catalog path, so no row is written for the refusal),ServiceUnavailablefor aCreateBackupwhose snapshot open or row insert meets the cancellation of the jobs' token, which a client sees only after a registration loss (on SIGTERM the listener's 5 s drain closes the connection before the token is cancelled, so the maintainer's client saw a closed connection 8.2 s after sending; the row is not written either way), and a backup of a table holding anNvalue the wire did not validate (#618) failing on the store-backed path (CREATING, thenBackupNotFound; the reason inbackup list --failedand the log).DescribeBackup'sItemCountandTableSizeBytesare the stats pass's figures whileCREATINGand the stream's (the items written, the sum of their DynamoDB sizes) onceAVAILABLE. On SQLite on a platform where the serve lock is not taken (not Unix) CreateBackup stays in-catalog whatever[backup.store]says (backup_store::in_catalog_create_only): the job's claim is an in-process slot, sound only with one process per file.Core (
crates/core).validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits): one walk of the item, everyNandNSmember parsed once on the accepting path with the wire's number rule, the nesting depth counted on the way, the key attributes (RestorableKeySchema::resolvefrom the key schema and attribute definitions, multi-part keys included) checked first with the format's owncheck_key_value(the declared type compared before the value is parsed), then the key sizes against the instance's hash and range key limits, every attribute name againstmax_attribute_name_bytes, then the size (RestorableLimitsfrom&LimitsConfig);RestorableItemError { key, path, reason }withDisplaykey {key}, attribute {path}: {reason}, every key value, name and path segment bounded on its own so a legal key appears whole whatever its bytes (a key value atSHOWN_KEY_VALUE_BYTES, 12,288 escaped bytes, six times the hash key limit; names, path segments and echoed numbers atSHOWN_VALUE_BYTES, 4096), and every such text written as a JSON string's content with U+0085, U+2028 and U+2029 escaped beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log (a reader usingstr.splitlinessees one line too);RestorableItemReason::{InvalidNumber, KeyShape, AttributeNameTooLong, TooLarge, TooDeep, LineTooLong}, the last carrying the cap.backup_format::encode_item_line_len_bounded(item, cap)measures an item's line through a counting writer that stops at the cap, so the feeder's measurement of an oversized item holds no line;limits::DEFAULT_MAX_ITEM_SIZE_BYTESis the feeder's threshold.backup_format::check_key_valueandshownarepub. Metrics:MetricName::{BackupJobs, BackupJobDuration, BackupBytesUploaded, BackupSnapshotAge, BackupStoreErrors},BackupJobOutcome,QuerySource::BackupJob, the recorders onMetricsCollector;/metricsemits them under those names as JSON like every other metric (the endpoint is not Prometheus exposition format),BackupJobDurationin microseconds from the snapshot's open to the job's last catalog or store call (for a FAILED job its FAILED write anddelete_prefixincluded),BackupBytesUploadedfor a job that ended AVAILABLE,WorkerErrorCountunderBackupJobfor a FAILED job (a cancelled one records neither worker metric).Storage (
crates/storage).JobClaim::heartbeat_within(bound), a provided method that boundsheartbeatover its whole call and answersNoneat the bound; the engine's timer and its claim re-check call it instead of wrappingheartbeatin a timeout, so a backend can place the bound (SQLite, below).BackupCatalog::claim_dead_backup(key, stale_after) -> Option<Box<dyn JobClaim>>(the reconciler's takeover of a dead CREATING row under a fresh generation,heartbeat_atuntouched;Nonefor a live owner, a fresh heartbeat, a row that is not CREATING or missing, or a claim another task holds),sweep_restore_registrations(a default empty body),StatusFields.source_table_details_json; the trait doc of #672'slist_backup_rows_older_thannames the reconciler's two listings beside the purge's.snapshot::paging::SNAPSHOT_FETCH_BYTES(64 MiB) withfetch_bytes(),record_page_bytes,record_rows_past_cutand thetest-hooksreaders and override;snapshot::source_table_schema(description, ttl)builds the manifest'ssource_tablefrom a snapshot's description (round-trips through the manifest reader, absent members as explicit nulls);snapshot_conformance::run_page_bounds(a one-byte budget measures the row and proves the knob took effect, then an eight-row budget: every row once, pages at most the budget plus one row, the cut fired; a stream that has yielded more items than the table holds fails the check at once, naming the key yielded twice, so an inclusive seek fails in seconds rather than running until the hooks lock's wait gives out).PostgreSQL (
crates/storage-postgres). The cursor is replaced by keyset pages in the sameREPEATABLE READtransaction: three layers (LIMIT nafter the last key, the running byte sum and the row count, the rows under the budget in key order), the key columns under qualified aliases and bound as their column types,COLLATE "C"only where the probe found it, the rows pulled from the connection one at a time and parsed as they arrive (the page's text is never held whole),n16 for the first page and then the budget divided by the largest row of the last pages that together hold at least a row bound's worth of rows (RowBoundMemory), never under 4 (MIN_NEXT_PAGE_ROWS) nor over 1000: the server serializes and spills every row the innerLIMITreads, so a page costs itnrows of text and not the rows it returns, and sized from the average row a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only (5.9 GB of server temp files, 18 to 44 s for the 3 rows it returned), where the largest row asks for 4 (208 MiB, 1.1 s); a page of 27 MB rows reads one row past its cut rather than thirteen; sized from the previous page alone the bound oscillated on a table whose large rows are scattered among small ones (4 after a large row, 1000 after four small ones, the 1000-row page meeting large rows again every other page: 5,555 rows read for 1,000 kept and 13.4 GB of temp files on a 1,000-row table with 30 such rows), where the memory keeps it at 4 while large rows keep arriving within a row bound's worth of rows (1,901 read, 3.9 GB; a memory of four pages read the same 5,555 as one); the bound that remains, stated as it is: the second page, and then the page that follows a row bound's worth of small rows, reads up tonrows of up to the line cap, at most once per row bound's worth of small rows, and nothing in this form stops the server at the budget (a walk that does was measured and not taken, below); the floor's cost is 250 pages of four after every large row of a table whose large rows are rarer than one per row bound (2,585 page queries against 38 with one page of memory on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, the record); the stream ends when fewer rows than asked came back and none was cut;page_sql_for(taking the collation flagopen_streamdecides),MIN_PAGE_ROWSandMIN_NEXT_PAGE_ROWSexported for the tests; the slot refusal throughlimit_exceeded_messagenaming[storage.postgres] pool_size snapshot-slot.claim_dead_backuptakes the row's advisory lock on a dedicated connection and thenUPDATE ... WHERE status = 'CREATING' AND heartbeat_at < now() - make_interval(secs => $stale);set_statuswritessource_table_details_json = COALESCE($10, ...);try_claimsetstcp_keepalives_idle = 30,tcp_keepalives_interval = 10,tcp_keepalives_count = 3(CLAIM_KEEPALIVES), each failure a warning, before #672's retried lock.SQLite (
crates/storage-sqlite).SqliteJobClaim::heartbeat_withintakes the engine write lock first and bounds only theUPDATE: the beat queues behind every writer (a GSI backfill, a large write), and that queue is this process's own, not a sign the claim is lost, so a writer holding the lock past a period delays the beat and the claim re-check and ends no job (the maintainer saw two backups fail beside a GSI build on a 400,000-item table; the end-to-end test holds the lock for three periods and the backup ends AVAILABLE; a second cell,a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_it, puts a data file per item through a store whose first put takes four periods and holds the lock until the second data file's put has started, so that put's check runs with the first beat parked and no beat confirmed since the insert, and the backup ends AVAILABLE with its three data files and the manifest; with the in-flight clause removed the second data file is never put and the row is left CREATING).store::connect_optionsbuilds every pool connection's options (row_buffer_sizefour), so the run-ahead test opens a connection with the engine's options and a counting collation.claim_dead_backupis the slot rule:JobSlots::takeis the liveness test and the exclusion in one step,stale_afterunread, then a conditionalUPDATEunderBEGIN IMMEDIATE; the byte cut infetch_pageas a row stream (the stream pulls no row past the cut; the driver's worker reads up to five ahead of it, below);backup::pool_size_for_jobs(pool_size, max_concurrent_jobs)andSqliteEngine::new_for_server: a file-backed pool is raised to2 * (min(jobs, 16) + 1)with a warning naming both numbers so the snapshot slot refuses no job the permit admitted (10 at the defaults, the default pool; over 16 jobs the warning names the jobs the slot cap refuses, 17 tomax_concurrent_jobs,pool_raise_message); the rule is stated once onSqliteConfig::pool_size. The pool's connections are built with sqlx'srow_buffer_sizeat four (store.rs): the driver's worker runs each statement ahead of the task reading it, and its default of 50 buffered rows let a snapshot of large rows hold up to 50 lines the page never asked for; the rows ahead of the reader are five (the four the channel holds and the one the worker has stepped and copied while itssendwaits), 160 MiB at the line cap.MongoDB (
crates/storage-mongodb).claim_dead_backupis onefind_one_and_updateonstatus: CREATINGandheartbeat_at < $$NOW - stale, the generation set through a$literalpipeline (literal_set, now one function for every pipeline write);set_statuswrites the member; the snapshot stream applies no cut (the 16 MiB reply bound is under the budget), stated in its module doc.Cassandra (
crates/storage-cassandra).claim_dead_backup: aLOCAL_SERIALread, staleness against the coordinator clock, thenUPDATE ... SET owner_generation = <fresh> IF owner_generation = <read> AND backup_status = 'CREATING' AND heartbeat_at = <read>(the heartbeat in the condition puts the liveness test inside the write), anUnknownoutcome settled by a re-read;set_statuswrites the member; atest-hookspause between the read and the LWT (test_hooks::pause_before_takeover_write) lets the direct test land the owner's beat in that gap and see the LWT refuse;sweep_restore_registrationsremovesactive_restoresentries whose target is not a CREATING restore target of that backup, by LWT on the entry as read. Cassandra has noSnapshotEngine, so these run only on rows the reconciler meets.Engine (
crates/engine/src/backup/).create_backupbranches onconfigured_store().is_some() && !legacy_create_only(): the store-backed path requires a healthy store, checks the maintenance claim (refuse_while_maintenance_held, before the permit as on the in-catalog path: a row written FAILEDmaintenancefor the refusal carried the insert's heartbeat andbackup purge's wait counted it as a live job for the stale window; the maintainer's eight purge attempts beside four creating clients all gave up, leaving 16,054 such rows), takes the permit, and spawnsjob::startthrough the runtime's task tracker, awaiting its one-shot reply;refuse_backup_of_vector_tablestays in front of the in-catalog create alone.job.rs:startopens the snapshot and inserts the CREATING row throughclaim_backup, both selected against the shutdown token (an open waiting behind a table'sACCESS EXCLUSIVElock no longer holds the drain for its whole bound; the request answersServiceUnavailable, which reaches a client only when the token is cancelled while the listener still serves, a registration loss: on SIGTERM the listener's 5 s drain closes the connection first), mints the id from the open's instant, reads the TTL, builds thesource_table(AlreadyExistson the insert is an internal error), answersCREATING, and runs the job:writer.rs(a feeder task validating every item into a four-item channel and, for an item over the default item limit, validating it and measuring its line againstMAX_LINE_BYTESon the blocking pool (the walk's cost grows with the item) through a counting writer that holds no line, a line over the cap refused through the same path with a reason naming the cap; onespawn_blockingencoder per data file writing 1 MiB chunks into a four-chunk channel that is the put's body; a write into the sink after the pipeline stopped isBrokenPipe, which the encoder does not retry, so a job cancelled mid-file ends within milliseconds rather than spinning through the drain; a file closes at the first item that takes it to or pastdata_file_target_bytes;KeyRangeTrackerper item on a key-ordered table; the first refusal stops the encoding and the uploads but not the validation, each refused item logged with key, path and reason up to 1,000, the rest counted; a put that fails ends the feeder at once, the feeder's token cancelled before the encoder is awaited (an encoder waiting on the item channel returns only once the feeder closes it, so the old order left a mid-file failure running for as long as the stream was held), so the FAILED write does not wait for the snapshot stream's next page; a data file whose stored size (meta.size) differs from the encoder's compressed count fails the job, since the reconciler's dead-owner probe refuses that difference and a live job must not commit a manifest the probe would call incomplete; before every put the job reads the store as the runtime holds it (Pipeline::store_check: healthy, and its identity the row's), and a store the reconciler marked unhealthy mid-job, or whose identity file now names another store (a root swapped under the job), fails the row with the store's reason and makes no store call, the prefix's deletion included (the FAILED-prefix duty deletes it once the store is healthy again); a stream that fails after a refusal gives the reason both, the refusals' count and first item and thenthe snapshot then failed: ..., sobackup list --failedstill names the item; a put is started only when a file's first chunk exists and only when a heartbeat confirmed the claim within two periods, the claim's insert standing for the first beat (the watch starts from the instant taken beforeclaim_backup); one info line at the start with the table's count as the stats pass last counted it and one at the end with the items written, the bytes and the duration), thenBackupDocuments::buildwith the stream's count and the sum of its items' sizes (the manifest'sitem_countandtable_size_bytes), the three documents, a claim re-check, the manifest and the AVAILABLE write with the manifest's size, count andsource_table;heartbeat.rs(a timer throughheartbeat_withineverymax(heartbeat_interval_secs, 15)s;starttakes the claim's instant; a lost, failed or unreturned beat raises the abort;confirms_the_claimis the pre-put check, inclusive at exactly two periods, and passes as well while a beat is in flight and has not failed (the maintainer's second round: a 180,000-item backup beside a GSI build stopped twice at the check before data file 3 or 4, its beat parked on the SQLite write lock past two periods, and the reconciler then failed each row and deleted its objects; the flag is set around every beat, the timer's and the re-check's, and cleared on the way out by a drop guard, so a timer that stops mid-beat leaves the two-period clause; the module doc argues why the clause is safe: on SQLite the beat parks only on this process's lock, one process serves the file and the reconciler's liveness test is the in-process slot, so no takeover coincides with a parked beat; on the other backends the bound covers the whole call, a beat is in flight at most a period after its tick and the clause admits no put the two-period rule refuses; a loss is read before the flight);recheckis the claim re-check before the manifest, whose confirmation counts as a beat; the module doc states the outcome for a process stopped and resumed: on PostgreSQL no takeover while its session lives, a job with no beat in flight resumes under its own generation once its next beat succeeds however long the stop was (a job frozen 160 s finished AVAILABLE), and it stops as a lost claim only when a beat in flight outlives its bound or a put comes before the next beat);failure.rs(write_status: each unknown outcome followed by a re-read, the write sent once more at most, and after the second write a re-read that shows the row in the target status counted as matched and anything else as matching no row, the second write being for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra; on PostgreSQL it fails with the session and the row is the reconciler's);fail_row: FAILED first,delete_prefixonly after it matched). A lost claim ends the job with no catalog write.KillPoint::{AfterDataFile, BeforeManifest, AfterManifest}pause the job underbackup-test-hooks(EXTENDDB_BACKUP_PAUSE, released by the fileEXTENDDB_BACKUP_PAUSE_RELEASEnames).runtime.rs:JobSettings,with_metrics,with_shutdown(the jobs run on a child of the server's token),cancel_jobs(cancels the jobs running now and gives the next ones a fresh child token; the server's workers' stop calls it, so a registration lost while serving cancels the jobs rather than letting their heartbeats and status writes go on beside a migration),with_source,spawn_job(public, for the server's shutdown test),running_jobs,drain_jobs(bound), whose give-up line counts job tasks, whether or not a row exists for one yet; the configured store is held in anArcthe jobs share (into_configured_storereturns it);ConfiguredStoregainsbucketandkey_prefixfor the manifests.reconciler.rs: after the maintenance check, the dead restore jobs, the abandoned in-catalog backups and the store check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (owner dead ...), nothing deleted there; then, without a maintenance claim: FAILED prefixes deleted once the heartbeat is stale (the same cycle's for a row it just failed: in that cycle on PostgreSQL, MongoDB and Cassandra, where the stale heartbeat made the owner dead, and once the dead owner's last beat is that old on SQLite, where the slot did), DELETED prefixes swept, tombstones pastfailed_row_retention_secsremoved once their prefix is empty, the registrations swept;CycleReportand the cycle's log line carry the counts, a dead-CREATING resolution whose write matched no row counted as left to its owner; a duty's store call that fails (getandlistof the manifest probe and the deleting duties,delete_prefix; not an absent object, not the store's check) recordsBackupStoreErrorsunder the call's name, so a store that refuses one row's calls is counted though the cycle completes.Server and app.
serve.rscreates the shutdown token before the runtime, gives it to the runtime with the metrics and the build's backend and version, and runs the shutdown sequence (drain_then_release:drain_jobs(30 s) after the listener stops, then the registration supervisor's return, thenstop_then_release(#666), which takes whether every job ended and keeps the serving registration, to expire by itself, while a job may still run; one function, which the server's test drives with a job that outlives a 100 ms drain and asserts no release);BackgroundWorkers::with_jobsmakes the workers' stop (the supervisor's, at a registration loss) cancel the jobs throughcancel_jobs;lib.rsbounds the plain-HTTP listener's drain atCONNECTION_DRAIN_TIMEOUTas the TLS listener's is, sostop's 76 s sum holds for adev-modebuild too;backup_store::in_catalog_create_only(backend)names why a backend's CreateBackup stays in-catalog (Cassandra; SQLite where the serve lock is not taken) and the startup log says so;cmd_backup.rs'sresolve_dead_ownersprints the CREATING rows the purge's cycle resolved, with their new status, and the ones another owner took over;JOB_DRAIN_TIMEOUT,WORKER_STOP_TIMEOUT(the supervisor's 10 s, the workers' stop's 16 s and the release's 10 s) and the listener'sCONNECTION_DRAIN_TIMEOUTare exported, andextenddb stop(cmd_stop.rs) waits for their sum and the runtime'sRUNTIME_SHUTDOWN_TIMEOUT, 76 s, where it gave up at 10 s while the server was still inside the job drain, its timeout message naming the four parts;backup_store.rsbuildsJobSettingsfrom[backup]and the manifest names from the store configuration (S3: bucket and prefix; filesystem: the root label and""), and its startup lines say CreateBackup writes to the store.crates/app/Cargo.tomlandcrates/server/Cargo.tomlchain thebackup-test-hooksfeature, andextenddb versionprintsbackup-test-hooks enabledon such a build (thetest-hooksline stays the MongoDB hooks' line thatdevtools/run-mongodb-testsreads).crates/config/src/backup.rsdocs and the floor's error text name the in-catalog jobs' constant, andfailed_row_retention_secsis refused understale_job_after_secs(with a retention of 0 one cycle failed a killed job's row and removed row and prefix within 29 ms, and an object the stopped owner landed afterwards was never deleted: the FAILED-prefix duty needs the row).Harness and CI.
tests/test_backup_restore_fidelity.py:wait_for_backup_available, used by_create_backup,_cleanupand every cell that reads a backup it just made; a session fixture reads/healthonce and checks that the server'sbackup_storecomponent is the kindEXTENDDB_TEST_BACKUP_STOREnames and healthy (absent fornone), so the variable describes the server under test rather than being taken on trust; the vector test reads the variable (filesystem,s3,none), fails unset ("is unset"), prints which branch ran, and runs the refusal (capturing the ARN of a create accepted against expectation) or the store-backed create (CREATING, AVAILABLE with the GSI inSourceTableFeatureDetails, listed, the restore'sValidationExceptionnaming the store until PR 9a); the 24 restore cells skip on a server with a store, naming PR 9a.tests/test_auth_permissions.py: the backup fixture's cleanup waits for AVAILABLE before DeleteBackup, and the restore-authorization cell skips on a server with a store for the same reason;tests/test_abac.py's backup cell waits the same way before its DeleteBackup (deleting at once answeredBackupInUseExceptionand left an AVAILABLE backup on every store-backed run), andtests/test_authz_resource_arn.py's module fixture, whose cells deny DeleteBackup to other users, deletes its backup once AVAILABLE and its table when the module ends (it left a complete backup under the store on every store-backed run); the fidelity suite's skip of its 24 restore cells is gated on the target as well as the variable, so a staleEXTENDDB_TEST_BACKUP_STOREin a real-DynamoDB run skips nothing.tests/test_backup_job.py(new, its own servers, run by name): a secondCreateBackupovermax_concurrent_jobs = 1waits for the slot (#671's 30 s wait) and is answered only once the first job has ended, the first backup AVAILABLE at its answer (one answered while the first is still CREATING fails the cell; a first backup AVAILABLE before the second is sent asks for more items; a wait that outlives the bound on a slow host isLimitExceededExceptionwith the bound's text, which the cell accepts and reports), and a third is accepted at once afterwards; one cell per #618 write path (BatchWriteItem,TransactWriteItems,UpdateItemon an absent item in the key it creates,ImportTable) with three refused items: three log lines with the keys, the FAILED reason's count and first item naming #618,BackupNotFound, no manifest; a refused item with a UUID hash key and a ten-digit epoch range key named by its whole key, and the wire's reason for its value, in the log line and inbackup list --failed(whose RustDebugescapes the harness decodes); an item overmax_item_size_bytesafter a restart with a lower limit: a reason naming the size and the limit and not #618.tests/test_backup_store_config.py: the create refused on an unhealthy or checking store, the objects under the root and in the bucket once healthy, the start and completion log lines with their two counts.tests/test_backup_job.py's deployment fixture gains a MongoDB branch (a replica set of the suite's own throughEXTENDDB_TEST_MONGODB_URI; the catalog's database names are fixed, so each test destroys and initializes the catalog),tests/test_mongodb_backup_restore.py's three cells skip on a server with a store (they read and restore the in-catalog create's documents; PR 9a brings them back),devtools/run-mongodb-tests --backup-store-root DIRconfigures a filesystem store on the server it starts, andintegration-mongodb.ymlgainsmongodb-backup-store: the backup selection of the pytest suite withfilesystem(the fidelity suite's store branch) and the job suite on a second replica set..github/workflows/clippy.ymllints--features extenddb-engine/backup-test-hooksbeside the default build, and the kill-point scripts are committed asdevtools/backup-kill-points.shanddevtools/backup-kill-points.py(the API half, which #740 extends with the restore points), run from a checkout against a build with the hooks..github/workflows/integration.yml: the PostgreSQL job appends a filesystem store afterinitand setsEXTENDDB_TEST_BACKUP_STORE=filesystem, the SQLite job setsfilesystem(the defaulted root), a newrun-integration-sqlite-s3job runs the backup tests against an S3 store on moto withs3(itsrust-cachestep saves only on main, as #680 made every other job's, its actions are pinned to the commits main's jobs use (#752), and it dumps moto's log on failure; its by-name step runs the store configuration suite's S3 cells and the job suite against the build's defaulted filesystem root, which the step says), and the PostgreSQL and SQLite jobs run the two store suites by name; the three start steps wait for thebackup_storehealth component to behealthyrather than for the word anywhere in the reply, so a store that failed its check or is stillcheckingis named at startup; the pytest lines carry-rPso a passed cell's printed branch reaches the job log;integration-mongodb.ymlandintegration-cassandra.ymlsetnone; the Rust integration suite (tests/rust/src/backup_restore.rs) waits for AVAILABLE after every create that goes on to describe, delete or restore (create_available_backup,wait_for_backup_available), deletes the backups its tests make, and readsEXTENDDB_TEST_BACKUP_STOREas the Python suite does (backup_store:filesystemands3expect the restore'sValidationExceptionnaming the store,nonethe round trip, unset fails with the fixture's text),backup_arn_scoping.rsandrestore_active_completeness.rsthrough it, since the SQLite Rust integration job's file-backed server has the defaulted root and its CreateBackup is the asynchronous one (five tests assumed the synchronous create and the suite left 12 complete backups under that root); the five Rust integration jobs set the variable (filesystemonrun-rust-integration-sqlite,noneon the PostgreSQL, MongoDB and Cassandra jobs);devtools/run-tests --pytestrefuses when the variable is unset (its--help,CONTRIBUTING.mdand the developer test guide name the variable, its three values and the refusal), naming the three values, ignores the two store suites in its line, and itshealth_checkwaits for thebackup_storecomponent to behealthywhen the variable isfilesystemors3(the server it restarts for[import]is the one the suites run against); the start steps'python3readscurl's reply with its stderr quiet, so the seconds before the listener answers print no traceback;tests/test_backup_job.py's_startedstops a server whose health wait or provisioning failed.crates/storage-sqlite/tests/backup_job_panic.rs(new;extenddb-engineandextenddb-backup-storeas dev-dependencies): a job whose storeputpanics leaves a CREATING row that one reconciler cycle in the same process resolves to FAILED.Docs. The differences document (the vector row, the restore-interruption row with the backup duties and the measured kill points, where backups live, backup creation with the record's durations, the CREATING
ItemCountandTableSizeBytesthat can be 0 and 0 and the stream's figures once AVAILABLE, theBackupInUseException"Backup is being created: " that DeleteBackup and RestoreTableFromBackup answer while CREATING lasts (the whole upload; the service's backups measured were AVAILABLE 12 to 90 ms after CreateBackup), and the snapshot's instant stated as the table at the snapshot's open, after the request's checks and after any wait for a permit (up to 30 s), so a write acknowledged during that wait is in the backup (the maintainer's second round: a PutItem acknowledged 1.01 s after a CreateBackup that waited for a permit was in it, 2 of 2 runs, where the row said no write acknowledged after the request was received), against the maintainer's measurement of the service (every write acknowledged before CreateBackup was sent and the 18 in the 0.1 s up toBackupCreationDateTime, none of the 11,754 in the next 75 s) in place of the "one-minute window" clause, the shutdown refusal's reach (a registration loss, not SIGTERM) in the same row and in the admin guide's Stopping paragraph, a new #618 row), the limits document (the SQLite pool raise and PostgreSQL's slot, the store row, a data-file row with the line cap and the file size, the contents row), the usage guide's backups row (Partial: CreateBackup to the store, RestoreTableFromBackup for in-catalog backups only this release), the upgrade manual (the asynchronous create, a store-backed backup listed, described and deleted but not restored until the restore job lands, the #618 paragraph, the rollback paragraph), the admin guide's Stopping paragraph (the connection drain, on the TLS and the plain-HTTP listener alike, the job drain's 30 s, the supervisor's 10 s, the workers' stop's 16 s, the release's 10 s, the runtime's 5 s, andstop's 76 s), its[backup.store]sentence on a parked store check and its statement of the different-roots gap (two host-local roots at one path with a copied identity file are not told apart before the first AVAILABLE backup, and one instance's reconciler fails the other's complete backup; one shared root from the first backup on), itsfailed_row_retention_secsrow (at leaststale_job_after_secs), the troubleshooting entry forstop's timeout, the deployment guide's container paragraph (a grace period under 35 s kills the process during the job drain; the three compose files setstop_grace_period: 80s, above the 76 s sum, where they set 15 s) and the architecture guide's shutdown step, the admin guide's[backup]rows (the memory figures, their scope at the defaultmax_item_size_bytes, andMALLOC_ARENA_MAX=1with its measured cost besidemax_concurrent_jobs, the SQLite pool raise, the job's two log lines quoted whole, the/healthexamples' key order as the build's) and[backup.store]section, the deployment guide's service unit (MALLOC_ARENA_MAX=1as a commented line with its condition and cost) and its SQLite sentence pointing at the admin guide's row,extenddb.sample.tomland the init template, the measurement record committed asdocs/design/15-backup-restore-bench-pr8.md, design revision items 31 and 32 (the review's changes: the backend-placed heartbeat bound and, from the second round, the pre-put check's in-flight clause with its safety argument in 5.1 step 4, the step 2 text on the snapshot's open after the permit wait and on the shutdown refusal's reach, the maintenance check before the permit in 5.1 step 2 and 4(f), the store check before every put, 4(c)'s gap before the first AVAILABLE row,TableSizeBytes, the retention floor, the kill-point scripts indevtools/, the MongoDB job) with the body sentences they correct (section 3's byte budget as the three-layer page with up ton - 1rows read past the cut, 4(e)'s unknown-outcome rule aswrite_statushas it (the second write for a backend whose claim connection survives; on PostgreSQL it fails with the session) and its PostgreSQL liveness rule (a stale heartbeat and a free advisory lock; a stopped owner whose session lives keeps its row), 5.1 step 4's SIGSTOP outcome per backend (on PostgreSQL no takeover while the session lives; on resume the job completes, or after more than two periods stops as a lost claim with no catalog write and the reconciler resolves the row) and step 6's cancelled job completing AVAILABLE after its manifest put, 5.1 step 1's store-health check and step 2's unreadable maintenance claim, section 7 as a sizing rule scoped to the default item limit with the one-arena factor measured on both backends and the setting's request-path cost, section 9's metrics as emitted (the reconciler'sBackupStoreErrorswith the job's), section 10's table with PR 8's rows, the duty list's resume marked PR 9a's, the testing summary's kill points run by hand, the PR 8 acceptance's bounds and its paging cells).Why
#396 section 5.1, 4(a) and 4(e), section 7's memory bound and pool rule, section 9, and the PR 8 section. A job without its reconciler leaves rows no one resolves, so both ship together (item 24). Decisions taken while implementing, each stated in the module doc of the code it concerns and in revision item 31: the job and the reconciler live in the engine's backup module and nothing crosses a channel between the request and the job but the response; the takeover is a method of its own,
claim_dead_backup, sinceclaim_backupinserts; the ages are backend predicates (list_backup_rows_older_than, which #672 now carries for its purge as well), so the engine holds no clock; the pre-put check is two heartbeat periods, since a beat sent at the tick returns up to a period later, or a beat in flight, since on SQLite a beat parked on the write lock behind a long writer is this process's own queue and not a loss; a lost heartbeat ends the job with no catalog write, the claim connection being the one that failed; the pipeline ends its puts before the FAILED write, so the failure path awaits nothing; the FAILED-prefix duty of the same cycle deletes the prefix of a row the cycle just failed once the dead owner's beat is stale; SQLite's snapshot slot stays the backend's admission bound and the pool is raised for it; the PostgreSQL page is three layers because a window function runs beforeLIMIT; the vector round trip's create half lands here and its restore half with PR 9a; the restore cells of the fidelity suite skip on a server with a store until then;EXTENDDB_TEST_BACKUP_STOREis set by every CI job that runs a suite creating backups (the pytest jobs and the Rust integration jobs:filesystemon the SQLite jobs,s3on the moto job,noneelsewhere) and refused unset. Three decisions came from the measurements: section 7's memory paragraph is the sizing rule the worst-expansion runs measured (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite: the page's decoded rows, the items in the feeder, the channel and the encoder, one row's parse, the line buffer, the chunks and SQLite's five-row run-ahead, times the allocator's factor, timesmax_concurrent_jobs), since the earlier "budget plus one row plus its decoded form" did not describe the 880 MiB peak of 27.7 MB rows; the allocator's factor is documented rather than set from the binary: with glibc's default arenas the peak grows with the number of large rows a job reads (466, 588, 760 MiB over idle at 12, 24, 48 rows on a quiet host; 707, 700, 1,281 MiB on a loaded one) and underMALLOC_ARENA_MAX=1it is flat (452 to 481 MiB in both series), so the admin guide carries that setting for hosts that back up large items and the service unit carries the line commented out, since one arena costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%); and the item channel stays at four items after a one-item channel was measured 75% slower on a 1M-row backup for 50 MiB less at that peak, while the SQLite pool'srow_buffer_sizegoes from sqlx's 50 to four, which bounded the driver's run-ahead at five rows at no cost to the 1M-row duration. The PostgreSQL page's row bound follows the largest row of the last pages rather than the average because the server serializes every row the innerLIMITreads: the average sized a page after a mixed page at 810 large rows for the 3 it returned; the memory covers a row bound's worth of rows because one page of memory oscillated on a table whose large rows are scattered among small ones (and four pages read the same rows as one); a page query that stops at the budget on the server, a recursive walk of the primary key one index probe per row, read exactly the rows it returned (the interleaved table in 38.8 s with 2.5 GB of temp files against 54.6 s and 3.2 GB for the memory and 72.0 s and 9.2 GB for one page) and was not taken because the probe per row cost the 1M-row backup 95% of its duration (23.1 s against 11.9 s); the record carries both.Testing done
Third review round: the in-flight clause of
confirms_the_claimis bounded where the beat does not park before its bound.JobClaimgains a providedheartbeat_parks_before_its_bound(false; SQLite's claim answerstrue, its beat waiting behind the engine write lock outside the bound), the handle records the beat's start, and a beat in flight confirms the claim for as long as it waits where the beat parks and within one period of its start otherwise, so a process stopped mid-beat and resumed on PostgreSQL, MongoDB or Cassandra passes no check in the window before its timer raises the loss.review_round_3::a_beat_in_flight_past_its_bound_confirms_the_claim_only_where_the_beat_parksruns the same parked beat under both declarations (within the bound both confirm; past it only the parking one; the loss read first when the beat returns Lost); with the clause unbounded again itsparks=falsehalf fails. The module doc, design 5.1 step 4 and item 32, and the admin guide'sheartbeat_interval_secsrow say so. Enginebackup::198 passed, storage-sqlitewrite_lock_tests15, clippy 1.97 clean. The SQLite write-lock cell's doc says what the cell itself sees without the clause (one put started, the row left CREATING: no reconciler runs there); the job-bound cell's comment describes the slot-wait branches, and its admitted branch asserts that the first backup was seen CREATING while the second request waited.The store jobs' runs of
tests/test_backup_job.py's job-bound cell found the first backup AVAILABLE once the second CreateBackup was answered, on MongoDB and SQLite alike: since #671's review round a request that finds every slot held waits up to 30 s for one rather than refusing at once, so the second request was admitted when the first job ended, and the cell's expectation (an immediateLimitExceededException) was the earlier contract. The cell (test_second_create_backup_over_the_job_bound_waits_for_the_slot) now sends the second request from its own thread while watching the first backup, and asserts that it is answered only once the first is AVAILABLE and then succeeds; the refusal at the bound is kept as the branch for a first job that outlives the wait. The design's PR 8 acceptance says the same.The maintainer's second round (three inline items), on this head:
cargo test -p extenddb-engine --lib -- backup::194 passed (re-measured,--list: 194;review_round_3::a_beat_parked_on_the_write_lock_confirms_the_claim_until_it_returns_lostnew),cargo test -p extenddb-storage-sqlite --lib267 passed, 1 ignored (re-measured: 268;write_lock_tests::a_put_during_a_writers_hold_passes_on_the_beat_parked_behind_itnew, the two write-lock cells run five more times, 1.7 s each run) andbackup_job_panic1 passed;cargo check --workspace --tests;cargo fmt --all -- --check;cargo +1.97.0 clippy -p extenddb-engine -p extenddb-storage-sqlite --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings, clean. The mutant the maintainer's item implies, the in-flight clause removed fromconfirms_the_claim: the new SQLite cell fails (one put started under the hold, the row left CREATING: the second data file's check refused the put) and the new engine cell fails at its first assertion (the parked beat does not confirm the claim); both pass after the revert. The other counts and runs below are carried from the previous head; nothing else in the code changed.After the rebase onto the review rounds of #671 and #672, on this head:
cargo clippy --all-targetswithextenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks,extenddb-engine/backup-test-hooksunder-D warnings, clean;cargo test -p extenddb-engine --lib -- backup::193 passed (a_store_backed_create_refuses_a_name_no_store_prefix_can_carryanda_store_backed_restore_checks_the_target_name_after_the_row_and_before_its_refusalamong them); theextenddb-app(47),extenddb-config(24),extenddb-server(307),extenddb-storage(233) andextenddb-storage-sqlite(266 passed, 1 ignored) lib suites;backup_restoreon PostgreSQL at127.0.0.1:55432, 56 passed;backup_engine::live_tests::on the MongoDB replica set at127.0.0.1:27117, 31 passed;direct_integration -- backup_engine::withtest-hookson a fresh Cassandra 4.1 node, 41 passed (a first run against the node earlier rounds had filled with test keyspaces failed 24 cells on connection timeouts while the node sat in old-generation collections; recreated, every cell passed). The wire, kill-point and memory runs below are from the previous heads and were not repeated; CI runs the wire suites.The review round (the maintainer's 30 inline items). Re-measured on the previous head (
--list):extenddb-corelib 797,extenddb-storage230,extenddb-engine331 (backup::tests161),extenddb-server307,extenddb-app47,extenddb-config24,extenddb-storage-sqlitelib 260 andbackup_job_panic1,extenddb-storage-postgreslib 78 andbackup_restore50,extenddb-storage-cassandralib 129 anddirect_integration212 withtest-hooks,extenddb-storage-mongodblib 220; the other counts below are carried from the previous head. Run on this head:cargo test -p extenddb-engine --lib -- backup::tests(161 passed), theextenddb-core,extenddb-storage,extenddb-config,extenddb-app,extenddb-serverandextenddb-storage-sqlitelib suites andbackup_job_panic(all passed),backup_restore::backup_job::the_claim_session_sets_tcp_keepaliveson PostgreSQL 16 at127.0.0.1:55432,backup_engine::test_a_beat_between_the_takeovers_read_and_its_write_refuses_the_takeoveron Cassandra at127.0.0.1:9042withtest-hooks(7.4 s),tests/test_backup_job.pyon a MongoDB release build against a replica set of its own (7 passed, 63 s), anddevtools/run-mongodb-tests --backup-store-root ... -- --pytest --parallel --filter backupwithEXTENDDB_TEST_BACKUP_STORE=filesystemon that build (40 passed, 29 skipped: the fidelity suite's 25 restore cells, the three MongoDB in-catalog cells, one authorization cell; the two collection errors were this host's Python 3.7 on unrelated files that use:=). Each test the maintainer asked for was run against the mutation he named and fails there: the SQLite write-lock test with the beat bounded over its whole call (the row stays CREATING); the mid-file put failure with the encoder awaited before the feeder's cancel (the job did not end within 10 s); the mid-job store check withstore_checkremoved; the key-ordered stream with the tracker's refusal ignored; theBackupStoreErrorscells with thedelete_prefixrecording removed fromfailure.rsand thelistrecording removed fromlist_sizes; the re-read cell with the generation check removed; the two-period sample with<=changed to<; the nesting cell with the list depth's>=changed to>; the keepalive cell withtcp_keepalives_idleat 7200; the run-ahead cell withROW_BUFFER_ROWSat 50 (51 rows stepped for one pulled); the Cassandra cell withAND heartbeat_at = ?removed from the LWT;pool_size_for_jobs(11, 4)with>=changed to>; the shutdown-sequence cell withjobs_left == 0replaced bytrue.cargo fmt --all -- --check;cargo +1.97.0 clippy --all-targets -- -D warningson the touched crates withextenddb-engine/backup-test-hooksand onextenddb-storage-cassandrawithtest-hooks(the host's 1.98 toolchain flags a pre-existingresult_large_errin the console pages, which CI's 1.97.0 does not);cargo +1.88.0 check --workspace --all-targets --features extenddb-engine/backup-test-hooks;cargo check --workspace --tests;shellcheckondevtools/backup-kill-points.sh(clean) anddevtools/run-mongodb-tests(no warnings, as before). The MongoDB CI job, the clippy step and the compose grace periods are not run here.The rounds before. Counts are of the whole tree on the previous head, measured after the rebase onto #672's head
9434f34cand re-measured after the move ontomainat e8ca7df (the workspace run on this tree; the MongoDB and Cassandra counts listed on this tree,--list, with their suites run on the top of the stack, #740); "new" is against #672's head. Runs marked "before the rebase" were made on this PR before the stack's rebases onto #666 and then main, and were not repeated.extenddb-core: lib 795 passed, 1 ignored (30 new:validate_restorable_itemper shape,1E126andabcin a non-key attribute with the wire's reason text, the nested patha.b[2].cand theNSmember position and member, validation past the first valid number, size at and over the limit, depth at and under the limit agreeing withvalidate_item_nesting_depth, a missing, mistyped and empty key attribute,abcand1E126in the range key asInvalidNumberand1.0asKeyShape,N("abc")under anSkey asKeyShape, multi-part keys attribute by attribute, key attributes before the others,RestorableKeySchema::resolve's refusals, every key attribute of a legal key shown whole and only an oversized value cut, the attribute path and the echoed number bounded segment by segment, the key sizes and attribute name sizes at and over their limits,RestorableLimitsfromLimitsConfig,LineTooLong's text naming the cap, a control character in a name or a number escaped in the message, a legal key of control characters shown whole, the bounded line measurement stopping at the cap without a buffer;check_key_valuecomparing the type before parsing; the backup job recorders landing under the outcome and the store call; the three line separators escaped in a key, a name and a number; a 4097-byte key attribute name cut atSHOWN_VALUE_BYTESand a 20,000-byte list under a key attribute cut to a JSON string);data_file_line_bomb1,data_file_nested_decode_memory1,data_file_decode_peak1 ignored.extenddb-storage: 220 (4 new:source_table_schemaround-trips throughExtenddbManifest::from_json_bytes, absent members as explicit nulls, on-demand drops the throughputs, theTableClassSummaryerror);snapshot_conformance::run_page_boundsis run by the PostgreSQL and SQLite crates and fails in 3 s on a stream that repeats rows (an inclusive-seek mutant, which ran until another test's 600 s lock wait before).extenddb-engine: 325 (35 new: a snapshot stream that fails after a refused item keeping both in the FAILED reason, and the plain stream error keeping its own; the reconciler's failedget,listanddelete_prefixeach recorded once asBackupStoreErrorsunder its name while the cycle completes, and nothing recorded once the store answers; atracingcallsite first met by a thread without a scoped subscriber still reaching the test'sLogLines(the suite's global default answers "sometimes" for every callsite, so a test running beside a log-asserting test cannot cache the job's callsites as "never"; the CI run of the previous push lost the lost-claim line that way); a 400 ms maintenance check at a 100 ms period delaying the request and not the first put, the backup AVAILABLE; the store-backed create answering CREATING and its job writing the backup with the put order, the manifest's instant, count, size, order and key range, the AVAILABLE row and the metrics; the maintenance claim refusing the request before any row; an open that fails leaving nothing; a healthy store making the create store-backed and a vector table backed up; an unhealthy store refusing create, delete and restore unless bound to the in-catalog create; the data-file close rule at the knob and at its 32 MiB minimum; the manifest recording the snapshot's order; the validator failing the backup after the rest is validated, with and without BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618; a put failure failing the row then deleting the prefix; a FAILED write that does not match making no store call; anUnknownwrite re-read, applied and lost; a lost, stalled or erroring heartbeat stopping the job with no put; the handle confirming the claim within two periods; cancellation running the failure path; a second create over the bound; the reconciler resolving a dead CREATING row with a manifest to AVAILABLE and one without to FAILED; the FAILED and DELETED prefixes and the tombstones on their clocks under and without the maintenance claim; a write after the stop answeringBrokenPipeand an encoder closing at once after the stop; a put failure ending the job while the stream is held, the start and completion lines' counts, 1,001 refusals through the log cap, a line over the cap refused and named, a status write error leaving the manifest and making no store call, a second unknown outcome, a stopped heartbeat timer starting no put, the claim re-check before the manifest, the probe's seven incomplete shapes and a store error during the probe, a row failed by the cycle losing its prefix in that cycle, a takeover lost to another reconciler left to it, the tombstone guard (an old AVAILABLE row with an empty prefix kept) and the registrations sweep gate, a second unknown outcome whose write landed matched, a dead CREATING row whose status write errors left for the next cycle, a FAILED job's duration covering its FAILED write and deletion, an eighth incomplete-manifest shape (an unparseable manifest), the resolved row'ssource_table_details_jsonequal to the manifest's; the validate-the-rest cell fed one item per 20 ms;BackupSnapshotAgeasserted recorded per confirmed beat; the probe-error cell asserting no row left to an owner). feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's held-branch test (a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_window) also asserts here that the holder's zero-window cycle leaves a stale FAILED row's prefix and runs no registrations sweep, and that the same cycle reading the claim runs both; it fails on a held branch that runs the deleting duties. The two slow job tests wait under a bound of their own (wait_for_jobs_within: 300 s for the line-cap test, about 20 s alone, 22.8 s on two cores beside a busy loop; 120 s for the data-file test, 5 s alone, 10.1 s so loaded), where the 30 s guard failed the line-cap test three times in forty loaded runs;cargo test -p extenddb-engine --lib -- backup::testsundertaskset -c 0-1 nice -n 5beside one busy loop passed 20 runs of 20 (155 tests each, 26 to 27 s).extenddb-server: 305 (2 new: the backup job metric names render from the store; a backup job still running at the drain's bound keeps the serving registration while the workers stop, which fails when the release ignores the jobs).extenddb-app47 (1 new:stopwaits for the sum of the server's shutdown bounds, 76 s, the bounds after the job drain 36 s; it fails on the old 10 s and on a sum without the supervisor's and the release's bounds).extenddb-config23,extenddb3,extenddb-auth274,extenddb-backup-storelib 37 andconformance123,extenddb-cache27: unchanged.extenddb-storage-postgres(127.0.0.1:55432): lib 78 (3 new:the_row_bound_adapts_within_its_clamps, sizing a page of 984 small rows and 3 large ones from the large row (under 8, not about 800) and pinning the floor at four,the_memory_covers_a_row_bound_of_rows,the_page_query_binds_the_key_by_type);backup_restore50 (3 new:claim_dead_backuptaking over a stale row on the database's clock and refusing a live one, with the successor's AVAILABLE write carryingsource_table_details_json; a terminated claim session reportingUnknownand the row re-read; the claim session's TCP keepalives; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's cell);table_snapshot89 passed, 2 ignored (4 new, each on both database layouts:pages_are_cut_at_the_byte_budgetwithrun_page_boundsand the rows past the cut asserted exactly, 8 over 77 pages, all on the first page;the_row_bound_remembers_large_rows_among_small_ones: a 300-row table with a 27 MB row at every 33rd key streamed whole in order, 201 rows read past the cut, all on the second page, over 53 round trips (306 over 7 with one page of memory, 103 round trips with a floor of 2);the_page_query_is_an_index_scan_on_every_key_shape:EXPLAINof the first page and a page after a key atn16 and 1000 on 20,000-row tables of the six key shapes (hash only,S,NandBrange keys, a multi-part range key, a legacy table recollated to the database collation),Limitover oneIndex Scanon the primary key with no sort on each, then each table streamed whole in 21 round trips;the_last_page_sorts_the_rows_after_the_seek: the index scan ten rows from the end of the 100k table and the bitmap scan with aSortnear the end of a 1,060-row table, the sorted page's rows in key order); the other 20 binaries as before.extenddb-storage-sqlite: lib 257 passed, 1 ignored (3 new: the takeover of a row whose slot no task holds, the pool raise and its slots, the pool-raise warning's text at 8 and 20 jobs; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's);backup_job_panic1 (new: a job whose storeputpanics leaves a CREATING row one reconciler cycle in the same process resolves to FAILED).extenddb-storage-mongodb(replica set at127.0.0.1:27117): lib 167 passed, 52 ignored (1 filtered),interpreter_selftests18,pushdown_parity20, the migration binariesmigrations_fresh8,migrations_guards7,migrations_holders10,migrations_locks11,migrations_released5 (1 new unit: the$literalset; the older-than filter's cell is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's);--lib -- --ignored --test-threads=1 live_52 (1 new: the takeover on the server's clock with the fence on the old claim's write, its window 3 s; the listing on the named clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's). Before the rebase two runs under a host running a 1M-item load beside them failed 5 and 3 restore cells on their 90 s ACTIVE wait and the takeover cell on its then 1 s window, and passed when rerun.extenddb-storage-cassandra(127.0.0.1:9042, one binary at a time): lib 129,migrations22,migrations_stopped8;direct_integration211 with--features test-hooks(2 new: the takeover with the heartbeat in the LWT's condition, the registrations sweep keeping the live entry; the listing on the coordinator's clock is feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's), run on03baa321in three parts of 53, 72 and 73 tests named--exact(the second part's first run met connection timeouts while the node was still dropping the first part's 45 keyspaces, and passed when rerun alone); a container holding the keyspaces earlier runs of the TTL suite left (165extenddb_ttl_test_account_*) stalled two runs before the rebase in old-generation collections, and the runs here drop what they leave;ttl_integration26 passed, 1 failed, 1 ignored,test_ttl_enable_rejects_asynchronous_gsibeingmain's failure PR 6 and PR 7 recorded (the file is not touched here).run-integration-sqlitejob's steps on the SQLite release build (check-sqlite-catalog-lockandcheck-sqlite-older-binaryagainstextenddb/extenddb-dev:0.1.14: all checks passed;initin the checkout's root and the schema check: no difference fromdocs/schema/sqlite.txt;serve --foreground --write-pid-filewaited for a healthy store;devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export"withEXTENDDB_TEST_BACKUP_STORE=filesystem: pytest 1641 passed, 46 skipped, 1 xfailed, comprehensive 502 passed; the two store suites by name: 18 passed, 5 skipped); therun-integration-sqlite-s3job's steps (a bucket on moto, the job's[backup.store.s3]block, the wait for a healthy S3 store, the backup selection--pytest --parallel --filter "backup"withs3: 40 passed, 28 skipped; the two store suites with the S3 endpoint: 19 passed, 4 skipped;extenddb stopof that server:extenddb stoppedin 0.26 s); the catalog-lock checks of the PostgreSQL, MongoDB and Cassandra jobs (check-postgres-catalog-lock16 ok without its pooler pass,check-mongodb-catalog-lock29 ok,check-cassandra-catalog-lock42 ok on a node of its own: all checks passed), which stop servers and wait for their registrations;cassandra-rustandmongodb-rust(run-cassandra-tests -- --rust-integrationandrun-mongodb-tests --rust-live -- --rust-integrationwithnone:tests/rust514 passed on each, MongoDB's live and migration binaries green);docs/schema/*.txtagainst a freshinitof each backend: identical; a catalog created bymain's release build at migration 5, on SQLite and PostgreSQL: this build'sserverefuses it (exit 3, naming migration 6),migrate --yesapplies 6,servestarts andextenddb stopof it on PostgreSQL ends it in 0.26 s with its 16 workers drained and no session left, andmain'sserverefuses the migrated catalog (exit 4). Before the rebase onto1234179b,devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export"on the PostgreSQL release build with a filesystem store appended afterinitandEXTENDDB_TEST_BACKUP_STORE=filesystem: 1439 passed, 52 skipped, 6 failed; withkind = "disabled"andnone: 1464 passed, 27 skipped, 6 failed; on the SQLite release build with the defaulted root andfilesystem: 1446 passed, 45 skipped, 6 failed; the comprehensive suite 336 passed, 4 failed on each. (These three full runs are on the binaries before the sink'sBrokenPipechange and the SQLite row buffer, neither of which a wire test reaches; the backup selection and the two store suites were rerun on the final binaries.) The ten failures are the same on every run and on62d32f0f: the host's botocore (Python 3.7) refusesReturnValuesOnConditionCheckFailure(test_conditional_writes.py::TestConditionalPutItem, 4 cells;tests/python/test_item_edge_cases.py, the 4rv_on_ccfcells) andOnDemandThroughput(test_config_fields.py::TestOnDemandThroughput, 2 cells) as unknown parameters before a request is sent; none is a backup test. The skipped cells include the 24 restore cells oftest_backup_restore_fidelity.pyon the servers with a store (PR 9a) andtest_auth_permissions.py's restore-authorization cell for the same reason;test_create_backup_of_a_vector_table_is_refusedran itsfilesystembranch there and itsnonebranch on the PostgreSQL run without a store. The S3 job's selection (--pytest --parallel --filter "backup") on the SQLite build against an S3 store on moto (127.0.0.1:9200) withs3: 27 passed, 28 skipped; the same selection on the SQLite build with the defaulted root: 27 passed, 28 skipped.tests/rust, the Rust integration suite as the jobs run it (devtools/run-tests --extenddb --rust-integration --release): the whole suite on the SQLite release build with the defaulted root andEXTENDDB_TEST_BACKUP_STORE=filesystem, 512 passed, and afterwards no complete backup under the root (find <root> -name extenddb-manifest.jsonfinds none;extenddb-store.jsonalone); thebackupandrestore_active_completenessfilters on the PostgreSQL release build without a store andnone: 18 and 2 passed, the restore round trip taken. With the variable unset the restore cell fails withEXTENDDB_TEST_BACKUP_STORE is unset; set it to one of ["filesystem", "s3", "none"] ..., and withnoneagainst the filesystem server it fails on the store'sValidationException.tests/test_backup_store_config.pyandtests/test_backup_job.py(7 cells) by name on this head: on the PostgreSQL release build 11 passed (feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's exit-code cell among them), 12 skipped (the SQLite class); on the SQLite release build 19 passed, 4 skipped (the PostgreSQL class) with moto reachable at127.0.0.1:9200for the S3 cells. feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672's in-memory cell asserts this PR's create there: with[backup.store.filesystem] pathset the backup is CREATING, then AVAILABLE with its manifest under the root, and its restore is refused naming the store until PR 9a; without[backup.store]the in-catalog create warns. Before the rebase, the fidelity, ABAC and auth-permissions suites against a server of each release build with a filesystem store andEXTENDDB_TEST_BACKUP_STORE=filesystem: 144 passed, 25 skipped on each, the fixture printingbackup store under test: {'kind': 'filesystem', 'status': 'healthy'}and the vector cell itsfilesystembranch; the ABAC cell's backup deleted.kill-points.sh, a server built with--features extenddb-app/backup-test-hooks,reconcile_interval_secs = 2,stale_job_after_secs = 45, a 100k-item composite-key table with two GSIs; the SIGTERM point on a 300k-item table without GSIs), on SQLite with the defaulted root, on PostgreSQL with a filesystem store, and on the SQLite build with an S3 store on moto:after_data_file: after the SIGKILL the prefix holdsdata/000001.json.gzand no manifest, the row CREATING; after the restart the row is FAILEDowner dead ...after 0 s on SQLite and the S3 store (the slot rule, the first cycle) and 46 s on PostgreSQL (the 45 s window), DescribeBackup BackupNotFound, the prefix deleted 45 s after the kill on SQLite and S3 (the dead owner's heartbeat stale) and in the same cycle on PostgreSQL;before_manifest(run on SQLite and PostgreSQL): the data file and the three documents (manifest-files.json,manifest-summary.json,extenddb-manifest.sha256) under the prefix after the kill and no manifest, the row CREATING, then FAILEDowner dead ...after 0 s on SQLite and 47 s on PostgreSQL, the prefix deleted 46 s after the kill on SQLite and 48 s on PostgreSQL (its cycle), so a prefix withoutextenddb-manifest.jsonis never resolved AVAILABLE whatever else is under it;after_manifest: the five objects under the prefix after the kill, the row AVAILABLE after 0 s on SQLite and S3 and 47 to 48 s on PostgreSQL with the manifest's size (847,689) and count (100,000), the prefix whole, DescribeBackup AVAILABLE;sigterm:waiting up to 30 s for 1 backup job(s) to end,FAILED (shutdown); its objects were deletedwithin 15 ms,drained, the row FAILEDshutdown, the prefix empty, the process gone 55 ms after the signal (polled at 50 ms; the first record's "5 s" was the script's own sleep), and on the S3 storelist-multipart-uploadsempty. The first SIGTERM runs on SQLite found the job spinning through the whole 30 s drain with its row left CREATING: the chunk sink answered the encoder's write after the stop withInterrupted, which flate2's writers retry; it answersBrokenPipenow (writer.rs, two unit tests), and the three stores pass on the rebuilt binary.bench-rss.sh, the release binary under aVmHWMsampler at 200 ms, restarted after each load so the high-water mark covers the backup alone; 16 cores, 30 GiB, PostgreSQL 16.15, bundled SQLite 3.46 via libsqlite3-sys 0.30.1, MongoDB 7.0.43;BENCH-pr8.mdhas every figure and caveat): 100k and 1M items of a composite-key table with two GSIs: peak over idle 11.3 and 20.6 MiB on PostgreSQL (+9.5 MiB), 45.0 and 50.5 MiB on SQLite (+7.9 MiB; the 45 is the pool's SQLite page caches), 9.6 MiB at 100k and 15.5 MiB at 500k on MongoDB (+4.8 MiB; the 1M load would have taken close to an hour through the wire, so 500k stands in its place), each under the 112 MiB bound. Durations: 1.85 s and 36.1 s on PostgreSQL (a loaded host), 1.44 s and 12.45 s on SQLite, 1.85 s and 8.81 s (500k) on MongoDB. The incompressible run (64 MiB of randomBattributes, a 68 MB backup gzip did not shrink): 120.5, 74.8 and 90.2 MiB over idle, so the encoder streamed the file through its 4 MiB chunk window. The worst-expansion run (1,000 small rows then 12 of feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's 27.7 MB worst lines, three to a page) at the first head: 880 MiB over idle on PostgreSQL, 582 to 762 MiB on SQLite across runs, 474 MiB on MongoDB with 13.5 MB lines (its 16 MiB document cap refuses the full shape). These peaks are not the design's earlier formula ("the budget plus one row of text, plus its decoded form"): the page's decoded rows, the items in the feeder, the four-item channel and the encoder, one row's parse through the deserializer's intermediate value, the line buffer, the chunks and on SQLite the driver's five-row run-ahead are terms of their own (545 MiB for this shape on SQLite, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite at the 32 MiB line cap), and what glibc's allocator keeps on top of them depends on how many large rows the job reads: the series at 12, 24 and 48 such rows on one SQLite binary peaked at 466, 588 and 760 MiB over idle on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill points running beside it (0.86 to 2.35 of the terms, growing with the rows), and underMALLOC_ARENA_MAX=1at 481, 452, 480 and 452, 480, 455 MiB (0.83 to 0.88 of the terms, flat), so section 7 states the terms as a sizing rule (the terms at the line cap, times 1.2 under one arena, timesmax_concurrent_jobs, at the defaultmax_item_size_bytes), the admin guide carriesMALLOC_ARENA_MAX=1for hosts that back up large items and the service unit the line commented out, and the acceptance bounds the one-arena peak at 1.2 times the terms and records the default-arena peak beside it; the record isdocs/design/15-backup-restore-bench-pr8.md. On PostgreSQL the build with the row bound from the largest row and the rows parsed as they arrive took the worst shape to aVmHWMof 790 MiB in 16.5 s against 905 MiB in 18.2 s, its 1M run +4.0 MiB over the 100k run (13.3 s), and the rows read past the cut in the paging test are 8 over 77 pages, all on the first page (602 at the head); the same shape underMALLOC_ARENA_MAX=1on the final PostgreSQL build peaked at 340 MiBVmHWMagainst 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB), so the one-arena factor is measured on both backends. The setting's cost on the request path (a 50k-item table, one boto3 client per thread, three alternating repeats of 20 s per operation): GetItem −3.8%, Query −3.5%, Scan −6.9%, paged Scan −9.2% on SQLite; +1.1%, −0.6%, −2.1%, −2.1% on PostgreSQL. The row bound's memory on the release binary (two 20,000-row tables with 30 worst-expansion items, scattered among the first 1,000 keys and grouped after them, both binaries back to back): the interleaved table's backup 54.6 s with 3.18 GB of server temp files against 72.0 s and 9.21 GB with one page of memory, the grouped table 38.9 against 39.8 s and 2.85 GB both; the 1M run 13.7 against 12.1 s (VmHWM36.7 against 36.3 MiB, +4.1 MiB over the 100k run; the memory changes no page of a small-row table, and the tree's binary carries the validator's and feeder's changes of the same day). The server-side walk that was measured and not taken: 38.8 s and 2.51 GB interleaved, 38.0 s and 2.51 GB grouped, 23.07 s against 11.85 s for the 1M run, 13.4 s and 658 MiB against 17.2 s and 553 MiB for the worst shape. A one-item channel was measured and not taken: 50 MiB less at the worst-shape peak for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders). The SQLite pool'srow_buffer_sizeis set to four (store.rs; sqlx's default of 50 let the driver's worker read every row of a 24-row worst-shape table ahead of the first page, 844 MiB over idle against 696 with four, back to back; a 1M-row backup took 11.4 s with either).Not changed here
RestoreTableFromBackupof a row of the configured store answersValidationException"Backup is in a backup store; this release restores in-catalog backups only"; the fidelity suite's restore cells skip on a server with a store and come back with it; the vector wire test asserts that refusal where PR 9a asserts the round trip.restore_progresssweep (PR 9a adds the records they read); the duty that removed a CREATING restore target with no job row is withdrawn (design item 31).backup syncand its options,remove-target,migrate-legacy(PR 10); export and import through the store (PR 13); the legacy reader's removal (PR 14).restore_jobs_total,restore_job_duration_seconds) are PR 9a's.Sortof the rows after the seek, pinned by a test and stated in the module doc); the query is not changed to force the index scan. A page query that stops at the budget on the server (a recursive walk of the primary key, one index probe per row) was measured and not taken: it reads no row past the cut on any shape and costs the 1M-row backup 95% of its duration; the record carries its figures, and the window form's bound is stated as it is.Checklist
cargo test --workspace), the pre-existing Cassandra TTL failure excepted (the review round ran the touched crates' suites and the named backend tests; CI runs the rest on push)cargo fmt --check)cargo clippy --workspace --all-targets -- -D warnings)Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).ADR / RFC: #396, sections 4(a), 4(c), 4(e), 4(f), 5.1, 7, 9 and the PR 8 section, revision items 31 and 32.
Storagetrait:BackupCataloggainsclaim_dead_backupandsweep_restore_registrations(the second with a default body);JobClaimgainsheartbeat_within(a provided method);StatusFieldsgainssource_table_details_json;SnapshotEngineis unchanged. No wire protocol or on-disk format change (the data files and manifests are #394's).Breaking changes
[backup.store]on PostgreSQL, SQLite and MongoDB,CreateBackupreturns the backupCREATINGand it becomesAVAILABLEonce the upload ends, as on the service; while it isCREATING, which lasts the whole upload,DeleteBackupandRestoreTableFromBackupof it answerBackupInUseException"Backup is being created: " (the service's backups measured wereAVAILABLE12 to 90 ms afterCreateBackup), so a client that deletes or restores a backup right after creating it pollsDescribeBackupfirst. The backup is the table at its snapshot's open, which comes after the request's checks and after any wait for a job permit (up to 30 s), so a write acknowledged during that wait is in the backup. On file-backed SQLite the defaulted root makes this the default behaviour. The restore of a store-backed backup is feat(backup): RestoreTableFromBackup job and cross-backend matrix [backup/restore 9] #740's, which merges with this PR (Merge order above): amaincarrying this PR alone would refuseRestoreTableFromBackupof the backups it takes withValidationException, so the two land together; the usage guide's backups row and the upgrade manual describe the pair.DescribeBackup'sItemCountandTableSizeBytesare the stats pass's figures whileCREATING(0 and 0 for a table it has not reached) and the stream's onceAVAILABLE; the previous release reported the stats pass's for both.CreateBackupis refused withLimitExceededExceptionwhilebackup purgeholds the maintenance claim, checked before the permit, so no FAILED row is written for the refusal (the previous head wrote one with reasonmaintenance).CreateBackupwrites no row when the server's shutdown meets its snapshot open or row insert; itsServiceUnavailableanswer reaches the client only when the jobs' token is cancelled while the listener still serves (a registration loss), since on SIGTERM the listener's 5 s drain closes the connection first.shutdownand deletes its objects); the jobs started under the new registration run on.[backup] failed_row_retention_secsmust be at leaststale_job_after_secs; a smaller value is refused at startup.CreateBackupwrites the in-catalog format whatever[backup.store]says, and the store serves restores and deletes.CreateBackupis refused withValidationExceptionnaming the store's kind while a configured store is unhealthy or its first check is pending (PR 7 refused only the restore and delete).CreateBackupof a table with vector indexes succeeds on the store-backed path (refused as before without a store).Nvalue the wire did not validate (BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618) fails on the store-backed path where the previous release backed it up; the reason reaches the operator throughextenddb backup list --failedand the log.[storage.sqlite] pool_sizeto2 * (max_concurrent_jobs + 1)with a warning when it is smaller (no change at the defaults).claim_dead_backup, writeStatusFields.source_table_details_jsoninJobClaim::set_status, and may overridesweep_restore_registrationsandJobClaim::heartbeat_within(a backend whose beat queues behind a lock of its own process places the bound after the lock, as SQLite does).BackupRuntime::shutdownreturns a token (the jobs' current one) rather than a reference, andinto_configured_storereturns theArcthe jobs share.devtools/run-tests --extenddb --pytestrequiresEXTENDDB_TEST_BACKUP_STORE(filesystem,s3ornone).extenddb stopwaits up to 76 s for the server to exit (it waited 10 s); a supervisor that runs it, or a container runtime's grace period, should allow that long for a server that may be running backup jobs. The repository's compose files setstop_grace_period: 80s(they set 15 s). Adev-modebuild's plain-HTTP listener drains for at most 5 s, as the TLS listener does.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.