Repository navigation
fix(backup): report point-in-time recovery as unsupported - #335
Merged
Merged
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
September 10, 2026 19:03
jcshepherd
reviewed
Sep 10, 2026
| /// a table whose recovery is disabled. Owned by the engine rather than the | ||
| /// backends: the answer does not depend on stored state, so no backend is | ||
| /// consulted and all three report identically. | ||
| fn disabled_continuous_backups_description() -> extenddb_core::types::ContinuousBackupsDescription { |
Collaborator
There was a problem hiding this comment.
Confusing name. default_continuous_backups_description?
Collaborator
Author
There was a problem hiding this comment.
Renamed to default_continuous_backups_description and updated the call sites.
jcshepherd
reviewed
Sep 10, 2026
| /// writes the same settings row the files write and the gate reads, so the | ||
| /// statement text is pinned against a rename of the key or the table. | ||
| #[test] | ||
| fn the_runner_convergence_write_targets_the_catalog_version_cell() { |
Collaborator
There was a problem hiding this comment.
:-/ I'm not sure this test is worth the CPU cycles needed to compile it.
jcshepherd
reviewed
Sep 10, 2026
| @@ -0,0 +1,311 @@ | |||
| # Design Spec: Off-Host Backup and Restore for ExtendDB | |||
Collaborator
There was a problem hiding this comment.
I'd prefer this be a separate PR.
jcshepherd
reviewed
Sep 10, 2026
| | 06 | [Server](06-component-server.md) | HTTP server, routing, middleware pipeline, response formatting, TLS, rate limiting, throughput tracking (`extenddb-server` crate) | | ||
| | 07 | [Streams](07-component-streams.md) | DynamoDB Streams design space — capture mechanism, shard management, retention (high-level; detailed design deferred) | | ||
| | 08 | [Configuration](08-component-config.md) | TOML config, env vars, CLI, logging, metrics, health checks, deployment (VM, Kubernetes, Docker) | | ||
| | 09 | [Testing](09-testing.md) | Test strategy, reference suites, golden files, multi-language test suites, coverage tracking | |
jcshepherd
reviewed
Sep 10, 2026
| - **Data database** (e.g., `extenddb_catalog_data`): Stores user items, GSI/LSI data, and stream records. Each table gets its own PostgreSQL table. | ||
|
|
||
| The catalog version is 0.0.3 on PostgreSQL and SQLite, stored in the `settings` table under the key `catalog_version` and checked at startup. | ||
| The catalog version is 0.0.4 on PostgreSQL and SQLite, stored in the `settings` table under the key `catalog_version` and checked at startup. |
Collaborator
There was a problem hiding this comment.
Can we remove the explicit version here? "The catalog version is stored in the settings table under the key catalog_version and checked at startup." Reduce the number of places where the version has to be updated.
jcshepherd
reviewed
Sep 10, 2026
| ## Current Status | ||
|
|
||
| Catalog 0.0.3 is current. The 0.0.2 to 0.0.3 upgrade is the first in-place catalog upgrade ExtendDB has, and **every existing PostgreSQL deployment must run it**, including deployments that never use vector indexes: the server refuses to start against a catalog version it was not built for. | ||
| Catalog 0.0.4 is current. **Every existing PostgreSQL and SQLite deployment must run the upgrade**: the server refuses to start against a catalog version it was not built for. |
Collaborator
There was a problem hiding this comment.
Same. Our tooling should say what the current and installed versions are, not the docs.
jcshepherd
reviewed
Sep 10, 2026
| ``` | ||
| 001_schema.sql ← the complete initial schema | ||
| 002_vector_indexes.sql ← vector index metadata, catalog 0.0.3 | ||
| 001_schema.sql ← the complete initial schema |
Collaborator
There was a problem hiding this comment.
Prefer to just let folks look in the migrations files, and let the files document their purpose rather than repeating it here.
jcshepherd
reviewed
Sep 10, 2026
jcshepherd
left a comment
jcshepherd
left a comment
Collaborator
There was a problem hiding this comment.
A few comments. I realize my complaining about having to update the catalog version in 10 different files is whining in the age of AI ... nonetheless, I'm not sure it's helping to expect to have real up-to-date catalog version numbers sprinkled through documentation. Teach a user to fish ...
8 tasks done
5 of 8 tasks
robinnsc
added a commit
that referenced
this pull request
Oct 5, 2026
…andoned restores Follows the sort-key fix. Restores on PostgreSQL and SQLite now reproduce what the service reproduces, and the copy is bounded and crash-safe. Restored, from a new per-backup table definition: - global and local secondary indexes, with their key schemas, projections, and GSI throughput, filled during the copy; - billing mode and provisioned throughput (was always 5/5); - table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions`, one row per backup holding that definition as JSON in the wire's shape behind a version marker (`extenddb_storage::backup_definition`). Backups taken before the upgrade have no row and restore as before: keys and items, no secondary indexes, 5/5 for a provisioned table. The migration runner now writes the compiled catalog version after every walk, as #335 also does, so a replayed earlier migration cannot leave the version behind the schema. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and locks the source data table before releasing it. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose catalog row exists but whose data table is not yet built is left out, with any attribute definition only it used. - SQLite CreateBackup reads the table, its definition, and its items in one write transaction. - DeleteBackup removes a backup's rows and marks it DELETED in one transaction; a restore reads from a snapshot (PostgreSQL) or re-checks under the write lock per batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a restore target returns ResourceInUseException, as the service does for a table it is still creating, instead of deleting it out from under the copy. Memory: items are streamed and buffered up to 500 items or 4 MiB of stored JSON, whichever comes first, on both sides of both backends. Measured on PostgreSQL: 100,000 items of 4 KB (about 400 MB), peak server RSS 87 MB during the backup and 109 MB during the restore; 150 items of about 2.9 MB JSON each (435 MB), 147 MB and 168 MB with MALLOC_ARENA_MAX=1 (glibc's per-thread arenas otherwise retain freed memory and RSS reads higher). Concurrency: SQLite restores commit per batch, so other writers wait for a batch rather than the whole restore (PutItem p50 137 ms, max 276 ms during a 60,000-item restore). SQLite backups still hold the write lock for their duration, as they must to be consistent. PostgreSQL allows four concurrent restores per process; a fifth waits up to 30 s and is then refused with LimitExceededException. Failure and crash: - A failed copy removes the target by table id, synchronously: it is claimed CREATING -> DELETING in one statement (so removal and the flip to ACTIVE are mutually exclusive), its data and index tables are dropped, and its row deleted. If a drop fails the row stays DELETING and the control plane finishes the job. - PostgreSQL: a restore holds a session advisory lock keyed by account and target name from before the target is created until the copy ends, on a dedicated connection with idle_session_timeout disabled. The control-plane pass removes targets that are CREATING with no scheduled transition, older than 60 s, and whose lock is free. Verified with kill -9 mid-restore: the target is removed about 60 s later and the name can be restored into again. - SQLite: the same targets are removed at startup, before serving; this assumes one server process per database file, which the backend's in-process write lock already requires. Verified with kill -9. Refused rather than restored wrongly: backups of tables with vector indexes (as before), backups of multi-part-key tables (a preview the item paths address only by their first parts), and definitions with an unknown version. RestoreTableFromBackup now validates TargetTableName. Tests: - tests/test_backup_restore_fidelity.py: hash-only tables with S, N, and B keys; composite keys with S, N, and B sort keys; a provisioned table with four GSIs (ALL, INCLUDE, KEYS_ONLY; S and N hash keys, N and B range keys; sparse) and an LSI, comparing definitions, throughput, IndexStatus, every index's contents, projected attribute sets against the projection definitions, and ordered ranged index reads in both directions. Waits for backups and restores as the service needs. 7/7 on PostgreSQL and SQLite; the index test fails on the previous commit. - crates/storage-postgres/tests/backup_restore.rs (14): legacy row shape, pre-0.0.4 backup after replaying migration 003, failed-copy cleanup under a non-zero control-plane delay, duplicate rows, multi-part refusal, indexes and throughput, table class/SSE/on-demand, >1,000 rows with oversized items, the abandoned-restore sweep (grace, held lock, unowned), a name whose lock is held, the backup barrier against an uncommitted UpdateTable (fails on the previous approach), a half-built GSI, and DeleteTable during a restore. Run one at a time, and scratch databases are dropped even when a test panics. - crates/storage-sqlite/src/backup.rs restore_tests (9): indexes and throughput, the startup sweep, failed-copy cleanup, multi-part refusal, table class/SSE/on-demand, 730 rows across batches with physical row and index-row equality against the write path, a database with the exact 0.0.3 schema (testdata/schema_0_0_3.sql) holding a 0.0.3-format backup, byte-bounded batch cuts, and DeleteTable during a restore. - tests/test_cli_vector_catalog_migration.py covers 0.0.2 -> 0.0.4 and the replay case. Not changed here: RestoreSummary appears only in the RestoreTableFromBackup response and not in later DescribeTable output (pre-existing; needs a persisted column). DeleteBackup during a restore fails the restore with BackupNotFoundException rather than refusing with BackupInUseException. Vector index restore is still refused. Readiness assessment P0-2. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 9, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 12, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This was referenced Oct 13, 2026
Merged
robinnsc
added a commit
that referenced
this pull request
Oct 13, 2026
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
This was referenced Nov 4, 2026
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming #618 for an `N` value, and then the stream's error when the snapshot failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming #618 for an `N` value, and then the stream's error when the snapshot failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 4, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
6 of 8 tasks
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
The `[backup.store]` section (the backup store crate's shape nested under `[backup]`, `deny_unknown_fields` on both), with the per-backend default for an absent section (a filesystem store at `backups/` beside a file-backed SQLite database, created 0700, whose validation or identity-check failure is a warning that leaves the store unconfigured while the catalog has no rows for it, and an unhealthy store or a startup error once it has; nothing on PostgreSQL, MongoDB, Cassandra and an in-memory database), `kind = "disabled"`, the spool path rule (required with an S3 store except beside a database file; refused inside or containing a filesystem root, or when it is or contains the database directory; a filesystem root that is or contains the database directory is refused the same way; an existing spool must be owner-only, parents included), relative store and spool paths made absolute at load, and the spool's `.lock`. Startup opens the configured store, validates it and checks its identity: `extenddb-store.json` at the root, created with the new `BackupStore::put_if_absent` when the root has no file and the catalog no rows for it (S3: `PutObject`/`CompleteMultipartUpload` with `If-None-Match: *`, a 412 is present, a 409 retried; filesystem: the temporary created O_EXCL and `link`ed to the key), re-read and compared with the catalog's id and with the rows' ids (`read_store_ids`), then the newest AVAILABLE row's manifest probed. A configuration error that needs no request stops startup naming the path or bucket; a filesystem store's checks run before the listener (under a 5 s bound that leaves the store `checking` when a mount stops answering after the root is opened; every later check runs under the same bound, 30 s for a bucket, the cycle selects it against shutdown, a filesystem root's abandoned call parks one thread until the mount answers and no check starts while it is, and the process exits within 5 s of SIGTERM with it parked), an S3 store's after it in the reconciler's first cycle (the component is `checking` until then); a transport, validation, identity or probe failure, a filesystem root's I/O failure at the probe write, or a check that does not finish within its bound leaves the server running with the store unhealthy: logged once, `/health` reports a `backup_store` component (status `degraded`, HTTP 200 kept, `reason` the problem's category alone), the restore and delete of a store-backed backup refuse with a ValidationException naming the kind and the category and not the root (or that the first check is pending), the reconciler acts on no row of the store, and the check runs again each cycle until it passes; the in-catalog create, which touches no store, is not refused. A check that cannot read the catalog changes nothing. DeleteBackup of a configured store's row now deletes the manifest, then the prefix, then marks the row DELETED; the reconciler finishes DELETING rows the same way. `BackupCatalog` gains `read_store_ids`, `maintenance_claim`, `remove_dead_creating_backup_row` and `list_backup_rows_older_than` (the rows of a store with a status whose heartbeat, or completion, is older than an age, compared in the backend's query on the database's clock), a `MaintenanceClaim` (heartbeat, release) held on `backup_maintenance` with each backend's claim mechanics (PostgreSQL advisory lock and row, the take trying the lock again over about 10 ms so a job's one-statement shared lock does not refuse it; SQLite row under the write lock, taken over whatever its heartbeat by a command under the serve lock; MongoDB conditional upsert; Cassandra LWT, its claim-row reads at LOCAL_SERIAL) and the stale rule; `check_maintenance_claim` reports a claim held while its heartbeat is within `stale_job_after_secs` (PostgreSQL: while the holder's session holds the advisory lock, tested with a transaction-scoped shared lock, then by the heartbeat; on SQLite a server's startup under the serve lock removes the record outright); the holder heartbeats it on a timer from the take to the release (`HeldClaim`, at `JOB_HEARTBEAT_INTERVAL`); `remove_dead_creating_backup_row` removes a CREATING row whose owner is dead. On this head CreateBackup and RestoreTableFromBackup (the in-catalog paths) refuse with LimitExceededException while the claim is held, after every other check and before the permit, the message naming a backup maintenance command rather than one command. `ServerComponentsOptions::beside_server` opens a catalog for a read beside a server, without the SQLite serve lock and the startup recovery; on PostgreSQL the factory starts the GSI queue workers only for a server, and on MongoDB it starts the repair of documents stored without their key (#473) only for a server. `ServerComponentsOptions::command` names the command that opens the catalog: its serving registration carries the name (PostgreSQL's session `application_name`; MongoDB's `extenddb_servers` document, in a `command` field; Cassandra's `extenddb_servers` row, after the release and ` as ` in `release`, the table having no column for it), so a migration waiting for the registrations names the command, and a refusal while a migration runs does not advise starting a server. The reconciler's cycle takes the caller's word that it holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) in place of reading it. `extenddb backup list [--failed]` (a read beside a server, on an unhealthy store too, that creates no identity file and reads it under the store's check bound) and `extenddb backup purge --confirm`: the latter takes the maintenance claim and heartbeats it for its whole run, on SQLite resolves the jobs the stopped server left (the reconciler's duties with a zero window, run as the claim's holder so the cycle takes the held branch without reading the claim back through that window: a RUNNING restore and the CREATING target of a FAILED one, an in-catalog CREATING backup, a DELETING row; a target that is no longer the restore target its record names is printed as left; a RUNNING `manifest` record, which no release before PR 9a writes, is left for its resumer and waited on), waits up to twice the stale window for CREATING or FAILED rows with a live owner (a heartbeat within the window on the catalog's clock) and RUNNING restores, deletes each row's prefix and the row outright (a CREATING row with a dead owner included, the row before its prefix), leaves `extenddb-store.json`, `exports/` and prefixes with no row, and audit-logs each deletion; a row a DeleteBackup moves to DELETING or DELETED, or removes, between the command's read and its turn is finished and counted as deleted; on SQLite the owner window is zero (under the serve lock every owner is dead), so a CREATING or FAILED row with a young heartbeat is deleted, not waited on, and a server starting on the file clears a maintenance claim a killed purge left, as a purge run after it takes the claim over; SIGINT or SIGTERM releases the claim at once during the wait and before the next row during the deletion, and the wait's failures (jobs still running, the claim lost, the catalog unreadable, a signal) are each reported as themselves. The dry run is a read like `list` and exits 0, with no store configured as with one; `--config` is accepted after the subcommand; both commands say why there is no store when there is none. On PostgreSQL, MongoDB and Cassandra both commands hold a serving registration named after the command for their run and release it when they exit, so `migrate --yes` does not wait for it to expire, and a catalog the backend refuses ends them with the message and exit code `serve` gives (3 for a catalog behind the binary, with the `migrate --yes` advice; 6 while a migration runs); on SQLite `purge --confirm` beside a running server is refused on the serve lock with exit 6, as a second `serve` is. `extenddb destroy` says it leaves the store and that a re-created catalog needs the old root's identity file removed. Docs: the admin guide (`[backup.store]`, `/health` components and the reason list with `catalog unreadable`, `backup purge`, the shared-root rule), the limits doc's backup store row, the upgrade manual (configure a store before the release that stops the in-catalog create; `disabled` as the opt-out; the SQLite default root rule; the re-init sentence), the deployment guide's multi-instance notes (the hard-link requirement, a root that refuses `link(2)` reported as failed validation, the empty-catalog rule), the init template and sample config, the store crate's README and module docs, the S3 root label built from the endpoint the client uses, in one spelling; the identity file read up to 64 KiB; the console banner naming the store, a filesystem root in the canonical spelling the log line and the rows use (the configured one beside it where they differ); the admin guide's purge paragraph on the commands' registration on the three backends, the SQLite serve lock's exit 6 and the refusal's text; the MongoDB design's startup paragraph; design revision item 28; and the licence notices (the server binaries now link the store crate's AWS SDK tree). Rebased onto PR 6 after main merged #335 and #681: MongoDB's and Cassandra's factories now register through the numbered runner, so the backup commands register there too, under their names, and MongoDB's key repair starts in `build_components`, after `register_and_check`, for a server only; the reconciler runs in #666's restartable worker set, and the PostgreSQL refusal cell forgets migration 6, PR 6's id. Moved with PR 6 onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` sets `beside_server` and `command` as the derived one did, and SQLite's factory opens its engine with #416's `options.max_item_size_bytes`, with the `beside_server` return after it. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, CreateBackup is the store-backed job of the design's 5.1: the request checks the table and the ARN, requires a healthy store, takes a job permit and spawns a detached task that opens the table snapshot, mints the backup id from the snapshot instant, builds the manifest's `source_table` from the snapshot's description and the table's TTL, inserts the CREATING row under the job's claim (`claim_backup`), checks the maintenance claim once the row exists (FAILED `maintenance` and LimitExceededException while one is held) and answers CREATING through a one-shot channel; the task then runs the job: items validated with the new `extenddb_core::validation::validate_restorable_item` (one walk, every `N` parsed once on the accepting path, the key attributes checked with the format's own rule and against the instance's key size limits, every attribute name against its limit, nesting depth and the instance's `max_item_size_bytes`; an item whose data file line would exceed the 32 MiB cap under a raised item limit is refused the same way, its line measured through a counting writer that holds no line, and an item over the default item limit is validated and measured on the blocking pool; every text an error carries is bounded on its own, a key value at 12,288 escaped bytes so a legal key appears whole whatever its bytes, and written as a JSON string's content with the Unicode line separators escaped too, so neither a control character nor a line separator in a tenant's item can add a line to the server's log), encoded by `DataFileEncoder` on a blocking task into 1 MiB chunks over a bounded channel that is the put's body, one data file closed at the first item that takes it to or past `data_file_target_bytes`, `KeyRangeTracker` on a key-ordered table, then the three documents, a claim re-check, the manifest, and the AVAILABLE write with the manifest's size, count and `source_table` (`StatusFields.source_table_details_json`, new on every backend). Every status write is conditioned on the generation and on CREATING; each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read showing the row in the target status counts as matched and anything else as matching no row; a write that matches no row ends the job with no store access; the second write is for a backend whose claim connection survives the unknown outcome (MongoDB, Cassandra): on PostgreSQL it fails with the session and the row is the reconciler's; an upload failure stops the feeder at once, so the FAILED write does not wait for the snapshot stream. A refused item stops the encoding but not the validation: the rest of the snapshot is read and checked, each refused item logged with its key, attribute path and reason beside the ARN (up to 1,000, then a count), and the backup FAILED with the count and the first item, naming failed after a refusal. The heartbeat is a timer through the claim (`max(heartbeat_interval_secs, 15)` s); a beat that is lost, errors or does not return within a period raises the abort, and no put starts unless a beat confirmed the claim within two periods, the claim's insert standing for the first beat; on PostgreSQL a stopped process keeps its row while its session lives and, resumed after more than two periods, stops as a lost claim with no catalog write. On any failure (a put error, a refused item, `SnapshotExpired`, shutdown) the FAILED write comes first and `delete_prefix` only after it matched; `BackupRuntime::drain_jobs` (30 s) cancels and awaits every job's failure path before the registration supervisor's return and the stop of the server's workers, a job still running at that bound keeps the serving registration as a worker that could not be stopped does, and `extenddb stop` waits 76 s, the sum of the server's shutdown bounds (connections 5 s, jobs 30 s, the supervisor, the workers' stop and the registration's release 36 s, runtime 5 s), where it gave up at 10 s; a write into the chunk sink after the stop is `BrokenPipe`, which the encoder does not retry, so a cancelled job ends at once. Without a store, and on Cassandra always, CreateBackup keeps the in-catalog create, with `refuse_backup_of_vector_table` in front of it alone: the store-backed path backs vector tables up. The reconciler's full duty list, one cycle: the maintenance check, the dead restore jobs, the abandoned in-catalog backups, the store's check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over through the new `BackupCatalog::claim_dead_backup(key, stale_after)` (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (`owner dead ...`), never continuing the upload; then, without a maintenance claim, the prefixes of FAILED rows whose heartbeat is stale deleted (the same cycle's for a row it just failed), DELETED prefixes swept, FAILED and DELETED tombstones past `failed_row_retention_secs` removed once their prefix is empty, and Cassandra's `active_restores` entries swept (`sweep_restore_registrations`). The ages are the backend's own predicates (PR 7's `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column); the engine holds no clock. `backup purge`'s own cycle runs as the claim's holder (PR 7's `ReconcilerState::for_maintenance_holder`), so it skips these deleting duties whatever its zero window. A duty's store call that fails records `BackupStoreErrors` under the call's name (`get`, `list`, `delete_prefix`), though the cycle completes. Per backend: PostgreSQL takes the row's advisory lock then a conditional update under `now() - stale`, sets TCP keepalives on the claim connection, and names `[storage.postgres] pool_size snapshot-slot` in its snapshot refusal; SQLite takes the row's in-process slot, so a row whose slot no task holds (a stopped process, a job task that panicked) is dead at once, and raises `pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) so the slot refuses no job the permit admitted; MongoDB's takeover is one `find_one_and_update` on `$$NOW`; Cassandra's an LWT carrying the generation, the status and the heartbeat it read. The snapshot streams take a byte budget: PostgreSQL's cursor is replaced by keyset pages in the same transaction, three layers (`LIMIT n` after the key, the running byte sum, the cut at `SNAPSHOT_FETCH_BYTES` = 64 MiB) with the key columns under qualified aliases and an outer `ORDER BY`, `n` 16 for the first page and then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows, never under 4 (the server serializes every row the inner `LIMIT` reads, so a page costs it `n` rows of text; sized from the average, the page after a mixed page read 810 large rows for the 3 it returned; sized from the previous page alone, the bound oscillated on a table whose large rows are scattered among small ones, 5,555 rows read for 1,000 kept, where the memory reads 1,901; the page that follows a row bound's worth of small rows still reads up to `n` rows of up to the line cap, which is the bound the record states; a walk that stops at the budget on the server was measured and not taken, one index probe per row costing a 1M-row backup 95% of its duration), the rows parsed as they arrive, the stream ending when fewer rows than asked came back and none was cut; SQLite pulls rows one at a time and stops after the row that reaches the budget, with the pool's `row_buffer_size` at four so the driver's worker reads at most five rows past it, the four it buffers and the one it holds (its default of 50 read a whole table of 27.7 MB rows ahead of the first page). `snapshot_conformance::run_page_bounds` checks that no page holds more than the budget plus one row and that the cut fired, on both backends and fails at once on a stream that repeats rows; an `EXPLAIN` test pins the index scan on PostgreSQL on the six key shapes a table can have, a second the bitmap scan and `Sort` the planner chooses near the end of a small table, and a third the row bound's memory on a table with 27 MB rows every 33rd key. Metrics: `BackupJobs` and `BackupJobDuration` (the outcome in the operation dimension; a FAILED job's duration covers its FAILED write and `delete_prefix`), `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors` (the job's `put` and `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`), `QuerySource::BackupJob`. Kill points behind `extenddb-engine/backup-test-hooks` (`EXTENDDB_BACKUP_PAUSE`); `extenddb version` prints `backup-test-hooks enabled` on such a build. Harness and CI: `_create_backup` and `_cleanup` wait for AVAILABLE; the vector wire test reads `EXTENDDB_TEST_BACKUP_STORE` and fails unset; the restore cells skip on a server with a store until PR 9a, as does `test_auth_permissions.py`'s restore-authorization cell, whose backup cleanup waits for AVAILABLE; the fidelity suite checks the server's `backup_store` health component against the variable; `tests/test_backup_job.py` (the job bound, the four #618 write paths, the oversize item, a refused item named by its whole key) and `tests/test_backup_store_config.py` (the create refused on an unhealthy store, the objects under the root) run by name on the PostgreSQL and SQLite jobs; the PostgreSQL job appends a filesystem store after `init`; a SQLite job against an S3 store on moto, whose cache is saved only on main and whose actions are pinned to main's commits, as every other job's; the start steps and `devtools/run-tests`' health check wait for a healthy store; `devtools/run-tests --pytest` refuses without the variable (its `--help`, `CONTRIBUTING.md` and the developer test guide name it); the Rust integration suite waits for AVAILABLE after every create it goes on to describe, delete or restore, deletes its backups and reads the variable the same way, and every Rust integration job sets it (`filesystem` on the SQLite job, whose server has the defaulted root; `none` elsewhere); `tests/test_abac.py`'s backup cell waits for AVAILABLE before its delete and `tests/test_authz_resource_arn.py`'s fixture deletes its backup and table when the module ends. Docs: the differences document (vector, restore-interruption, where backups live, backup creation, consistency, a #618 row), the limits document (the slot and pool raise, the store row, a data-file row, the contents row), the usage guide's backups line (the restore of a store-backed backup is a later release's), the upgrade manual (the asynchronous create, #618, the rollback paragraph), the admin guide's stop paragraph and the architecture guide's shutdown step (the job drain before the supervisor, the workers and the release), `[backup]` and `[backup.store]` (with `MALLOC_ARENA_MAX=1` for hosts that back up large items, which costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%)) and the deployment guide's service unit (the line commented out with that condition), the troubleshooting entry for `stop`'s timeout, the sample and init templates, the measurement record as `docs/design/15-backup-restore-bench-pr8.md`, design revision item 29, which makes section 7's memory paragraph the sizing rule the worst-expansion runs measured (the terms at the line cap times the allocator's factor, flat under one arena on both backends and growing with the large rows under glibc's default arenas, stated at the default item limit) and records the item channel kept at four, PostgreSQL's outcome for a stopped and resumed job (5.1 step 4, `heartbeat.rs`), the second status write's scope (4(e), `failure.rs`) and the reconciler's `BackupStoreErrors` (section 9). Rebased onto main through PR 7: main merged #335, #666 and #681 under the stack, so the catalog migration this PR's columns come from is 6, the job drain runs in #666's shutdown path (this PR's worker drain is gone, the reconciler being one of the workers the supervisor stops and starts) and `stop` sums that path's bounds, and the Cassandra Rust job's `none` sits on #681's `cassandra-rust` job. Moved with PRs 6 and 7 onto main at e8ca7df: SQLite's factory opens `new_for_server` with #416's `options.max_item_size_bytes`, and the item validator's module doc, the differences document's #618 row, the upgrade manual and the design's Drawbacks name a second class of item the backup refuses that earlier builds stored: an attribute name over the limit that UpdateItem and a TransactWriteItems Update did not check before #433. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Nov 5, 2026
Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, the `[storage.sqlite] pool_size` snapshot-slot bound, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 6` is the migration. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore checks the target name through `describe_table` after the ARN (a table of the name `TableAlreadyExistsException`, a CREATING restore target `TableInUseException` naming its source backup), then the backup, then takes the permit last and hands it to the backend; the backend's fenced name check under the registration stays. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 6 (`0006_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 6 as a code body (`m0006_backup_catalog_index.rs`) with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock, so a beat queues behind every writer holding or waiting for that lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays beside `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the `[storage.sqlite] pool_size` snapshot-slot bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections and a `restore_job` subdocument on a restore target's `tables` document, created by migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`), which also backfills the registrations and writes the catalog id; every request it sends is safe to send again (an existing collection accepted by its error code, the backfill limited to registrations without a job record, the id an upsert with `$setOnInsert`). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores and the startup sweep are removed. The factory registers and reads the catalog's status (`register_and_check`) before it builds the engine or starts #473's startup key repair, so a server that refuses a catalog without migration 6 has written nothing; a live test pins that refusal on scratch databases holding a document the repair would rewrite. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. Migration 6, a code body on the numbered runner (`m0006_backup_catalog_index.rs`): it executes `migrations/catalog/0006_backup_catalog_index.cql` as the runner executes a file body (the lease checked before each statement, schema agreement after each DDL statement), then backfills the registrations with lightweight transactions conditioned on the job status being unset and inserts the catalog id `IF NOT EXISTS`, so a run that fails part way leaves migration 6 unrecorded and the next `migrate --yes` runs the body again. The file adds the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, and `backup_maintenance`, every statement an `ALTER TABLE ... ADD IF NOT EXISTS` or `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace; the backfill and the id name the keyspace in their own statements, and no code or test issues `USE`. Code bodies get the lease check and the file execution on `CassandraStores` (`check`, `progress`, `run_file`, which moves there from the runner). An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration. One change, one number: migration 6, `backup_catalog_index`, on every backend, recorded in each store's ledger (migration 5 is servers first; the previous release refuses a catalog carrying it, exit 4) and convergent; a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Rebased onto main after #335 and #681 merged: the migration moved from id 5 to 6, Cassandra's V006 file and MongoDB's versioned step became migration 6 code bodies on #681's runner, the `0.0.5` catalog version and this PR's own serve-time version checks went (the runner's status check refuses the catalog on every backend), and the PITR code #335 removed is gone from the legacy reader. Moved onto main at e8ca7df: #416's hand-written default for `ServerComponentsOptions` gains `max_concurrent_jobs: 0`, and the differences document's backup rows keep #509's text beside this PR's. Docs. The upgrade manual (migration 6 and its rollback, with the per-backend objects and the Cassandra and MongoDB rerun sentence, and the SQLite startup sentence: the server ages every RUNNING restore record and the first reconciler cycle fails an in-catalog restore and removes its target), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 7, the sample config, `docs/schema/postgres.txt` (regenerated for the collation), `sqlite.txt`, `cassandra.txt` and `mongodb.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration number as implemented, the interface block with `list_dead_restore_jobs`, `fail_restore_job` and `abandoned_creating_backups`, revision log items 26 and 27). Tests. Counts on this tree; added and removed are test functions against main. The MongoDB and Cassandra counts are listed on this tree (`--list`); their suites ran on the top of the stack. extenddb-storage 215 (13 added; 216 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 added), extenddb-engine 255 (25 added: the target name before the backup and the permit, catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 75, tests/backup_restore 37 (27 kept, 1 rewritten, 2 removed, 9 added), every other binary green (fresh 6, released 12, golden 7, crash 9, guards 7, runs 7, owners 3, serving 5, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 248 (17 added and 4 removed, among them a target removed only under its full key and migration 6's golden test; 1 ignored benchmark). extenddb-storage-mongodb: lib 165 plus 46 live tests against a replica set (among them a catalog without migration 6 refused with no repair started), migrations_fresh 8, guards 7, holders 10, locks 10, released 5 (migration 6's golden test added), interpreter_selftests 18, pushdown_parity 20; 29 added and 27 removed with the slots, the sweep and the claim markers. extenddb-storage-cassandra: lib 128 (4 added, 1 removed), migrations 21, migrations_stopped 8 (migration 6's golden test added), direct_integration 205 (13 added, 1 rewritten, 1 removed; three of the added tests build with `--features test-hooks`), ttl_integration 26 passed and main's one failure below. The released-catalog fixture of tests/test_resource_tag_migration.py strips migration 6's objects before it clears the ledger, since the check after migration 1 refuses a `table_restores` carrying the job columns. Through the binary on every backend: a catalog at main's migration 5 is refused by `serve` (exit 3), migrated by `migrate --yes`, then served, and main's binary refuses it (exit 4); a fresh install's schema equals `docs/schema/<backend>.txt`. The MongoDB workflow's jobs pass the CI way (pytest 1636 passed, 50 skipped; comprehensive 502; `tests/rust` 514; the catalog checks), and so do the Cassandra workflow's jobs (`tests/rust` 514, comprehensive 502, the catalog-lock, older-binary, two-node and authorizer checks), apart from the pytest job's parallel leg on the review host, whose 30 s request timeouts on account creation main's binary meets the same way there and whose files pass run serially. Wire fidelity (tests/test_backup_restore_fidelity.py, 36 passed and 1 skipped) ran inside those pytest jobs on MongoDB and Cassandra; the PostgreSQL and SQLite runs (64 and 63 passed) are from the tree before its rebases. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure is main's: the test and `metadata_engine.rs` are as on main (the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reports point-in-time recovery as unsupported, served by the engine on all four backends, as section 6 of the backup and restore design (
docs/design/15-backup-restore.md) sets out for PR 1:DescribeContinuousBackupsreturnsContinuousBackupsStatus: ENABLED(the service reports this unconditionally) withPointInTimeRecoveryStatus: DISABLEDand no other member.UpdateContinuousBackupswithPointInTimeRecoveryEnabled: truereturnsContinuousBackupsUnavailableException(HTTP 400,__type com.amazonaws.dynamodb.v20120810#ContinuousBackupsUnavailableException), the exception the service models on that operation. Withfalseit returns the DISABLED description.RestoreTableToPointInTimechecks the source table and then returnsPointInTimeRecoveryUnavailableExceptionwith the service's message, "Point in time recovery is not enabled for table ''".TableNotFoundException, "Table not found: ", on all three operations.DescribeContinuousBackupsandUpdateContinuousBackupsansweredResourceNotFoundException.The request checks before those answers, and their order, follow the service (measured on DynamoDB, two samples of each request):
TableNameon the two continuous-backups operations, or noSourceTableName(orSourceTableArn) orTargetTableNameon the restore:ValidationException, "The parameter 'TableName' is required but was not present in the request".TableNameon the two continuous-backups operations is theValidationExceptionthat lists thetableNamelength and pattern constraints ("2 validation errors detected: ..."), checked before the table is looked up; the length bounds are the deployment's. ATableNamethat starts witharn:is looked up as before (see Related behaviour).UpdateContinuousBackupswithoutPointInTimeRecoverySpecification, or with one that does not holdPointInTimeRecoveryEnabled:ValidationExceptionnaming the missing member, checked before the table is looked up. It was treated as false. The flag given as the JSON string"true"or"false"is taken as the boolean; the service answers 200 for"false".SourceTableArnnames the source asSourceTableNamedoes; an index ARN there is "Invalid Request: Table ARN is invalid."; both members together are "Invalid Request: Must provide exactly one of: sourceTableArn, sourceTableName". ASourceTableNameoutside the length bounds (an empty one, for example) is "TableName must be at least 3 characters long and at most 255 characters long", with the deployment's bounds in the text, and one that fails the[a-zA-Z0-9_.-]+pattern, such as a table ARN, is theValidationExceptionnaming that pattern atsourceTableName; both come before the source table is looked up. Neither time member, orUseLatestRestorableTimefalse without aRestoreDateTime, is "Invalid Request: Either one of RestoreDateTime or UseLatestRestorableTime must be specified, but not both"; both is "Invalid Request: Both RestoreDateTime and UseLatestRestorableTime cannot be set ...". The target table is never looked up: the service makes the recovery check first, and here it refuses every restore.The three handlers live in
crates/engine/src/backup/pitr.rs.BackupEnginelosesdescribe_continuous_backupsandupdate_continuous_backups, with their implementations on all four backends, and the per-tablepitr_enabledflag they stored, which only the description read and no restore ever used, loses its storage:004_drop_continuous_backups.sqldrops thecontinuous_backupstable and writes catalog 0.0.5.extenddb migratebrings a migrated database to the fresh shape.V006__drop_continuous_backups.cqldrops the table and writes 0.0.5. The migration runner acceptsDROP TABLE IF EXISTS, qualified with the keyspace like the other statement forms; a bareDROP TABLEis still refused. DeleteTable, account deletion and the DELETING-table worker no longer write to the table.continuous_backupscollection, andextenddb migrate --yesdrops it from a catalog an earlier build created;extenddb migratewithout--yesreports it ascatalog: drop the continuous_backups collection. The catalog version stays 0.0.2, andservedrops nothing.Documentation: the differences doc's point-in-time recovery row and the limits doc's row describe the new surface and point at on-demand backups; the limits doc's shard iterator row now says the 15-minute expiry is enforced, which it is. The upgrade manual no longer states which catalog version is current or what each migration file does: the tooling reports the installed and expected versions (
extenddb verify,extenddb migratewithout--yes, and the startup mismatch error, quoted with placeholders), the migration files' headers document their purpose, and the point-in-time recovery change is under Behavior Changes by Release with whatextenddb migratedoes on each backend. The architecture guide loses its catalog version literals the same way. The manual's version history has a catalog 0.0.5 entry that points at the Behavior Changes entry, and that entry says what an earlier build does against a migrated catalog on Cassandra and MongoDB. The usage guide's feature table gives on-demand backups and point-in-time recovery a row each, the SQLite design notes and the MongoDB storage design stop listing the dropped table and collection as current, and the TODO index loses the point-in-time recovery item.Why
Point-in-time recovery reported a capability that did not exist.
UpdateContinuousBackupsaccepted an enable request and stored a flag,DescribeContinuousBackupsthen answered ENABLED with a 35-day window, andRestoreTableToPointInTimeanswered every request with a genericValidationException. A recovery window that cannot be restored from is worse than a refusal the caller can act on. This is PR 1 of the backup and restore series.Rebase on main
Rebased onto
f49c5f823. The PR's own code changed because of these main PRs and the design:restore_table_to_point_in_timefromBackupEngine, and addedTableNotFoundExceptionfor CreateBackup. The PR's drop is now migration 004 and catalog 0.0.5 on PostgreSQL, SQLite and Cassandra; the PR's ownTableNotFoundExceptionvariant is dropped in favour of the existing one. fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 also brought the migration runner's write of the compiled catalog version after every walk, which this PR's third commit used to add; that commit now carries only its tests (a replay of the final migration, and a check that the replayed deployment serves) and the upgrade manual's line about the convergence.continuous_backupstable and its storage-level implementation, which this PR now removes with catalog migration V006.SnapshotEngine) and fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 rewrote the backends' backup files, so the removals were redone against the new files.docs/design/15-backup-restore.mdon main, so this PR no longer carries it or its README line. Its section 6 asks forTableNotFoundExceptionon all three operations, aValidationExceptionfor a missingPointInTimeRecoverySpecification, and MongoDB's stale collection dropped byextenddb migrate --yes; the rebase adds all three, with the request checks above.Merging
These open PRs take the same numbers, and whichever lands second renumbers:
004_control_chars_encoding.sql.V006__backup_catalog_index.cql; its body already says that whichever of PR 1 and feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671 lands second takes the next numbers.Related behaviour
Measured on DynamoDB, with ExtendDB's answer in parentheses:
TableNameonDescribeContinuousBackupsorUpdateContinuousBackupsresolves the table (TableNotFoundExceptionnaming the whole ARN). An index ARN or a malformed ARN there is aValidationExceptionabout the ARN format (the sameTableNotFoundException). A table ARN of another account isAccessDeniedException(the sameTableNotFoundException).SourceTableArnis "One or more parameter values were invalid: Fifth colon (namespace/relative-id delimiter) not found: " ("Invalid Request: Table ARN is invalid."). ASourceTableArnof another account is "This action is only supported by accounts that match the resource owner's account." (the account in the ARN is ignored and the caller's own table of that name is looked up, soPointInTimeRecoveryUnavailableExceptionwhen it exists).RestoreTableToPointInTimeondynamodb:Scanfor the target table, and ExtendDB ondynamodb:RestoreTableToPointInTimefor the source table (from the order ofauthz_targets, not run).Testing done
cargo +1.97.0 fmt --all --check: clean.cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings: clean.cargo +1.97.0 check --locked --workspace --all-targetsandcargo +1.88.0 check --locked --workspace: clean.cargo +1.97.0 test --locked --workspace --no-fail-fast --exclude extenddb-backup-storewith no database configured: 2,534 passed, 0 failed, 59 ignored, across 47 test binaries including the doc tests. The library tests in that run:extenddb-core715,extenddb-engine193,extenddb-storage139,extenddb-storage-postgres63,extenddb-storage-sqlite162,extenddb-storage-mongodb152 (44 ignored, the live MongoDB tests),extenddb-storage-cassandra76.extenddb-backup-storeis left out because its conformance tests refuse a backup root whose parent directories are not owned by root or by the extenddb user, and on the machine these ran on the temporary directory sits under a directory owned by another user; its library tests passed 33 of 33.cargo test -p extenddb-storage-cassandra --lib, 76 passed, and--test direct_integration -- migrations, 2 passed. WithV006__drop_continuous_backups.cqlleft out of the migration list, the new library test and both migration tests fail; with the drop step removed from the MongoDB bootstrapper, the new MongoDB test fails.tests/test_continuous_backups.pyagainst a PostgreSQL and a SQLite server built from this branch: 13 passed on each. Its three newest tests (the empty names, a table ARN asSourceTableName, and the flag as the string"false") fail on a SQLite server built from the previous revision of this branch.04468b4c7.cargo test -p extenddb-storage-postgresagainst PostgreSQL 16: every test binary passed, includingcatalog_converge(5),backup_restore(30),table_snapshot(81) and the documentation version guard (3).test_cli_migrate_concurrency.py,test_cli_vector_catalog_migration.py), which refuse to run without a PostgreSQL connection string and passed on the PostgreSQL run.backup_restoretests against SQLite and PostgreSQL servers: 13 passed on each.test_continuous_backups.py, then 10 tests, passed on each, and on MongoDB the backup and restore pytest files gave 49 passed, 1 skipped (a vector index case). Upgrade on both from a main deployment with point-in-time recovery enabled on a table: on MongoDBextenddb migratereportscatalog: drop the continuous_backups collectionand exits 3,--yesdrops the collection, and the catalog stays 0.0.2; on Cassandra it reportscatalog 0.0.4 -> 0.0.5and exits 3, and--yesapplies V006, after which the catalog reads 0.0.5 and the table is gone. The branch then reports the table DISABLED and refuses enabling on both.test_continuous_backups.py, then 10 tests, against a SQLite server built from main: 8 fail, one for each answer this PR changes; the two that pass are the describe shape and disabling, which main already answers this way.extenddb migratereportscatalog 0.0.4 -> 0.0.5andcatalog 004_drop_continuous_backups.sqland exits 3;--yesapplies it), reads back 0.0.5 withcontinuous_backupsgone; the branch then reports the table DISABLED and refuses enabling, and the main binary refuses the migrated catalog withCatalog version mismatch: expected 0.0.4, found 0.0.5. Run 'extenddb migrate'. The same on a file-backed SQLite database, whereextenddb migratereportscatalog 0.0.4 -> 0.0.5.Checklist
cargo test --workspace --exclude extenddb-backup-store, and that crate's library tests; see Testing done)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC:
docs/design/15-backup-restore.md, section 6 (Point-in-time recovery in v1) and PR 1 in its planBreaking changes
BackupEnginelosesdescribe_continuous_backupsandupdate_continuous_backups. Out-of-tree backends implementing them must drop them; the engine now answers the three point-in-time recovery operations without consulting storage.UpdateContinuousBackupswithPointInTimeRecoveryEnabled: truepreviously succeeded andDescribeContinuousBackupsthen reported ENABLED with a 35-day window; both now report DISABLED and the enable request fails withContinuousBackupsUnavailableException.RestoreTableToPointInTimemoves from a genericValidationExceptiontoTableNotFoundExceptionfor a missing source table andPointInTimeRecoveryUnavailableExceptionfor an existing one. A missing table isTableNotFoundExceptioninstead ofResourceNotFoundExceptionon the two continuous-backups operations, andUpdateContinuousBackupswithoutPointInTimeRecoverySpecificationis refused. Clients or templates that enable point-in-time recovery (for example Terraformpoint_in_time_recovery = true) now receive an error and must stop requesting it.continuous_backupstable is dropped), and MongoDB's stale collection is dropped byextenddb migrate --yeswith its version unchanged. Existing PostgreSQL and SQLite deployments must runextenddb migratebefore the new binary serves; Cassandra refuses backups until it has. See the upgrade manual.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.