[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework (merged) · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · #700 the CreateBackup job · #740 the restore job, the cross-backend matrix and backup remove-target · #825 backup sync, --prune-missing, --adopt-store · this PR: the CI filters, the open measurement cells and the backup and restore manual · next: #820 the store-maintenance options, #821 migrate-legacy, #748 export and import through the store.
Closes #747 (part of #624).

Stacked on #825 (feat/backup-sync, head 358ff8bd), opened as a draft against it. The diff is one workflow file, documentation, and one source change: the help text of init --data-db and --catalog-db (crates/app/src/cmd_init.rs), which named the old defaults. The binaries every figure and check below used are release builds of this tree (358ff8bd plus this diff); the help text is the only source line the diff changes, so builds before and after it serve the same requests.

What

CI filters (.github/workflows/integration.yml). The PostgreSQL job's and the SQLite job's "Run integration tests" steps drop --filter "not import_export". History: the filter came with the PostgreSQL job's first version (687e46f, "ci: add integration test workflow with PostgreSQL") and was copied into the SQLite job (556c501); neither commit gives a reason. A foreground server wrote a PID file in every tree up to e0d6a0a (crates/bin/src/cmd_serve.rs at 687e46f, crates/server/src/serve.rs at e0d6a0a^), so stop could signal the job's server. e0d6a0a made foreground mode write no PID file by default and added --write-pid-file to the job's start in the same commit. 556c501 came after e0d6a0a (git merge-base --is-ancestor e0d6a0af 556c501e), and its SQLite job passes the flag. The MongoDB and Cassandra jobs already run tests/test_import_export.py unfiltered (--pytest --comprehensive --parallel). What the expression selects out (collect-only, Python 3.10, the job's own ignore list): the 10 tests of tests/test_import_export.py and nothing else (0 of 502 comprehensive, 0 of the 52 CLI lifecycle tests, 0 of the 18 backup-store step tests). Run the job's way (release build, init, serve --foreground --write-pid-file, the job's environment, run-tests --extenddb --pytest --parallel --filter import_export): the main pytest step passed 10 of 10 on SQLite and on PostgreSQL. That command exits 1, because its later steps select nothing under the filter (on SQLite the backup-store step's 18 tests are deselected, on PostgreSQL the CLI lifecycle list's 52) and pytest exits 5 there. No test fails, so both filters go and no narrower filter is kept.

The open measurement cells (docs/design/15-backup-restore-bench-pr11.md, new; row 15d of docs/design/README.md beside 15b and 15c). In the PR 9 record's method: release builds, a filesystem store, [backup] max_concurrent_jobs = 2, the table loaded through the wire from four processes of eight threads, the script's control_plane_delay_seconds = 0.05 and index_propagation_delay_ms = 0, the server restarted before each phase, VmHWM read one second after the start and one second after the phase and sampled every 200 ms. The plain items' 200-byte pad is one repeated character, which the record states. Host: 16 cores (Xeon Platinum 8124M), 30 GiB, Linux 5.10, glibc 2.26; PostgreSQL 16.15 with pgvector 0.8.7, MongoDB 7.0.43, Cassandra 4.1.12, each in a container on loopback; one run each, load average beside each figure.

Cell	Duration	Peak over idle
MongoDB restore, 1M × 200 B (own backup, unordered)	33.81 s	51.8 MiB
Cassandra restore, 1M × 200 B (SQLite backup adopted with backup sync)	48.99 s	33.1 MiB
PostgreSQL vector table, 1M × 128 dimensions: backup / restore	243.17 s / 246.41 s	104.2 / 111.1 MiB
SQLite vector table, 1M × 128 dimensions: backup / restore	209.72 s / 214.23 s	137.6 / 552.1 MiB
the same under MALLOC_ARENA_MAX=1	222.48 s / 282.86 s	57.8 / 203.9 MiB

The 100k cells for each shape are in the record. From 100k to 1M the restore's peak moved +16.7 MiB on MongoDB, +13.2 on Cassandra, +10.5 on PostgreSQL with a vector index, and +198 MiB on SQLite with a vector index under the default arenas (the 1M run levelled off after about 150 s; no other size was measured) against +8.6 MiB under one arena. One arena cost that restore 32% in duration. A whole-table SearchVectors over the 1M restored vectors answered in 0.24 s on PostgreSQL and 14.7 s on SQLite (an exact scan). No cell is left open: the 1M Cassandra restore finished. It ran on a Cassandra 4.1 node of its own with the Cassandra integration workflow's service settings (PasswordAuthenticator, MAX_HEAP_SIZE=1024M), under the Cassandra lock, removed afterwards: the shared node on this host runs AllowAllAuthenticator, where extenddb init cannot create its role, and it was not reconfigured. The record says the node was used by these runs alone.

The record's "Later runs on the same host" section holds a second set of runs on release builds of 30553b3 (this tree's sources before the help-text line), on the same host, with a separate harness and the server's default settings. They reproduce every cell in kind (within 12% on the peak, 20% on the durations; the SQLite 1M vector restore 522.8 MiB, 218.7 s) and add:

• SQLite restores of 100k items with a 128-number list and no vector index: 407.5 MiB over idle (184.0 MiB under one arena), against 373.3 and 376.5 MiB with the index. The memory is the items' lists of numbers, not the index. 20,000 items of 1,536 numbers with a vector index: 668.6 MiB, and under one arena 301.0 MiB (load 1.6) and 315.9 MiB (load 2.8, the run on this tree that confirms it), 48.9 and 48.4 s; their backup 520.9 MiB. On PostgreSQL the index adds about 50 MiB (112.5 against 60.0 MiB at 100k) and 47% of the restore's duration (29.7 against 15.6 s); on SQLite about 20% (22.7 to 23.3 against 18.3 s).
• Two 100k vector restores at once: 24.9 s and 696.7 MiB under the default arenas, 57.5 s and 278.9 MiB under one arena.
• The SQLite 100k vector restore's peak over five runs: 354, 373 and 377 MiB unpinned, 409 and 376 MiB pinned to two cores (up to 16% apart); the record no longer says the peaks do not depend on load.

The manual (docs/manuals/14-backup-and-restore.md, new), for operators and derived from the design, which remains the record. Its first sentence: a filesystem store keeps the backups in the host's failure domain, so S3 is the production recommendation. Sections: strategy (what a table backup holds and does not: the catalog, scheduling with a crontab line that escapes % and names AWS_CA_BUNDLE, retention, PITR; Cassandra's in-catalog create); RPO and RTO per store kind (RPO equals the backup interval, no PITR; the restore and backup durations and the memory figures from the three records and the later runs, with the host class and what was not measured; MALLOC_ARENA_MAX=1 recommended on a SQLite host that restores tables whose items carry long lists of numbers, with or without a vector index, with its duration cost for one job and for two); configuration (filesystem and S3 examples, the root that must exist, pointing at the admin guide's [backup] and [backup.store] rows and crates/backup-store/README.md); securing the store (including that the check does not probe kms:Decrypt); what a running backup costs each database; a restore drill (with the copy modes an off-host drill needs); catalog-loss recovery with backup sync (a catalog restored from a dump, a re-created catalog with --adopt-store in both SQLite shapes, the in-memory startup sync and the accounts it needs re-created); legacy backups (they stay restorable; migrate-legacy is #821, so the manual gives today's path: restore, back up again with a store, check, delete); what a restore preserves (the table the differences document carries, with its one gap); and moving data to and from Amazon DynamoDB (the format is the S3 export layout; the service's ImportTable parameters and that it creates no LSI; ExtendDB's ImportTable reads one uncompressed local file named by its canonical path; the export's FilePath; a store-prefix import is #748, not in this release).

Each statement of the design's PR 11 acceptance is in the manual: the first-sentence failure domain and S3 recommendation; encryption at rest is the store's job; a copy tool runs as the server's user or as root, and a root under a storage administrator's directory, an NFS mount owned by nobody or a group-writable parent without the sticky bit is refused; a bind mount or a filesystem without Unix permission semantics is outside what the store observes, and a root mounted with a mask that reports 0700 enforces nothing; AbortIncompleteMultipartUpload, and for a versioned bucket NoncurrentVersionExpiration and ExpiredObjectDeleteMarker; the least-privilege IAM policy (pointed at); a PostgreSQL snapshot retains every dead row version and the SQLite WAL grows during a backup, with figures measured for the manual (below); the claim connection's keepalive bound (30 s idle, three probes 10 s apart: about a minute) and pg_terminate_backend on the claim's session, with the query that finds it by its lock key in the catalog database; a transaction-mode pooler is unsupported; the SDK's trace logs carry the session token; write access to an account's prefix is backup-administrator authority for that account, with per-account IAM prefix policies; MongoDB's minSnapshotHistoryWindowInSeconds.

Figures measured for the manual, same host class, release builds, 100k-item vector tables as the backup's source: on PostgreSQL a 25.2 s backup beside 12,156 PutItem overwrites of 1,000 200-byte items in another table grew that table from 0.4 to 4.4 MiB, and a VACUUM (VERBOSE) 10 s into the backup reported 4,948 row versions "dead but not yet removable"; the same writer for the same time with no backup left the table at 0.5 MiB, and the first VACUUM after the backup removed every dead version. On SQLite the WAL grows by the pages each write changes, which depends on the table's indexes. A 22.8 s backup beside 15,472 such writes to a table with no secondary index grew the -wal file from 4.0 to 61.1 MiB, about 3.9 KB per write. The same writer on a table with two GSIs grew it from 4.1 to 1,294.1 MiB over 13,996 writes in a 25.6 s backup (96.6 KB per write), 58.7 to 1,277.6 MiB over 13,953 writes in a second run on a WAL already grown (91.6 KB), and 4.1 to 1,261.8 MiB over 13,891 writes in 26.0 s in the run on this tree (94.9 KB, load 3.7). With no backup the writers left it at 4.0 and 4.1 MiB, and the file kept its size after the backup and after the server stopped. The manual's sizing rule names the indexes: about one 4 KiB page per write for a table with no index and about 24 for a table with two GSIs, since an asynchronous GSI write goes through gsi_pending and is applied in later transactions.

Binary checks of the manual's process-level statements (release builds of this tree, SQLite and PostgreSQL unless named):

• A root under a 0775 parent: startup error, "has mode 0775 and is writable by group or others without the sticky bit"; a 0755 root: "chmod it to 0700 (on a filesystem without Unix permissions, mount it with a mask that yields 0700)"; a 0700 root under a 1777 parent starts healthy; an absent configured root stops startup, "backup root is not usable: No such file or directory (os error 2)".
• The drill on SQLite and on PostgreSQL: CREATING from both requests, AVAILABLE, ACTIVE, the count equal; backup list beside a running server. An off-host copy with 0755/0644 modes makes the drill deployment's store unhealthy and refuses the restore; with the modes kept the copy adopts and restores.
• Catalog loss on SQLite with [backup.store] configured explicitly: the database removed and init run again; /health reports "reason":"owned by another catalog", degraded; DescribeBackup is BackupNotFoundException; plain backup sync exits 1; --adopt-store adopts (its output quoted in the manual), the store is healthy and the backup restores. Plain sync on an owned root reports the prefix as having a row.
• Catalog loss on SQLite with the default root (no [backup.store] section): the server logs "the default root ... failed its check (extenddb-store.json names catalog ; this catalog is ); the catalog has no backup rows for it, so the store is unconfigured and CreateBackup writes in-catalog backups", /health is {"components":{},"status":"healthy"}, and a CreateBackup answers AVAILABLE with the in-catalog warning. The account re-created with manage create-account --account-id 123456789012, its user and policy, and the old key pair imported with manage import-access-key ... --yes: the old key pair signs requests again. --adopt-store beside the running server exits 6 ("another extenddb process is already using ..."); with it stopped, plain sync exits 1 and --adopt-store exits 0; restarted, backup_store is healthy, the old backup and the in-catalog one are AVAILABLE, and the old one restores with 300 of 300 items.
• The in-memory server: the startup sync's log line, DescribeBackup AVAILABLE in the new catalog once its account exists, the restore, backup list refused with exit 1. On the sqlite-memory,dev-mode build: a backup taken in the first start's default account (995720651603); the second start's default account was 823675310910, ListBackups listed nothing and DescribeBackup answered AccessDeniedException after the line "startup sync of the in-memory catalog: 1 backup(s) adopted"; after manage create-account --account-id 995720651603, a user, a policy and a key, DescribeBackup answered AVAILABLE and the restore reached ACTIVE with 300 of 300 items.
• kind = "disabled": the in-catalog create answers AVAILABLE and logs the deprecation warning; with the store back, ListBackups lists both kinds, the legacy backup restores, the restored table backs up to the store, backup list lists only store rows; backup migrate-legacy is an unrecognized subcommand.
• An S3 store at moto (a bucket of its own, removed): healthy, the objects under <prefix><account>/<table>/<id>/ beside extenddb-store.json, a restore through backup-spool/ beside the database file.
• backup sync beside a running SQLite server exits 6 ("another extenddb process is already using ...").
• ImportTable of a backup's data file: the gzip file is refused, "I/O error reading import file at line 1"; decompressed, 1,000 of 1,000 items imported. A FileSource.Path through a symbolic link (/home to /local/home on this host) is refused, "Path must resolve under one of the configured allowed paths", whether the [import] paths entry is written through the link or not; the canonical path imports with either entry. Export writes one uncompressed {"Item": ...} file under the account's directory, at FilePath, or at S3Prefix when FilePath is absent; S3Bucket alone is refused, "Either FilePath or S3Prefix must be provided".
• manage create-account --help names --account-id; manage import-access-key --help names --account-id, --user-name, --access-key-id, --secret-access-key and --yes; init --help gives the two new defaults.
• PostgreSQL init --catalog-db <name>_catalog creates the data database <name> and records data_database_name. The claim session is found by hashtextextended('backup:<backup id>', 4539474) in pg_locks (and a restore's by <account id>/<table>), distinct from the extenddb serve registration session; with two catalogs on one cluster holding the same key, the query without a database predicate returned both sessions, and with l.database limited to the current database (as the manual gives it) only its own. Terminating the claim stopped the job 13 s later at its heartbeat ("stopped with no further store or catalog access"), and with stale_job_after_secs = 45 the reconciler failed the row and deleted its partial objects 47 s after the termination. Terminating the registration session as well (a first run that ended every advisory holder) makes the server answer ServiceUnavailable while it registers again, which the manual warns against.
• A SQLite catalog restored from a copy taken before its newest backup was deleted: /health "reason":"manifest probe failed", CreateBackup and the restore of an older backup refused, plain sync and --prune-missing exit 1 ("sync changes no row while the store is unhealthy"), still unhealthy after a restart. The manual says this release has no command that clears it without a copy of the prefix, and to take the catalog dump after deletions.

Other documentation. The admin guide's Data Recovery entry named the PostgreSQL data database extenddb_catalog_data; init derives it from the catalog's name without _catalog (extenddb by default) and records it in settings as data_database_name. The entry now has a table of each backend's databases, says that a catalog name without the suffix needs --data-db, and gives the pg_dump lines with the server stopped (each pg_dump takes its own snapshot of one database, so two dumps beside a running server disagree when a table is created or deleted between them), or a base backup of the PostgreSQL server holding both. The same wrong name is corrected in the architecture, design and security-model guides; the quickstart's --data-db row gave <catalog>_data and now gives the derived name; the design guide's settings key was data_database_url and is data_database_connection_string (and data_database_name); init --help said "Data database name (default: extenddb)" and "Catalog database name (default: _catalog)", and now says "the catalog's name without its _catalog suffix" and "extenddb_catalog, or the config file's" (bootstrapper.rs: the catalog never derives from --data-db). git grep '<catalog>_data\|<data-db>_catalog\|data_database_url' finds nothing. The admin guide's [backup] section and the upgrade manual's backup paragraph point at the manual; the upgrade manual's shared-root and re-created-catalog sentences move to the manual, which states both. The admin guide's max_concurrent_jobs row adds the vector restore's figures on SQLite and points at the manual's. The deployment guide's PostgreSQL checklist has "Configure [backup.store] (S3 for production) per the Backup and Restore manual", and its backup mentions and service-unit comment point at the manual. The differences document's restore row gives the 1M Cassandra (49.0 s) and MongoDB (33.8 s) restores with a pointer to the PR 11 record. ADR-0019 (status and index row) and the Cassandra RFC's "Backup and restore" section carry a short note pointing at the manual and the design: superseded in part, since CreateBackup on Cassandra still writes their in-catalog format, before it returns. The ADR's note adds that the reconciler now removes an in-catalog backup whose CreateBackup did not finish, so its Negative consequence about a future reconciliation worker no longer holds for the create. The RFC's section 4 line on synchronous data movement and its Known Gaps lines on synchronous backup and restore and on restored indexes carry "Superseded in part" notes. docs/adr/README.md says an ADR is never edited after acceptance except to update its Status, while design section 10 asks for a note at the top of ADR-0019. The maintainer chooses which rule wins: a "Superseded in part by" status allowed in the README with the note kept, or the ADR back to a Status-only change and the note moved to the design. docs/build-docs.py lists the manual as document 19; the full build ran on this host (Python 3.10, WeasyPrint 70.0): 19 HTML, 19 PDF, 0 failed. The README's and AGENTS.md's documentation tables have a row for it.

Design (docs/design/15-backup-restore.md). The testing summary's "still to be measured" sentence now states the measured cells, what the vector restore's memory belongs to and the vector index's share of the restore's duration (earlier revision items keep their wording); the PR 11 section's line says what shipped; section 10's closing paragraph says what the ADR, the RFC and the two manuals now carry; revision item 35 records the filter history, the measurements, the manual's place and checks, and the database-name correction. The design's revision items 32 to 34 already exist, so this PR's item is 35. The bench record's Durations section gives the one-arena SQLite backup (222 s) and restore (283 s) separately.

Cross-references with #737

#737 (open, another author's) touches four files this PR also edits: 05-admin-guide.md, 07-upgrade-manual.md, 10-security-model.md and 11-deployment-guide.md. It is not on this branch and nothing it adds is edited here. The overlaps:

• The admin guide's "Data Recovery" entry: docs: back up and restore SQLite and PostgreSQL with their own tools #737 replaces its body with a pointer to its new section, and this PR rewrites the body (the databases table, the derived data-database name, the pg_dump lines with the server stopped or a base backup). The manual links to that entry.
• The security model's "Backup and Recovery" block: docs: back up and restore SQLite and PostgreSQL with their own tools #737 replaces it with a pointer to its admin guide section, and this PR rewrites the same lines.
• docs: back up and restore SQLite and PostgreSQL with their own tools #737's new section opens with "CreateBackup stores each table backup in the deployment's own database, in the catalog's backups and backup_items tables", which is false of this series once a backup store is configured.

Whichever of the two lands second reconciles these: the entry's anchor and body, the security model's block, its CreateBackup sentence, and the manual's links.

Verification

• cargo fmt --all -- --check: clean.
• cargo build -p extenddb-app and cargo clippy -p extenddb-app --all-targets --no-deps -- -D warnings: clean. No test or snapshot holds the init --help text.
• cargo test -p extenddb-storage-postgres --test doc_version_literals: 3 passed (the manual carries no binary version literal; the record's service versions are not flagged).
• No other workflow runs a documentation check. actionlint 1.7.7 on integration.yml: only the pre-existing custom runner-label findings.
• docs/build-docs.py on an exported copy of the tree (Python 3.10, WeasyPrint 70.0): 19 HTML, 19 PDF, 0 failed.
• The CI-way runs and the binary checks above. Every workflow runs on this PR in CI.
• git grep '<catalog>_data\|<data-db>_catalog\|data_database_url': no hit; extenddb_catalog_data only in revision item 35, as history.
• No non-inclusive term in the diff.

Not here

• migrate-legacy (Backup/restore: backup migrate-legacy #821), the store-maintenance options (Backup/restore: backup sync --move-store, --forget-store, --forget-backup #820), export and import through the store (Backup/restore: export and import through the store #748).
• A live check of an ExtendDB backup imported into Amazon DynamoDB, and of a DynamoDB export imported into ExtendDB from the service: the manual states that the import into the service has not been run.
• An S3 store's upload and download durations, and a database on another host: not measured, and the manual says so.
• init --catalog-db with a name that lacks the _catalog suffix derives the catalog's own name for the data database, creates the database once, fails on the second create with "Run 'destroy' first", and leaves an empty database behind with no config file for destroy to read. An existing init defect; the admin guide now says such a name needs --data-db.
• A catalog restored from a dump whose newest AVAILABLE backup was deleted after the dump keeps its store unhealthy (every store operation refused, across restarts), and this release has no remedy without a copy of that backup's prefix: --prune-missing refuses while the store is unhealthy. The manual states it and says to take the dump after deletions.