Sync upstream dev into re-dev (2026-11-05): upstream's source filter, typed conflicts and 25 more

.github/workflows/checks.yml

@@ -47,7 +47,9 @@ jobs:
   # migrations 再编一遍才真的跑。给这个 job 也起一个 Postgres，一次编译全部跑完，
   # migrations job 只留迁移本身的检查。连库测试由此进了 backend 汇总，才真正成为合并门槛
   test:
-    runs-on: ubuntu-latest
+    # 钉在 24.04：ubuntu-latest 换到 26.04 后 apt 里没有 postgresql-client-16，
+    # 装测试工具这一步直接失败，dev 与所有 PR 一起红（2026-10-29）
+    runs-on: ubuntu-24.04
     services:
       postgres:
         image: pgvector/pgvector:pg16
@@ -66,8 +68,10 @@ jobs:
       - uses: Swatinem/rust-cache@v2
         with:
           save-if: ${{ github.event_name == 'push' }}
-      - name: Install PDF text extraction tools
-        run: sudo apt-get update && sudo apt-get install -y poppler-utils poppler-data
+      - name: Install PDF and PostgreSQL test tools
+        run: |
+          sudo apt-get update && sudo apt-get install -y poppler-utils poppler-data postgresql-client-16
+          echo '/usr/lib/postgresql/16/bin' >> "$GITHUB_PATH"
       # 库要先迁移好：store 的一百多个集成测试和 server 的多数 fixture 不自己跑迁移，
       # 对着空库并发起几百个测试会成片撞 relation "organizations" does not exist（#869）。
       # 不用 sqlx-cli——装它要一分多钟，而这个 job 在关键路径上；这个 example 复用

.roadmap-proposals/backup-restore.md

@@ -81,11 +81,40 @@ $ utopia restore --from <PATH> [flags]
   --from <PATH>              # archive to restore from (required)
   --target-data-dir <PATH>   # override the data_dir path
   --pg-restore <PATH>        # path to pg_restore binary
-  --dry-run                  # print the plan, don't write anything
-  --force                    # allow restore into a non-empty database
+  --dry-run                  # validate and print the plan; don't change targets
+  --force                    # allow overwriting a non-empty database or data dir
   --yes                      # skip the "are you sure?" prompt
 ```
 
+Before changing either target, restore checks required archive components,
+checksums, version, target path types and overwrite permission. Declared data
+must exist in the archive. `--force` is required for a database containing user
+objects (even empty tables, schemas, functions or types), and for a non-empty
+data directory when the archive includes data. `--yes` only skips confirmation.
+`--dry-run` uses the same preflight, including a read-only database check; it
+extracts temporary staging files in the current working directory but never
+restores the database or writes the target data directory. After confirmation
+and data preparation, restore repeats component, target-path and overwrite
+checks immediately before running `pg_restore`.
+
+Database-only archives leave the data directory untouched. Otherwise, data is
+prepared separately and replaces the old directory after `pg_restore` succeeds;
+files and keys are never merged with the old directory. A database restore
+failure preserves the old files. With `--force`, `pg_restore --clean --if-exists`
+replaces objects in the archive without removing unrelated database objects.
+Every restore uses `--single-transaction --exit-on-error`: a SQL execution error
+rolls back the database restore instead of leaving partial SQL changes.
+
+Stop all writers and retain a backup of both targets before restoring. Preflight
+does not prevent concurrent changes or later permission and disk failures.
+Database and filesystem restoration are **not atomic**: the database may be
+committed before a later directory installation fails. Failed installation
+attempts to put the old directory back; a failed rollback reports its retained
+location. After an interruption, also check the target's parent for
+`.utopia-restore-data-*/previous`. Keep the service stopped, resolve the error,
+and retry the archive with `--force`, or restore both targets from the previous
+backup. Moving old files back alone does not undo database changes.
+
 Both halves are implemented. `backup` landed first, and `restore`
 followed in the same crate, for two reasons worth keeping on the record:
 
@@ -150,50 +179,45 @@ binary first, the image second.
 
 - No docker image change (see Open question #3).
 - No `UTOPIA_BACKUP_DIR` config (see Open question #5).
-- No automatic migration of older manifests — `restore` will only
-  read `schema_version == current`.
+- No automatic migration of older manifests — older versions are accepted
+  with a warning; newer versions are refused.
 - **The sealing key is left out of the archive by default.** The dump carries
   credentials sealed with `data/secret.key`, and an archive is the artifact that
   gets copied between hosts and handed to whoever runs the restore, so shipping
   both halves in one file is not an unencrypted archive — it is no sealing at
-  all. `--include-secret-key` puts it in on purpose, the manifest records
-  whether it is there, and a restore without it says so rather than letting the
-  server fail to open its own credentials later.
+  all. `--include-secret-key` opts in to copying an existing key file. The
+  manifest records that flag even when an environment-key deployment has no
+  key file; restore remains compatible with those archives. When replacing
+  data, an absent key is never taken from the old directory: restore warns to
+  set `UTOPIA_SECRET_KEY` to the original key, or re-enter sealed credentials
+  after starting the server.
 - No encryption-at-rest. Out of scope; the operator's filesystem
   encryption (LUKS, EBS encryption, etc.) is the right layer.
 - No streaming upload to S3. Out of scope; the operator can pipe
   `utopia backup --output -` to `aws s3 cp - s3://…` today.
 
 ## Test plan
 
-8 unit tests, all pure (no live DB, no Docker):
-
-1. `parses_backup_minimal` — `utopia backup --dry-run` produces a
-   `BackupArgs` with all defaults.
-2. `parses_backup_full` — every flag set, every field populated.
-3. `parses_restore_requires_from` — `--from` is mandatory.
-4. `parses_restore_full` — every flag set, every field populated.
-5. `rejects_unknown_subcommand` — `utopia frobnicate` fails clearly.
-6. `redact_url_host_keeps_userinfo_at_host` — `postgres://u:***@h/db`
-   stays redacted on round-trip.
-7. `redact_url_host_handles_no_at` — `postgres://h/db` redacts
-   nothing (no userinfo).
-8. `hex_encode_known_value` — `hex_encode(&[0xde, 0xad]) == "dead"`.
-
-Live integration (smoke test, not part of CI):
-
-```
-$ createdb utopia_test_restore
-$ UTOPIA_DATABASE_URL=postgres://utopia:utopia@localhost:1543/utopia_test \
-    utopia backup --output /tmp/utopia.tar.gz --include-data-dir
-$ dropdb utopia_test_restore && createdb utopia_test_restore
-$ UTOPIA_DATABASE_URL=postgres://utopia:utopia@localhost:1543/utopia_test \
-    utopia restore --from /tmp/utopia.tar.gz --yes
+Unit tests cover parsing, manifests, versions, checksums and key-copy policy.
+The CLI integration suite runs in `cargo test --workspace`: a recording
+`pg_restore` double checks preflight ordering, post-confirmation validation,
+overwrite guards, replacement, rollback, dry runs and database-only archives.
+Real `pg_dump`/`pg_restore` tests exercise database restoration and SQL rollback
+using disposable test databases.
+CI provides PostgreSQL 16 clients; the double also needs `rustc`, and
+archive operations need `tar`.
+
+Tests use `utopia_store::test_db::url()` and create uniquely named databases on
+that test cluster; they never modify the guard database's contents. Use only a
+dedicated test cluster with `CREATEDB` and catalog-fixture privileges, never a
+business database. Without a URL, tests skip unless `UTOPIA_TEST_REQUIRE_DB=1`
+requires them to run, as in CI.
+
+```bash
+UTOPIA_DATABASE_URL=postgres://test_role:test_password@localhost:1543/test_admin \
+UTOPIA_TEST_REQUIRE_DB=1 cargo test -p utopia-cli
 ```
 
-Smoke-tested manually on the maintainer's local docker-compose stack
-once the restore side lands.
-
 ## Migration concern: existing data is already there
 
 Backups are forward-looking. Anything written before this lands has

crates/utopia-cli/Cargo.toml

@@ -18,3 +18,9 @@ tracing.workspace = true
 tracing-subscriber.workspace = true
 sha2.workspace = true
 dotenvy.workspace = true
+sqlx.workspace = true
+tokio.workspace = true
+tempfile.workspace = true
+
+[dev-dependencies]
+utopia-store.workspace = true