From 2a3b5c48fc3542453cf71401ecf501ef2397ff0f Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:15:08 -0400 Subject: [PATCH 01/42] refactor(export): rename message-crate-pull to message-crate-export CONTEXT.md names the operation Export and lists Pull under its Avoid, but the crate that runs an Export from a server, its types, the desktop command, the state file and the recorded tool name all said Pull. - The crate is message-crate-export at crates/libs/export/ (library message_crate_export), with ExportConfig, ExportReport, the private Export type, ExportJournalEvent, ExportJournalState and EXPORT_JOURNAL_NAME. - The state file is .message-crate-export-state.jsonl. The old file is not read. - The server records the tool as message-crate-export. The old name is not mapped. - The desktop command is export (src-tauri/src/commands/export.rs, ExportArgs), and the web app calls it as invokeExport. - CLAUDE.md, AGENTS.md, the developer pages, the Export guide, the OpenAPI document and the crate READMEs name the new crate. ADR 0001 and ADR 0012 keep their text and gain a dated note. Closes #1906 Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 2 +- AGENTS.md | 2 +- CHANGELOG.md | 8 ++ CLAUDE.md | 4 +- Cargo.lock | 30 +++---- Cargo.toml | 2 +- crates/libs/api-types/README.md | 2 +- crates/libs/api-types/src/lib.rs | 8 +- crates/libs/{pull => export}/Cargo.toml | 4 +- crates/libs/{pull => export}/README.md | 4 +- crates/libs/{pull => export}/src/http.rs | 6 +- crates/libs/{pull => export}/src/journal.rs | 80 +++++++++---------- crates/libs/{pull => export}/src/lib.rs | 8 +- crates/libs/{pull => export}/src/part_file.rs | 2 +- crates/libs/{pull => export}/src/project.rs | 6 +- crates/libs/{pull => export}/src/run.rs | 55 +++++++------ .../tests/export_mock.rs} | 64 +++++++-------- crates/libs/http/Cargo.toml | 2 +- crates/libs/http/README.md | 4 +- crates/libs/http/src/lib.rs | 4 +- crates/libs/http/src/response.rs | 2 +- crates/libs/http/src/session.rs | 2 +- crates/libs/ir/README.md | 2 +- crates/libs/journal/README.md | 4 +- .../server/src/db/conversation_messages.rs | 2 +- crates/server/server/src/exports_api.rs | 4 +- .../0001-no-command-line-except-the-server.md | 7 ++ ...0012-four-crates-in-the-export-pipeline.md | 7 ++ docs/architecture/http-api.md | 2 +- docs/src/assets/openapi.json | 10 +-- .../src/content/docs/docs/developer/design.md | 6 +- .../docs/docs/developer/message-transfer.md | 4 +- .../docs/docs/developer/reference/api.md | 4 +- .../docs/docs/developer/rustdoc-style.md | 2 +- .../docs/user/features/messages/export.md | 2 +- scripts/coverage.sh | 2 +- src-tauri/Cargo.lock | 26 +++--- src-tauri/Cargo.toml | 2 +- src-tauri/src/commands/{pull.rs => export.rs} | 24 +++--- src-tauri/src/commands/jobs.rs | 2 +- src-tauri/src/commands/mod.rs | 2 +- src-tauri/src/export_directories.rs | 4 +- src-tauri/src/export_directories/tests.rs | 6 +- src-tauri/src/lib.rs | 2 +- src-tauri/src/main.rs | 2 +- src-tauri/src/state.rs | 2 +- web/src/lib/serverApi.ts | 2 +- web/src/lib/serverApi.types.ts | 10 +-- web/src/lib/serverApiOpenapi.test.ts | 2 +- web/src/lib/tauri.ts | 8 +- web/src/screens/ExportScreen.test.tsx | 40 +++++----- web/src/screens/ExportScreen.tsx | 8 +- 52 files changed, 263 insertions(+), 238 deletions(-) rename crates/libs/{pull => export}/Cargo.toml (86%) rename crates/libs/{pull => export}/README.md (90%) rename crates/libs/{pull => export}/src/http.rs (98%) rename crates/libs/{pull => export}/src/journal.rs (85%) rename crates/libs/{pull => export}/src/lib.rs (60%) rename crates/libs/{pull => export}/src/part_file.rs (97%) rename crates/libs/{pull => export}/src/project.rs (99%) rename crates/libs/{pull => export}/src/run.rs (97%) rename crates/libs/{pull/tests/pull_mock.rs => export/tests/export_mock.rs} (96%) rename src-tauri/src/commands/{pull.rs => export.rs} (86%) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b8f432ed2..293c2da66 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -610,7 +610,7 @@ jobs: --notes "## Highlights - Tauri v2 desktop app with all exporters included - - Push and pull against a server, format conversion, contacts preview + - Import into and export from a server, format conversion, contacts preview - Server Docker image: \`bitrealm/message-crate:${TAG#v}\` - User Guide and developer docs: https://messagecrate.app/docs/user/ and https://messagecrate.app/docs/developer/ diff --git a/AGENTS.md b/AGENTS.md index d7ad7e796..5c75344a6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -262,7 +262,7 @@ message-crate │ ├── exporters/ # backup parsers (iMessage, WhatsApp, SMS, experimental) │ ├── helpers/ # imessage-reader (GPL helper process the app spawns) and its protocol │ ├── libs/ # shared libraries (ir, ir-format, reexport, contacts, media, -│ │ # message-crate-push, message-crate-pull, …) +│ │ # message-crate-push, message-crate-export, …) │ └── server/ # message-crate-server (HTTP API + SQLite) and demo-seed ├── docker/ # Dockerfile and Compose for a release-shaped server image ├── docs/ # Astro Starlight site (messagecrate.app) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b30258ba..80dd40217 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,14 @@ released versions carry their date on the heading. belongs only to images built by hand from a branch, which show the commit in their version, so one `sha-` tag always names one image. Pull a release by its version, such as `0.11.0`, or by `latest`. +- 2026-10-05: **An Export's state file is now + `.message-crate-export-state.jsonl`.** The desktop app's Export keeps this + file in the directory it writes, to remember which attachments it already + fetched. It was named `.message-crate-pull-state.jsonl`. The old file is + no longer read and can be deleted; attachments already in the directory + are still kept rather than fetched again. The server now records an Export + Run the desktop app starts with the tool name `message-crate-export`, and + the library behind it is the `message-crate-export` crate. ### Fixes diff --git a/CLAUDE.md b/CLAUDE.md index 744c96d53..5afc3373a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,12 +24,12 @@ vendor backup (chat.db, SMS XML, WhatsApp crypt15, …) - **`crates/libs/ir`** (`message-ir`) is the shared conversation model every exporter writes: `ConversationDocument` holds export metadata, participants, and messages. `schema_version` is `SCHEMA_VERSION` in `crates/libs/ir/src/lib.rs`, independent of the product version. `check_schema_version` in `crates/libs/ir/src/schema_version.rs` refuses a file at any other version, naming its version and the current one. Nothing is upgraded. - **`crates/libs/ir-format`** reads/writes the formats Message Crate emits itself (JSON, JSONL, CSV, EML, MBOX) to/from IR; **`crates/libs/staging`** (`message-staging`) is the resumable write path every exporter and the desktop app go through (`ExportWriter`, the write queue, the transcode pass, the staging summary); SBR XML belongs to `crates/exporters/sms-backup-restore-exporter` in both directions; **`crates/libs/reexport`** converts between existing export formats, which is how Export writes anything other than JSONL. One job per crate, and why: `docs/adr/0012-four-crates-in-the-export-pipeline.md`. -- **No command line except the server.** Every exporter, `message-reexport`, `message-crate-push`, and `message-crate-pull` are library crates with no binary; the desktop app calls them in process. Only `message-crate-server`, `demo-seed`, and `imessage-reader` build binaries, and the last is not a command line: it is the GPL helper the desktop app spawns to read Apple Messages (below). Why: `docs/adr/0001-no-command-line-except-the-server.md`. +- **No command line except the server.** Every exporter, `message-reexport`, `message-crate-push`, and `message-crate-export` are library crates with no binary; the desktop app calls them in process. Only `message-crate-server`, `demo-seed`, and `imessage-reader` build binaries, and the last is not a command line: it is the GPL helper the desktop app spawns to read Apple Messages (below). Why: `docs/adr/0001-no-command-line-except-the-server.md`. - **GPL only behind a process boundary.** `imessage-database` and `crabapple` are GPL-3.0-or-later and the repository is under the Fair Core License, so `crates/helpers/imessage-reader` (GPL) is the only crate that links them. `crates/libs/ios-backup` starts it as a process (for the Apple Messages exporter, the WhatsApp import from an encrypted iPhone backup, and the desktop app's backup checks) and talks JSON lines over stdin/stdout through `crates/helpers/imessage-reader-protocol` (MIT OR Apache-2.0, so both sides can link it). `src-tauri/build.rs` builds the helper and Tauri ships it beside the app as an `externalBin`; `cargo tree --manifest-path src-tauri/Cargo.toml -i imessage-database` must match nothing. `cargo deny check licenses bans` in `audit.yml` enforces the rule. Why, and the rules: `docs/adr/0014-gpl-code-only-behind-a-process-boundary.md`. - **One way to fetch data in `web/`** — TanStack Query over route functions in `web/src/lib/serverApi.ts`, with response types generated from `docs/src/assets/openapi.json`. Do not write a new cache, change-notification event, or fetching hook for a screen. This is **built** (PRs #290–#293): `useResource`, `usePagedList`, and `contactDetailCache` are gone, the `mv-*-changed` browser events with them; `nameCollection`, `savedSearches`, and `useAccountProfile` remain only as thin wrappers over TanStack Query, not as mechanisms of their own. Every cache entry is named with the logged-in account, so nothing has to be cleared when the account changes. Why, and what replaces what: `docs/adr/0002-one-way-to-fetch-data-in-the-web-app.md`. - **`crates/core/message-crate-core`** — shared export pipeline, jobs, form model. The form's validation reports problems as a `Vec` so the desktop app can show them as they are; the pipeline itself returns `anyhow` errors like every other crate. - **`crates/server/server`** — each `*_api.rs` file is one Axum route group; `db/` modules mirror the table sources in the repo-root `schema/sql/*.sql`, which the server embeds at compile time (`db/schema.rs`) — change tables there, not in a live db file. Import path: `jsonl.rs` reads the records, then `imports_api/` runs `staging` (which calls `contact_name`) → `promote`, calling `dedupe.rs`; demo mode is a seed action, not a runtime mode — `reset_demo.rs` writes one account row, `demo` at the fixed `DEMO_ACCOUNT_ID` with a NULL password hash, which is why `demo` logs in with an empty password, and no owner. `serve` runs it with the medium set on a database that does not exist yet, before it listens, so every new Message Crate starts with the Demo Account; `reset-demo --size medium|large` rebuilds it; `create-database` makes an empty one. The generator's inputs are compiled into the server, so seeding needs no files beside the binary. Nothing at request time knows a Message Crate is a demo. The Demo Account itself is fixed by its id: `is_demo_account` reports `is_demo` on the profile, and the server refuses a password, a status, permission, identity, display name or time zone change, deleting its messages for good, and an address book load, from the account and from the owner alike. Starting an import and every delete for good are refused by its id in the import and delete guards (`server::require_import_access`, `server::require_delete_access`), whatever its permission row says, and `account_profile::load_account_auth` gives it `DEMO_ACCOUNT_PERMISSIONS` (export, neither import nor delete) from its id, so the profile and the account list never read the row either. Why: `docs/adr/0016-the-demo-account-is-fixed-not-configured.md`. A test module over a few hundred lines lives beside its source as `/tests.rs` (declared `mod tests;`), so the source file stays readable; small ones stay inline. -- **`src-tauri/`** is **not a workspace member** (own `Cargo.toml`, listed in the root workspace `exclude`). Its `commands/` wrap the exporter crates and push/pull for the desktop app. Format/build it with `--manifest-path`. +- **`src-tauri/`** is **not a workspace member** (own `Cargo.toml`, listed in the root workspace `exclude`). Its `commands/` wrap the exporter crates, `message-crate-push`, and `message-crate-export` for the desktop app. Format/build it with `--manifest-path`. - **`web/src/lib/api.ts`** holds the server URL, the session token, and the `apiClient` fetch wrapper with its `ApiError`; the route functions built on it are in `serverApi.ts`; `web/src/lib/tauri.ts` wraps desktop-only commands; `desktopFeatures.ts` gates them. Tests sit next to sources as `*.test.ts(x)` (Vitest + Testing Library). - **Not the product path**: `web-next/` (legacy Next.js browse UI). New features go in `web/` + `src-tauri/` + `crates/server/server/`. **It is kept on purpose — do not propose deleting it.** It stays until the functionality worth keeping has been ported into `web/`, and that porting work is not yet defined or scoped: nobody has named which screens or behaviours would come across. Being outside CI, unserved, and excluded from dependabot are all true and none of them are an argument for removing it. Its screens and a feature-by-feature comparison with `web/` are recorded in `docs/superpowers/reference/web-next.md`. The old Slint GUI is gone; its screens are recorded in `docs/superpowers/reference/legacy-slint-gui.md`. diff --git a/Cargo.lock b/Cargo.lock index a1d882788..94388b61c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1926,40 +1926,40 @@ dependencies = [ ] [[package]] -name = "message-crate-http" +name = "message-crate-export" version = "0.1.0" dependencies = [ "anyhow", + "chrono", + "hex", "httpmock", + "journal", + "media", "message-crate-api-types", - "rcgen", + "message-crate-core", + "message-crate-http", + "message-ir", + "message-ir-format", "reqwest", - "rustls", "serde", "serde_json", - "thiserror 2.0.21", + "sha2 0.11.0", + "tempfile", ] [[package]] -name = "message-crate-pull" +name = "message-crate-http" version = "0.1.0" dependencies = [ "anyhow", - "chrono", - "hex", "httpmock", - "journal", - "media", "message-crate-api-types", - "message-crate-core", - "message-crate-http", - "message-ir", - "message-ir-format", + "rcgen", "reqwest", + "rustls", "serde", "serde_json", - "sha2 0.11.0", - "tempfile", + "thiserror 2.0.21", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index 1ca72a3e1..67baac17f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -32,7 +32,7 @@ members = [ "crates/helpers/imessage-reader", "crates/core/message-crate-core", "crates/libs/push", - "crates/libs/pull", + "crates/libs/export", "crates/server/demo-seed", "crates/server/server", ] diff --git a/crates/libs/api-types/README.md b/crates/libs/api-types/README.md index d02fa98af..e8b329f59 100644 --- a/crates/libs/api-types/README.md +++ b/crates/libs/api-types/README.md @@ -29,7 +29,7 @@ Workspace setup: [CONTRIBUTING.md](../../../CONTRIBUTING.md). ## Docs This crate is a library shared by the server, `message-crate-push`, -`message-crate-pull`, and `message-crate-http`. It builds no binary. +`message-crate-export`, and `message-crate-http`. It builds no binary. ## License diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 6fff20fbc..6cb3ed665 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -2,15 +2,15 @@ //! server that writes them and the client crates that read them. //! //! Two crates sit on either side of these shapes: `message-crate-server` -//! serializes them, and `message-crate-pull` deserializes them on its way to +//! serializes them, and `message-crate-export` deserializes them on its way to //! `message-ir`. While each kept its own struct, the two could disagree -//! silently and did, three times: `message-crate-pull` declared a +//! silently and did, three times: `message-crate-export` declared a //! participant's address a `String` after the server started sending `null` //! for a participant a backup named without an address, kept //! `#[serde(default)]` on a field the server had removed, and read a //! `service` off the conversation the server has never sent there. Each of //! those was a pull that failed at runtime, or quietly produced worse data, -//! with nothing in either crate's tests to catch it — `message-crate-pull`'s +//! with nothing in either crate's tests to catch it — `message-crate-export`'s //! "real export page" was a JSON literal it wrote itself, so it agreed with //! whatever the mirror said. //! @@ -310,7 +310,7 @@ api_shape! { pub id: i64, /// What the run asked for, as given. pub scope: ExportScope, - /// Exporting tool, e.g. `message-crate-pull`, when the client named one. + /// Exporting tool, e.g. `message-crate-export`, when the client named one. pub tool: Option, /// Lifecycle status. pub status: ExportStatus, diff --git a/crates/libs/pull/Cargo.toml b/crates/libs/export/Cargo.toml similarity index 86% rename from crates/libs/pull/Cargo.toml rename to crates/libs/export/Cargo.toml index f76354b1a..b02b63ab5 100644 --- a/crates/libs/pull/Cargo.toml +++ b/crates/libs/export/Cargo.toml @@ -1,8 +1,8 @@ [package] -name = "message-crate-pull" +name = "message-crate-export" version = "0.1.0" edition = "2024" -description = "Pull messages from a Message Crate export API into a message-ir directory" +description = "Export messages from a Message Crate export API into a message-ir directory" license = "LicenseRef-FCL-1.0-ALv2" [dependencies] diff --git a/crates/libs/pull/README.md b/crates/libs/export/README.md similarity index 90% rename from crates/libs/pull/README.md rename to crates/libs/export/README.md index 7ee81ceb9..b6a5e98e9 100644 --- a/crates/libs/pull/README.md +++ b/crates/libs/export/README.md @@ -1,4 +1,4 @@ -# message-crate-pull +# message-crate-export Export messages from a running server into a local JSON Lines directory (`*.jsonl` plus `attachments/`), fetching each Asset the messages' attachments name. @@ -7,7 +7,7 @@ The desktop app's **Export** screen uses this crate as a library, with the logge ## Build and test ```bash -cargo test -p message-crate-pull +cargo test -p message-crate-export ``` Workspace setup: [CONTRIBUTING.md](../../../CONTRIBUTING.md). diff --git a/crates/libs/pull/src/http.rs b/crates/libs/export/src/http.rs similarity index 98% rename from crates/libs/pull/src/http.rs rename to crates/libs/export/src/http.rs index f2cb5fbb9..5a359213e 100644 --- a/crates/libs/pull/src/http.rs +++ b/crates/libs/export/src/http.rs @@ -332,7 +332,7 @@ mod tests { &format!("http://127.0.0.1:{port}"), "mc_test", &ExportScope::Everything, - "message-crate-pull", + "message-crate-export", ) .expect_err("nothing listens on that port"); assert_eq!(err.to_string(), "Export Run start failed"); @@ -405,12 +405,12 @@ mod tests { }; let body = serde_json::to_value(CreateExportBody { scope: &scope, - tool: "message-crate-pull", + tool: "message-crate-export", }) .unwrap(); assert_eq!( body, - serde_json::json!({ "scope": { "kind": "query", "list": "conversations", "q": "from:me" }, "tool": "message-crate-pull" }) + serde_json::json!({ "scope": { "kind": "query", "list": "conversations", "q": "from:me" }, "tool": "message-crate-export" }) ); } diff --git a/crates/libs/pull/src/journal.rs b/crates/libs/export/src/journal.rs similarity index 85% rename from crates/libs/pull/src/journal.rs rename to crates/libs/export/src/journal.rs index 5b729da6c..357a9c3eb 100644 --- a/crates/libs/pull/src/journal.rs +++ b/crates/libs/export/src/journal.rs @@ -1,6 +1,6 @@ //! Local log of which Assets an Export from a server already fetched. //! -//! The file is `.message-crate-pull-state.jsonl`. JSON Lines means one JSON object per +//! The file is `.message-crate-export-state.jsonl`. JSON Lines means one JSON object per //! line. A later Export Run can skip Assets that are already on disk. use std::collections::HashSet; @@ -13,12 +13,12 @@ pub use jsonl_journal::ServerTarget; use serde::{Deserialize, Serialize}; /// Filename of the journal, written in the output directory. -pub const PULL_JOURNAL_NAME: &str = ".message-crate-pull-state.jsonl"; +pub const EXPORT_JOURNAL_NAME: &str = ".message-crate-export-state.jsonl"; #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(tag = "event", rename_all = "snake_case")] -/// One row in `.message-crate-pull-state.jsonl`. -pub enum PullJournalEvent { +/// One row in `.message-crate-export-state.jsonl`. +pub enum ExportJournalEvent { /// One Asset is on disk, so a later run can skip fetching it. AssetOk { /// Server and account the Asset came from. @@ -44,16 +44,16 @@ pub enum PullJournalEvent { #[derive(Debug, Default)] /// Skip sets rebuilt from the journal for one [`ServerTarget`]. -pub struct PullJournalState { +pub struct ExportJournalState { /// SHA-256 fingerprints (hex of the file bytes) of attachments already on disk. pub assets: HashSet, /// True if the last run finished cleanly (an `export_complete` event was written). pub export_complete: bool, } -/// Path of `.message-crate-pull-state.jsonl` inside the output directory. +/// Path of `.message-crate-export-state.jsonl` inside the output directory. pub fn journal_path(out_dir: &Path) -> PathBuf { - out_dir.join(PULL_JOURNAL_NAME) + out_dir.join(EXPORT_JOURNAL_NAME) } /// The Export's log line for a line of the journal that could not be read. @@ -83,18 +83,18 @@ pub fn load( path: &Path, target: &ServerTarget, on_unreadable: &mut dyn FnMut(String), -) -> Result { - let mut state = PullJournalState::default(); - let events: Vec = - jsonl_journal::load_events("pull journal", path, &mut |line, error| { +) -> Result { + let mut state = ExportJournalState::default(); + let events: Vec = + jsonl_journal::load_events("export journal", path, &mut |line, error| { on_unreadable(unreadable_line_sentence(path, line, error)); })?; for event in events.into_iter().filter(|event| event.target() == target) { match event { - PullJournalEvent::AssetOk { sha256, .. } => { + ExportJournalEvent::AssetOk { sha256, .. } => { state.assets.insert(sha256); } - PullJournalEvent::ExportComplete { .. } => state.export_complete = true, + ExportJournalEvent::ExportComplete { .. } => state.export_complete = true, } } Ok(state) @@ -106,8 +106,8 @@ pub fn load( /// /// Returns an error when the parent directory cannot be created, the file cannot /// be opened, or the write fails. -pub fn append(path: &Path, event: &PullJournalEvent) -> Result<()> { - jsonl_journal::append("pull journal", path, event) +pub fn append(path: &Path, event: &ExportJournalEvent) -> Result<()> { + jsonl_journal::append("export journal", path, event) } /// Rewrite the lines of `target` from in-memory `state`, and keep every line @@ -120,16 +120,16 @@ pub fn append(path: &Path, event: &PullJournalEvent) -> Result<()> { /// # Errors /// /// Returns an error when the temporary file cannot be written or the rename fails. -pub fn compact(path: &Path, target: &ServerTarget, state: &PullJournalState) -> Result<()> { - jsonl_journal::compact_with::("pull journal", path, |read| { - let mut events: Vec = read +pub fn compact(path: &Path, target: &ServerTarget, state: &ExportJournalState) -> Result<()> { + jsonl_journal::compact_with::("export journal", path, |read| { + let mut events: Vec = read .into_iter() .filter(|event| event.target() != target) .collect(); let mut assets: Vec<_> = state.assets.iter().collect(); assets.sort_unstable(); for sha in assets { - events.push(PullJournalEvent::AssetOk { + events.push(ExportJournalEvent::AssetOk { target: target.clone(), sha256: sha.clone(), }); @@ -137,7 +137,7 @@ pub fn compact(path: &Path, target: &ServerTarget, state: &PullJournalState) -> if state.export_complete { // A later Export Run ignores the counts; an `export_complete` row // only means the last Export Run finished. - events.push(PullJournalEvent::ExportComplete { + events.push(ExportJournalEvent::ExportComplete { target: target.clone(), conversations: 0, messages: 0, @@ -148,7 +148,7 @@ pub fn compact(path: &Path, target: &ServerTarget, state: &PullJournalState) -> }) } -impl PullJournalEvent { +impl ExportJournalEvent { /// The server and account of the run that wrote this line. fn target(&self) -> &ServerTarget { match self { @@ -169,7 +169,7 @@ mod tests { #[test] fn loads_asset_and_export_complete_events() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); fs::write( &path, concat!( @@ -193,7 +193,7 @@ mod tests { #[test] fn filters_by_url_and_username() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); fs::write( &path, concat!( @@ -218,8 +218,8 @@ mod tests { #[test] fn compact_sorts_assets_and_rewrites() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); - let mut state = PullJournalState::default(); + let path = dir.path().join(EXPORT_JOURNAL_NAME); + let mut state = ExportJournalState::default(); state.assets.insert("ccc".into()); state.assets.insert("aaa".into()); state.assets.insert("bbb".into()); @@ -235,18 +235,18 @@ mod tests { assert!(reloaded.export_complete); } - /// `append` is what a pull actually calls, once per Asset, and nothing + /// `append` is what an Export Run actually calls, once per Asset, and nothing /// called it: every test here wrote the file by hand or went through /// `compact`. Replacing it with a no-op made a later Export Run start /// from nothing and fetch every Asset again, with the suite green. #[test] fn appended_events_are_on_disk_and_load_back() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); append( &path, - &PullJournalEvent::AssetOk { + &ExportJournalEvent::AssetOk { target: alice(), sha256: "aaa".into(), }, @@ -261,7 +261,7 @@ mod tests { append( &path, - &PullJournalEvent::AssetOk { + &ExportJournalEvent::AssetOk { target: alice(), sha256: "bbb".into(), }, @@ -284,17 +284,17 @@ mod tests { ); } - /// The parent directory is created on the way. A pull writing its first + /// The parent directory is created on the way. An Export Run writing its first /// journal into a fresh output directory would otherwise fail on the very /// first asset. #[test] fn appending_creates_the_directory_it_needs() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join("not-yet").join(PULL_JOURNAL_NAME); + let path = dir.path().join("not-yet").join(EXPORT_JOURNAL_NAME); append( &path, - &PullJournalEvent::AssetOk { + &ExportJournalEvent::AssetOk { target: alice(), sha256: "aaa".into(), }, @@ -318,7 +318,7 @@ mod tests { #[test] fn an_unreadable_line_is_skipped_and_named_by_its_line_in_the_file() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); fs::write( &path, concat!( @@ -354,15 +354,15 @@ mod tests { #[test] fn compact_keeps_the_lines_of_every_other_server_and_account() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); let alice_a = ServerTarget::new("http://server-a", "alice"); let bob_a = ServerTarget::new("http://server-a", "bob"); let alice_b = ServerTarget::new("http://server-b", "alice"); - let asset = |target: &ServerTarget, sha256: &str| PullJournalEvent::AssetOk { + let asset = |target: &ServerTarget, sha256: &str| ExportJournalEvent::AssetOk { target: target.clone(), sha256: sha256.into(), }; - let complete = |target: &ServerTarget| PullJournalEvent::ExportComplete { + let complete = |target: &ServerTarget| ExportJournalEvent::ExportComplete { target: target.clone(), conversations: 1, messages: 1, @@ -377,7 +377,7 @@ mod tests { ] { append(&path, &event).unwrap(); } - let mut state = PullJournalState::default(); + let mut state = ExportJournalState::default(); state.assets.insert("ccc".into()); state.assets.insert("ddd".into()); state.export_complete = true; @@ -405,10 +405,10 @@ mod tests { #[test] fn each_event_writes_its_target_as_url_and_username_keys() { let dir = tempfile::tempdir().unwrap(); - let path = dir.path().join(PULL_JOURNAL_NAME); + let path = dir.path().join(EXPORT_JOURNAL_NAME); append( &path, - &PullJournalEvent::AssetOk { + &ExportJournalEvent::AssetOk { target: alice(), sha256: "aaa".into(), }, @@ -416,7 +416,7 @@ mod tests { .unwrap(); append( &path, - &PullJournalEvent::ExportComplete { + &ExportJournalEvent::ExportComplete { target: alice(), conversations: 2, messages: 3, diff --git a/crates/libs/pull/src/lib.rs b/crates/libs/export/src/lib.rs similarity index 60% rename from crates/libs/pull/src/lib.rs rename to crates/libs/export/src/lib.rs index dc816fb87..7acde622b 100644 --- a/crates/libs/pull/src/lib.rs +++ b/crates/libs/export/src/lib.rs @@ -1,4 +1,4 @@ -//! Pulls messages out of a running server as one Export Run: `POST /v1/exports` +//! Exports messages out of a running server as one Export Run: `POST /v1/exports` //! records what is asked for, `GET /v1/exports/{id}/messages` pages the rows, //! and `complete` or `cancel` closes the run. The messages are written as //! chat files. @@ -11,10 +11,10 @@ mod part_file; mod project; mod run; -pub use journal::{PULL_JOURNAL_NAME, PullJournalEvent, PullJournalState, journal_path}; +pub use journal::{EXPORT_JOURNAL_NAME, ExportJournalEvent, ExportJournalState, journal_path}; pub use message_crate_api_types::{ExportQueryList, ExportRun, ExportScope, Message}; pub use message_crate_http::{AuthError, AuthInfo, auth_check as authenticate}; pub use run::{ - DEFAULT_ASSET_FETCH_WORKERS, DEFAULT_PAGE_LIMIT, MAX_PAGE_LIMIT, ProgressEvent, ProgressFn, - PullConfig, PullReport, run, + DEFAULT_ASSET_FETCH_WORKERS, DEFAULT_PAGE_LIMIT, ExportConfig, ExportReport, MAX_PAGE_LIMIT, + ProgressEvent, ProgressFn, run, }; diff --git a/crates/libs/pull/src/part_file.rs b/crates/libs/export/src/part_file.rs similarity index 97% rename from crates/libs/pull/src/part_file.rs rename to crates/libs/export/src/part_file.rs index 689882cf9..bae4bf089 100644 --- a/crates/libs/pull/src/part_file.rs +++ b/crates/libs/export/src/part_file.rs @@ -30,7 +30,7 @@ fn part_path(dest: &Path, sha256: &str) -> PathBuf { /// file ([`part_path`]), with [`message_ir::write_atomic_via`]: `write` fills /// the temporary file, which is synced and renamed onto `dest` only when /// `write` succeeds, and removed when it fails. A synced rename matters, -/// because the pull journal records the Asset next, and a later Export Run +/// because the export journal records the Asset next, and a later Export Run /// skips an Asset the journal names whose file exists. /// /// # Errors diff --git a/crates/libs/pull/src/project.rs b/crates/libs/export/src/project.rs similarity index 99% rename from crates/libs/pull/src/project.rs rename to crates/libs/export/src/project.rs index 45304b3e8..8b4b21140 100644 --- a/crates/libs/pull/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -27,7 +27,7 @@ pub fn build_document( seed: &Message, messages: Vec, ) -> ConversationDocument { - // The server says whether the conversation is a group; the pull does + // The server says whether the conversation is a group; the Export Run does // not read `conversation_type` to decide it again. It reads it only to // tell a conversation of orphaned messages from a one-to-one // conversation, as the API type says: written back as one-to-one, its @@ -370,7 +370,7 @@ mod tests { /// builds the `message_crate_api_types` shapes in Rust, which is what let /// three of them drift away from what the server sends without the /// compiler or the suite noticing: `handle: String` rejected `"handle": null` and aborted every - /// pull of a conversation holding an address-less participant, and + /// Export Run of a conversation holding an address-less participant, and /// `conversation.service` read a field the server has never sent, so every /// pulled message came out `IrService::Unknown`. const EXPORT_PAGE_JSON: &str = r#"{ @@ -520,7 +520,7 @@ mod tests { ); } - /// The message a reply quotes comes through the pull into the message's + /// The message a reply quotes comes through the Export Run into the message's /// `reply_to`, and the stored reactions into the message's `reactions`, each /// under the person who reacted. #[test] diff --git a/crates/libs/pull/src/run.rs b/crates/libs/export/src/run.rs similarity index 97% rename from crates/libs/pull/src/run.rs rename to crates/libs/export/src/run.rs index 3e115cad9..f30feb350 100644 --- a/crates/libs/pull/src/run.rs +++ b/crates/libs/export/src/run.rs @@ -14,7 +14,7 @@ use message_ir_format::mark_export_directory; use serde::Serialize; use crate::http::{CloseAction, ExportMessagesArgs, HttpSession}; -use crate::journal::{self, PullJournalEvent, PullJournalState, ServerTarget}; +use crate::journal::{self, ExportJournalEvent, ExportJournalState, ServerTarget}; use crate::part_file::write_asset; use crate::project::{ExportPath, build_document, conversation_key, export_path, to_ir_message}; use message_crate_api_types::{ExportQueryList, ExportRun, ExportScope, Message}; @@ -24,7 +24,7 @@ pub const DEFAULT_PAGE_LIMIT: usize = 500; /// The largest page the server will hand back for `GET /v1/exports/{id}/messages`. pub const MAX_PAGE_LIMIT: usize = 500; /// The `tool` every run this crate creates is recorded under. -pub const TOOL_NAME: &str = "message-crate-pull"; +pub const TOOL_NAME: &str = "message-crate-export"; /// Default number of workers that fetch Assets in parallel. pub const DEFAULT_ASSET_FETCH_WORKERS: usize = 8; /// Extra tries for transient HTTP failures, matching the message-crate-push default. @@ -32,7 +32,7 @@ const MAX_RETRIES: u32 = 3; /// Settings for one Export Run (output directory, URL, search, flags). #[derive(Debug, Clone)] -pub struct PullConfig { +pub struct ExportConfig { /// Directory the JSON Lines files and attachments are written into. pub out_dir: PathBuf, /// Server base URL, e.g. `http://127.0.0.1:8080`. @@ -61,7 +61,7 @@ pub struct PullConfig { /// Final summary of an Export Run: conversations, messages, and the Assets /// fetched and kept. #[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct PullReport { +pub struct ExportReport { /// Account id the token resolved to. pub account: i64, /// The Export Run the server recorded for this export. @@ -104,7 +104,7 @@ pub enum ProgressEvent { total_so_far: u64, }, /// The run finished; the report is final. - Done(PullReport), + Done(ExportReport), } /// Callback type for live progress (desktop log panel, tests). @@ -163,23 +163,26 @@ fn prepare_out_dir(out_dir: &Path, skip_attachments: bool) -> Result<()> { /// Export the matching messages into `cfg.out_dir` as JSON Lines plus attachments. /// /// JSON Lines means one JSON object per line. A local journal -/// (`.message-crate-pull-state.jsonl`) records which Assets were already fetched so a +/// (`.message-crate-export-state.jsonl`) records which Assets were already fetched so a /// later run can skip them. /// /// # Errors /// /// Returns an error when the session token or output directory is missing, login fails, a /// page or Asset fetch fails, or a conversation file cannot be written. -pub fn run(cfg: &PullConfig, mut on_progress: Option<&mut ProgressFn<'_>>) -> Result { +pub fn run( + cfg: &ExportConfig, + mut on_progress: Option<&mut ProgressFn<'_>>, +) -> Result { if cfg.token.trim().is_empty() { bail!("session token is required"); } if cfg.out_dir.as_os_str().is_empty() { bail!("output directory is required"); } - let pull = Pull::login(cfg, &mut on_progress)?; + let exporter = Export::login(cfg, &mut on_progress)?; prepare_out_dir(&cfg.out_dir, cfg.skip_attachments)?; - if pull.journal.export_complete { + if exporter.journal.export_complete { emit( &mut on_progress, ProgressEvent::Log( @@ -190,8 +193,8 @@ pub fn run(cfg: &PullConfig, mut on_progress: Option<&mut ProgressFn<'_>>) -> Re // Nothing is recorded for a run the caller already gave up on. check_cancel(cfg.cancel.as_ref())?; - let export = pull.start_export(&mut on_progress)?; - let outcome = pull.export_into_directory(&export, &mut on_progress); + let export = exporter.start_export(&mut on_progress)?; + let outcome = exporter.export_into_directory(&export, &mut on_progress); // The client closes the run either way, so the server's record says how // it ended. A close that fails after the files are written is a warning, // not a failed export: the directory is complete, only the record is not. @@ -200,7 +203,7 @@ pub fn run(cfg: &PullConfig, mut on_progress: Option<&mut ProgressFn<'_>>) -> Re } else { CloseAction::Cancel }; - if let Err(error) = pull.close_export(export.id, action) { + if let Err(error) = exporter.close_export(export.id, action) { // A refused completion follows a run whose files are all written; a // refused cancellation follows the failure the run returns below. let line = match action { @@ -216,10 +219,10 @@ pub fn run(cfg: &PullConfig, mut on_progress: Option<&mut ProgressFn<'_>>) -> Re refused_paths, } = outcome?; - let report = PullReport { - account: pull.account, + let report = ExportReport { + account: exporter.account, export_id: export.id, - query: pull.query, + query: exporter.query, conversations, messages, assets_fetched: assets.fetched, @@ -273,8 +276,8 @@ struct Written { /// One authenticated Export Run: the connection, the account it resolved /// to, and the local journal of files already on disk. -struct Pull<'a> { - cfg: &'a PullConfig, +struct Export<'a> { + cfg: &'a ExportConfig, session: HttpSession, account: i64, /// The server and account this Export Run's journal lines belong to. @@ -282,16 +285,16 @@ struct Pull<'a> { /// The search query with surrounding whitespace removed. query: String, journal_path: PathBuf, - journal: PullJournalState, + journal: ExportJournalState, } -impl<'a> Pull<'a> { +impl<'a> Export<'a> { /// Check the token, announce the account and query, and load the journal. /// /// # Errors /// /// Returns an error when login fails or the journal cannot be read. - fn login(cfg: &'a PullConfig, out: &mut Option<&mut ProgressFn<'_>>) -> Result { + fn login(cfg: &'a ExportConfig, out: &mut Option<&mut ProgressFn<'_>>) -> Result { let auth = authenticate(&cfg.base_url, &cfg.token).map_err(|e| anyhow::anyhow!("{e}"))?; let account = auth.account_id; let username = auth.username; @@ -561,7 +564,7 @@ impl<'a> Pull<'a> { }; for sha in assets.keys() { if !self.journal.assets.contains(sha) { - let event = PullJournalEvent::AssetOk { + let event = ExportJournalEvent::AssetOk { target: self.target.clone(), sha256: sha.clone(), }; @@ -571,7 +574,7 @@ impl<'a> Pull<'a> { ProgressEvent::Log(format!( "Asset {sha} could not be added to {}, the record of \ fetched Assets: {error:#}", - journal::PULL_JOURNAL_NAME + journal::EXPORT_JOURNAL_NAME )), ); } @@ -631,7 +634,7 @@ impl<'a> Pull<'a> { assets: &AssetCounts, seen_assets: HashMap, ) { - let event = PullJournalEvent::ExportComplete { + let event = ExportJournalEvent::ExportComplete { target: self.target.clone(), conversations, messages, @@ -642,13 +645,13 @@ impl<'a> Pull<'a> { out, ProgressEvent::Log(format!( "Export Run {export_id} could not be recorded as finished in {}: {error:#}", - journal::PULL_JOURNAL_NAME + journal::EXPORT_JOURNAL_NAME )), ); } let mut recorded_assets = self.journal.assets.clone(); recorded_assets.extend(seen_assets.into_keys()); - let final_state = PullJournalState { + let final_state = ExportJournalState { assets: recorded_assets, export_complete: true, }; @@ -657,7 +660,7 @@ impl<'a> Pull<'a> { out, ProgressEvent::Log(format!( "{} could not be rewritten in its shortest form: {error:#}", - journal::PULL_JOURNAL_NAME + journal::EXPORT_JOURNAL_NAME )), ); } diff --git a/crates/libs/pull/tests/pull_mock.rs b/crates/libs/export/tests/export_mock.rs similarity index 96% rename from crates/libs/pull/tests/pull_mock.rs rename to crates/libs/export/tests/export_mock.rs index 890bd2ea4..d9ad9acdf 100644 --- a/crates/libs/pull/tests/pull_mock.rs +++ b/crates/libs/export/tests/export_mock.rs @@ -1,4 +1,4 @@ -//! Mock server tests for one pull: login, the Export Run it records, two +//! Mock server tests for one Export: login, the Export Run it records, two //! pages of messages, Asset fetches, the journal a second run reads, and //! the progress a caller sees. //! @@ -7,7 +7,7 @@ //! `POST /v1/exports/{id}/complete` or `/cancel`, and `GET /v1/assets/{sha256}` //! — with the JSON the server serializes (`message-crate-api-types`, //! `docs/src/assets/openapi.json`). Every request derives from -//! `PullConfig::base_url`, so the mock's address is the only seam. +//! `ExportConfig::base_url`, so the mock's address is the only seam. use std::collections::HashSet; use std::fs; @@ -16,8 +16,8 @@ use std::sync::Arc; use std::sync::atomic::AtomicBool; use httpmock::prelude::*; -use message_crate_pull::{ - ExportQueryList, ProgressEvent, PullConfig, PullJournalState, PullReport, journal, run, +use message_crate_export::{ + ExportConfig, ExportJournalState, ExportQueryList, ExportReport, ProgressEvent, journal, run, }; use message_ir::{Deletion, EarlierVersion, Reaction}; use message_ir_format::{EXPORT_SENTINEL, read_conversation_jsonl}; @@ -33,7 +33,7 @@ const PHOTO_SHA: &str = "be04c407026cf352d54a051993ef2fec8153cc554f8ff7c697b225c const MENU_BYTES: &[u8] = b"%PDF-1.4 menu"; /// 9 bytes. const PHOTO_BYTES: &[u8] = b"PNG photo"; -/// The file a pull of `+15555550101` from `sms-backup-restore` writes. +/// The file an Export Run of `+15555550101` from `sms-backup-restore` writes. const CONVERSATION_FILE: &str = "+15555550101__sms-backup-restore.jsonl"; /// The id the mock server gives every run it records. const EXPORT_ID: i64 = 7; @@ -49,7 +49,7 @@ fn menu_attachment(path: Value) -> Value { }) } -/// The photo attachment: no `path`, so the pull files it under its fingerprint. +/// The photo attachment: no `path`, so the Export Run files it under its fingerprint. fn photo_attachment() -> Value { json!({ "path": null, @@ -108,7 +108,7 @@ fn export_run(scope: Value, status: &str) -> Value { json!({ "id": EXPORT_ID, "scope": scope, - "tool": "message-crate-pull", + "tool": "message-crate-export", "status": status, "started_at": "2026-09-08T12:00:00Z", "finished_at": if status == "running" { Value::Null } else { json!("2026-09-08T12:00:09Z") }, @@ -129,9 +129,9 @@ fn mock_auth(server: &MockServer) -> httpmock::Mock<'_> { }) } -/// `POST /v1/exports` for exactly `scope`, recorded as `message-crate-pull`'s run. +/// `POST /v1/exports` for exactly `scope`, recorded as `message-crate-export`'s run. fn mock_create<'a>(server: &'a MockServer, scope: Value) -> httpmock::Mock<'a> { - let body = json!({ "scope": scope.clone(), "tool": "message-crate-pull" }); + let body = json!({ "scope": scope.clone(), "tool": "message-crate-export" }); server.mock(move |when, then| { when.method(POST) .path("/v1/exports") @@ -245,10 +245,10 @@ fn part_files_in(dir: &Path) -> Vec { .collect() } -/// A pull of every message into `out_dir`: two messages a page, one fetch +/// An Export Run of every message into `out_dir`: two messages a page, one fetch /// worker so the counts in the log are fixed. -fn config(out_dir: &Path, base_url: String) -> PullConfig { - PullConfig { +fn config(out_dir: &Path, base_url: String) -> ExportConfig { + ExportConfig { out_dir: out_dir.to_path_buf(), base_url, token: "mc_test".into(), @@ -264,7 +264,7 @@ fn config(out_dir: &Path, base_url: String) -> PullConfig { /// The journal a run against `server` wrote in `out_dir` for `alice`. The /// run writes every line itself, so the test fails on any line /// `journal::load` could not read. -fn load_journal(out_dir: &Path, server: &MockServer) -> PullJournalState { +fn load_journal(out_dir: &Path, server: &MockServer) -> ExportJournalState { let mut unreadable = Vec::new(); let state = journal::load( &journal::journal_path(out_dir), @@ -277,8 +277,8 @@ fn load_journal(out_dir: &Path, server: &MockServer) -> PullJournalState { } /// The report `run` returns for the three-message fixture. -fn report_for(out_dir: &Path, fetched: u64, kept: u64) -> PullReport { - PullReport { +fn report_for(out_dir: &Path, fetched: u64, kept: u64) -> ExportReport { + ExportReport { account: 1, export_id: EXPORT_ID, query: String::new(), @@ -292,7 +292,7 @@ fn report_for(out_dir: &Path, fetched: u64, kept: u64) -> PullReport { } #[test] -fn a_pull_records_one_run_and_writes_the_conversation_and_every_asset_once_across_two_pages() { +fn an_export_records_one_run_and_writes_the_conversation_and_every_asset_once_across_two_pages() { let server = MockServer::start(); let _auth = mock_auth(&server); let (create, complete) = mock_run(&server); @@ -555,7 +555,7 @@ fn a_second_run_over_the_same_directory_fetches_nothing_it_already_has() { assert_eq!(report, report_for(&out, 0, 2)); assert_eq!(menu.calls(), 1); assert_eq!(photo.calls(), 1); - // Every pull is its own run, whether or not it fetches anything. + // Every Export is its own Export Run, whether or not it fetches anything. assert_eq!(create.calls(), 2); assert_eq!(complete.calls(), 2); assert_eq!( @@ -571,7 +571,7 @@ fn a_second_run_over_the_same_directory_fetches_nothing_it_already_has() { ); } -/// A line of the pull-state file the Export cannot read is named in the +/// A line of the export-state file the Export cannot read is named in the /// Export's log, by its line in the file, and costs no fetch: the Assets it /// might have recorded are on disk and are kept (#1910). #[test] @@ -647,7 +647,7 @@ fn a_file_the_journal_lists_but_the_disk_lost_is_fetched_again() { } /// A run that stopped after writing its Assets but before recording them in -/// the pull-state file leaves files the next run keeps without fetching. The +/// the export-state file leaves files the next run keeps without fetching. The /// "Fetching" line counts each of them as already on disk, as the fetch does, /// so it agrees with the "Fetched" line after it, and the "Fetched" line /// counts only the bytes this run fetched. @@ -694,8 +694,8 @@ fn a_file_on_disk_the_journal_does_not_list_is_kept_and_counted_as_on_disk() { } /// With every Asset already on disk, a run logs no "Fetching" or "Fetched" -/// line whether or not the pull-state file lists the Assets, because what is -/// on disk, not the pull-state file, decides what is fetched. +/// line whether or not the export-state file lists the Assets, because what is +/// on disk, not the export-state file, decides what is fetched. #[test] fn every_file_on_disk_logs_no_fetch_lines_whether_or_not_the_journal_lists_it() { let server = MockServer::start(); @@ -742,7 +742,7 @@ fn a_cancel_requested_before_the_run_records_nothing_on_the_server() { let (first, _second) = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); let out = dir.path().join("pulled"); - let cfg = PullConfig { + let cfg = ExportConfig { cancel: Some(Arc::new(AtomicBool::new(true))), ..config(&out, server.base_url()) }; @@ -827,7 +827,7 @@ fn two_groups_with_one_title_are_both_written() { .unwrap() .filter_map(|e| e.ok()) .filter(|e| { - // The pull's own journal is a hidden JSONL file beside them. + // The Export Run's own journal is a hidden JSONL file beside them. let name = e.file_name().to_string_lossy().to_string(); name.ends_with(".jsonl") && !name.starts_with('.') }) @@ -856,7 +856,7 @@ fn skipping_attachments_writes_messages_without_files_or_fetches() { let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); let out = dir.path().join("pulled"); - let cfg = PullConfig { + let cfg = ExportConfig { skip_attachments: true, ..config(&out, server.base_url()) }; @@ -886,7 +886,7 @@ fn a_query_becomes_the_runs_query_scope_and_progress_narrates_the_run() { let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); let out = dir.path().join("pulled"); - let cfg = PullConfig { + let cfg = ExportConfig { query: " from:sam ".into(), ..config(&out, server.base_url()) }; @@ -944,7 +944,7 @@ fn a_query_for_the_conversations_list_names_that_list_in_the_scope() { let _pages = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); let out = dir.path().join("pulled"); - let cfg = PullConfig { + let cfg = ExportConfig { query: "messages:>100".into(), list: ExportQueryList::Conversations, skip_attachments: true, @@ -1073,7 +1073,7 @@ fn a_scope_the_server_refuses_fails_the_run_with_the_servers_sentence() { let (first, _second) = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); let out = dir.path().join("pulled"); - let cfg = PullConfig { + let cfg = ExportConfig { query: "wibble:yes".into(), ..config(&out, server.base_url()) }; @@ -1134,11 +1134,11 @@ fn a_refused_completion_is_a_warning_that_names_the_run_once() { fn a_blank_token_or_output_directory_is_refused_before_login() { let dir = tempdir().unwrap(); let base_url = "http://127.0.0.1:1".to_string(); - let blank_token = PullConfig { + let blank_token = ExportConfig { token: " ".into(), ..config(dir.path(), base_url.clone()) }; - let blank_out_dir = PullConfig { + let blank_out_dir = ExportConfig { out_dir: Path::new("").to_path_buf(), ..config(dir.path(), base_url) }; @@ -1176,9 +1176,9 @@ fn a_refused_session_says_to_log_in_again() { /// Staging names a file by date and fingerprint, so one menu sent on two days /// has two paths on the server. The menu is fetched once, and every path a -/// message names exists after the pull holding the menu's bytes. +/// message names exists after the Export Run holding the menu's bytes. #[test] -fn every_path_a_message_names_exists_after_a_pull() { +fn every_path_a_message_names_exists_after_an_export() { let server = MockServer::start(); let _auth = mock_auth(&server); let (_create, _complete) = mock_run(&server); @@ -1216,7 +1216,7 @@ fn every_path_a_message_names_exists_after_a_pull() { assert_eq!( fs::read(out.join(rel)).ok().as_deref(), Some(MENU_BYTES), - "message {} names {rel}, which the pull never wrote", + "message {} names {rel}, which the Export Run never wrote", msg.guid ); } diff --git a/crates/libs/http/Cargo.toml b/crates/libs/http/Cargo.toml index fb4a81339..cb7a8590a 100644 --- a/crates/libs/http/Cargo.toml +++ b/crates/libs/http/Cargo.toml @@ -2,7 +2,7 @@ name = "message-crate-http" version = "0.1.0" edition = "2024" -description = "Blocking HTTP client helpers and typed retry classification for the push and pull crates" +description = "Blocking HTTP client helpers and typed retry classification for the push and export crates" license = "LicenseRef-FCL-1.0-ALv2" [dependencies] diff --git a/crates/libs/http/README.md b/crates/libs/http/README.md index a67b9ac05..3e387e38f 100644 --- a/crates/libs/http/README.md +++ b/crates/libs/http/README.md @@ -9,7 +9,7 @@ probe; `ok_json` reads every server answer, turning a failure into the `detail` sentence of the server's RFC 7807 problem document rather than a status code; and `classify_retry` decides which failures are worth trying again. -`message-crate-push` and `message-crate-pull` use this crate, and the desktop app reaches +`message-crate-push` and `message-crate-export` use this crate, and the desktop app reaches `AuthError` through their re-exports. ## Build and test @@ -22,7 +22,7 @@ Workspace setup: [CONTRIBUTING.md](../../../CONTRIBUTING.md). ## Docs -This crate is a library used by the push and pull crates. It builds no binary. +This crate is a library used by the push and export crates. It builds no binary. ## License diff --git a/crates/libs/http/src/lib.rs b/crates/libs/http/src/lib.rs index c724db528..7b650378d 100644 --- a/crates/libs/http/src/lib.rs +++ b/crates/libs/http/src/lib.rs @@ -1,7 +1,7 @@ -//! Blocking HTTP client helpers and retry classification for the push and pull +//! Blocking HTTP client helpers and retry classification for the push and export //! crates. //! -//! `message-crate-push` and `message-crate-pull` both talk to the server through one +//! `message-crate-push` and `message-crate-export` both talk to the server through one //! [`HttpSession`] (built on [`build_client`]), log in through //! [`auth_check`], share [`truncate`] for error snippets, and classify //! retryable failures through `classify_retry` / `with_retries`. diff --git a/crates/libs/http/src/response.rs b/crates/libs/http/src/response.rs index 5d9d02f93..4d43b285b 100644 --- a/crates/libs/http/src/response.rs +++ b/crates/libs/http/src/response.rs @@ -4,7 +4,7 @@ //! Every route the server serves answers a failure with an RFC 7807 problem //! document (`docs/architecture/http-api.md`), and its `detail` is written for the person to read. //! Both client crates were reading it themselves — `message-crate-push` with an -//! `ok_json` helper, `message-crate-pull` with an `error_sentence` one — over two +//! `ok_json` helper, `message-crate-export` with an `error_sentence` one — over two //! private copies of the same struct. One copy of the reading lives here, over //! the shared [`Problem`] type, so a change to the server's failure shape is //! one edit rather than a hunt. diff --git a/crates/libs/http/src/session.rs b/crates/libs/http/src/session.rs index 6e8ee8e32..0d53c653d 100644 --- a/crates/libs/http/src/session.rs +++ b/crates/libs/http/src/session.rs @@ -1,6 +1,6 @@ //! The shared blocking HTTP session and the `GET /v1/session` login call. //! -//! `message-crate-push` and `message-crate-pull` both talk to the server through one +//! `message-crate-push` and `message-crate-export` both talk to the server through one //! [`HttpSession`]. The session owns base-URL trimming and bearer-header //! construction so no caller formats `Authorization` by hand. diff --git a/crates/libs/ir/README.md b/crates/libs/ir/README.md index ed0e74a80..53d627bc1 100644 --- a/crates/libs/ir/README.md +++ b/crates/libs/ir/README.md @@ -2,7 +2,7 @@ Shared conversation types for Message Crate: `ConversationDocument`, messages, attachments, and participants. This crate has no I/O and no formatting. Attachment bytes are never serialized to JSON; paths and hashes point at sidecar files. -Exporters, `message-ir-format`, `message-crate-push`, `message-crate-pull`, and the server use this crate. +Exporters, `message-ir-format`, `message-crate-push`, `message-crate-export`, and the server use this crate. ## Build and test diff --git a/crates/libs/journal/README.md b/crates/libs/journal/README.md index 9fb347786..a9ad7b421 100644 --- a/crates/libs/journal/README.md +++ b/crates/libs/journal/README.md @@ -5,7 +5,7 @@ journal is one JSON object per line, and readers rebuild skip-sets from it, so a run that stops partway can pick up where it left off rather than starting over. -`message-crate-push` and `message-crate-pull` use this crate to resume a transfer. +`message-crate-push` and `message-crate-export` use this crate to resume a transfer. ## Build and test @@ -17,7 +17,7 @@ Workspace setup: [CONTRIBUTING.md](../../../CONTRIBUTING.md). ## Docs -This crate is a library used by the push and pull crates, which the desktop app +This crate is a library used by the push and export crates, which the desktop app calls in process. It builds no binary. ## License diff --git a/crates/server/server/src/db/conversation_messages.rs b/crates/server/server/src/db/conversation_messages.rs index e040a58c4..3926bed7f 100644 --- a/crates/server/server/src/db/conversation_messages.rs +++ b/crates/server/server/src/db/conversation_messages.rs @@ -2,7 +2,7 @@ //! conversation, attachments, tapbacks and earlier versions, joined and //! grouped. //! -//! The row shapes themselves live in `message-crate-api-types`, where `message-crate-pull` +//! The row shapes themselves live in `message-crate-api-types`, where `message-crate-export` //! reads them from the same definition rather than a hand-written mirror. //! //! `load_messages` takes an already-compiled `WHERE` fragment and its bound diff --git a/crates/server/server/src/exports_api.rs b/crates/server/server/src/exports_api.rs index 7672e5395..8a4a29347 100644 --- a/crates/server/server/src/exports_api.rs +++ b/crates/server/server/src/exports_api.rs @@ -42,7 +42,7 @@ pub(crate) struct OwnerExportRun { id: i64, /// The form of the scope the run asked for. scope_kind: ExportScopeKind, - /// Exporting tool, e.g. `message-crate-pull`, when the client named one. + /// Exporting tool, e.g. `message-crate-export`, when the client named one. tool: Option, /// Lifecycle status. status: ExportStatus, @@ -228,7 +228,7 @@ pub async fn scope_filter( pub(crate) struct CreateExportRequest { /// What to export. pub(crate) scope: ExportScope, - /// Client/tool name recorded on the run, e.g. `message-crate-pull`. + /// Client/tool name recorded on the run, e.g. `message-crate-export`. #[serde(default)] pub(crate) tool: Option, } diff --git a/docs/adr/0001-no-command-line-except-the-server.md b/docs/adr/0001-no-command-line-except-the-server.md index 475bb2104..c1d824769 100644 --- a/docs/adr/0001-no-command-line-except-the-server.md +++ b/docs/adr/0001-no-command-line-except-the-server.md @@ -106,3 +106,10 @@ The binary is a convenience for working on the repository, run through database schemas, not for stored data, and not for URLs — so those addresses return 404 rather than pointing somewhere that does not answer the question they were bookmarked for. + +## Note, 2026-10-05: `message-crate-pull` is now `message-crate-export` + +The text above is kept as it was decided. Since #1906, the crate it calls +`message-crate-pull` is `message-crate-export`, at `crates/libs/export/`, +because CONTEXT.md names the operation Export and avoids Pull. Nothing else +in this decision changed. diff --git a/docs/adr/0012-four-crates-in-the-export-pipeline.md b/docs/adr/0012-four-crates-in-the-export-pipeline.md index 957f78215..8b1d18dda 100644 --- a/docs/adr/0012-four-crates-in-the-export-pipeline.md +++ b/docs/adr/0012-four-crates-in-the-export-pipeline.md @@ -212,3 +212,10 @@ case into Convert would have to be undone. - Nothing here is kept for compatibility. Public items are renamed, moved between crates and removed wherever the result is simpler, and tests are rewritten to match rather than preserved. + +## Note, 2026-10-05: `message-crate-pull` is now `message-crate-export` + +The text above is kept as it was decided. Since #1906, the crate it calls +`message-crate-pull` is `message-crate-export`, at `crates/libs/export/`, +because CONTEXT.md names the operation Export and avoids Pull. Its journal is +now `.message-crate-export-state.jsonl`. Nothing else in this decision changed. diff --git a/docs/architecture/http-api.md b/docs/architecture/http-api.md index c22509765..4532cb629 100644 --- a/docs/architecture/http-api.md +++ b/docs/architecture/http-api.md @@ -440,7 +440,7 @@ security scheme with its scopes, so every route says which it accepts. with a session) or by its expiry, never by the program holding it. - `GET /v1/session` answers whose credential the caller holds — the account's id and username — for a session or a token. Why: a program holding a token - needs to know which account it writes to before it starts, and push and pull + needs to know which account it writes to before it starts, and push and export label their work with it. `DELETE /v1/session` refuses a token with `403`, because a token is not a Session and a `204` would say something ended when nothing did. diff --git a/docs/src/assets/openapi.json b/docs/src/assets/openapi.json index 9169a7b53..225ef6f4b 100644 --- a/docs/src/assets/openapi.json +++ b/docs/src/assets/openapi.json @@ -14279,7 +14279,7 @@ "string", "null" ], - "description": "Client/tool name recorded on the run, e.g. `message-crate-pull`." + "description": "Client/tool name recorded on the run, e.g. `message-crate-export`." } } }, @@ -14773,7 +14773,7 @@ "string", "null" ], - "description": "Exporting tool, e.g. `message-crate-pull`, when the client named one." + "description": "Exporting tool, e.g. `message-crate-export`, when the client named one." }, "total_bytes": { "type": "integer", @@ -15982,7 +15982,7 @@ "string", "null" ], - "description": "Exporting tool, e.g. `message-crate-pull`, when the client named one." + "description": "Exporting tool, e.g. `message-crate-export`, when the client named one." }, "total_bytes": { "type": "integer", @@ -17185,7 +17185,7 @@ "string", "null" ], - "description": "Exporting tool, e.g. `message-crate-pull`, when the client named one." + "description": "Exporting tool, e.g. `message-crate-export`, when the client named one." }, "total_bytes": { "type": "integer", @@ -18020,7 +18020,7 @@ "string", "null" ], - "description": "Exporting tool, e.g. `message-crate-pull`, when the client named one." + "description": "Exporting tool, e.g. `message-crate-export`, when the client named one." }, "total_bytes": { "type": "integer", diff --git a/docs/src/content/docs/docs/developer/design.md b/docs/src/content/docs/docs/developer/design.md index b24ef7e2d..8d57d0dee 100644 --- a/docs/src/content/docs/docs/developer/design.md +++ b/docs/src/content/docs/docs/developer/design.md @@ -20,7 +20,7 @@ message-crate │ ├── core/ # shared import/export job settings used by the desktop app │ ├── exporters/ # parse iMessage, WhatsApp, SMS, and other backups into JSONL │ ├── libs/ # shared code the exporters and the server use (format, contacts, -│ │ # media, message-crate-push, message-crate-pull) +│ │ # media, message-crate-push, message-crate-export) │ └── server/ # message-crate-server (API + SQLite) and demo-seed (sample inbox) ├── docker/ # image and Compose file that look like a published install ├── docs/ # messagecrate.app (User Guide, Developer docs, landing page) @@ -58,7 +58,7 @@ and its amendment. | `go-sms-pro-exporter`, `imazing-exporter`, `openextract-exporter`, `sms-backup-plus-exporter` | `crates/exporters/` | Rescue / experimental extract | | `ios-backup` | `crates/libs/ios-backup/` | Check an iPhone backup before an import and decrypt one of its domains, through `imessage-reader` | | `message-reexport` | `crates/libs/reexport/` | Convert an existing export directory | -| `message-crate-push` / `message-crate-pull` | `crates/libs/` | JSONL → running server / server → JSONL | +| `message-crate-push` / `message-crate-export` | `crates/libs/` | JSONL → running server / server → JSONL | C4 PlantUML sources and SVG exports live in [`docs/src/assets/architecture/`](https://github.com/messagecrate/message-crate/tree/main/docs/src/assets/architecture). Edit the `.puml` file, export SVG into the same directory, and commit both in one change. @@ -190,7 +190,7 @@ sequenceDiagram #### How an export runs -Messages and attachments are downloaded from the server using the `message-crate-pull` library. +Messages and attachments are downloaded from the server using the `message-crate-export` library. ```mermaid sequenceDiagram diff --git a/docs/src/content/docs/docs/developer/message-transfer.md b/docs/src/content/docs/docs/developer/message-transfer.md index 8bb6bce6b..78965d276 100644 --- a/docs/src/content/docs/docs/developer/message-transfer.md +++ b/docs/src/content/docs/docs/developer/message-transfer.md @@ -23,7 +23,7 @@ flowchart LR ## How chats come back out -Export copies chats from a running server into a new directory of the same chat files. In the desktop app this is the **Export** screen, which uses the `message-crate-pull` library. +Export copies chats from a running server into a new directory of the same chat files. In the desktop app this is the **Export** screen, which uses the `message-crate-export` library. ```mermaid flowchart LR @@ -75,5 +75,5 @@ These do not read a phone backup. They load or save the chat-file directory, or | Library | What it does | |---------|----------------| | `message-crate-push` | Loads a chat-file directory into a running server. Used by **Import**. | -| `message-crate-pull` | Writes a chat-file directory from a running server. Used by **Export**. | +| `message-crate-export` | Writes a chat-file directory from a running server. Used by **Export**. | | `message-reexport` | Turns an existing Message Crate export directory into another format, such as CSV or mail. Used by **Export** for any format other than JSON Lines. See [Convert an existing export](/docs/developer/formats/convert/). | diff --git a/docs/src/content/docs/docs/developer/reference/api.md b/docs/src/content/docs/docs/developer/reference/api.md index f61f9a4aa..e2d919bb6 100644 --- a/docs/src/content/docs/docs/developer/reference/api.md +++ b/docs/src/content/docs/docs/developer/reference/api.md @@ -9,7 +9,7 @@ Three places describe the server's `/v1` interface, and this page is the smalles - The [HTTP interface rules](https://github.com/messagecrate/message-crate/blob/main/docs/architecture/http-api.md) state what every route must do, each rule with its reason: route shape, lists and paging, failures as problem documents, credentials and what each reaches, and runs. - This page walks through an Import Run and an Export Run from start to finish, and lists the words of the search language. -Day-to-day import uses the desktop [Import](/docs/user/features/messages/import/) screen and download uses [Export](/docs/user/features/messages/export/). Both call this API with [JSONL](/docs/developer/reference/export-structure/) and attachment bytes keyed by SHA-256, through the `message-crate-push` and `message-crate-pull` libraries. +Day-to-day import uses the desktop [Import](/docs/user/features/messages/import/) screen and download uses [Export](/docs/user/features/messages/export/). Both call this API with [JSONL](/docs/developer/reference/export-structure/) and attachment bytes keyed by SHA-256, through the `message-crate-push` and `message-crate-export` libraries. ## Tokens @@ -62,7 +62,7 @@ A batch holds one pooled database connection for the whole of its work: parsing Every export is an Export Run, and there is no unrecorded export. `POST /v1/exports` creates one and answers `201 Created` with the run: what was asked for, and the four counts the server computed for it at creation — messages, conversations, distinct attachments, and their bytes. The body names a `scope` in one of three forms, stored as given, and an optional `tool`: ```json title="POST /v1/exports" -{ "scope": { "kind": "everything" }, "tool": "message-crate-pull" } +{ "scope": { "kind": "everything" }, "tool": "message-crate-export" } { "scope": { "kind": "query", "list": "messages", "q": "from:me date:>2024" } } { "scope": { "kind": "query", "list": "conversations", "q": "messages:>100" } } { "scope": { "kind": "selection", "conversation_ids": [12, 40], "message_ids": [913] } } diff --git a/docs/src/content/docs/docs/developer/rustdoc-style.md b/docs/src/content/docs/docs/developer/rustdoc-style.md index 2284a0ce3..932ac6dcb 100644 --- a/docs/src/content/docs/docs/developer/rustdoc-style.md +++ b/docs/src/content/docs/docs/developer/rustdoc-style.md @@ -59,7 +59,7 @@ Keep `# Errors` rustdoc sections out of handler docs that become OpenAPI descrip Document every public item — type, variant, field, const, function. The workspace `Cargo.toml` and `src-tauri/Cargo.toml` set `missing_docs = "warn"` under `[lints.rust]`, and Clippy runs with `-D warnings`, so an undocumented item on a crate's public surface fails CI. The lint does not see `pub` items in a private module or `pub(crate)` items, so those still need a reader. - `crates/server/demo-seed/src/personas.rs`, `const EMPTY_GROUP_HANDLE`, `const EMPTY_THREAD_HANDLE`, `const ORPHAN_SENDER` — no doc on any of the three, and `struct Roster` just above them documents the struct but none of its fields. Bad: the module is private, so the lint is silent, and nothing says why an empty group, an empty thread, and an orphaned sender exist in the demo data. -- `crates/libs/pull/src/http.rs`, `struct ExportMessagesArgs` — `export_id` is documented and `base_url`, `key`, `limit`, `offset` are not. Bad: forgotten rather than deliberate, and `pub(crate)` keeps it out of the lint. +- `crates/libs/export/src/http.rs`, `struct ExportMessagesArgs` — `export_id` is documented and `base_url`, `key`, `limit`, `offset` are not. Bad: forgotten rather than deliberate, and `pub(crate)` keeps it out of the lint. - `crates/libs/sbr/src/read.rs`, `MmsPart::filename_attr` — "Filename from the XML `fn` attribute (not a function attribute)." — Good: the field used to be `fn_attr` with no doc; the rename and the doc together say what it holds and head off the misreading. - `crates/core/message-crate-core/src/exporters.rs`, `struct Form` — every one of its fields carries a doc, for example "Packaging format projected from the common message (`json` default)." on `output_format`. Good: the GUI-facing form is where an undocumented field costs the most, because the desktop app is built against it. diff --git a/docs/src/content/docs/docs/user/features/messages/export.md b/docs/src/content/docs/docs/user/features/messages/export.md index 6bd23edb3..251ca0cdd 100644 --- a/docs/src/content/docs/docs/user/features/messages/export.md +++ b/docs/src/content/docs/docs/user/features/messages/export.md @@ -108,7 +108,7 @@ The disk that holds the Export Directory needs room for a second copy of the exp A directory chosen under **Save to** gets the result instead, and the export's directory in the Export Directory holds only the JSON Lines copy while the conversion runs. A JSON Lines export writes straight into the chosen directory. -It also keeps a file named `.message-crate-pull-state.jsonl` there, which records the attachments already fetched from each server and account. +It also keeps a file named `.message-crate-export-state.jsonl` there, which records the attachments already fetched from each server and account. A later JSON Lines export into the same directory, from the same server and account, skips those attachments. An export into its own directory in the Export Directory deletes that file when it finishes, since nothing exports into that directory again. diff --git a/scripts/coverage.sh b/scripts/coverage.sh index 9a136384f..418b8da08 100755 --- a/scripts/coverage.sh +++ b/scripts/coverage.sh @@ -22,7 +22,7 @@ # ffmpeg on PATH matters: the transcode and media tests skip themselves # without it, and what they would have called then shows as uncovered. # src-tauri is not a workspace member and is not measured; its commands are -# thin wrappers over the exporter and push/pull crates, which are. +# thin wrappers over the exporter, push, and export crates, which are. # Test code itself (tests/ directories and the /tests.rs files) is # left out of the numbers. Coverage is a report, never a gate: # docs/adr/0007-ci-is-the-only-gate.md. The Coverage workflow runs this diff --git a/src-tauri/Cargo.lock b/src-tauri/Cargo.lock index a5e494808..466107a3b 100644 --- a/src-tauri/Cargo.lock +++ b/src-tauri/Cargo.lock @@ -2346,8 +2346,8 @@ dependencies = [ "ios-backup", "media", "message-crate-core", + "message-crate-export", "message-crate-http", - "message-crate-pull", "message-crate-push", "message-crate-serve-protocol", "message-ir", @@ -2370,35 +2370,35 @@ dependencies = [ ] [[package]] -name = "message-crate-http" +name = "message-crate-export" version = "0.1.0" dependencies = [ "anyhow", + "chrono", + "hex", + "journal", + "media", "message-crate-api-types", + "message-crate-core", + "message-crate-http", + "message-ir", + "message-ir-format", "reqwest 0.12.28", "serde", "serde_json", - "thiserror 2.0.21", + "sha2 0.11.0", ] [[package]] -name = "message-crate-pull" +name = "message-crate-http" version = "0.1.0" dependencies = [ "anyhow", - "chrono", - "hex", - "journal", - "media", "message-crate-api-types", - "message-crate-core", - "message-crate-http", - "message-ir", - "message-ir-format", "reqwest 0.12.28", "serde", "serde_json", - "sha2 0.11.0", + "thiserror 2.0.21", ] [[package]] diff --git a/src-tauri/Cargo.toml b/src-tauri/Cargo.toml index f5aa761e3..a03a1cc40 100644 --- a/src-tauri/Cargo.toml +++ b/src-tauri/Cargo.toml @@ -40,7 +40,7 @@ imessage-ir-exporter = { path = "../crates/exporters/imessage-ir-exporter", defa ios-backup = { path = "../crates/libs/ios-backup" } message-reexport = { path = "../crates/libs/reexport" } message-crate-push = { path = "../crates/libs/push" } -message-crate-pull = { path = "../crates/libs/pull" } +message-crate-export = { path = "../crates/libs/export" } message-crate-http = { path = "../crates/libs/http" } message-crate-serve-protocol = { path = "../crates/libs/serve-protocol" } # The temporary file an attachment download is written to before it is diff --git a/src-tauri/src/commands/pull.rs b/src-tauri/src/commands/export.rs similarity index 86% rename from src-tauri/src/commands/pull.rs rename to src-tauri/src/commands/export.rs index ef7f578bd..3f236a123 100644 --- a/src-tauri/src/commands/pull.rs +++ b/src-tauri/src/commands/export.rs @@ -1,30 +1,30 @@ -//! `pull` command — export messages from a Message Crate server. +//! `export` command — export messages from a Message Crate server. use std::path::PathBuf; use std::sync::{Arc, Mutex}; use message_crate_core::count_of; -use message_crate_pull::{ - DEFAULT_ASSET_FETCH_WORKERS, DEFAULT_PAGE_LIMIT, ExportQueryList, ProgressEvent, PullConfig, - run as run_pull, +use message_crate_export::{ + DEFAULT_ASSET_FETCH_WORKERS, DEFAULT_PAGE_LIMIT, ExportConfig, ExportQueryList, ProgressEvent, + run as run_export, }; use super::events; use super::jobs::{spawn_job, start_job}; use crate::state::{AppState, JobName}; -/// User-facing parameters for the `pull` command. +/// User-facing parameters for the `export` command. #[derive(Debug, serde::Deserialize)] #[serde(rename_all = "camelCase")] -pub struct PullArgs { +pub struct ExportArgs { /// Base URL of the server, for example `http://127.0.0.1:8080`. pub base_url: String, /// The logged-in Session's token, sent as the bearer token. Never an API /// Token, and never a password. pub token: String, - /// Directory the pulled conversation files are written into. + /// Directory the exported conversation files are written into. pub out_dir: String, - /// Search query selecting what to pull. Blank pulls everything. + /// Search query selecting what to export. Blank exports everything. pub query: String, /// The list the query is for, `conversations` or `messages`: every /// message of the conversations the query shows, or the messages it @@ -46,17 +46,17 @@ pub struct PullArgs { /// while holding the shared state lock. Failures during the Export are /// sent as `extract:error`. #[tauri::command(async)] -pub fn pull( +pub fn export( state: tauri::State<'_, Arc>>, app: tauri::AppHandle, - args: PullArgs, + args: ExportArgs, ) -> Result<(), String> { let job = start_job(&state, JobName::Export)?; let cancel = job.cancel_flag(); let app_handle = app.clone(); spawn_job(app, job, move || { - let cfg = PullConfig { + let cfg = ExportConfig { out_dir: PathBuf::from(&args.out_dir), base_url: args.base_url, token: args.token, @@ -86,7 +86,7 @@ pub fn pull( ProgressEvent::Done(_) => {} }; - let report = run_pull(&cfg, Some(&mut progress))?; + let report = run_export(&cfg, Some(&mut progress))?; Ok(finished_line(report.messages, report.conversations)) }); diff --git a/src-tauri/src/commands/jobs.rs b/src-tauri/src/commands/jobs.rs index 5c917e86f..8015e9d09 100644 --- a/src-tauri/src/commands/jobs.rs +++ b/src-tauri/src/commands/jobs.rs @@ -1,5 +1,5 @@ //! Shared scaffolding for the background job commands (`extract`, `format`, -//! `pull`, `upload`, `transcode_staging`). +//! `export`, `upload`, `transcode_staging`). //! //! One job runs at a time in this process. Every job reports on the same //! `extract:*` events, which do not say which job sent them, so a second job diff --git a/src-tauri/src/commands/mod.rs b/src-tauri/src/commands/mod.rs index 9a9699d87..fef16339e 100644 --- a/src-tauri/src/commands/mod.rs +++ b/src-tauri/src/commands/mod.rs @@ -12,6 +12,7 @@ pub mod download; pub mod events; +pub mod export; pub mod exports; pub mod extract; pub mod ffmpeg; @@ -19,7 +20,6 @@ pub mod format; pub mod jobs; pub mod local_server; pub mod paths; -pub mod pull; pub mod staging; pub mod upload; diff --git a/src-tauri/src/export_directories.rs b/src-tauri/src/export_directories.rs index 051c31492..ea4623940 100644 --- a/src-tauri/src/export_directories.rs +++ b/src-tauri/src/export_directories.rs @@ -260,7 +260,7 @@ impl ExportDirectories { /// Finish the directory `dir` after its Export or Convert succeeded: move /// the converted files up out of [`CONVERTING`], delete the JSON Lines it - /// pulled and the pull's journal, and delete the directory when nothing is + /// pulled and the export journal, and delete the directory when nothing is /// left in it because the result went elsewhere. Returns the directory /// when the result is in it. /// @@ -301,7 +301,7 @@ impl ExportDirectories { } // The journal lets a later pull into the same directory skip what it // downloaded. Nothing pulls into this directory again. - let journal = dir.join(message_crate_pull::PULL_JOURNAL_NAME); + let journal = dir.join(message_crate_export::EXPORT_JOURNAL_NAME); if journal.exists() { std::fs::remove_file(&journal).map_err(|e| left_over(&journal, e))?; } diff --git a/src-tauri/src/export_directories/tests.rs b/src-tauri/src/export_directories/tests.rs index c4fc90b95..af79a75d3 100644 --- a/src-tauri/src/export_directories/tests.rs +++ b/src-tauri/src/export_directories/tests.rs @@ -33,7 +33,7 @@ fn pull_into(dir: &Path) { sink.write_document(message_ir::testutil::sample_document("hello export")) .unwrap(); sink.finish(&mut ExportReport::default()).unwrap(); - std::fs::write(dir.join(message_crate_pull::PULL_JOURNAL_NAME), "{}\n").unwrap(); + std::fs::write(dir.join(message_crate_export::EXPORT_JOURNAL_NAME), "{}\n").unwrap(); } /// Convert `input` into `output` as CSV, as Export's format step does. @@ -86,7 +86,7 @@ fn an_export_writes_its_result_to_its_own_directory_and_leaves_no_in_between_fil assert!( !left.iter().any(|name| name == PULLED || name == CONVERTING - || name == message_crate_pull::PULL_JOURNAL_NAME + || name == message_crate_export::EXPORT_JOURNAL_NAME || name.ends_with(".jsonl")), "only the result is left: {left:?}" ); @@ -112,7 +112,7 @@ fn a_json_lines_export_keeps_its_files_and_drops_the_pull_journal() { assert!( !left .iter() - .any(|name| name == message_crate_pull::PULL_JOURNAL_NAME), + .any(|name| name == message_crate_export::EXPORT_JOURNAL_NAME), "{left:?}" ); } diff --git a/src-tauri/src/lib.rs b/src-tauri/src/lib.rs index b800c8f85..bd837f335 100644 --- a/src-tauri/src/lib.rs +++ b/src-tauri/src/lib.rs @@ -6,7 +6,7 @@ //! ffmpeg. Those jobs need a program that talks to the operating system. //! //! This crate is that program. It owns the window, loads the UI, and exposes -//! commands the UI can call (`extract`, `format`, `upload`, `pull`, and the +//! commands the UI can call (`extract`, `format`, `upload`, `export`, and the //! ffmpeg helpers). It also starts the Message Crate server it ships with, //! when nothing answers at the app's own address (`local_server`). Progress and errors go back to the UI as Tauri events. //! diff --git a/src-tauri/src/main.rs b/src-tauri/src/main.rs index e4411c2a9..28fc9590d 100644 --- a/src-tauri/src/main.rs +++ b/src-tauri/src/main.rs @@ -76,7 +76,7 @@ fn main() { commands::local_server::set_open_to_network, commands::local_server::open_data_directory, commands::upload::upload, - commands::pull::pull, + commands::export::export, commands::exports::export_directory, commands::exports::create_export_dir, commands::exports::finish_export_dir, diff --git a/src-tauri/src/state.rs b/src-tauri/src/state.rs index 6f64d4177..bf9c92b5a 100644 --- a/src-tauri/src/state.rs +++ b/src-tauri/src/state.rs @@ -24,7 +24,7 @@ pub enum JobName { Media, /// The `upload` command: the Upload Stage of an Import Run. Upload, - /// The `pull` command, and the `format` command when it is the second + /// The `export` command, and the `format` command when it is the second /// step of an Export. Export, /// The `format` command started from Settings → Convert. diff --git a/web/src/lib/serverApi.ts b/web/src/lib/serverApi.ts index c9268c0c5..b763538dc 100644 --- a/web/src/lib/serverApi.ts +++ b/web/src/lib/serverApi.ts @@ -910,7 +910,7 @@ export function getImportContacts( // ── Export Runs ───────────────────────────────────────────────────────────── // -// The desktop app pages a run's messages from its Rust side (`message-crate-pull`), +// The desktop app pages a run's messages from its Rust side (`message-crate-export`), // so `GET /v1/exports/{id}/messages` has no function here. export function getExport(id: number, opts?: RequestOptions): Promise { diff --git a/web/src/lib/serverApi.types.ts b/web/src/lib/serverApi.types.ts index 1b70b933a..9bb8d9cea 100644 --- a/web/src/lib/serverApi.types.ts +++ b/web/src/lib/serverApi.types.ts @@ -2408,7 +2408,7 @@ export interface components { CreateExportRequest: { /** @description What to export. */ scope: components["schemas"]["ExportScope"]; - /** @description Client/tool name recorded on the run, e.g. `message-crate-pull`. */ + /** @description Client/tool name recorded on the run, e.g. `message-crate-export`. */ tool?: string | null; }; /** @description Import result: the import counts plus optional dedupe counts. */ @@ -2720,7 +2720,7 @@ export interface components { started_at: string; /** @description Lifecycle status. */ status: components["schemas"]["ExportStatus"]; - /** @description Exporting tool, e.g. `message-crate-pull`, when the client named one. */ + /** @description Exporting tool, e.g. `message-crate-export`, when the client named one. */ tool: string | null; /** * Format: int64 @@ -3468,7 +3468,7 @@ export interface components { started_at: string; /** @description Lifecycle status. */ status: components["schemas"]["ExportStatus"]; - /** @description Exporting tool, e.g. `message-crate-pull`, when the client named one. */ + /** @description Exporting tool, e.g. `message-crate-export`, when the client named one. */ tool: string | null; /** * Format: int64 @@ -4068,7 +4068,7 @@ export interface components { started_at: string; /** @description Lifecycle status. */ status: components["schemas"]["ExportStatus"]; - /** @description Exporting tool, e.g. `message-crate-pull`, when the client named one. */ + /** @description Exporting tool, e.g. `message-crate-export`, when the client named one. */ tool: string | null; /** * Format: int64 @@ -4500,7 +4500,7 @@ export interface components { started_at: string; /** @description Lifecycle status. */ status: components["schemas"]["ExportStatus"]; - /** @description Exporting tool, e.g. `message-crate-pull`, when the client named one. */ + /** @description Exporting tool, e.g. `message-crate-export`, when the client named one. */ tool: string | null; /** * Format: int64 diff --git a/web/src/lib/serverApiOpenapi.test.ts b/web/src/lib/serverApiOpenapi.test.ts index 740749691..6899efd63 100644 --- a/web/src/lib/serverApiOpenapi.test.ts +++ b/web/src/lib/serverApiOpenapi.test.ts @@ -251,7 +251,7 @@ const EXERCISED: Record unknown> = { // Exports getExport: () => serverApi.getExport(2), createExport: () => - serverApi.createExport({ scope: { kind: "everything" }, tool: "message-crate-pull" }), + serverApi.createExport({ scope: { kind: "everything" }, tool: "message-crate-export" }), completeExport: () => serverApi.completeExport(2), cancelExport: () => serverApi.cancelExport(2), diff --git a/web/src/lib/tauri.ts b/web/src/lib/tauri.ts index b7da98f62..00f964774 100644 --- a/web/src/lib/tauri.ts +++ b/web/src/lib/tauri.ts @@ -278,7 +278,7 @@ export async function invokeUpload(config: UploadConfig): Promise { */ export type ExportQueryList = components["schemas"]["ExportQueryList"]; -export interface PullConfig { +export interface ExportConfig { base_url: string; token: string; out_dir: string; @@ -290,8 +290,8 @@ export interface PullConfig { } /** Download conversations from a server into a directory. */ -export async function invokePull(config: PullConfig): Promise { - return invoke("pull", { +export async function invokeExport(config: ExportConfig): Promise { + return invoke("export", { args: { baseUrl: config.base_url, token: config.token, @@ -632,7 +632,7 @@ export function parseTauriJobResult(summary: string): TauriJobResult { }; } } catch { - // Format and pull jobs send a plain sentence, not JSON. + // Format and export jobs send a plain sentence, not JSON. } return { summary }; } diff --git a/web/src/screens/ExportScreen.test.tsx b/web/src/screens/ExportScreen.test.tsx index 3f3c872c8..115853eac 100644 --- a/web/src/screens/ExportScreen.test.tsx +++ b/web/src/screens/ExportScreen.test.tsx @@ -8,7 +8,7 @@ import { fill, setupUser } from "../test/user"; import ExportScreen from "./ExportScreen"; import { ConvertSection } from "./settings/ConvertSection"; -const invokePull = vi.hoisted(() => vi.fn()); +const invokeExport = vi.hoisted(() => vi.fn()); const invokeFormat = vi.hoisted(() => vi.fn()); const invokeFinishExportDir = vi.hoisted(() => vi.fn()); const invokeDiscardExportDir = vi.hoisted(() => vi.fn()); @@ -24,7 +24,7 @@ vi.mock("../lib/tauri", async (importOriginal) => { const actual = await importOriginal(); return { EXPORT_FORMATS: actual.EXPORT_FORMATS, - invokePull: (...args: unknown[]) => invokePull(...args), + invokeExport: (...args: unknown[]) => invokeExport(...args), invokeFormat: (...args: unknown[]) => invokeFormat(...args), invokeCreateExportDir: (...args: unknown[]) => invokeCreateExportDir(...args), invokeFinishExportDir: (...args: unknown[]) => invokeFinishExportDir(...args), @@ -111,10 +111,10 @@ describe("ExportScreen", () => { it("pulls straight into the chosen directory for JSON Lines", async () => { await exportTo("/home/demo/out"); - await waitFor(() => expect(invokePull).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); // Everything is the scope the screen opens in without a query, and it - // sends a blank query, which message-crate-pull reads as the whole account. - expect(invokePull.mock.calls[0][0]).toMatchObject({ out_dir: "/home/demo/out", query: "" }); + // sends a blank query, which message-crate-export reads as the whole account. + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: "/home/demo/out", query: "" }); // JSONL is what pull already writes, so there is nothing to convert. The // export's own directory is finished, which deletes it when it is empty. expect(invokeFormat).not.toHaveBeenCalled(); @@ -128,9 +128,9 @@ describe("ExportScreen", () => { expect(screen.getByRole("button", { name: "Export" })).toBeEnabled(); await user.click(screen.getByRole("button", { name: "Export" })); - await waitFor(() => expect(invokePull).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); expect(invokeCreateExportDir).toHaveBeenCalledWith("export", "jsonl", ""); - expect(invokePull.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.dir }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.dir }); await waitFor(() => expect(invokeFinishExportDir).toHaveBeenCalledWith(EXPORT_DIR.dir)); expect(await screen.findByText(/Export complete/)).toHaveTextContent( `Export complete. JSON Lines (.jsonl) saved to ${EXPORT_DIR.dir}.`, @@ -146,7 +146,7 @@ describe("ExportScreen", () => { await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); expect(invokeCreateExportDir).toHaveBeenCalledWith("export", "csv", ""); - expect(invokePull.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.pulled }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.pulled }); // The conversion may not write into the directory that holds its input, // so it writes beside it and the finish moves the result up. expect(invokeFormat.mock.calls[0][0]).toMatchObject({ @@ -164,7 +164,7 @@ describe("ExportScreen", () => { await exportAs("/home/demo/out", "CSV (.csv)"); await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); - expect(invokePull.mock.calls[0][0]).toMatchObject({ out_dir: pulled }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: pulled }); expect(invokeFormat.mock.calls[0][0]).toEqual({ input_dir: pulled, output_dir: "/home/demo/out", @@ -176,7 +176,7 @@ describe("ExportScreen", () => { it("hands the conversion the time the Export Run started, before the pull", async () => { let pulled = 0; - invokePull.mockImplementation(async () => { + invokeExport.mockImplementation(async () => { pulled = Date.now(); }); const before = Date.now(); @@ -198,7 +198,7 @@ describe("ExportScreen", () => { await screen.findByText("/home/demo holds the Export Directory, where the export works."), ).toBeTruthy(); expect(invokeCreateExportDir).toHaveBeenCalledWith("export", "csv", "/home/demo"); - expect(invokePull).not.toHaveBeenCalled(); + expect(invokeExport).not.toHaveBeenCalled(); expect(invokeDiscardExportDir).not.toHaveBeenCalled(); }); @@ -293,7 +293,7 @@ describe("ExportScreen", () => { // The pull has ended and the format step has not started. await waitFor(() => expect(awaitTauriJob).toHaveBeenCalledTimes(2)); - expect(invokePull).toHaveBeenCalledTimes(1); + expect(invokeExport).toHaveBeenCalledTimes(1); expect(invokeFormat).not.toHaveBeenCalled(); expect(currentDesktopJob()).toBe("Export"); expect(convert).toBeDisabled(); @@ -344,7 +344,7 @@ describe("ExportScreen", () => { releasePull(); await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); - expect(invokePull).toHaveBeenCalledTimes(1); + expect(invokeExport).toHaveBeenCalledTimes(1); expect(invokeCreateExportDir).toHaveBeenCalledTimes(1); }); @@ -368,10 +368,10 @@ describe("ExportScreen", () => { await fill(user, screen.getByRole("textbox", { name: "Search" }), " in:#19,#22 "); await user.click(screen.getByRole("button", { name: "Export" })); - await waitFor(() => expect(invokePull).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); // A search typed here, with no hand-off, is for the Messages list. expect(screen.getByRole("button", { name: /Search in/ })).toHaveTextContent("Messages"); - expect(invokePull.mock.calls[0][0]).toMatchObject({ query: "in:#19,#22", list: "messages" }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ query: "in:#19,#22", list: "messages" }); }); it("opens in Search with the query it was given, and sends it", async () => { @@ -391,9 +391,9 @@ describe("ExportScreen", () => { await fill(user, screen.getByPlaceholderText("The Export Directory"), "/home/demo/out"); await user.click(screen.getByRole("button", { name: "Export" })); - await waitFor(() => expect(invokePull).toHaveBeenCalledTimes(1)); + await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); // Sent as a Messages query, the server would refuse `messages:` (#959). - expect(invokePull.mock.calls[0][0]).toMatchObject({ + expect(invokeExport.mock.calls[0][0]).toMatchObject({ query: "messages:>100 tag:Work", list: "conversations", }); @@ -409,12 +409,12 @@ describe("ExportScreen", () => { await fill(user, screen.getByPlaceholderText("The Export Directory"), "/home/demo/out"); await user.click(screen.getByRole("button", { name: "Export" })); - await waitFor(() => expect(invokePull).toHaveBeenCalledTimes(1)); - expect(invokePull.mock.calls[0][0]).toMatchObject({ query: "tag:Work", list: "messages" }); + await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ query: "tag:Work", list: "messages" }); }); it("will not export a Search scope with a blank query", async () => { - // message-crate-pull reads a blank query as the whole account, which is not what + // message-crate-export reads a blank query as the whole account, which is not what // someone who chose Search and left the box empty asked for. const user = setupUser(); renderScreen("from:me"); diff --git a/web/src/screens/ExportScreen.tsx b/web/src/screens/ExportScreen.tsx index 18e56ed36..1a65186bb 100644 --- a/web/src/screens/ExportScreen.tsx +++ b/web/src/screens/ExportScreen.tsx @@ -17,14 +17,14 @@ import { EXPORT_FORMATS, type ExportFormat, type ExportQueryList, + invokeExport, invokeFormat, - invokePull, } from "../lib/tauri"; const FORMAT_IDS = EXPORT_FORMATS.map((f) => f.id); /** - * What an export covers. `everything` sends a blank query, which message-crate-pull + * What an export covers. `everything` sends a blank query, which message-crate-export * reads as the whole account; `search` sends the text of the query box. */ type ExportScope = "everything" | "search"; @@ -64,7 +64,7 @@ function formatLabel(id: ExportFormat): string { } /** - * Desktop export: `message-crate-pull` downloads the account's conversations as + * Desktop export: `message-crate-export` downloads the account's conversations as * JSON Lines, and for any other format `message-reexport` rewrites them into * the chosen format. * @@ -154,7 +154,7 @@ export default function ExportScreen() { const pullInto = (outDir: string) => run( exportCancel.guard(() => - invokePull({ + invokeExport({ base_url: getBaseUrl(), token, out_dir: outDir, From 19ac8e12a6657245d7a1cd15bf8a15ccdf42e7be Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:23:56 -0400 Subject: [PATCH 02/42] docs(changelog): say the Export record rename in plain words AGENTS.md keeps file paths, tool names and crate names out of CHANGELOG entries. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1c2605fe..e48caec16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -56,14 +56,12 @@ released versions carry their date on the heading. belongs only to images built by hand from a branch, which show the commit in their version, so one `sha-` tag always names one image. Pull a release by its version, such as `0.11.0`, or by `latest`. -- 2026-10-05: **An Export's state file is now - `.message-crate-export-state.jsonl`.** The desktop app's Export keeps this - file in the directory it writes, to remember which attachments it already - fetched. It was named `.message-crate-pull-state.jsonl`. The old file is - no longer read and can be deleted; attachments already in the directory - are still kept rather than fetched again. The server now records an Export - Run the desktop app starts with the tool name `message-crate-export`, and - the library behind it is the `message-crate-export` crate. +- 2026-10-05: **An Export keeps its record of fetched attachments under a + new name.** The desktop app's Export keeps a small record in the directory + it writes, so a later Export there does not fetch the same attachments + again. That record now has a new name; one with the old name is ignored and + can be deleted, and attachments already in the directory are still not + fetched again. ### Fixes From 95e1b71ef4997a1b4e23019749af2cb66177be34 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:23:56 -0400 Subject: [PATCH 03/42] docs(adr): head the rename notes Amended, like ADR 0011 Co-Authored-By: Claude Opus 5.5 --- docs/adr/0001-no-command-line-except-the-server.md | 2 +- docs/adr/0012-four-crates-in-the-export-pipeline.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/0001-no-command-line-except-the-server.md b/docs/adr/0001-no-command-line-except-the-server.md index c1d824769..676d0232c 100644 --- a/docs/adr/0001-no-command-line-except-the-server.md +++ b/docs/adr/0001-no-command-line-except-the-server.md @@ -107,7 +107,7 @@ The binary is a convenience for working on the repository, run through addresses return 404 rather than pointing somewhere that does not answer the question they were bookmarked for. -## Note, 2026-10-05: `message-crate-pull` is now `message-crate-export` +## Amended 2026-10-05: `message-crate-pull` is now `message-crate-export` The text above is kept as it was decided. Since #1906, the crate it calls `message-crate-pull` is `message-crate-export`, at `crates/libs/export/`, diff --git a/docs/adr/0012-four-crates-in-the-export-pipeline.md b/docs/adr/0012-four-crates-in-the-export-pipeline.md index 8b1d18dda..229a1fb0a 100644 --- a/docs/adr/0012-four-crates-in-the-export-pipeline.md +++ b/docs/adr/0012-four-crates-in-the-export-pipeline.md @@ -213,7 +213,7 @@ case into Convert would have to be undone. crates and removed wherever the result is simpler, and tests are rewritten to match rather than preserved. -## Note, 2026-10-05: `message-crate-pull` is now `message-crate-export` +## Amended 2026-10-05: `message-crate-pull` is now `message-crate-export` The text above is kept as it was decided. Since #1906, the crate it calls `message-crate-pull` is `message-crate-export`, at `crates/libs/export/`, From 53ee12a970c1fa644c9afcc6f9e6d0655eb284c0 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:23:56 -0400 Subject: [PATCH 04/42] docs(agents): the layout tree names export, not pull Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 5c75344a6..3419ae209 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -277,7 +277,7 @@ message-crate ├── src-tauri/ # Tauri v2 native shell (not a workspace member) │ ├── capabilities/ # Tauri permission manifests │ ├── icons/ # desktop app icons -│ └── src/ # Tauri commands wrapping exporters / push / pull +│ └── src/ # Tauri commands wrapping exporters / push / export ├── staging/ # empty; the release Compose file mounts it for JSONL imports ├── tests/ │ └── fixtures/ # committed schema and search fixtures (no personal backups) From a08c6aa2ec0dd54f4bef15774a5bd1ce36f37001 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:23:56 -0400 Subject: [PATCH 05/42] refactor(export): the JSON Lines an Export writes go in .exported The Export's own working directory and the step that fills it said pull, which CONTEXT.md avoids for Export. The directory is .exported (EXPORTED, the exported field on ExportDir), and the step is the fetch. Co-Authored-By: Claude Opus 5.5 --- CONTEXT.md | 2 +- src-tauri/src/export_directories.rs | 24 +++---- src-tauri/src/export_directories/tests.rs | 26 +++---- web/src/lib/desktopJob.ts | 2 +- web/src/lib/runCancel.ts | 2 +- web/src/lib/tauri.ts | 4 +- web/src/screens/ExportScreen.test.tsx | 68 +++++++++---------- web/src/screens/ExportScreen.tsx | 18 ++--- .../screens/settings/ConvertSection.test.tsx | 2 +- 9 files changed, 74 insertions(+), 74 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 35dbd1c7c..6d2dd5753 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -489,7 +489,7 @@ chosen, gets a directory of its own, named for what it is, when it started and its format, such as `export-2026-10-04-1430-mbox`. That directory is where the result lands unless the person chose another destination. While the run goes, it also holds the in-between files, such as the JSON Lines an -Export pulls before converting them; they are deleted when the run +Export fetches before converting them; they are deleted when the run finishes, leaving only the result. A run that fails or is cancelled deletes its directory, and the app deletes one it did not see to its end the next time it starts. Message Crate never deletes a finished export from it. It belongs in the Message diff --git a/src-tauri/src/export_directories.rs b/src-tauri/src/export_directories.rs index ea4623940..3f3fae530 100644 --- a/src-tauri/src/export_directories.rs +++ b/src-tauri/src/export_directories.rs @@ -5,7 +5,7 @@ //! it started and the format it writes: `export-2026-10-04-1430-mbox`. It is //! where the result lands unless the person chose another destination, and it //! is where an Export keeps its in-between files while it runs: the JSON Lines -//! it pulls from the server before converting them ([`PULLED`]), and the +//! it exports from the server before converting them ([`EXPORTED`]), and the //! converted files while they are written ([`CONVERTING`]), since a conversion //! may not write into a directory that holds its input. //! [`ExportDirectories::finish`] moves the result up and deletes the @@ -35,8 +35,8 @@ use std::sync::Mutex; pub const EXPORT_DIRECTORY_NAME: &str = "exports"; /// The directory inside an Export's directory that holds the JSON Lines it -/// pulled, while it converts them. -pub const PULLED: &str = ".pulled"; +/// exported, while it converts them. +pub const EXPORTED: &str = ".exported"; /// The directory inside an Export's directory that the conversion writes /// into, before [`ExportDirectories::finish`] moves its files up. @@ -84,8 +84,8 @@ pub struct ExportDir { /// The directory, where the result lands unless another destination is /// chosen. pub dir: String, - /// Where an Export pulls its JSON Lines to before converting them. - pub pulled: String, + /// Where an Export writes its JSON Lines before converting them. + pub exported: String, /// Where an Export's conversion writes when the result lands in `dir`. pub converting: String, } @@ -195,7 +195,7 @@ impl ExportDirectories { }; self.running_markers().insert(dir.clone(), marker); Ok(ExportDir { - pulled: dir.join(PULLED).display().to_string(), + exported: dir.join(EXPORTED).display().to_string(), converting: dir.join(CONVERTING).display().to_string(), dir: dir.display().to_string(), }) @@ -260,7 +260,7 @@ impl ExportDirectories { /// Finish the directory `dir` after its Export or Convert succeeded: move /// the converted files up out of [`CONVERTING`], delete the JSON Lines it - /// pulled and the export journal, and delete the directory when nothing is + /// exported and the export journal, and delete the directory when nothing is /// left in it because the result went elsewhere. Returns the directory /// when the result is in it. /// @@ -295,12 +295,12 @@ impl ExportDirectories { path.display() ) }; - let pulled = dir.join(PULLED); - if pulled.exists() { - std::fs::remove_dir_all(&pulled).map_err(|e| left_over(&pulled, e))?; + let exported = dir.join(EXPORTED); + if exported.exists() { + std::fs::remove_dir_all(&exported).map_err(|e| left_over(&exported, e))?; } - // The journal lets a later pull into the same directory skip what it - // downloaded. Nothing pulls into this directory again. + // The journal lets a later Export into the same directory skip what it + // fetched. Nothing exports into this directory again. let journal = dir.join(message_crate_export::EXPORT_JOURNAL_NAME); if journal.exists() { std::fs::remove_file(&journal).map_err(|e| left_over(&journal, e))?; diff --git a/src-tauri/src/export_directories/tests.rs b/src-tauri/src/export_directories/tests.rs index af79a75d3..2907bf963 100644 --- a/src-tauri/src/export_directories/tests.rs +++ b/src-tauri/src/export_directories/tests.rs @@ -6,7 +6,7 @@ use message_crate_core::{ }; use message_ir_format::{EXPORT_SENTINEL, FormatSink}; -use super::{CONVERTING, EXPORT_DIRECTORY_NAME, ExportDirectories, ExportKind, PULLED, sweep}; +use super::{CONVERTING, EXPORT_DIRECTORY_NAME, EXPORTED, ExportDirectories, ExportKind, sweep}; /// An Export Directory in its own temporary app-data directory. fn exports() -> (tempfile::TempDir, ExportDirectories) { @@ -25,8 +25,8 @@ fn names(dir: &Path) -> Vec { names } -/// Write one conversation into `dir` as JSON Lines, as a pull does. -fn pull_into(dir: &Path) { +/// Write one conversation into `dir` as JSON Lines, as an Export does. +fn export_into(dir: &Path) { std::fs::create_dir_all(dir).unwrap(); message_ir_format::mark_export_directory(dir).unwrap(); let mut sink = FormatSink::open(dir, OutputFormat::Jsonl, ExportTransforms::none()).unwrap(); @@ -73,9 +73,9 @@ fn an_export_writes_its_result_to_its_own_directory_and_leaves_no_in_between_fil .join("export-2026-10-04-1430-csv") ); - pull_into(Path::new(&made.pulled)); + export_into(Path::new(&made.exported)); convert( - Path::new(&made.pulled), + Path::new(&made.exported), Path::new(&made.converting), &app_data.path().join("scratch"), ); @@ -84,7 +84,7 @@ fn an_export_writes_its_result_to_its_own_directory_and_leaves_no_in_between_fil assert_eq!(finished.as_deref(), Some(dir.as_path())); let left = names(&dir); assert!( - !left.iter().any(|name| name == PULLED + !left.iter().any(|name| name == EXPORTED || name == CONVERTING || name == message_crate_export::EXPORT_JOURNAL_NAME || name.ends_with(".jsonl")), @@ -98,12 +98,12 @@ fn an_export_writes_its_result_to_its_own_directory_and_leaves_no_in_between_fil } #[test] -fn a_json_lines_export_keeps_its_files_and_drops_the_pull_journal() { +fn a_json_lines_export_keeps_its_files_and_drops_the_export_journal() { let (_app_data, exports) = exports(); let made = exports .create(ExportKind::Export, "jsonl", "2026-10-04-1430", None) .unwrap(); - pull_into(Path::new(&made.dir)); + export_into(Path::new(&made.dir)); exports.finish(&made.dir).unwrap(); @@ -124,9 +124,9 @@ fn an_export_to_another_destination_leaves_no_directory_behind() { .create(ExportKind::Export, "csv", "2026-10-04-1430", None) .unwrap(); let chosen = app_data.path().join("chosen"); - pull_into(Path::new(&made.pulled)); + export_into(Path::new(&made.exported)); convert( - Path::new(&made.pulled), + Path::new(&made.exported), &chosen, &app_data.path().join("scratch"), ); @@ -242,11 +242,11 @@ fn the_start_up_sweep_deletes_an_interrupted_export_and_keeps_a_running_one() { let interrupted = exports .create(ExportKind::Export, "csv", "2026-10-04-1430", None) .unwrap(); - pull_into(Path::new(&interrupted.pulled)); + export_into(Path::new(&interrupted.exported)); let finished = exports .create(ExportKind::Export, "jsonl", "2026-10-04-1430", None) .unwrap(); - pull_into(Path::new(&finished.dir)); + export_into(Path::new(&finished.dir)); exports.finish(&finished.dir).unwrap(); // The app quits mid-export: its hold on the marker ends with it. drop(exports); @@ -307,7 +307,7 @@ fn a_finished_run_is_never_swept_even_with_files_left_in_it() { std::fs::write(Path::new(&made.converting).join("a.csv"), "x").unwrap(); exports.finish(&made.dir).unwrap(); // Something the finish could not delete, as an open file on Windows. - std::fs::create_dir_all(&made.pulled).unwrap(); + std::fs::create_dir_all(&made.exported).unwrap(); sweep(exports.root()); diff --git a/web/src/lib/desktopJob.ts b/web/src/lib/desktopJob.ts index 15cae4e01..242521fa9 100644 --- a/web/src/lib/desktopJob.ts +++ b/web/src/lib/desktopJob.ts @@ -13,7 +13,7 @@ export type DesktopJobName = "Import Run" | "Export" | "Convert"; * desktop refuses. * * Holds nest: an Import Run holds it from its first stage to its end and an - * Export from its pull to the end of its format step, and `awaitTauriJob` + * Export from its fetch to the end of its format step, and `awaitTauriJob` * holds it again for each desktop job call inside them. A call ending * releases only its own hold, so the run's hold covers the gaps between its * jobs, when the desktop itself has nothing running to refuse a Convert with. diff --git a/web/src/lib/runCancel.ts b/web/src/lib/runCancel.ts index 8f0405614..8a160ecc6 100644 --- a/web/src/lib/runCancel.ts +++ b/web/src/lib/runCancel.ts @@ -8,7 +8,7 @@ import { invokeCancel } from "./tauri"; export const CANCELLED_MESSAGE = "cancelled"; /** - * The Cancel of one run of desktop jobs: an Import Run, or an export's pull + * The Cancel of one run of desktop jobs: an Import Run, or an export's fetch * and conversion. * * The desktop side stops only the job that is running, and each job command diff --git a/web/src/lib/tauri.ts b/web/src/lib/tauri.ts index 00f964774..a3b94c7ef 100644 --- a/web/src/lib/tauri.ts +++ b/web/src/lib/tauri.ts @@ -328,8 +328,8 @@ export type ExportFormat = (typeof EXPORT_FORMATS)[number]["id"]; export interface ExportDir { /** Where the result lands unless another destination is chosen. */ dir: string; - /** Where an Export pulls its JSON Lines before converting them. */ - pulled: string; + /** Where an Export writes its JSON Lines before converting them. */ + exported: string; /** Where an Export's conversion writes when the result lands in `dir`. */ converting: string; } diff --git a/web/src/screens/ExportScreen.test.tsx b/web/src/screens/ExportScreen.test.tsx index 115853eac..ca287d6f9 100644 --- a/web/src/screens/ExportScreen.test.tsx +++ b/web/src/screens/ExportScreen.test.tsx @@ -56,8 +56,8 @@ afterEach(() => { /** The directory the desktop makes for the export in the Export Directory. */ const EXPORT_DIR = { dir: "/home/demo/.local/share/app.messagecrate.desktop/exports/export-2026-10-04-1430-csv", - pulled: - "/home/demo/.local/share/app.messagecrate.desktop/exports/export-2026-10-04-1430-csv/.pulled", + exported: + "/home/demo/.local/share/app.messagecrate.desktop/exports/export-2026-10-04-1430-csv/.exported", converting: "/home/demo/.local/share/app.messagecrate.desktop/exports/export-2026-10-04-1430-csv/.converting", }; @@ -108,21 +108,21 @@ async function exportAs(directory: string, formatLabel: string) { } describe("ExportScreen", () => { - it("pulls straight into the chosen directory for JSON Lines", async () => { + it("fetches straight into the chosen directory for JSON Lines", async () => { await exportTo("/home/demo/out"); await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); // Everything is the scope the screen opens in without a query, and it // sends a blank query, which message-crate-export reads as the whole account. expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: "/home/demo/out", query: "" }); - // JSONL is what pull already writes, so there is nothing to convert. The + // JSONL is what an Export already writes, so there is nothing to convert. The // export's own directory is finished, which deletes it when it is empty. expect(invokeFormat).not.toHaveBeenCalled(); await waitFor(() => expect(invokeFinishExportDir).toHaveBeenCalledWith(EXPORT_DIR.dir)); expect(invokeDiscardExportDir).not.toHaveBeenCalled(); }); - it("pulls JSON Lines into its own directory in the Export Directory when no directory is chosen", async () => { + it("fetches JSON Lines into its own directory in the Export Directory when no directory is chosen", async () => { const user = setupUser(); renderScreen(); expect(screen.getByRole("button", { name: "Export" })).toBeEnabled(); @@ -146,11 +146,11 @@ describe("ExportScreen", () => { await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); expect(invokeCreateExportDir).toHaveBeenCalledWith("export", "csv", ""); - expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.pulled }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: EXPORT_DIR.exported }); // The conversion may not write into the directory that holds its input, // so it writes beside it and the finish moves the result up. expect(invokeFormat.mock.calls[0][0]).toMatchObject({ - input_dir: EXPORT_DIR.pulled, + input_dir: EXPORT_DIR.exported, output_dir: EXPORT_DIR.converting, }); await waitFor(() => expect(invokeFinishExportDir).toHaveBeenCalledWith(EXPORT_DIR.dir)); @@ -159,14 +159,14 @@ describe("ExportScreen", () => { ); }); - it("pulls into the export's own directory and converts into the chosen directory for CSV", async () => { - const pulled = EXPORT_DIR.pulled; + it("fetches into the export's own directory and converts into the chosen directory for CSV", async () => { + const exported = EXPORT_DIR.exported; await exportAs("/home/demo/out", "CSV (.csv)"); await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); - expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: pulled }); + expect(invokeExport.mock.calls[0][0]).toMatchObject({ out_dir: exported }); expect(invokeFormat.mock.calls[0][0]).toEqual({ - input_dir: pulled, + input_dir: exported, output_dir: "/home/demo/out", output_format: "csv", started_from: "export", @@ -174,10 +174,10 @@ describe("ExportScreen", () => { }); }); - it("hands the conversion the time the Export Run started, before the pull", async () => { - let pulled = 0; + it("hands the conversion the time the Export Run started, before the fetch", async () => { + let fetched = 0; invokeExport.mockImplementation(async () => { - pulled = Date.now(); + fetched = Date.now(); }); const before = Date.now(); await exportAs("/home/demo/out", "CSV (.csv)"); @@ -185,7 +185,7 @@ describe("ExportScreen", () => { await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); const started = invokeFormat.mock.calls[0][0].run_started_ms as number; expect(started).toBeGreaterThanOrEqual(before); - expect(started).toBeLessThanOrEqual(pulled); + expect(started).toBeLessThanOrEqual(fetched); }); it("starts nothing when the desktop refuses Save to for holding the Export Directory", async () => { @@ -214,7 +214,7 @@ describe("ExportScreen", () => { // disk, in a directory the person never chose and will not think to look in. awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "pulled" }; + return { summary: "fetched" }; }); awaitTauriJob.mockImplementationOnce(async () => { throw new Error("unsupported output format"); @@ -227,7 +227,7 @@ describe("ExportScreen", () => { expect(await screen.findByText("unsupported output format")).toBeTruthy(); }); - it("does not start the conversion when Cancel is pressed after the pull finished", async () => { + it("does not start the conversion when Cancel is pressed after the fetch finished", async () => { // A Cancel sent while no job runs stops nothing, and invokeFormat starts // its job with a cancel flag of its own, so the screen must not start it. let releaseFormat: () => void = () => {}; @@ -236,7 +236,7 @@ describe("ExportScreen", () => { }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "pulled" }; + return { summary: "fetched" }; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await formatHeld; @@ -253,8 +253,8 @@ describe("ExportScreen", () => { expect(invokeFormat).not.toHaveBeenCalled(); }); - it("keeps Convert off between the pull and the format step, and lets it start once the export ends", async () => { - // Each job holds the desktop only while it runs; between the pull and the + it("keeps Convert off between the fetch and the format step, and lets it start once the export ends", async () => { + // Each job holds the desktop only while it runs; between the fetch and the // format step the desktop has nothing running, so a Convert started there // would make it refuse the format step (#1407). let releaseFormat: () => void = () => {}; @@ -263,7 +263,7 @@ describe("ExportScreen", () => { }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "pulled" }; + return { summary: "fetched" }; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await formatHeld; @@ -291,7 +291,7 @@ describe("ExportScreen", () => { await user.click(await screen.findByRole("option", { name: "CSV (.csv)" })); await user.click(screen.getByRole("button", { name: "Export" })); - // The pull has ended and the format step has not started. + // The fetch has ended and the format step has not started. await waitFor(() => expect(awaitTauriJob).toHaveBeenCalledTimes(2)); expect(invokeExport).toHaveBeenCalledTimes(1); expect(invokeFormat).not.toHaveBeenCalled(); @@ -317,13 +317,13 @@ describe("ExportScreen", () => { it("ignores a second Export while one is already under way", async () => { // The desktop backend runs one job at a time (src-tauri/src/commands/jobs.rs), - // and between the pull and the conversion it has nothing running to refuse. - let releasePull: () => void = () => {}; - const pullStarted = new Promise((resolve) => { - releasePull = resolve; + // and between the fetch and the conversion it has nothing running to refuse. + let releaseFetch: () => void = () => {}; + const fetchStarted = new Promise((resolve) => { + releaseFetch = resolve; }); invokeCreateExportDir.mockImplementation(async () => { - await pullStarted; + await fetchStarted; return EXPORT_DIR; }); @@ -341,7 +341,7 @@ describe("ExportScreen", () => { // itself 80 ms later. On a busy machine that lands after the first export // has ended and the button is live again, and starts a second one. fireEvent.click(exportButton); - releasePull(); + releaseFetch(); await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); expect(invokeExport).toHaveBeenCalledTimes(1); @@ -453,19 +453,19 @@ describe("ExportScreen", () => { }); it("locks the directory field while an export runs", async () => { - let releasePull: () => void = () => {}; - const pullHeld = new Promise((resolve) => { - releasePull = resolve; + let releaseFetch: () => void = () => {}; + const fetchHeld = new Promise((resolve) => { + releaseFetch = resolve; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { - await pullHeld; + await fetchHeld; await invokeFn(); - return { summary: "pulled" }; + return { summary: "fetched" }; }); await exportTo("/a"); expect(screen.getByPlaceholderText("The Export Directory")).toBeDisabled(); - releasePull(); + releaseFetch(); await screen.findByText(/Export complete/); expect(screen.getByPlaceholderText("The Export Directory")).toBeEnabled(); }); diff --git a/web/src/screens/ExportScreen.tsx b/web/src/screens/ExportScreen.tsx index 1a65186bb..01d76bdc8 100644 --- a/web/src/screens/ExportScreen.tsx +++ b/web/src/screens/ExportScreen.tsx @@ -71,7 +71,7 @@ function formatLabel(id: ExportFormat): string { * Every export gets a directory of its own in the Export Directory, named for * when it started and its format (`export-2026-10-04-1430-mbox`). The result * lands there unless the person chose another directory under **Save to**. - * The JSON Lines a non-JSONL export pulls wait in that directory while they + * The JSON Lines a non-JSONL export fetches wait in that directory while they * are converted, since `message-reexport` refuses to write into a directory * that holds its input, and the conversion writes beside them. When the export * finishes, the desktop deletes the JSON Lines and moves the result up, so the @@ -105,7 +105,7 @@ export default function ExportScreen() { const [log, setLog] = useState([]); // `running` only turns true once a job starts, which leaves two windows // where the Export button would be live mid-export: while the export's - // directory is made, and between the pull and the conversion. The desktop + // directory is made, and between the fetch and the conversion. The desktop // refuses a second job while one runs (`jobs.rs`), but between two jobs it // has nothing to refuse. This covers the whole run. const [busy, setBusy] = useState(false); @@ -114,7 +114,7 @@ export default function ExportScreen() { const { running, finished, run } = useTauriJob<{ savePath: string; format: ExportFormat }>({ job: "Export", }); - // The Cancel of the export under way. A Cancel pressed after the pull and + // The Cancel of the export under way. A Cancel pressed after the fetch and // before the conversion starts must stop the conversion, and the desktop // alone would not: with no job running, its Cancel stops nothing, and // `format` starts with a cancel flag of its own. @@ -131,7 +131,7 @@ export default function ExportScreen() { return; } setBusy(true); - // The export holds the desktop from its pull to the end of its format + // The export holds the desktop from its fetch to the end of its format // step. Each job holds it too, but only while it runs, which would leave // a gap between the two where Settings → Convert could start a job the // desktop then runs instead of the format step (#1407). @@ -151,7 +151,7 @@ export default function ExportScreen() { chosen, async (exportDir) => { const request = { savePath: chosen || exportDir.dir, format }; - const pullInto = (outDir: string) => + const fetchInto = (outDir: string) => run( exportCancel.guard(() => invokeExport({ @@ -167,14 +167,14 @@ export default function ExportScreen() { { onLog: appendLog }, ); if (format === "jsonl") { - await pullInto(chosen || exportDir.dir); + await fetchInto(chosen || exportDir.dir); return; } - await pullInto(exportDir.pulled); + await fetchInto(exportDir.exported); await run( exportCancel.guard(() => invokeFormat({ - input_dir: exportDir.pulled, + input_dir: exportDir.exported, output_dir: chosen || exportDir.converting, output_format: format, started_from: "export", @@ -220,7 +220,7 @@ export default function ExportScreen() { } success={ // `busy` hides it from the moment the next export starts, and between - // the pull and the conversion, when the pull alone has finished. + // the fetch and the conversion, when the fetch alone has finished. finished && !busy && !error ? (
Export complete. {formatLabel(finished.format)} saved to {finished.savePath}. diff --git a/web/src/screens/settings/ConvertSection.test.tsx b/web/src/screens/settings/ConvertSection.test.tsx index e262b1fad..b5d718a01 100644 --- a/web/src/screens/settings/ConvertSection.test.tsx +++ b/web/src/screens/settings/ConvertSection.test.tsx @@ -50,7 +50,7 @@ beforeEach(() => { tauriState.isTauri = true; invokeCreateExportDir.mockResolvedValue({ dir: CONVERT_DIR, - pulled: `${CONVERT_DIR}/.pulled`, + exported: `${CONVERT_DIR}/.exported`, converting: `${CONVERT_DIR}/.converting`, }); invokeFinishExportDir.mockResolvedValue(CONVERT_DIR); From 1599291ba596906c4a53c85b64f8cc23dabc8510 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:23:56 -0400 Subject: [PATCH 06/42] refactor(export): say Export where comments and tests still said pull Co-Authored-By: Claude Opus 5.5 --- crates/libs/api-types/README.md | 2 +- crates/libs/api-types/src/lib.rs | 2 +- crates/libs/export/src/project.rs | 6 +-- crates/libs/export/src/run.rs | 4 +- crates/libs/export/tests/export_mock.rs | 52 +++++++++---------- crates/libs/http/src/auth_error.rs | 2 +- crates/libs/journal/src/lib.rs | 2 +- .../server/src/accounts_api/api_tokens.rs | 10 ++-- crates/server/server/src/exports_api/tests.rs | 4 +- crates/server/server/src/session_api/tests.rs | 2 +- 10 files changed, 43 insertions(+), 43 deletions(-) diff --git a/crates/libs/api-types/README.md b/crates/libs/api-types/README.md index e8b329f59..d9c51ff9d 100644 --- a/crates/libs/api-types/README.md +++ b/crates/libs/api-types/README.md @@ -6,7 +6,7 @@ that writes them and the client crates that read them: `Message`, Two crates sit on either side of these shapes. While each kept its own copy, the two could disagree silently, and did — three defects shipped that way, each -a pull that failed at runtime or quietly produced worse data. One definition +an Export that failed at runtime or quietly produced worse data. One definition makes the compiler the check instead. `skip_serializing_if` and `default` come as a pair here and only as a pair: a diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 6cb3ed665..01d558657 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -9,7 +9,7 @@ //! for a participant a backup named without an address, kept //! `#[serde(default)]` on a field the server had removed, and read a //! `service` off the conversation the server has never sent there. Each of -//! those was a pull that failed at runtime, or quietly produced worse data, +//! those was an Export that failed at runtime, or quietly produced worse data, //! with nothing in either crate's tests to catch it — `message-crate-export`'s //! "real export page" was a JSON literal it wrote itself, so it agreed with //! whatever the mirror said. diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index 8b4b21140..e92cd285e 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -372,7 +372,7 @@ mod tests { /// compiler or the suite noticing: `handle: String` rejected `"handle": null` and aborted every /// Export Run of a conversation holding an address-less participant, and /// `conversation.service` read a field the server has never sent, so every - /// pulled message came out `IrService::Unknown`. + /// exported message came out `IrService::Unknown`. const EXPORT_PAGE_JSON: &str = r#"{ "items": [ { @@ -644,8 +644,8 @@ mod tests { ); } - /// A conversation of orphaned messages is pulled as one, so importing - /// the pulled file again does not make its key a person (#1095). + /// A conversation of orphaned messages is exported as one, so importing + /// the exported file again does not make its key a person (#1095). #[test] fn a_document_of_orphaned_messages_stays_orphaned() { let mut seed = seed_message_with_participant(Participant { diff --git a/crates/libs/export/src/run.rs b/crates/libs/export/src/run.rs index f30feb350..a9131e104 100644 --- a/crates/libs/export/src/run.rs +++ b/crates/libs/export/src/run.rs @@ -902,7 +902,7 @@ mod out_dir_tests { // clean an exported directory, so an export that staged into one would // leave the run directory behind. let dir = tempfile::tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); prepare_out_dir(&out, false).unwrap(); @@ -926,7 +926,7 @@ mod out_dir_tests { fn runs_again_over_a_directory_it_already_prepared() { // A second export into the same directory is a later Export Run over it. let dir = tempfile::tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); prepare_out_dir(&out, false).unwrap(); prepare_out_dir(&out, false).unwrap(); diff --git a/crates/libs/export/tests/export_mock.rs b/crates/libs/export/tests/export_mock.rs index d9ad9acdf..6c35abc6c 100644 --- a/crates/libs/export/tests/export_mock.rs +++ b/crates/libs/export/tests/export_mock.rs @@ -300,7 +300,7 @@ fn an_export_records_one_run_and_writes_the_conversation_and_every_asset_once_ac let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let report = run(&config(&out, server.base_url()), None).unwrap(); @@ -371,7 +371,7 @@ fn an_attachment_path_that_leaves_the_directory_with_no_sha256_is_not_written() let _auth = mock_auth(&server); let (_create, complete) = mock_run(&server); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let climbing = "../escape.pdf"; let _page = server.mock(|when, then| { when.method(GET) @@ -432,7 +432,7 @@ fn an_attachment_path_that_leaves_the_output_directory_is_written_under_its_fing let _auth = mock_auth(&server); let (_create, complete) = mock_run(&server); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let climbing = "../escape.pdf"; let absolute = dir.path().join("absolute.png").display().to_string(); let page = server.mock(|when, then| { @@ -517,7 +517,7 @@ fn the_journal_lists_every_asset_and_marks_the_run_finished() { let _menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -538,7 +538,7 @@ fn a_second_run_over_the_same_directory_fetches_nothing_it_already_has() { let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = config(&out, server.base_url()); run(&cfg, None).unwrap(); @@ -583,7 +583,7 @@ fn an_unreadable_journal_line_is_a_sentence_in_the_exports_log() { let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = config(&out, server.base_url()); run(&cfg, None).unwrap(); let path = journal::journal_path(&out); @@ -630,7 +630,7 @@ fn a_file_the_journal_lists_but_the_disk_lost_is_fetched_again() { let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = config(&out, server.base_url()); run(&cfg, None).unwrap(); fs::remove_file(out.join("attachments/menu.pdf")).unwrap(); @@ -660,7 +660,7 @@ fn a_file_on_disk_the_journal_does_not_list_is_kept_and_counted_as_on_disk() { let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = config(&out, server.base_url()); run(&cfg, None).unwrap(); fs::remove_file(journal::journal_path(&out)).unwrap(); @@ -705,7 +705,7 @@ fn every_file_on_disk_logs_no_fetch_lines_whether_or_not_the_journal_lists_it() let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = config(&out, server.base_url()); run(&cfg, None).unwrap(); fs::remove_file(journal::journal_path(&out)).unwrap(); @@ -741,7 +741,7 @@ fn a_cancel_requested_before_the_run_records_nothing_on_the_server() { let cancel = mock_cancel(&server); let (first, _second) = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = ExportConfig { cancel: Some(Arc::new(AtomicBool::new(true))), ..config(&out, server.base_url()) @@ -773,7 +773,7 @@ fn a_source_name_with_spaces_and_brackets_becomes_a_file_safe_suffix() { let _menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -819,7 +819,7 @@ fn two_groups_with_one_title_are_both_written() { })); }); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -855,7 +855,7 @@ fn skipping_attachments_writes_messages_without_files_or_fetches() { let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = ExportConfig { skip_attachments: true, ..config(&out, server.base_url()) @@ -885,7 +885,7 @@ fn a_query_becomes_the_runs_query_scope_and_progress_narrates_the_run() { let _menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = ExportConfig { query: " from:sam ".into(), ..config(&out, server.base_url()) @@ -943,7 +943,7 @@ fn a_query_for_the_conversations_list_names_that_list_in_the_scope() { let complete = mock_complete(&server); let _pages = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = ExportConfig { query: "messages:>100".into(), list: ExportQueryList::Conversations, @@ -982,7 +982,7 @@ fn an_asset_the_server_does_not_have_fails_the_run_and_cancels_it_on_the_server( }); let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let error = run(&config(&out, server.base_url()), None).unwrap_err(); @@ -1018,7 +1018,7 @@ fn bytes_whose_sha256_is_not_the_one_asked_for_fail_the_run_and_are_not_kept() { b"Sign in to continue", ); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let error = run(&config(&out, server.base_url()), None).unwrap_err(); @@ -1072,7 +1072,7 @@ fn a_scope_the_server_refuses_fails_the_run_with_the_servers_sentence() { }); let (first, _second) = mock_pages(&server, "sms-backup-restore"); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let cfg = ExportConfig { query: "wibble:yes".into(), ..config(&out, server.base_url()) @@ -1112,7 +1112,7 @@ fn a_refused_completion_is_a_warning_that_names_the_run_once() { let _menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let _photo = mock_asset(&server, PHOTO_SHA, PHOTO_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); let mut events = Vec::new(); { @@ -1205,7 +1205,7 @@ fn every_path_a_message_names_exists_after_an_export() { }); let menu = mock_asset(&server, MENU_SHA, MENU_BYTES); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -1227,7 +1227,7 @@ fn every_path_a_message_names_exists_after_an_export() { /// serializes them, is written with both in its `reactions`, each under the /// person who reacted, in the shape an import reads back. #[test] -fn a_pulled_message_keeps_its_reactions_under_each_reactor() { +fn an_exported_message_keeps_its_reactions_under_each_reactor() { let server = MockServer::start(); let _auth = mock_auth(&server); let _run = mock_run(&server); @@ -1255,7 +1255,7 @@ fn a_pulled_message_keeps_its_reactions_under_each_reactor() { })); }); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -1300,7 +1300,7 @@ fn a_pulled_message_keeps_its_reactions_under_each_reactor() { /// are written with `deletion` set to the same mark, and left out for the /// last, in the shape an import reads back. #[test] -fn a_pulled_message_keeps_its_deletion_mark() { +fn an_exported_message_keeps_its_deletion_mark() { let server = MockServer::start(); let _auth = mock_auth(&server); let _run = mock_run(&server); @@ -1354,7 +1354,7 @@ fn a_pulled_message_keeps_its_deletion_mark() { })); }); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); @@ -1384,7 +1384,7 @@ fn a_pulled_message_keeps_its_deletion_mark() { /// returns is written in the message's `edits` with its part, text and time, /// in the server's order, and the file reads back as the same versions. #[test] -fn a_pulled_message_keeps_its_earlier_versions() { +fn an_exported_message_keeps_its_earlier_versions() { let server = MockServer::start(); let _auth = mock_auth(&server); let _run = mock_run(&server); @@ -1412,7 +1412,7 @@ fn a_pulled_message_keeps_its_earlier_versions() { })); }); let dir = tempdir().unwrap(); - let out = dir.path().join("pulled"); + let out = dir.path().join("exported"); run(&config(&out, server.base_url()), None).unwrap(); diff --git a/crates/libs/http/src/auth_error.rs b/crates/libs/http/src/auth_error.rs index 4940faf3d..3a5696656 100644 --- a/crates/libs/http/src/auth_error.rs +++ b/crates/libs/http/src/auth_error.rs @@ -1,7 +1,7 @@ //! Typed login failures from `GET /v1/session`. //! //! Each variant has a stable `kind()` string for tests, and its `Display` -//! text is the message the desktop app shows. The push and the pull send a +//! text is the message the desktop app shows. The push and the Export send a //! session token, so no text names an API token. use crate::retry::HttpError; diff --git a/crates/libs/journal/src/lib.rs b/crates/libs/journal/src/lib.rs index d44340da5..7b4d0c972 100644 --- a/crates/libs/journal/src/lib.rs +++ b/crates/libs/journal/src/lib.rs @@ -156,7 +156,7 @@ pub fn unreadable_line_reason(error: &serde_json::Error) -> String { /// them. /// /// Unreadable lines are skipped silently during the read, in the push journal -/// and the pull journal alike. +/// and the export journal alike. /// /// # Errors /// diff --git a/crates/server/server/src/accounts_api/api_tokens.rs b/crates/server/server/src/accounts_api/api_tokens.rs index 3e07c4d9d..4610a2cda 100644 --- a/crates/server/server/src/accounts_api/api_tokens.rs +++ b/crates/server/server/src/accounts_api/api_tokens.rs @@ -405,7 +405,7 @@ mod tests { &state, &tokens, &alice.token, - serde_json::json!({ "label": "pull", "can_import": false }), + serde_json::json!({ "label": "export", "can_import": false }), ) .await; @@ -518,7 +518,7 @@ mod tests { &state, &alices, &alice.token, - serde_json::json!({ "label": "pull", "can_import": false }), + serde_json::json!({ "label": "export", "can_import": false }), ) .await; // Use the token once, so it has a last use to show. @@ -531,7 +531,7 @@ mod tests { let listed: serde_json::Value = get_json(&state, &alices, &owner.token).await; let item = &listed["items"][0]; assert_eq!(item["id"], created["id"], "{listed}"); - assert_eq!(item["label"], "pull", "{listed}"); + assert_eq!(item["label"], "export", "{listed}"); assert_eq!(item["can_import"], false, "{listed}"); assert_eq!(item["can_export"], true, "{listed}"); assert_eq!(item["created_at"], created["created_at"], "{listed}"); @@ -716,7 +716,7 @@ mod tests { &state, &format!("/v1/accounts/{}/api-tokens", alice.account_id), &alice.token, - serde_json::json!({ "label": "pull" }), + serde_json::json!({ "label": "export" }), ) .await; assert_eq!(created["can_export"], true); @@ -787,7 +787,7 @@ mod tests { &fixture.state, &collection, &alice.token, - serde_json::json!({ "label": "pull" }), + serde_json::json!({ "label": "export" }), ) .await; assert_eq!(created["can_export"], true, "{created}"); diff --git a/crates/server/server/src/exports_api/tests.rs b/crates/server/server/src/exports_api/tests.rs index 59440ce75..f43790815 100644 --- a/crates/server/server/src/exports_api/tests.rs +++ b/crates/server/server/src/exports_api/tests.rs @@ -769,7 +769,7 @@ async fn api_token(fixture: &TestFixture, user: &RegisteredAccount, can_export: &fixture.state, &format!("/v1/accounts/{}/api-tokens", user.account_id), &user.token, - json!({ "label": "pull", "can_import": false, "can_export": can_export }), + json!({ "label": "export", "can_import": false, "can_export": can_export }), ) .await; created["token"].as_str().unwrap().to_string() @@ -1309,7 +1309,7 @@ async fn an_export_token_reads_messages_only_through_a_run() { /// landing while a run is being read. /// Each message of a run carries the account holder's own address the server /// stores for it (`messages.owner_handle_id`), and none when it stores none. -/// Without it a pull writes no owner, and an import of that export files no +/// Without it an Export writes no owner, and an import of that export files no /// message under the holder's addresses (#1098). #[tokio::test] async fn a_run_returns_the_owner_address_of_each_message() { diff --git a/crates/server/server/src/session_api/tests.rs b/crates/server/server/src/session_api/tests.rs index 9ea2c26ad..2d3cee822 100644 --- a/crates/server/server/src/session_api/tests.rs +++ b/crates/server/server/src/session_api/tests.rs @@ -379,7 +379,7 @@ async fn export_token( state, &format!("/v1/accounts/{}/api-tokens", account.account_id), &account.token, - serde_json::json!({ "label": "pull", "can_import": false, "can_export": true }), + serde_json::json!({ "label": "export", "can_import": false, "can_export": true }), ) .await; ( From cbbece2c6203c68aeac0c2213af4d55aa22692c7 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:25:26 -0400 Subject: [PATCH 07/42] fix(server): keep a message's time to the millisecond The server stored a message's time as whole seconds, dropping the milliseconds WhatsApp, Apple Messages and SMS Backup & Restore record, so two messages in one second were ordered by sort_order alone. messages.timestamp (and staging_messages.timestamp and message_versions.edited_at) now hold RFC 3339 UTC with three fractional digits, such as 2015-03-12T18:04:22.250Z, always in that one form so the text still sorts in time order. Lists order by it as before, and the API returns it in Message.timestamp and edited_at. The content key and the near-time pass still match at whole seconds. A search's day bounds are written in the same millisecond form, since a bound without a fraction sorts after a stored time of the same second. The search for the end of a time zone's gap now steps in whole seconds, because a half second left in it would now show in the bound. Tests: a fixture with sub-second times keeps them through import and the API, two messages 300 ms apart list in time order on both message routes, and a day begins at its first millisecond in search. Each fails without the change. The schema change rebuilds an existing database. Closes #1096 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 15 +++ crates/libs/api-types/src/lib.rs | 13 ++- .../server/server/src/accounts_api/tests.rs | 20 ++-- crates/server/server/src/assets_api/tests.rs | 2 +- .../server/server/src/contacts_api/tests.rs | 30 ++--- .../server/src/conversations_api/tests.rs | 42 +++---- crates/server/server/src/db/staging/tests.rs | 2 +- crates/server/server/src/db/write_guard.rs | 2 +- crates/server/server/src/dedupe.rs | 6 +- crates/server/server/src/dedupe/tests.rs | 58 +++++----- crates/server/server/src/exports_api/tests.rs | 12 +- .../server/server/src/messages_api/tests.rs | 106 ++++++++++++++++-- crates/server/server/src/models.rs | 29 +++-- .../server/src/openapi/credential_matrix.rs | 2 +- crates/server/server/src/reset_demo/tests.rs | 8 +- crates/server/server/src/search/tests.rs | 37 ++++++ crates/server/server/src/search/value.rs | 23 ++-- crates/server/server/src/server_api/tests.rs | 2 +- crates/server/server/src/session_api/tests.rs | 2 +- crates/server/server/src/test_support.rs | 17 +-- crates/server/server/src/trash_api/tests.rs | 2 +- .../apple-messages-sub-second-times.jsonl | 3 + .../contacts-identities-and-messages.md | 2 +- docs/src/assets/openapi.json | 6 +- schema/sql/messages.sql | 11 +- schema/sql/staging.sql | 4 +- web/src/lib/serverApi.types.ts | 18 ++- 27 files changed, 317 insertions(+), 157 deletions(-) create mode 100644 crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b30258ba..c1c2561f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -84,6 +84,18 @@ released versions carry their date on the heading. the conversation again. An Error about a conversation written before the stop stays, because the resumed Staging does not read it again. +#### Browsing and search + +- 2026-10-05: **A message keeps the milliseconds of its time.** Message + Crate kept a message's time to the whole second and dropped the + milliseconds that WhatsApp, Apple Messages and SMS Backup & Restore + record. Two messages sent within one second could then show + in the wrong order. The time is now kept to the millisecond, a + conversation lists its messages in the order they were sent, and an + export written from Message Crate keeps the milliseconds too. A backup + that records whole seconds, such as iMazing or OpenExtract, lists the + messages of one second in the order the backup gives them, as before. + ### Upgrading - The database format changed. **An existing Message Crate is rebuilt empty @@ -101,6 +113,9 @@ released versions carry their date on the heading. in place of `num_replies`. - A Saved Search that uses `deleted:yes` finds fewer messages than before: it leaves unsent messages out. Add `or unsent:yes` to it to find both. +- If you have a program that reads messages from the HTTP API, a message's + `timestamp` and an earlier version's `edited_at` now carry three digits of + milliseconds, such as `2015-03-12T18:04:22.250Z`, where they had none. ## [0.10.1] - 2026-10-05 diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 6fff20fbc..8906a2b07 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -372,8 +372,11 @@ api_shape! { /// Export GUID for replies and grouping. Every message has one, /// because the import refuses a message without one. pub guid: String, - /// The instant the message was sent: RFC 3339 in UTC with a `Z` - /// suffix. A caller shows it in the account's time zone + /// The instant the message was sent, to the millisecond: RFC 3339 + /// in UTC with three fractional digits and a `Z` suffix + /// (`2015-03-12T18:04:22.250Z`; `.000` when the source records + /// whole seconds). Messages are listed in the order of this time. A + /// caller shows it in the account's time zone /// (`Account.time_zone`); the database stores nothing /// about where the phone was. pub timestamp: String, @@ -450,8 +453,8 @@ api_shape! { pub part_index: i64, /// The part's text in this version. pub text: String, - /// When this version was written: RFC 3339 in UTC with a `Z` suffix, - /// as `Message.timestamp`. The original's is when it was sent, a + /// When this version was written, to the millisecond in the form + /// `Message.timestamp` takes. The original's is when it was sent, a /// later version's is when the edit that wrote it was made. `None` /// when the source does not record it. pub edited_at: Option, @@ -603,7 +606,7 @@ mod tests { source: "imessage".into(), service: None, guid: "g1".into(), - timestamp: "2024-01-01T00:00:00Z".into(), + timestamp: "2024-01-01T00:00:00.000Z".into(), sort_order: 0, is_from_me: false, sender: None, diff --git a/crates/server/server/src/accounts_api/tests.rs b/crates/server/server/src/accounts_api/tests.rs index c243e51e6..f50d01dff 100644 --- a/crates/server/server/src/accounts_api/tests.rs +++ b/crates/server/server/src/accounts_api/tests.rs @@ -2069,13 +2069,13 @@ async fn the_identities_route_counts_the_direct_and_group_messages_held_at_each_ let two = [ SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "a", }, SeedMessage { source: "imessage", - timestamp: "2020-01-02T00:00:00Z", + timestamp: "2020-01-02T00:00:00.000Z", is_from_me: false, body: "b", }, @@ -2095,19 +2095,19 @@ async fn the_identities_route_counts_the_direct_and_group_messages_held_at_each_ let three = [ SeedMessage { source: "imessage", - timestamp: "2020-02-01T00:00:00Z", + timestamp: "2020-02-01T00:00:00.000Z", is_from_me: true, body: "c", }, SeedMessage { source: "imessage", - timestamp: "2020-02-02T00:00:00Z", + timestamp: "2020-02-02T00:00:00.000Z", is_from_me: false, body: "d", }, SeedMessage { source: "imessage", - timestamp: "2020-02-03T00:00:00Z", + timestamp: "2020-02-03T00:00:00.000Z", is_from_me: false, body: "e", }, @@ -2192,8 +2192,8 @@ async fn the_identities_route_counts_the_direct_and_group_messages_held_at_each_ { "address": "+15555550100", "service": "phone", - "start_date": "2020-01-01T00:00:00Z", - "end_date": "2020-02-02T00:00:00Z", + "start_date": "2020-01-01T00:00:00.000Z", + "end_date": "2020-02-02T00:00:00.000Z", "conversations": 2, "direct_messages": 2, "group_messages": 2 @@ -2201,8 +2201,8 @@ async fn the_identities_route_counts_the_direct_and_group_messages_held_at_each_ { "address": "alice@example.com", "service": "email", - "start_date": "2020-02-03T00:00:00Z", - "end_date": "2020-02-03T00:00:00Z", + "start_date": "2020-02-03T00:00:00.000Z", + "end_date": "2020-02-03T00:00:00.000Z", "conversations": 1, "direct_messages": 0, "group_messages": 1 @@ -2243,7 +2243,7 @@ async fn the_storage_route_sums_attachment_bytes_and_lists_the_largest_first() { source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "photos", }], diff --git a/crates/server/server/src/assets_api/tests.rs b/crates/server/server/src/assets_api/tests.rs index 7596faa9f..fbb2b0ea3 100644 --- a/crates/server/server/src/assets_api/tests.rs +++ b/crates/server/server/src/assets_api/tests.rs @@ -1489,7 +1489,7 @@ pub(crate) async fn seed_attachment_with_preview( source_file: "seed.jsonl", messages: &[crate::test_support::SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "two photos", }], diff --git a/crates/server/server/src/contacts_api/tests.rs b/crates/server/server/src/contacts_api/tests.rs index d173bc55d..b5bc66637 100644 --- a/crates/server/server/src/contacts_api/tests.rs +++ b/crates/server/server/src/contacts_api/tests.rs @@ -562,8 +562,8 @@ async fn get_contact_detail_counts_direct_group_and_messages() { .await .unwrap(); for (body, ts) in [ - ("hi", "2024-06-01T12:00:00Z"), - ("there", "2024-06-01T13:00:00Z"), + ("hi", "2024-06-01T12:00:00.000Z"), + ("there", "2024-06-01T13:00:00.000Z"), ] { MessageRow { timestamp: ts, @@ -576,7 +576,7 @@ async fn get_contact_detail_counts_direct_group_and_messages() { } // A reply of yours: not a message Sam sent, so not in `total_messages`. MessageRow { - timestamp: "2024-06-01T14:00:00Z", + timestamp: "2024-06-01T14:00:00.000Z", is_from_me: true, body: Some("back at you"), ..MessageRow::new(account, 1) @@ -612,7 +612,7 @@ async fn get_contact_detail_counts_direct_group_and_messages() { .await .unwrap(); MessageRow { - timestamp: "2024-07-01T12:00:00Z", + timestamp: "2024-07-01T12:00:00.000Z", sender_handle_id: Some(peer), body: Some("group hi"), ..MessageRow::new(account, 2) @@ -636,7 +636,7 @@ async fn get_contact_detail_counts_direct_group_and_messages() { .await .unwrap(); MessageRow { - timestamp: "2024-08-01T12:00:00Z", + timestamp: "2024-08-01T12:00:00.000Z", body: Some("nope"), ..MessageRow::new(account, 9) } @@ -665,11 +665,11 @@ async fn get_contact_detail_counts_direct_group_and_messages() { assert_eq!(detail.identities[0].group_messages, 1); assert_eq!( detail.identities[0].start_date.as_deref(), - Some("2024-06-01T12:00:00Z") + Some("2024-06-01T12:00:00.000Z") ); assert_eq!( detail.identities[0].end_date.as_deref(), - Some("2024-07-01T12:00:00Z") + Some("2024-07-01T12:00:00.000Z") ); } @@ -719,8 +719,8 @@ async fn get_contact_summaries_counts_two_contacts_in_one_query() { .await .unwrap(); for (body, ts) in [ - ("hi", "2024-06-01T12:00:00Z"), - ("there", "2024-06-01T13:00:00Z"), + ("hi", "2024-06-01T12:00:00.000Z"), + ("there", "2024-06-01T13:00:00.000Z"), ] { MessageRow { timestamp: ts, @@ -758,7 +758,7 @@ async fn get_contact_summaries_counts_two_contacts_in_one_query() { .await .unwrap(); MessageRow { - timestamp: "2024-07-01T12:00:00Z", + timestamp: "2024-07-01T12:00:00.000Z", sender_handle_id: Some(sam_handle), body: Some("group hi"), ..MessageRow::new(account, 2) @@ -806,7 +806,7 @@ async fn get_contact_summaries_counts_two_contacts_in_one_query() { .await .unwrap(); MessageRow { - timestamp: "2024-05-01T09:00:00Z", + timestamp: "2024-05-01T09:00:00.000Z", sender_handle_id: Some(pat_handle), body: Some("hey"), ..MessageRow::new(account, 3) @@ -827,11 +827,11 @@ async fn get_contact_summaries_counts_two_contacts_in_one_query() { assert_eq!(summaries[0].group_message_count, 1); assert_eq!( summaries[0].start_date.as_deref(), - Some("2024-06-01T12:00:00Z") + Some("2024-06-01T12:00:00.000Z") ); assert_eq!( summaries[0].end_date.as_deref(), - Some("2024-07-01T12:00:00Z") + Some("2024-07-01T12:00:00.000Z") ); assert_eq!(summaries[1].id, pat_id); @@ -842,11 +842,11 @@ async fn get_contact_summaries_counts_two_contacts_in_one_query() { assert_eq!(summaries[1].group_message_count, 0); assert_eq!( summaries[1].start_date.as_deref(), - Some("2024-05-01T09:00:00Z") + Some("2024-05-01T09:00:00.000Z") ); assert_eq!( summaries[1].end_date.as_deref(), - Some("2024-05-01T09:00:00Z") + Some("2024-05-01T09:00:00.000Z") ); } diff --git a/crates/server/server/src/conversations_api/tests.rs b/crates/server/server/src/conversations_api/tests.rs index ced4408b5..fb1943a1b 100644 --- a/crates/server/server/src/conversations_api/tests.rs +++ b/crates/server/server/src/conversations_api/tests.rs @@ -98,7 +98,7 @@ async fn conversations_setup() -> (sqlx::SqlitePool, TestFixture, i64) { .await .unwrap(); MessageRow { - timestamp: "2024-06-01T12:00:00Z", + timestamp: "2024-06-01T12:00:00.000Z", body: Some("hello"), ..MessageRow::new(account, 1) } @@ -189,7 +189,7 @@ async fn list_conversations_finds_a_handle_across_platforms() { .unwrap(); MessageRow { source: "whatsapp", - timestamp: "2024-08-01T12:00:00Z", + timestamp: "2024-08-01T12:00:00.000Z", body: Some("wa hello"), ..MessageRow::new(account, 10) } @@ -251,7 +251,7 @@ async fn list_conversations_sorts_by_date_or_message_count() { .await .unwrap(); MessageRow { - timestamp: "2024-07-01T12:00:00Z", + timestamp: "2024-07-01T12:00:00.000Z", body: Some("newest"), ..MessageRow::new(account, 2) } @@ -349,7 +349,7 @@ async fn list_conversations_paginates() { .await .unwrap(); MessageRow { - timestamp: "2024-07-01T12:00:00Z", + timestamp: "2024-07-01T12:00:00.000Z", body: Some("later"), ..MessageRow::new(account, 2) } @@ -490,7 +490,7 @@ async fn list_conversations_filters_by_contact_and_type() { .await .unwrap(); MessageRow { - timestamp: "2024-08-01T12:00:00Z", + timestamp: "2024-08-01T12:00:00.000Z", body: Some("group"), ..MessageRow::new(account, 9) } @@ -521,7 +521,7 @@ async fn list_conversations_filters_by_contact_and_type() { .await .unwrap(); MessageRow { - timestamp: "2024-09-01T12:00:00Z", + timestamp: "2024-09-01T12:00:00.000Z", body: Some("hi group"), ..MessageRow::new(account, 3) } @@ -689,7 +689,7 @@ async fn list_conversations_filters_by_participant_count() { .await .unwrap(); MessageRow { - timestamp: "2024-10-01T12:00:00Z", + timestamp: "2024-10-01T12:00:00.000Z", body: Some("hi"), ..MessageRow::new(account, 10) } @@ -764,7 +764,7 @@ async fn list_conversations_participants_eq_three_on_built_fixture() { .await .unwrap(); MessageRow { - timestamp: "2024-11-01T12:00:00Z", + timestamp: "2024-11-01T12:00:00.000Z", body: Some("hi trio"), ..MessageRow::new(account, 20) } @@ -865,7 +865,7 @@ async fn list_conversations_filters_by_import_id() { .unwrap(); MessageRow { - timestamp: "2024-06-01T12:00:00Z", + timestamp: "2024-06-01T12:00:00.000Z", body: Some("hello"), import_id: Some(import_a), ..MessageRow::new(account, 1) @@ -873,7 +873,7 @@ async fn list_conversations_filters_by_import_id() { .insert(&mut conn) .await; MessageRow { - timestamp: "2024-07-01T12:00:00Z", + timestamp: "2024-07-01T12:00:00.000Z", body: Some("later"), import_id: Some(import_b), ..MessageRow::new(account, 2) @@ -994,7 +994,7 @@ async fn duplicate_only_threads_have_no_last_message_date_and_sort_last() { // Conversation 4 keeps a real message, and it belongs to the import. let winner_id = MessageRow { - timestamp: "2024-05-01T12:00:00Z", + timestamp: "2024-05-01T12:00:00.000Z", body: Some("canonical"), import_id: Some(import_a), ..MessageRow::new(account, 4) @@ -1005,7 +1005,7 @@ async fn duplicate_only_threads_have_no_last_message_date_and_sort_last() { // Conversation 3's only message is a duplicate, so its last_message_at // is NULL even though its timestamp is the later of the two. MessageRow { - timestamp: "2024-06-01T12:00:00Z", + timestamp: "2024-06-01T12:00:00.000Z", body: Some("dup"), import_id: Some(import_a), duplicate_of: Some(winner_id), @@ -1208,7 +1208,7 @@ async fn list_conversations_import_id_includes_duplicate_only_thread() { .await .unwrap(); let winner_id = MessageRow { - timestamp: "2024-05-01T12:00:00Z", + timestamp: "2024-05-01T12:00:00.000Z", body: Some("canonical"), ..MessageRow::new(account, 4) } @@ -1217,7 +1217,7 @@ async fn list_conversations_import_id_includes_duplicate_only_thread() { // Only message in conversation 3 from import A is a duplicate. MessageRow { - timestamp: "2024-06-01T12:00:00Z", + timestamp: "2024-06-01T12:00:00.000Z", body: Some("dup"), import_id: Some(import_a), duplicate_of: Some(winner_id), @@ -1279,7 +1279,7 @@ async fn the_conversation_list_labels_each_thread_by_its_sources() { let (fixture, user) = fixture_with_account().await; let message = |source| SeedMessage { source, - timestamp: "2024-01-01T00:00:00Z", + timestamp: "2024-01-01T00:00:00.000Z", is_from_me: true, body: "hi", }; @@ -1450,7 +1450,7 @@ async fn conversation_detail_shows_no_chat_id_as_a_group_participant() { source_file: "_import.jsonl", messages: &[crate::test_support::SeedMessage { source: "imessage", - timestamp: "2024-02-01T10:00:00Z", + timestamp: "2024-02-01T10:00:00.000Z", is_from_me: false, body: "hello all", }], @@ -1841,7 +1841,7 @@ async fn conversation_delete_removes_files_only_the_deleted_conversation_used() source_file: "seed.jsonl", messages: &[crate::test_support::SeedMessage { source: "imessage", - timestamp: "2020-01-02T00:00:00Z", + timestamp: "2020-01-02T00:00:00.000Z", is_from_me: false, body: "hi", }], @@ -2062,7 +2062,7 @@ async fn conversation_messages_say_which_were_sent_and_which_received() { ) .await; MessageRow { - timestamp: "2024-01-02T00:00:00Z", + timestamp: "2024-01-02T00:00:00.000Z", body: Some("received"), ..MessageRow::new(user.account_id, conversation_id) } @@ -2181,7 +2181,7 @@ async fn insert_many_messages( for i in 0..count { let body = format!("msg{i}"); MessageRow { - timestamp: "2024-01-01T00:00:00Z", + timestamp: "2024-01-01T00:00:00.000Z", is_from_me: true, sort_order: i, body: Some(&body), @@ -2411,7 +2411,7 @@ async fn a_page_starts_at_one_place_beside_a_message_of_this_conversation() { source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "elsewhere", }], @@ -2737,7 +2737,7 @@ async fn the_list_labels_a_group_with_its_trimmed_title_and_a_blank_title_with_n let (fixture, alice) = fixture_with_account().await; let message = SeedMessage { source: "imessage", - timestamp: "2024-01-01T10:00:00Z", + timestamp: "2024-01-01T10:00:00.000Z", is_from_me: false, body: "hello", }; diff --git a/crates/server/server/src/db/staging/tests.rs b/crates/server/server/src/db/staging/tests.rs index f038cac7e..98ed3d733 100644 --- a/crates/server/server/src/db/staging/tests.rs +++ b/crates/server/server/src/db/staging/tests.rs @@ -30,7 +30,7 @@ async fn reset_for_account_leaves_other_accounts() { account_id: account, source: "sms", guid: "g1", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: 0, sender_handle_id: None, owner_handle_id: None, diff --git a/crates/server/server/src/db/write_guard.rs b/crates/server/server/src/db/write_guard.rs index 0edcb758f..8fd2bcc1e 100644 --- a/crates/server/server/src/db/write_guard.rs +++ b/crates/server/server/src/db/write_guard.rs @@ -173,7 +173,7 @@ mod tests { source_file: "guard.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: false, body: "kept", }], diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index b99cb36bc..d749e6bd7 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -655,11 +655,13 @@ fn pick_winner(cands: &[Cand], prio: &HashMap<&str, usize>) -> i64 { .map_or(cands[0].id, |c| c.id) } -/// Parse an RFC3339 timestamp into Unix UTC seconds, honoring Z / ±HH:MM offsets. +/// Parse an RFC3339 timestamp into Unix UTC seconds, honoring Z / ±HH:MM +/// offsets and dropping the milliseconds: the content key and the near-time +/// pass match at whole seconds. /// /// Strict RFC3339 is sufficient: `messages.timestamp` /// are only ever written by `models::format_utc_timestamp` (chrono's -/// `to_rfc3339_opts(SecondsFormat::Secs, true)`), so no lenient spellings reach +/// `to_rfc3339_opts(SecondsFormat::Millis, true)`), so no lenient spellings reach /// this path. Unparseable input yields `None`. fn parse_rfc3339_utc_secs(ts: &str) -> Option { chrono::DateTime::parse_from_rfc3339(ts.trim()) diff --git a/crates/server/server/src/dedupe/tests.rs b/crates/server/server/src/dedupe/tests.rs index 5fa840769..cc212dd8f 100644 --- a/crates/server/server/src/dedupe/tests.rs +++ b/crates/server/server/src/dedupe/tests.rs @@ -99,7 +99,7 @@ fn parallel_content_keys_match_serial() { chat_id: "+14075550106".into(), conversation_type: "individual".into(), is_from_me: 1, - timestamp: "2015-03-12T18:04:22Z".into(), + timestamp: "2015-03-12T18:04:22.000Z".into(), body: Some("hi".into()), sender_normalized: None, }, @@ -109,7 +109,7 @@ fn parallel_content_keys_match_serial() { chat_id: "chat-group".into(), conversation_type: "group".into(), is_from_me: 0, - timestamp: "2015-03-12T18:04:23Z".into(), + timestamp: "2015-03-12T18:04:23.000Z".into(), body: Some("yo".into()), sender_normalized: Some("+15555550128".into()), }, @@ -154,7 +154,7 @@ fn a_group_is_keyed_as_a_group_whatever_the_case_of_its_type() { chat_id: "chat-group".to_string(), conversation_type: conversation_type.to_string(), is_from_me: 0, - timestamp: "2015-03-12T18:04:23Z".to_string(), + timestamp: "2015-03-12T18:04:23.000Z".to_string(), body: Some("yo".to_string()), sender_normalized: Some("+15555550128".to_string()), }; @@ -237,7 +237,7 @@ async fn fill_missing_content_keys_skips_rows_that_already_have_keys() { MessageRow { source: "go-sms-pro", guid: Some("g-fill".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Need a key"), sort_order: 0, @@ -278,7 +278,7 @@ async fn fill_missing_content_keys_writes_multiple_rows_in_one_batch() { MessageRow { source: "go-sms-pro", guid: Some(guid.into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some(body), sort_order, @@ -321,7 +321,7 @@ async fn dedupe_cross_source_does_not_rewrite_unchanged_keys() { MessageRow { source: "go-sms-pro", guid: Some("g-once".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Once"), sort_order: 0, @@ -347,7 +347,7 @@ async fn integration_exact_flags_cross_source() { let a = MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Running late"), sort_order: 0, @@ -394,7 +394,7 @@ async fn integration_near_flags_within_window() { let a = MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: false, body: Some("On my way"), sort_order: 0, @@ -405,7 +405,7 @@ async fn integration_near_flags_within_window() { let b = MessageRow { source: "sms-backup-plus", guid: Some("g2".into()), - timestamp: "2015-03-12T18:04:24Z", + timestamp: "2015-03-12T18:04:24.000Z", is_from_me: false, body: Some("On my way"), sort_order: 1, @@ -435,7 +435,7 @@ async fn integration_negative_far_apart_not_flagged() { MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: false, body: Some("On my way"), sort_order: 0, @@ -446,7 +446,7 @@ async fn integration_negative_far_apart_not_flagged() { MessageRow { source: "sms-backup-plus", guid: Some("g2".into()), - timestamp: "2015-03-12T18:05:22Z", + timestamp: "2015-03-12T18:05:22.000Z", is_from_me: false, body: Some("On my way"), sort_order: 1, @@ -477,7 +477,7 @@ async fn integration_priority_prefers_first_imported_source() { let first_imported = MessageRow { source: "sms-backup-plus", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Hello"), sort_order: 0, @@ -488,7 +488,7 @@ async fn integration_priority_prefers_first_imported_source() { let second_imported = MessageRow { source: "go-sms-pro", guid: Some("g2".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Hello"), sort_order: 1, @@ -541,7 +541,7 @@ async fn a_twin_exactly_at_the_window_edge_is_flagged_and_one_past_it_is_not() { let first = MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: false, body: Some("On my way"), sort_order: 0, @@ -602,7 +602,7 @@ async fn two_near_messages_from_one_source_are_both_kept() { MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("ok"), sort_order: 0, @@ -613,7 +613,7 @@ async fn two_near_messages_from_one_source_are_both_kept() { let second = MessageRow { source: "go-sms-pro", guid: Some("g2".into()), - timestamp: "2015-03-12T18:04:23Z", + timestamp: "2015-03-12T18:04:23.000Z", is_from_me: true, body: Some("ok"), sort_order: 1, @@ -654,7 +654,7 @@ async fn identical_rows_from_one_source_are_both_kept() { MessageRow { source: "go-sms-pro", guid: Some(guid.into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("ok"), sort_order: 0, @@ -692,7 +692,7 @@ async fn an_exact_duplicate_across_three_sources_keeps_one() { MessageRow { source, guid: Some(guid.into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Running late"), sort_order: 0, @@ -849,7 +849,7 @@ async fn the_same_words_from_two_group_members_are_never_near_duplicates() { let from_ann = MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: false, sender_handle_id: Some(ann), body: Some("happy birthday!"), @@ -860,7 +860,7 @@ async fn the_same_words_from_two_group_members_are_never_near_duplicates() { let from_bo = MessageRow { source: "sms-backup-plus", guid: Some("g2".into()), - timestamp: "2015-03-12T18:04:23Z", + timestamp: "2015-03-12T18:04:23.000Z", is_from_me: false, sender_handle_id: Some(bo), body: Some("happy birthday!"), @@ -891,7 +891,7 @@ async fn a_write_that_commits_while_the_pass_reads_does_not_fail_it() { let first = MessageRow { source: "go-sms-pro", guid: Some("g1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Running late"), sort_order: 0, @@ -902,7 +902,7 @@ async fn a_write_that_commits_while_the_pass_reads_does_not_fail_it() { let second = MessageRow { source: "sms-backup-plus", guid: Some("g2".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, body: Some("Running late"), sort_order: 0, @@ -1046,7 +1046,7 @@ async fn an_attachment_added_after_the_first_dedupe_changes_the_content_key() { let first = MessageRow { source: "imessage", guid: Some("a-1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, sender_handle_id: None, body: Some("look"), @@ -1063,7 +1063,7 @@ async fn an_attachment_added_after_the_first_dedupe_changes_the_content_key() { let twin = MessageRow { source: "sms-backup-plus", guid: Some("b-1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, sender_handle_id: None, body: Some("look"), @@ -1103,7 +1103,7 @@ async fn a_participant_added_after_the_first_dedupe_changes_the_group_content_ke let first = MessageRow { source: "imessage", guid: Some("a-1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, sender_handle_id: None, body: Some("dinner at 7?"), @@ -1121,7 +1121,7 @@ async fn a_participant_added_after_the_first_dedupe_changes_the_group_content_ke let twin = MessageRow { source: "sms-backup-plus", guid: Some("b-1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, sender_handle_id: None, body: Some("dinner at 7?"), @@ -1163,7 +1163,7 @@ async fn the_holders_own_address_does_not_change_a_groups_content_key() { MessageRow { source, guid: Some(guid.into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: true, sender_handle_id: None, body: Some("dinner at 7?"), @@ -1195,7 +1195,7 @@ async fn received_notes_to_yourself( let before_link = MessageRow { source: "imazing", guid: Some("a-1".into()), - timestamp: "2015-03-12T18:04:22Z", + timestamp: "2015-03-12T18:04:22.000Z", is_from_me: false, sender_handle_id: Some(holder), body: Some("buy milk"), @@ -1444,7 +1444,7 @@ async fn generate_database(conn: &mut SqliteConnection, seed: u64) -> Vec { guid += 1; let timestamp = chrono::DateTime::from_timestamp(copy_secs, 0) .unwrap() - .to_rfc3339_opts(chrono::SecondsFormat::Secs, true); + .to_rfc3339_opts(chrono::SecondsFormat::Millis, true); let conversation_id = chat.conversations[rng.below(chat.conversations.len())]; let source = GEN_SOURCES[rng.below(GEN_SOURCES.len())]; let id = MessageRow { diff --git a/crates/server/server/src/exports_api/tests.rs b/crates/server/server/src/exports_api/tests.rs index 59440ce75..3a81ddb1c 100644 --- a/crates/server/server/src/exports_api/tests.rs +++ b/crates/server/server/src/exports_api/tests.rs @@ -169,7 +169,7 @@ async fn seeded_export_fixture() -> (TestFixture, i64, i64) { id: Some(2), source: "sms", service: Some("sms"), - timestamp: "2020-01-02T00:00:00Z", + timestamp: "2020-01-02T00:00:00.000Z", body: Some("hello two"), ..MessageRow::new(101, conv2) } @@ -296,7 +296,7 @@ async fn a_conversations_query_hides_what_the_conversations_list_hides() { id: Some(9), source: "sms", service: Some("sms"), - timestamp: "2020-01-05T00:00:00Z", + timestamp: "2020-01-05T00:00:00.000Z", body: Some("hello bob"), ..MessageRow::new(202, bobs) } @@ -407,7 +407,7 @@ async fn a_selection_refuses_ids_the_account_does_not_hold_naming_them() { id: Some(99), source: "sms", service: Some("sms"), - timestamp: "2020-02-01T00:00:00Z", + timestamp: "2020-02-01T00:00:00.000Z", body: Some("bob secret"), ..MessageRow::new(102, 99) } @@ -693,13 +693,13 @@ async fn fixture_with_two_conversations() -> (TestFixture, RegisteredAccount, i6 messages: &[ SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "pizza tonight", }, SeedMessage { source: "imessage", - timestamp: "2020-01-02T00:00:00Z", + timestamp: "2020-01-02T00:00:00.000Z", is_from_me: false, body: "salad tomorrow", }, @@ -717,7 +717,7 @@ async fn fixture_with_two_conversations() -> (TestFixture, RegisteredAccount, i6 source_file: "backup-a.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-03T00:00:00Z", + timestamp: "2020-01-03T00:00:00.000Z", is_from_me: false, body: "the menu", }], diff --git a/crates/server/server/src/messages_api/tests.rs b/crates/server/server/src/messages_api/tests.rs index e2d0091a7..1b231c393 100644 --- a/crates/server/server/src/messages_api/tests.rs +++ b/crates/server/server/src/messages_api/tests.rs @@ -24,13 +24,13 @@ async fn seeded() -> (TestFixture, RegisteredAccount, i64, i64) { messages: &[ SeedMessage { source: "imessage", - timestamp: "2024-01-01T10:00:00Z", + timestamp: "2024-01-01T10:00:00.000Z", is_from_me: false, body: "dentist on tuesday", }, SeedMessage { source: "imessage", - timestamp: "2024-01-02T10:00:00Z", + timestamp: "2024-01-02T10:00:00.000Z", is_from_me: true, body: "see you there", }, @@ -48,7 +48,7 @@ async fn seeded() -> (TestFixture, RegisteredAccount, i64, i64) { source_file: "t.json", messages: &[SeedMessage { source: "imessage", - timestamp: "2024-02-01T10:00:00Z", + timestamp: "2024-02-01T10:00:00.000Z", is_from_me: false, body: "the dentist called again", }], @@ -83,7 +83,7 @@ async fn seeded() -> (TestFixture, RegisteredAccount, i64, i64) { source_file: "t.json", messages: &[SeedMessage { source: "imessage", - timestamp: "2024-03-01T10:00:00Z", + timestamp: "2024-03-01T10:00:00.000Z", is_from_me: false, body: "bob's dentist", }], @@ -614,9 +614,9 @@ async fn an_edited_message_is_returned_with_its_earlier_versions_and_their_times edited["earlier_versions"], serde_json::json!([ {"part_index": 0, "text": "Meet at the library", - "edited_at": "2020-01-06T11:10:00Z", "matched": false}, + "edited_at": "2020-01-06T11:10:00.000Z", "matched": false}, {"part_index": 0, "text": "Meet at the museum", - "edited_at": "2020-01-06T11:10:30Z", "matched": false} + "edited_at": "2020-01-06T11:10:30.000Z", "matched": false} ]), "{page}" ); @@ -1605,12 +1605,14 @@ async fn date_today_is_the_day_on_the_accounts_clock() { .single() .unwrap() .with_timezone(&chrono::Utc) - .format("%Y-%m-%dT%H:%M:%SZ") + .format("%Y-%m-%dT%H:%M:%S%.3fZ") .to_string() }; let early_today = local(today, 0, 30); let late_yesterday = local(today.pred_opt().unwrap(), 23, 30); - let now = chrono::Utc::now().format("%Y-%m-%dT%H:%M:%SZ").to_string(); + let now = chrono::Utc::now() + .format("%Y-%m-%dT%H:%M:%S%.3fZ") + .to_string(); seed_conversation( &fixture.state, &SeedConversation { @@ -1680,25 +1682,25 @@ async fn seeded_for_relevance() -> (TestFixture, RegisteredAccount) { messages: &[ SeedMessage { source: "imessage", - timestamp: "2024-01-01T10:00:00Z", + timestamp: "2024-01-01T10:00:00.000Z", is_from_me: false, body: "dentist dentist dentist", }, SeedMessage { source: "imessage", - timestamp: "2024-01-02T10:00:00Z", + timestamp: "2024-01-02T10:00:00.000Z", is_from_me: false, body: "after work I will call the office of the dentist about next week", }, SeedMessage { source: "imessage", - timestamp: "2024-01-03T10:00:00Z", + timestamp: "2024-01-03T10:00:00.000Z", is_from_me: true, body: "the dentist moved it", }, SeedMessage { source: "imessage", - timestamp: "2024-01-04T10:00:00Z", + timestamp: "2024-01-04T10:00:00.000Z", is_from_me: true, body: "nothing to see", }, @@ -1895,3 +1897,83 @@ async fn a_conversations_messages_take_no_relevance() { .await; expect_problem(status, &text, ProblemType::ValidationFailed); } + +/// An Apple Messages conversation file whose two messages are 300 ms apart +/// in one second, the later one listed first, and whose earlier message has +/// an earlier version written 125 ms before it was sent (#1096). +fn apple_messages_sub_second_times() -> String { + at_current_schema_version(include_str!( + "../../tests/fixtures/apple-messages-sub-second-times.jsonl" + )) +} + +/// A message time the source records to the millisecond is stored and +/// returned to the millisecond, on the message and on its earlier version +/// (#1096). +#[tokio::test] +async fn a_message_time_keeps_its_milliseconds_through_import_and_the_api() { + let (fixture, alice) = fixture_with_account().await; + let counts = import_conversation_file( + &fixture, + alice.account_id, + "sub-second", + &apple_messages_sub_second_times(), + "imessage", + ) + .await; + assert_eq!(counts.messages, 2); + + let page: serde_json::Value = + get_json(&fixture.state, "/v1/messages?sort=date", &alice.token).await; + let first = message_by_guid(&page, "guid-first"); + assert_eq!(first["timestamp"], "2020-01-06T11:10:00.250Z", "{page}"); + assert_eq!( + first["earlier_versions"][0]["edited_at"], "2020-01-06T11:10:00.125Z", + "{page}" + ); + assert_eq!( + message_by_guid(&page, "guid-300-ms-later")["timestamp"], + "2020-01-06T11:10:00.550Z", + "{page}" + ); +} + +/// Two messages 300 ms apart in one conversation are listed in the order of +/// their times, though the file lists the later one first, on the messages +/// route and on the conversation's own (#1096). +#[tokio::test] +async fn two_messages_300_ms_apart_are_ordered_by_their_times() { + let (fixture, alice) = fixture_with_account().await; + import_conversation_file( + &fixture, + alice.account_id, + "sub-second", + &apple_messages_sub_second_times(), + "imessage", + ) + .await; + + let page: serde_json::Value = + get_json(&fixture.state, "/v1/messages?sort=date", &alice.token).await; + assert_eq!(guids(&page), ["guid-first", "guid-300-ms-later"], "{page}"); + let newest_first: serde_json::Value = + get_json(&fixture.state, "/v1/messages?sort=-date", &alice.token).await; + assert_eq!( + guids(&newest_first), + ["guid-300-ms-later", "guid-first"], + "{newest_first}" + ); + + let conversation_id = &page["items"][0]["conversation"]["id"]; + let thread: serde_json::Value = get_json( + &fixture.state, + &format!("/v1/conversations/{conversation_id}/messages"), + &alice.token, + ) + .await; + assert_eq!( + guids(&thread), + ["guid-first", "guid-300-ms-later"], + "{thread}" + ); +} diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index 927fea3dd..2b56a8bde 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -90,7 +90,9 @@ pub struct MessageRecord { /// The line the message is on in its file or batch, counted from 1 with /// blank lines included, so a refusal of one of its attachments names it. pub line: usize, - /// The instant the message was sent: RFC 3339 in UTC with a `Z` suffix. + /// The instant the message was sent, to the millisecond: RFC 3339 in UTC + /// with three fractional digits and a `Z` suffix + /// (`2015-03-12T18:04:22.250Z`). pub timestamp: String, /// True for messages sent by the account owner. pub is_from_me: bool, @@ -136,8 +138,9 @@ pub struct EarlierVersionRecord { pub part_index: i64, /// The part's text in this version. pub text: String, - /// When this version was written, RFC 3339 in UTC with a `Z` suffix as - /// `MessageRecord::timestamp`; `None` when the source does not record it. + /// When this version was written, to the millisecond in the form + /// `MessageRecord::timestamp` takes; `None` when the source does not + /// record it. pub edited_at: Option, } @@ -336,8 +339,7 @@ fn message_from_ir( header_owner: Option<&str>, line: usize, ) -> Result { - let secs = msg.timestamp_unix_ms.div_euclid(1000); - let timestamp = format_utc_timestamp(secs).with_context(|| { + let timestamp = format_utc_timestamp(msg.timestamp_unix_ms).with_context(|| { format!( "unrepresentable timestamp_unix_ms {}", msg.timestamp_unix_ms @@ -403,7 +405,7 @@ fn earlier_version_from_ir(version: &EarlierVersion) -> Result TapbackRecord { } } -/// The UTC RFC 3339 string (`Z` suffix) for a Unix timestamp, or `None` when -/// it cannot be represented. The server stores the instant and nothing about -/// where the phone was; the account's time zone turns it into a clock reading. -fn format_utc_timestamp(secs: i64) -> Option { +/// The UTC RFC 3339 string for a Unix time in milliseconds, or `None` when it +/// cannot be represented. It always has three fractional digits and a `Z` +/// suffix (`2015-03-12T18:04:22.000Z` for a whole second), so every stored +/// time has one form and sorts as text in time order. The server stores the +/// instant and nothing about where the phone was; the account's time zone +/// turns it into a clock reading. +fn format_utc_timestamp(ms: i64) -> Option { Some( - Utc.timestamp_opt(secs, 0) + Utc.timestamp_millis_opt(ms) .single()? - .to_rfc3339_opts(chrono::SecondsFormat::Secs, true), + .to_rfc3339_opts(chrono::SecondsFormat::Millis, true), ) } diff --git a/crates/server/server/src/openapi/credential_matrix.rs b/crates/server/server/src/openapi/credential_matrix.rs index 47c933398..a060ee33b 100644 --- a/crates/server/server/src/openapi/credential_matrix.rs +++ b/crates/server/server/src/openapi/credential_matrix.rs @@ -445,7 +445,7 @@ impl<'a> World<'a> { source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "hello", }], diff --git a/crates/server/server/src/reset_demo/tests.rs b/crates/server/server/src/reset_demo/tests.rs index 03a4faa63..01e1d7d46 100644 --- a/crates/server/server/src/reset_demo/tests.rs +++ b/crates/server/server/src/reset_demo/tests.rs @@ -1140,7 +1140,7 @@ async fn seed_reset_test_account(conn: &mut SqliteConnection, account_id: i64, g .expect("insert reset test conversation"); MessageRow { guid: Some(guid.into()), - timestamp: "2026-01-01T00:00:00Z", + timestamp: "2026-01-01T00:00:00.000Z", body: Some("keep me"), ..MessageRow::new(account_id, conversation_id) } @@ -1909,7 +1909,7 @@ async fn seed_previous_demo(db: &Path, data_dir: &Path) -> PathBuf { MessageRow { source: "whatsapp", guid: Some("previous-demo-message".into()), - timestamp: "2026-01-01T00:00:00Z", + timestamp: "2026-01-01T00:00:00.000Z", body: Some("from the previous demo"), ..MessageRow::new(DEMO_ACCOUNT_ID, conversation_id) } @@ -2705,7 +2705,7 @@ async fn seed_bulky_demo(db: &Path) { MessageRow { source: "whatsapp", guid: Some(format!("previous-{i}")), - timestamp: "2026-01-01T00:00:00Z", + timestamp: "2026-01-01T00:00:00.000Z", body: Some(&body), sort_order: i, ..MessageRow::new(DEMO_ACCOUNT_ID, conversation_id) @@ -2852,7 +2852,7 @@ async fn the_wipe_deletes_duplicates_before_the_messages_they_duplicate() { MessageRow { source: "sms", guid: Some(guid.into()), - timestamp: "2026-01-01T00:00:00Z", + timestamp: "2026-01-01T00:00:00.000Z", body: Some("hello"), duplicate_of, ..MessageRow::new(DEMO_ACCOUNT_ID, conversation_id) diff --git a/crates/server/server/src/search/tests.rs b/crates/server/server/src/search/tests.rs index 6629fd17a..4a7ee12b7 100644 --- a/crates/server/server/src/search/tests.rs +++ b/crates/server/server/src/search/tests.rs @@ -2617,6 +2617,43 @@ mod measure_words { ); } + /// A day begins at its first millisecond: a message sent then is on that + /// day, and one sent a millisecond before is on the day before. The + /// stored time has milliseconds, so the day's start is compared in the + /// same text form (#1096). + #[tokio::test] + async fn a_day_begins_at_its_first_millisecond() { + let (pool, _dir, f) = seeded().await; + let mut conn = pool.acquire().await.unwrap(); + let c = f.ana_direct; + let h = Some(f.ana_handle); + let last_of_april = message( + &mut conn, + ACCOUNT, + msg(c, "2013-04-30T23:59:59.999Z", false, h, "a"), + ) + .await; + let first_of_may = message( + &mut conn, + ACCOUNT, + msg(c, "2013-05-01T00:00:00.000Z", false, h, "b"), + ) + .await; + let m = ListKind::Messages; + assert_eq!( + run(&mut conn, m, "date:2013-05-01").await, + vec![first_of_may] + ); + assert_eq!( + run(&mut conn, m, "date:2013-04-30").await, + vec![last_of_april] + ); + assert_eq!( + run(&mut conn, m, "date:>=2013-05-01 date:<2013-06").await, + vec![first_of_may] + ); + } + /// Year 9999 ends at the start of year 10000, which in UTC or west of it /// is an instant chrono writes `+10000-…`. As text that sorts before every /// stored timestamp, so the comparison went the wrong way (#1205). diff --git a/crates/server/server/src/search/value.rs b/crates/server/server/src/search/value.rs index 372c675b6..686e9ae49 100644 --- a/crates/server/server/src/search/value.rs +++ b/crates/server/server/src/search/value.rs @@ -59,20 +59,20 @@ pub(crate) enum Value { } /// The instant `day` begins in `zone`, as the RFC 3339 UTC text the server -/// stores (`2024-01-01T05:00:00Z`), so a day or a year in the account's time +/// stores (`2024-01-01T05:00:00.000Z`), so a day or a year in the account's time /// zone compares against `messages.timestamp` as text. A day whose midnight /// falls in a daylight-saving gap starts at the first instant after the gap. /// A day the zone skipped whole (Pacific/Apia, 30 December 2011) starts where /// the next day starts, so it holds no instant. /// /// `None` when the instant falls after year 9999: RFC 3339 text for it has a -/// sign and five digits (`+10000-01-01T00:00:00Z`), and `+` sorts before +/// sign and five digits (`+10000-01-01T00:00:00.000Z`), and `+` sorts before /// every digit, so as text it would come before every stored timestamp /// instead of after them all. pub(crate) fn utc_instant(zone: chrono_tz::Tz, day: NaiveDate) -> Option { let midnight = day.and_hms_opt(0, 0, 0).expect("midnight is a valid time"); let instant = first_instant_at_or_after(zone, midnight); - (instant.year() <= 9999).then(|| instant.to_rfc3339_opts(chrono::SecondsFormat::Secs, true)) + (instant.year() <= 9999).then(|| instant.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)) } /// The first instant whose clock reading in `zone` is `local` or later. @@ -85,11 +85,14 @@ fn first_instant_at_or_after(zone: chrono_tz::Tz, local: NaiveDateTime) -> DateT // Every UTC offset is shorter than a day, so a day before `local` read as // UTC the clock shows an earlier time, and a day after it a later one. // The gap's end lies between. A binary search over whole seconds finds - // it, because every transition falls on a whole second. + // it, because every transition falls on a whole second. Each step is a + // whole number of seconds, so the instant found has no fraction of a + // second: halving an odd number of seconds once left one, and the + // millisecond text the day's start is compared in would carry it. let mut before = local.and_utc() - Duration::days(1); let mut after = local.and_utc() + Duration::days(1); while after - before > Duration::seconds(1) { - let mid = before + (after - before) / 2; + let mid = before + Duration::seconds((after - before).num_seconds() / 2); if mid.with_timezone(&zone).naive_local() >= local { after = mid; } else { @@ -481,19 +484,19 @@ mod tests { let zone = chrono_tz::Pacific::Apia; assert_eq!( utc_instant(zone, d(2011, 12, 29)).as_deref(), - Some("2011-12-29T10:00:00Z") + Some("2011-12-29T10:00:00.000Z") ); assert_eq!( utc_instant(zone, d(2011, 12, 30)).as_deref(), - Some("2011-12-30T10:00:00Z") + Some("2011-12-30T10:00:00.000Z") ); assert_eq!( utc_instant(zone, d(2011, 12, 31)).as_deref(), - Some("2011-12-30T10:00:00Z") + Some("2011-12-30T10:00:00.000Z") ); assert_eq!( utc_instant(zone, d(2012, 1, 1)).as_deref(), - Some("2011-12-31T10:00:00Z") + Some("2011-12-31T10:00:00.000Z") ); } @@ -520,7 +523,7 @@ mod tests { fn a_midnight_in_a_one_hour_gap_starts_the_day_after_the_gap() { assert_eq!( utc_instant(chrono_tz::America::Sao_Paulo, d(2018, 11, 4)).as_deref(), - Some("2018-11-04T03:00:00Z") + Some("2018-11-04T03:00:00.000Z") ); } } diff --git a/crates/server/server/src/server_api/tests.rs b/crates/server/server/src/server_api/tests.rs index 88b34572e..9575ecdbf 100644 --- a/crates/server/server/src/server_api/tests.rs +++ b/crates/server/server/src/server_api/tests.rs @@ -644,7 +644,7 @@ async fn the_owner_reads_the_server_totals_summed_over_every_account() { .iter() .map(|body| SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body, }) diff --git a/crates/server/server/src/session_api/tests.rs b/crates/server/server/src/session_api/tests.rs index 9ea2c26ad..d79f5f5a0 100644 --- a/crates/server/server/src/session_api/tests.rs +++ b/crates/server/server/src/session_api/tests.rs @@ -233,7 +233,7 @@ async fn seed_source(state: &crate::server::AppState, account_id: i64, source: & source_file: "seed.jsonl", messages: &[SeedMessage { source, - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "hello", }], diff --git a/crates/server/server/src/test_support.rs b/crates/server/server/src/test_support.rs index 480f157ee..cb84904d0 100644 --- a/crates/server/server/src/test_support.rs +++ b/crates/server/server/src/test_support.rs @@ -770,7 +770,8 @@ pub async fn delete_raw(state: &AppState, path: &str, token: &str) -> (StatusCod pub struct SeedMessage<'a> { /// The `messages.source` slug, such as `imessage`. pub source: &'a str, - /// RFC 3339 timestamp, stored as text the way the importer writes it. + /// RFC 3339 timestamp, stored as text the way the importer writes it: UTC + /// to the millisecond (`2020-01-01T00:00:00.000Z`). pub timestamp: &'a str, /// Whether the account sent it. pub is_from_me: bool, @@ -834,7 +835,7 @@ pub struct MessageRow<'a> { pub source: &'a str, /// `messages.guid`. `None` writes NULL, which the table refuses. pub guid: Option, - /// RFC 3339 in UTC, as the importer writes it. + /// RFC 3339 in UTC to the millisecond, as the importer writes it. pub timestamp: &'a str, /// Whether the account sent it. pub is_from_me: bool, @@ -870,7 +871,7 @@ pub struct MessageRow<'a> { impl MessageRow<'_> { /// A received `imessage` message in `conversation_id` of `account_id`, - /// with a fresh guid, at 2020-01-01T00:00:00Z, and nothing optional set. + /// with a fresh guid, at 2020-01-01T00:00:00.000Z, and nothing optional set. pub fn new(account_id: i64, conversation_id: i64) -> Self { Self { id: None, @@ -878,7 +879,7 @@ impl MessageRow<'_> { account_id, source: "imessage", guid: Some(unique_guid()), - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: false, sender_handle_id: None, owner_handle_id: None, @@ -1069,7 +1070,7 @@ pub async fn seed_one_message(state: &AppState, account_id: i64) { source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "hello", }], @@ -1250,13 +1251,13 @@ mod tests { messages: &[ SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "first", }, SeedMessage { source: "imessage", - timestamp: "2020-01-02T00:00:00Z", + timestamp: "2020-01-02T00:00:00.000Z", is_from_me: false, body: "second", }, @@ -1301,7 +1302,7 @@ mod tests { source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "hello, this is a message long enough to add up", }], diff --git a/crates/server/server/src/trash_api/tests.rs b/crates/server/server/src/trash_api/tests.rs index 92933b138..10d7f9fca 100644 --- a/crates/server/server/src/trash_api/tests.rs +++ b/crates/server/server/src/trash_api/tests.rs @@ -19,7 +19,7 @@ async fn seed(fixture: &TestFixture, account: &RegisteredAccount, handle: &str) source_file: "seed.jsonl", messages: &[SeedMessage { source: "imessage", - timestamp: "2020-01-01T00:00:00Z", + timestamp: "2020-01-01T00:00:00.000Z", is_from_me: true, body: "hello", }], diff --git a/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl b/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl new file mode 100644 index 000000000..04a008e8d --- /dev/null +++ b/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl @@ -0,0 +1,3 @@ +{"schema_version":10,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":2,"attachment_count":0,"first_timestamp_unix_ms":1578309000250,"last_timestamp_unix_ms":1578309000550}}} +{"guid":"guid-300-ms-later","timestamp_unix_ms":1578309000550,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"On my way","attachments":[],"imessage":null,"source":null} +{"guid":"guid-first","timestamp_unix_ms":1578309000250,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000125}],"imessage":null,"source":null} diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 53409dbef..19acb9caf 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -361,7 +361,7 @@ of a group, or one chat id written two ways, merges into the conversation already there. The merged conversation keeps the group title of the copy whose latest message is later. A copy with no title never clears a title, and when both copies' latest messages share one time, the title already stored stays. -Times are compared to the second, the precision a message's time is stored at. +Times are compared to the millisecond, the precision a message's time is stored at. The conversation stores the latest message time of the copy that gave its title (`group_title_at`), and an incoming copy is compared with that, not with the whole conversation: an untitled copy whose messages end last would diff --git a/docs/src/assets/openapi.json b/docs/src/assets/openapi.json index 9169a7b53..e16d3d703 100644 --- a/docs/src/assets/openapi.json +++ b/docs/src/assets/openapi.json @@ -14682,7 +14682,7 @@ "string", "null" ], - "description": "When this version was written: RFC 3339 in UTC with a `Z` suffix,\nas `Message.timestamp`. The original's is when it was sent, a\nlater version's is when the edit that wrote it was made. `None`\nwhen the source does not record it." + "description": "When this version was written, to the millisecond in the form\n`Message.timestamp` takes. The original's is when it was sent, a\nlater version's is when the edit that wrote it was made. `None`\nwhen the source does not record it." }, "matched": { "type": "boolean", @@ -15840,7 +15840,7 @@ }, "timestamp": { "type": "string", - "description": "The instant the message was sent: RFC 3339 in UTC with a `Z`\nsuffix. A caller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." + "description": "The instant the message was sent, to the millisecond: RFC 3339\nin UTC with three fractional digits and a `Z` suffix\n(`2015-03-12T18:04:22.250Z`; `.000` when the source records\nwhole seconds). Messages are listed in the order of this time. A\ncaller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." } } }, @@ -17867,7 +17867,7 @@ }, "timestamp": { "type": "string", - "description": "The instant the message was sent: RFC 3339 in UTC with a `Z`\nsuffix. A caller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." + "description": "The instant the message was sent, to the millisecond: RFC 3339\nin UTC with three fractional digits and a `Z` suffix\n(`2015-03-12T18:04:22.250Z`; `.000` when the source records\nwhole seconds). Messages are listed in the order of this time. A\ncaller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." } } }, diff --git a/schema/sql/messages.sql b/schema/sql/messages.sql index 373e1a671..f74d51d45 100644 --- a/schema/sql/messages.sql +++ b/schema/sql/messages.sql @@ -56,8 +56,11 @@ CREATE TABLE IF NOT EXISTS messages ( -- Source-native message id; used for exact dedupe with source. The import -- refuses a message without one. guid TEXT NOT NULL CHECK (guid != ''), - -- The instant the message was sent, RFC 3339 in UTC with a Z suffix. Shown, searched - -- and filed by day and year in the account's time zone (accounts.time_zone). + -- The instant the message was sent, to the millisecond: RFC 3339 in UTC + -- with three fractional digits and a Z suffix (2015-03-12T18:04:22.250Z; + -- .000 for a source that records whole seconds). One fixed form, so the + -- text sorts in time order and lists order by it. Shown, searched and + -- filed by day and year in the account's time zone (accounts.time_zone). timestamp TEXT NOT NULL, -- 1 = sent by the account holder; 0 = received from someone else. is_from_me INTEGER NOT NULL, @@ -229,8 +232,8 @@ CREATE TABLE IF NOT EXISTS message_versions ( part_index INTEGER NOT NULL DEFAULT 0, -- The part's text in this version. text TEXT NOT NULL, - -- When this version was written, RFC 3339 in UTC with a Z suffix, as - -- messages.timestamp: the send time for the original, the edit's time for + -- When this version was written, to the millisecond in the form + -- messages.timestamp holds: the send time for the original, the edit's time for -- a later one. NULL when the source does not record it. edited_at TEXT ); diff --git a/schema/sql/staging.sql b/schema/sql/staging.sql index 84fd99b08..5d015b579 100644 --- a/schema/sql/staging.sql +++ b/schema/sql/staging.sql @@ -46,8 +46,8 @@ CREATE TABLE IF NOT EXISTS staging_messages ( -- The message's id from the export: Apple's own for Apple Messages, otherwise -- the exporter's MessageGuid. Never empty; the import refuses a message without one. guid TEXT NOT NULL CHECK (guid != ''), - -- The instant the message was sent, RFC 3339 in UTC with a Z suffix. Shown, searched - -- and filed by day and year in the account's time zone (accounts.time_zone). + -- The instant the message was sent, in the form messages.timestamp holds: + -- RFC 3339 in UTC to the millisecond (2015-03-12T18:04:22.250Z). timestamp TEXT NOT NULL, -- 1 = sent by the account holder; 0 = received from someone else. is_from_me INTEGER NOT NULL, diff --git a/web/src/lib/serverApi.types.ts b/web/src/lib/serverApi.types.ts index 1b70b933a..ec6c38763 100644 --- a/web/src/lib/serverApi.types.ts +++ b/web/src/lib/serverApi.types.ts @@ -2650,8 +2650,8 @@ export interface components { /** @description One earlier version of one part of an edited message. */ EarlierVersion: { /** - * @description When this version was written: RFC 3339 in UTC with a `Z` suffix, - * as `Message.timestamp`. The original's is when it was sent, a + * @description When this version was written, to the millisecond in the form + * `Message.timestamp` takes. The original's is when it was sent, a * later version's is when the edit that wrote it was made. `None` * when the source does not record it. */ @@ -3367,8 +3367,11 @@ export interface components { /** @description Body text, when present. */ text: string | null; /** - * @description The instant the message was sent: RFC 3339 in UTC with a `Z` - * suffix. A caller shows it in the account's time zone + * @description The instant the message was sent, to the millisecond: RFC 3339 + * in UTC with three fractional digits and a `Z` suffix + * (`2015-03-12T18:04:22.250Z`; `.000` when the source records + * whole seconds). Messages are listed in the order of this time. A + * caller shows it in the account's time zone * (`Account.time_zone`); the database stores nothing * about where the phone was. */ @@ -4420,8 +4423,11 @@ export interface components { /** @description Body text, when present. */ text: string | null; /** - * @description The instant the message was sent: RFC 3339 in UTC with a `Z` - * suffix. A caller shows it in the account's time zone + * @description The instant the message was sent, to the millisecond: RFC 3339 + * in UTC with three fractional digits and a `Z` suffix + * (`2015-03-12T18:04:22.250Z`; `.000` when the source records + * whole seconds). Messages are listed in the order of this time. A + * caller shows it in the account's time zone * (`Account.time_zone`); the database stores nothing * about where the phone was. */ From 5c8cc8d70e198e41e3ff3644f77fb9040ceb5df6 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:26:47 -0400 Subject: [PATCH 08/42] refactor(export): call the step that writes JSON Lines the export step Fetch already names fetching an Asset, so the step before the format step is the export step, after the invokeExport call it makes. Co-Authored-By: Claude Opus 5.5 --- CONTEXT.md | 2 +- web/src/lib/desktopJob.ts | 2 +- web/src/lib/runCancel.ts | 2 +- web/src/screens/ExportScreen.test.tsx | 52 +++++++++++++-------------- web/src/screens/ExportScreen.tsx | 16 ++++----- 5 files changed, 37 insertions(+), 37 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 6d2dd5753..8ee8d28d3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -489,7 +489,7 @@ chosen, gets a directory of its own, named for what it is, when it started and its format, such as `export-2026-10-04-1430-mbox`. That directory is where the result lands unless the person chose another destination. While the run goes, it also holds the in-between files, such as the JSON Lines an -Export fetches before converting them; they are deleted when the run +Export writes before converting them; they are deleted when the run finishes, leaving only the result. A run that fails or is cancelled deletes its directory, and the app deletes one it did not see to its end the next time it starts. Message Crate never deletes a finished export from it. It belongs in the Message diff --git a/web/src/lib/desktopJob.ts b/web/src/lib/desktopJob.ts index 242521fa9..ede52b4be 100644 --- a/web/src/lib/desktopJob.ts +++ b/web/src/lib/desktopJob.ts @@ -13,7 +13,7 @@ export type DesktopJobName = "Import Run" | "Export" | "Convert"; * desktop refuses. * * Holds nest: an Import Run holds it from its first stage to its end and an - * Export from its fetch to the end of its format step, and `awaitTauriJob` + * Export from its export step to the end of its format step, and `awaitTauriJob` * holds it again for each desktop job call inside them. A call ending * releases only its own hold, so the run's hold covers the gaps between its * jobs, when the desktop itself has nothing running to refuse a Convert with. diff --git a/web/src/lib/runCancel.ts b/web/src/lib/runCancel.ts index 8a160ecc6..b5e6d8f47 100644 --- a/web/src/lib/runCancel.ts +++ b/web/src/lib/runCancel.ts @@ -8,7 +8,7 @@ import { invokeCancel } from "./tauri"; export const CANCELLED_MESSAGE = "cancelled"; /** - * The Cancel of one run of desktop jobs: an Import Run, or an export's fetch + * The Cancel of one run of desktop jobs: an Import Run, or an Export's export step * and conversion. * * The desktop side stops only the job that is running, and each job command diff --git a/web/src/screens/ExportScreen.test.tsx b/web/src/screens/ExportScreen.test.tsx index ca287d6f9..d0f68533a 100644 --- a/web/src/screens/ExportScreen.test.tsx +++ b/web/src/screens/ExportScreen.test.tsx @@ -108,7 +108,7 @@ async function exportAs(directory: string, formatLabel: string) { } describe("ExportScreen", () => { - it("fetches straight into the chosen directory for JSON Lines", async () => { + it("exports straight into the chosen directory for JSON Lines", async () => { await exportTo("/home/demo/out"); await waitFor(() => expect(invokeExport).toHaveBeenCalledTimes(1)); @@ -122,7 +122,7 @@ describe("ExportScreen", () => { expect(invokeDiscardExportDir).not.toHaveBeenCalled(); }); - it("fetches JSON Lines into its own directory in the Export Directory when no directory is chosen", async () => { + it("exports JSON Lines into its own directory in the Export Directory when no directory is chosen", async () => { const user = setupUser(); renderScreen(); expect(screen.getByRole("button", { name: "Export" })).toBeEnabled(); @@ -159,7 +159,7 @@ describe("ExportScreen", () => { ); }); - it("fetches into the export's own directory and converts into the chosen directory for CSV", async () => { + it("exports into the export's own directory and converts into the chosen directory for CSV", async () => { const exported = EXPORT_DIR.exported; await exportAs("/home/demo/out", "CSV (.csv)"); @@ -174,10 +174,10 @@ describe("ExportScreen", () => { }); }); - it("hands the conversion the time the Export Run started, before the fetch", async () => { - let fetched = 0; + it("hands the conversion the time the Export Run started, before the export step", async () => { + let exportStepAt = 0; invokeExport.mockImplementation(async () => { - fetched = Date.now(); + exportStepAt = Date.now(); }); const before = Date.now(); await exportAs("/home/demo/out", "CSV (.csv)"); @@ -185,7 +185,7 @@ describe("ExportScreen", () => { await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); const started = invokeFormat.mock.calls[0][0].run_started_ms as number; expect(started).toBeGreaterThanOrEqual(before); - expect(started).toBeLessThanOrEqual(fetched); + expect(started).toBeLessThanOrEqual(exportStepAt); }); it("starts nothing when the desktop refuses Save to for holding the Export Directory", async () => { @@ -214,7 +214,7 @@ describe("ExportScreen", () => { // disk, in a directory the person never chose and will not think to look in. awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "fetched" }; + return { summary: "exported" }; }); awaitTauriJob.mockImplementationOnce(async () => { throw new Error("unsupported output format"); @@ -227,7 +227,7 @@ describe("ExportScreen", () => { expect(await screen.findByText("unsupported output format")).toBeTruthy(); }); - it("does not start the conversion when Cancel is pressed after the fetch finished", async () => { + it("does not start the conversion when Cancel is pressed after the export step finished", async () => { // A Cancel sent while no job runs stops nothing, and invokeFormat starts // its job with a cancel flag of its own, so the screen must not start it. let releaseFormat: () => void = () => {}; @@ -236,7 +236,7 @@ describe("ExportScreen", () => { }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "fetched" }; + return { summary: "exported" }; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await formatHeld; @@ -253,8 +253,8 @@ describe("ExportScreen", () => { expect(invokeFormat).not.toHaveBeenCalled(); }); - it("keeps Convert off between the fetch and the format step, and lets it start once the export ends", async () => { - // Each job holds the desktop only while it runs; between the fetch and the + it("keeps Convert off between the export step and the format step, and lets it start once the export ends", async () => { + // Each job holds the desktop only while it runs; between the export step and the // format step the desktop has nothing running, so a Convert started there // would make it refuse the format step (#1407). let releaseFormat: () => void = () => {}; @@ -263,7 +263,7 @@ describe("ExportScreen", () => { }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await invokeFn(); - return { summary: "fetched" }; + return { summary: "exported" }; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { await formatHeld; @@ -291,7 +291,7 @@ describe("ExportScreen", () => { await user.click(await screen.findByRole("option", { name: "CSV (.csv)" })); await user.click(screen.getByRole("button", { name: "Export" })); - // The fetch has ended and the format step has not started. + // The export step has ended and the format step has not started. await waitFor(() => expect(awaitTauriJob).toHaveBeenCalledTimes(2)); expect(invokeExport).toHaveBeenCalledTimes(1); expect(invokeFormat).not.toHaveBeenCalled(); @@ -317,13 +317,13 @@ describe("ExportScreen", () => { it("ignores a second Export while one is already under way", async () => { // The desktop backend runs one job at a time (src-tauri/src/commands/jobs.rs), - // and between the fetch and the conversion it has nothing running to refuse. - let releaseFetch: () => void = () => {}; - const fetchStarted = new Promise((resolve) => { - releaseFetch = resolve; + // and between the export step and the conversion it has nothing running to refuse. + let releaseExport: () => void = () => {}; + const exportStarted = new Promise((resolve) => { + releaseExport = resolve; }); invokeCreateExportDir.mockImplementation(async () => { - await fetchStarted; + await exportStarted; return EXPORT_DIR; }); @@ -341,7 +341,7 @@ describe("ExportScreen", () => { // itself 80 ms later. On a busy machine that lands after the first export // has ended and the button is live again, and starts a second one. fireEvent.click(exportButton); - releaseFetch(); + releaseExport(); await waitFor(() => expect(invokeFormat).toHaveBeenCalledTimes(1)); expect(invokeExport).toHaveBeenCalledTimes(1); @@ -453,19 +453,19 @@ describe("ExportScreen", () => { }); it("locks the directory field while an export runs", async () => { - let releaseFetch: () => void = () => {}; - const fetchHeld = new Promise((resolve) => { - releaseFetch = resolve; + let releaseExport: () => void = () => {}; + const exportHeld = new Promise((resolve) => { + releaseExport = resolve; }); awaitTauriJob.mockImplementationOnce(async (invokeFn: () => Promise) => { - await fetchHeld; + await exportHeld; await invokeFn(); - return { summary: "fetched" }; + return { summary: "exported" }; }); await exportTo("/a"); expect(screen.getByPlaceholderText("The Export Directory")).toBeDisabled(); - releaseFetch(); + releaseExport(); await screen.findByText(/Export complete/); expect(screen.getByPlaceholderText("The Export Directory")).toBeEnabled(); }); diff --git a/web/src/screens/ExportScreen.tsx b/web/src/screens/ExportScreen.tsx index 01d76bdc8..a8c37d14c 100644 --- a/web/src/screens/ExportScreen.tsx +++ b/web/src/screens/ExportScreen.tsx @@ -71,7 +71,7 @@ function formatLabel(id: ExportFormat): string { * Every export gets a directory of its own in the Export Directory, named for * when it started and its format (`export-2026-10-04-1430-mbox`). The result * lands there unless the person chose another directory under **Save to**. - * The JSON Lines a non-JSONL export fetches wait in that directory while they + * The JSON Lines a non-JSONL export writes wait in that directory while they * are converted, since `message-reexport` refuses to write into a directory * that holds its input, and the conversion writes beside them. When the export * finishes, the desktop deletes the JSON Lines and moves the result up, so the @@ -105,7 +105,7 @@ export default function ExportScreen() { const [log, setLog] = useState([]); // `running` only turns true once a job starts, which leaves two windows // where the Export button would be live mid-export: while the export's - // directory is made, and between the fetch and the conversion. The desktop + // directory is made, and between the export step and the conversion. The desktop // refuses a second job while one runs (`jobs.rs`), but between two jobs it // has nothing to refuse. This covers the whole run. const [busy, setBusy] = useState(false); @@ -114,7 +114,7 @@ export default function ExportScreen() { const { running, finished, run } = useTauriJob<{ savePath: string; format: ExportFormat }>({ job: "Export", }); - // The Cancel of the export under way. A Cancel pressed after the fetch and + // The Cancel of the export under way. A Cancel pressed after the export step and // before the conversion starts must stop the conversion, and the desktop // alone would not: with no job running, its Cancel stops nothing, and // `format` starts with a cancel flag of its own. @@ -131,7 +131,7 @@ export default function ExportScreen() { return; } setBusy(true); - // The export holds the desktop from its fetch to the end of its format + // The export holds the desktop from its export step to the end of its format // step. Each job holds it too, but only while it runs, which would leave // a gap between the two where Settings → Convert could start a job the // desktop then runs instead of the format step (#1407). @@ -151,7 +151,7 @@ export default function ExportScreen() { chosen, async (exportDir) => { const request = { savePath: chosen || exportDir.dir, format }; - const fetchInto = (outDir: string) => + const exportInto = (outDir: string) => run( exportCancel.guard(() => invokeExport({ @@ -167,10 +167,10 @@ export default function ExportScreen() { { onLog: appendLog }, ); if (format === "jsonl") { - await fetchInto(chosen || exportDir.dir); + await exportInto(chosen || exportDir.dir); return; } - await fetchInto(exportDir.exported); + await exportInto(exportDir.exported); await run( exportCancel.guard(() => invokeFormat({ @@ -220,7 +220,7 @@ export default function ExportScreen() { } success={ // `busy` hides it from the moment the next export starts, and between - // the fetch and the conversion, when the fetch alone has finished. + // the export step and the conversion, when the export step alone has finished. finished && !busy && !error ? (
Export complete. {formatLabel(finished.format)} saved to {finished.savePath}. From 6c5703fdba56383a3b58126e46a864d8fa133c62 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:26:47 -0400 Subject: [PATCH 09/42] test(web): the Export job test says Export complete, not Pull Co-Authored-By: Claude Opus 5.5 --- web/src/lib/tauri.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/web/src/lib/tauri.test.ts b/web/src/lib/tauri.test.ts index 641600785..e87ac2bd5 100644 --- a/web/src/lib/tauri.test.ts +++ b/web/src/lib/tauri.test.ts @@ -206,7 +206,7 @@ describe("awaitTauriJob", () => { let seenWhileRunning: string | null = null; const done = awaitTauriJob("Export", async () => { seenWhileRunning = currentDesktopJob(); - queueMicrotask(() => listeners.get("extract:finished")?.({ payload: "Pull complete" })); + queueMicrotask(() => listeners.get("extract:finished")?.({ payload: "Export complete" })); }); await done; expect(seenWhileRunning).toBe("Export"); From 87c5cf08159369deec36b5d2447c272ace10c1e3 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:46:18 -0400 Subject: [PATCH 10/42] refactor(server): one function writes the stored message time form The search day bound, format_utc_timestamp and the tests that build stored times now all call models::utc_timestamp_text, so the form that text comparison relies on is defined once. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 2 +- crates/server/server/src/dedupe/tests.rs | 6 ++--- .../server/server/src/messages_api/tests.rs | 12 ++++------ crates/server/server/src/models.rs | 24 ++++++++++--------- crates/server/server/src/search/value.rs | 2 +- 5 files changed, 23 insertions(+), 23 deletions(-) diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index d749e6bd7..59ae45156 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -660,7 +660,7 @@ fn pick_winner(cands: &[Cand], prio: &HashMap<&str, usize>) -> i64 { /// pass match at whole seconds. /// /// Strict RFC3339 is sufficient: `messages.timestamp` -/// are only ever written by `models::format_utc_timestamp` (chrono's +/// are only ever written by `models::utc_timestamp_text` (chrono's /// `to_rfc3339_opts(SecondsFormat::Millis, true)`), so no lenient spellings reach /// this path. Unparseable input yields `None`. fn parse_rfc3339_utc_secs(ts: &str) -> Option { diff --git a/crates/server/server/src/dedupe/tests.rs b/crates/server/server/src/dedupe/tests.rs index cc212dd8f..7fe715a6a 100644 --- a/crates/server/server/src/dedupe/tests.rs +++ b/crates/server/server/src/dedupe/tests.rs @@ -1442,9 +1442,9 @@ async fn generate_database(conn: &mut SqliteConnection, seed: u64) -> Vec { _ => {} } guid += 1; - let timestamp = chrono::DateTime::from_timestamp(copy_secs, 0) - .unwrap() - .to_rfc3339_opts(chrono::SecondsFormat::Millis, true); + let timestamp = crate::models::utc_timestamp_text( + chrono::DateTime::from_timestamp(copy_secs, 0).unwrap(), + ); let conversation_id = chat.conversations[rng.below(chat.conversations.len())]; let source = GEN_SOURCES[rng.below(GEN_SOURCES.len())]; let id = MessageRow { diff --git a/crates/server/server/src/messages_api/tests.rs b/crates/server/server/src/messages_api/tests.rs index 1b231c393..081cb9e50 100644 --- a/crates/server/server/src/messages_api/tests.rs +++ b/crates/server/server/src/messages_api/tests.rs @@ -1601,18 +1601,16 @@ async fn date_today_is_the_day_on_the_accounts_clock() { let today = chrono::Utc::now().with_timezone(&zone).date_naive(); let local = |day: chrono::NaiveDate, h: u32, m: u32| { - zone.from_local_datetime(&day.and_hms_opt(h, m, 0).unwrap()) + let instant = zone + .from_local_datetime(&day.and_hms_opt(h, m, 0).unwrap()) .single() .unwrap() - .with_timezone(&chrono::Utc) - .format("%Y-%m-%dT%H:%M:%S%.3fZ") - .to_string() + .with_timezone(&chrono::Utc); + crate::models::utc_timestamp_text(instant) }; let early_today = local(today, 0, 30); let late_yesterday = local(today.pred_opt().unwrap(), 23, 30); - let now = chrono::Utc::now() - .format("%Y-%m-%dT%H:%M:%S%.3fZ") - .to_string(); + let now = crate::models::utc_timestamp_text(chrono::Utc::now()); seed_conversation( &fixture.state, &SeedConversation { diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index c81af8212..22661a5bf 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -1,7 +1,7 @@ //! Import-side records mapped from message-ir JSONL. use anyhow::{Context, Result}; -use chrono::{TimeZone, Utc}; +use chrono::{DateTime, TimeZone, Utc}; use message_ir::{ ConversationHeader, Deletion, EarlierVersion, HandleService, HandleType, IrAttachment, IrDirection, IrMessage, IrMessageKind, IrParticipant, Reaction, ReplyTo, @@ -503,17 +503,19 @@ fn tapback_from_reaction(reaction: &Reaction) -> TapbackRecord { } /// The UTC RFC 3339 string for a Unix time in milliseconds, or `None` when it -/// cannot be represented. It always has three fractional digits and a `Z` -/// suffix (`2015-03-12T18:04:22.000Z` for a whole second), so every stored -/// time has one form and sorts as text in time order. The server stores the -/// instant and nothing about where the phone was; the account's time zone -/// turns it into a clock reading. +/// cannot be represented, in the form `utc_timestamp_text` writes. The server +/// stores the instant and nothing about where the phone was; the account's +/// time zone turns it into a clock reading. fn format_utc_timestamp(ms: i64) -> Option { - Some( - Utc.timestamp_millis_opt(ms) - .single()? - .to_rfc3339_opts(chrono::SecondsFormat::Millis, true), - ) + Some(utc_timestamp_text(Utc.timestamp_millis_opt(ms).single()?)) +} + +/// The one text form of a stored message time: UTC RFC 3339 with three +/// fractional digits and a `Z` suffix (`2015-03-12T18:04:22.000Z` for a whole +/// second). Every stored time and every string compared with one, such as a +/// search day bound, is written here, so they all sort as text in time order. +pub(crate) fn utc_timestamp_text(instant: DateTime) -> String { + instant.to_rfc3339_opts(chrono::SecondsFormat::Millis, true) } #[cfg(test)] diff --git a/crates/server/server/src/search/value.rs b/crates/server/server/src/search/value.rs index 686e9ae49..2637923e5 100644 --- a/crates/server/server/src/search/value.rs +++ b/crates/server/server/src/search/value.rs @@ -72,7 +72,7 @@ pub(crate) enum Value { pub(crate) fn utc_instant(zone: chrono_tz::Tz, day: NaiveDate) -> Option { let midnight = day.and_hms_opt(0, 0, 0).expect("midnight is a valid time"); let instant = first_instant_at_or_after(zone, midnight); - (instant.year() <= 9999).then(|| instant.to_rfc3339_opts(chrono::SecondsFormat::Millis, true)) + (instant.year() <= 9999).then(|| crate::models::utc_timestamp_text(instant)) } /// The first instant whose clock reading in `zone` is `local` or later. From 2d1f6386e9cddf801a861a7eb23319a0722624a5 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:47:07 -0400 Subject: [PATCH 11/42] docs(server): name the stored time writer once, and where it lives Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 7 +++---- crates/server/server/src/models.rs | 4 +++- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 59ae45156..3a64ba25b 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -659,10 +659,9 @@ fn pick_winner(cands: &[Cand], prio: &HashMap<&str, usize>) -> i64 { /// offsets and dropping the milliseconds: the content key and the near-time /// pass match at whole seconds. /// -/// Strict RFC3339 is sufficient: `messages.timestamp` -/// are only ever written by `models::utc_timestamp_text` (chrono's -/// `to_rfc3339_opts(SecondsFormat::Millis, true)`), so no lenient spellings reach -/// this path. Unparseable input yields `None`. +/// Strict RFC3339 is sufficient: `messages.timestamp` is only ever written by +/// `models::utc_timestamp_text`, so no lenient spellings reach this path. +/// Unparseable input yields `None`. fn parse_rfc3339_utc_secs(ts: &str) -> Option { chrono::DateTime::parse_from_rfc3339(ts.trim()) .ok() diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index 22661a5bf..3a558d855 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -1,4 +1,6 @@ -//! Import-side records mapped from message-ir JSONL. +//! Import-side records mapped from message-ir JSONL, and the one text form +//! of a stored message time (`utc_timestamp_text`), which search day bounds +//! use too. use anyhow::{Context, Result}; use chrono::{DateTime, TimeZone, Utc}; From d8e0c554932d3b3658b49510b639ce6edca2572d Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:47:09 -0400 Subject: [PATCH 12/42] feat(ir): the conversation file says when its backup was made ExportMeta gains backup_taken_at_unix_ms, and schema_version goes from 10 to 11; a version-10 file is refused by its version. Each exporter fills the date from its source: an iPhone backup's Manifest.plist date for Apple Messages and WhatsApp, a Mac chat.db's modification time, the backup_date attribute of an SMS Backup & Restore file, the newest modification time of the files read for iMazing, OpenExtract, GO SMS Pro, SMS Backup+ and an Android WhatsApp database. The demo seed dates its backups at its reference time. ir-format carries the date in every format it writes and reads: JSON and JSON Lines through serde, CSV in a backup_taken_at_unix_ms column, EML and MBOX in an X-ME-Backup-Taken-At-Unix-Ms header. A value that is not a whole number is refused rather than read as no date. Part of #1924. Co-Authored-By: Claude Opus 5.5 --- Cargo.lock | 1 + crates/core/message-crate-core/Cargo.toml | 7 +- .../src/attachment_jobs/tests.rs | 2 + crates/core/message-crate-core/src/lib.rs | 4 +- .../core/message-crate-core/src/pipeline.rs | 50 ++- .../core/message-crate-core/src/testutil.rs | 41 +++ .../exporters/go-sms-pro-exporter/src/emit.rs | 17 +- .../tests/convert_smoke.rs | 38 +++ crates/exporters/imazing-exporter/src/emit.rs | 11 +- .../imazing-exporter/tests/convert_smoke.rs | 31 ++ .../imessage-ir-exporter/src/convert.rs | 37 ++- .../exporters/imessage-ir-exporter/src/run.rs | 16 + .../imessage-ir-exporter/src/run/tests.rs | 42 +++ .../fixtures/dated-backup/Manifest.plist | 12 + .../tests/helper_process.rs | 24 ++ .../openextract-exporter/src/emit.rs | 10 +- .../tests/convert_smoke.rs | 30 ++ .../sms-backup-plus-exporter/src/emit.rs | 7 + .../tests/convert_smoke.rs | 48 +++ .../sms-backup-restore-exporter/src/read.rs | 18 ++ .../src/read/tests.rs | 73 +++++ .../tests/fixtures/dated_backup.xml | 5 + .../exporters/whatsapp-exporter/src/emit.rs | 5 + crates/exporters/whatsapp-exporter/src/run.rs | 291 ++++++++++++------ .../whatsapp-exporter/tests/convert_smoke.rs | 6 + .../tests/fixtures/ios-backup/Manifest.plist | 12 + crates/libs/ios-backup/src/backup.rs | 39 ++- crates/libs/ios-backup/src/lib.rs | 5 +- .../fixtures/dated-backup/Manifest.plist | 12 + .../ir-format/src/export_transforms/tests.rs | 2 + crates/libs/ir-format/src/lib_tests.rs | 84 +++++ crates/libs/ir-format/src/read_csv.rs | 25 +- crates/libs/ir-format/src/read_mail.rs | 1 + crates/libs/ir-format/src/write.rs | 21 +- crates/libs/ir/src/lib.rs | 9 +- crates/libs/ir/src/projection.rs | 1 + crates/libs/ir/src/schema_version.rs | 27 +- crates/libs/ir/src/testutil.rs | 2 + crates/libs/mail/src/headers.rs | 2 + crates/libs/mail/src/lib.rs | 7 + crates/libs/mail/src/parse.rs | 17 + crates/libs/mail/src/tests.rs | 2 + crates/libs/push/src/project.rs | 3 +- crates/libs/push/tests/push_mock.rs | 1 + crates/libs/sbr/src/read.rs | 15 +- crates/server/demo-seed/src/conversations.rs | 48 ++- src-tauri/src/commands/upload.rs | 1 + 47 files changed, 1023 insertions(+), 139 deletions(-) create mode 100644 crates/exporters/imessage-ir-exporter/tests/fixtures/dated-backup/Manifest.plist create mode 100644 crates/exporters/sms-backup-restore-exporter/tests/fixtures/dated_backup.xml create mode 100644 crates/exporters/whatsapp-exporter/tests/fixtures/ios-backup/Manifest.plist create mode 100644 crates/libs/ios-backup/tests/fixtures/dated-backup/Manifest.plist diff --git a/Cargo.lock b/Cargo.lock index 94388b61c..71b2f2dde 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1920,6 +1920,7 @@ dependencies = [ "message-csv", "message-ir", "obfuscate", + "serde_json", "sha2 0.11.0", "tempfile", "thiserror 2.0.21", diff --git a/crates/core/message-crate-core/Cargo.toml b/crates/core/message-crate-core/Cargo.toml index e69ea83f4..b3096ad55 100644 --- a/crates/core/message-crate-core/Cargo.toml +++ b/crates/core/message-crate-core/Cargo.toml @@ -12,16 +12,19 @@ message-csv = { path = "../../libs/csv", optional = true } message-ir = { path = "../../libs/ir" } media = { path = "../../libs/media" } obfuscate = { path = "../../libs/obfuscate" } +serde_json = { workspace = true, optional = true } sha2 = { workspace = true } tempfile = { workspace = true } thiserror = { workspace = true } [features] -# The CSV reader is only for `testutil`, which reads exports back in tests. -testutil = ["dep:message-csv"] +# The CSV and JSON readers are only for `testutil`, which reads exports back +# in tests. +testutil = ["dep:message-csv", "dep:serde_json"] [dev-dependencies] message-csv = { path = "../../libs/csv" } +serde_json = { workspace = true } media = { path = "../../libs/media", features = ["testutil"] } [lints] diff --git a/crates/core/message-crate-core/src/attachment_jobs/tests.rs b/crates/core/message-crate-core/src/attachment_jobs/tests.rs index c10625fd6..67b69de56 100644 --- a/crates/core/message-crate-core/src/attachment_jobs/tests.rs +++ b/crates/core/message-crate-core/src/attachment_jobs/tests.rs @@ -468,6 +468,7 @@ fn staging_a_conversation_writes_the_files_counts_them_and_frees_the_bytes() { tool_version: "0.1.0".into(), owner_identity: Some("+15555550100".into()), owner_display_name: None, + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), @@ -710,6 +711,7 @@ fn staging_frees_the_bytes_the_documents_were_carrying() { tool_version: "0.1.0".into(), owner_identity: None, owner_display_name: None, + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), diff --git a/crates/core/message-crate-core/src/lib.rs b/crates/core/message-crate-core/src/lib.rs index 1f983bd4d..7ba72e40a 100644 --- a/crates/core/message-crate-core/src/lib.rs +++ b/crates/core/message-crate-core/src/lib.rs @@ -40,8 +40,8 @@ pub use exporters::{ }; pub use pipeline::{ CSV_NOT_READ, ExportReport, IssueSink, NAME_ONLY_CHAT_NOTE, NOTE, RunIssue, RunResult, - discover_files, emit_issue, export_meta, prepare_outputs, project_conversation, - unreadable_parts_note, + discover_files, emit_issue, export_meta, file_modified_unix_ms, newest_file_modified_unix_ms, + prepare_outputs, project_conversation, unreadable_parts_note, }; pub use process::{ CancelFlag, Cancelled, LogSink, check_cancel, emit_log, is_cancelled, parallel_for_each, diff --git a/crates/core/message-crate-core/src/pipeline.rs b/crates/core/message-crate-core/src/pipeline.rs index 38b25bce9..6dbc362f8 100644 --- a/crates/core/message-crate-core/src/pipeline.rs +++ b/crates/core/message-crate-core/src/pipeline.rs @@ -462,13 +462,15 @@ pub fn prepare_outputs( Ok((resolved, output)) } -/// Standard export metadata: source / tool / version plus the owner identity. +/// Standard export metadata: source / tool / version, the owner identity, +/// and when the backup was made. pub fn export_meta( source: &str, tool: &str, tool_version: &str, owner_identity: Option, owner_display_name: Option, + backup_taken_at_unix_ms: Option, ) -> message_ir::ExportMeta { message_ir::ExportMeta { source: source.to_string(), @@ -476,13 +478,59 @@ pub fn export_meta( tool_version: tool_version.to_string(), owner_identity, owner_display_name, + backup_taken_at_unix_ms, } } +/// When a file was last changed, in Unix milliseconds, or `None` when it +/// cannot be read or its time is before 1970. The backup date of a source +/// whose files record none of their own: the file was written when the +/// backup was made, or copied with its time kept. +pub fn file_modified_unix_ms(path: &std::path::Path) -> Option { + let modified = fs::metadata(path).ok()?.modified().ok()?; + let since = modified.duration_since(std::time::UNIX_EPOCH).ok()?; + i64::try_from(since.as_millis()).ok() +} + +/// The latest [`file_modified_unix_ms`] of `paths`, or `None` when none has +/// one: the backup date of a source read from several files, which is as +/// new as the newest of them. +pub fn newest_file_modified_unix_ms<'a>( + paths: impl IntoIterator, +) -> Option { + paths.into_iter().filter_map(file_modified_unix_ms).max() +} + #[cfg(test)] mod tests { use super::*; + /// A source with no date of its own is as new as its newest file, and + /// a file that is not there adds nothing. + #[test] + fn a_backup_read_from_files_is_as_new_as_the_newest() { + use crate::testutil::{TEST_BACKUP_TAKEN_AT_UNIX_MS, set_modified_unix_ms}; + let dir = tempfile::tempdir().unwrap(); + let older = dir.path().join("older.csv"); + let newer = dir.path().join("newer.csv"); + fs::write(&older, "a").unwrap(); + fs::write(&newer, "b").unwrap(); + set_modified_unix_ms(&older, TEST_BACKUP_TAKEN_AT_UNIX_MS - 86_400_000); + set_modified_unix_ms(&newer, TEST_BACKUP_TAKEN_AT_UNIX_MS); + + assert_eq!( + file_modified_unix_ms(&older), + Some(TEST_BACKUP_TAKEN_AT_UNIX_MS - 86_400_000) + ); + let missing = dir.path().join("missing.csv"); + assert_eq!(file_modified_unix_ms(&missing), None); + assert_eq!( + newest_file_modified_unix_ms([older.as_path(), newer.as_path(), missing.as_path()]), + Some(TEST_BACKUP_TAKEN_AT_UNIX_MS) + ); + assert_eq!(newest_file_modified_unix_ms([missing.as_path()]), None); + } + #[test] fn the_not_sms_or_mms_line_names_the_format_and_the_count() { let mut report = ExportReport::default(); diff --git a/crates/core/message-crate-core/src/testutil.rs b/crates/core/message-crate-core/src/testutil.rs index 65d9fb0ae..bdcc23038 100644 --- a/crates/core/message-crate-core/src/testutil.rs +++ b/crates/core/message-crate-core/src/testutil.rs @@ -370,3 +370,44 @@ pub fn assert_run_wrote_jsonl( .collect::>() .join("\n") } + +/// The backup date the exporters' tests give a source: 2026-09-30T18:45:12Z, +/// in Unix milliseconds. A date of the test's own, so a test that reads it +/// back cannot pass by reading the time it ran at. +pub const TEST_BACKUP_TAKEN_AT_UNIX_MS: i64 = 1_790_793_912_000; + +/// Set `path`'s modification time to `unix_ms`, the way a backup file keeps +/// the time it was written. +/// +/// # Panics +/// +/// Panics when the file cannot be opened or its time cannot be set. +pub fn set_modified_unix_ms(path: &Path, unix_ms: i64) { + let at = std::time::UNIX_EPOCH + + std::time::Duration::from_millis(u64::try_from(unix_ms).expect("a time after 1970")); + fs::File::options() + .write(true) + .open(path) + .expect("open the file to date") + .set_modified(at) + .expect("set the file's modification time"); +} + +/// Every JSON Lines file's `export.backup_taken_at_unix_ms` under `dir`, in +/// file name order. +/// +/// # Panics +/// +/// Panics when a file cannot be read or its first line is not JSON. +pub fn jsonl_backup_dates(dir: &Path) -> Vec> { + jsonl_names(dir) + .iter() + .map(|name| { + let text = fs::read_to_string(dir.join(name)).expect("read jsonl"); + let header: serde_json::Value = + serde_json::from_str(text.lines().next().expect("a header line")) + .expect("the header is JSON"); + header["export"]["backup_taken_at_unix_ms"].as_i64() + }) + .collect() +} diff --git a/crates/exporters/go-sms-pro-exporter/src/emit.rs b/crates/exporters/go-sms-pro-exporter/src/emit.rs index ea2c6cc59..3e0b03c76 100644 --- a/crates/exporters/go-sms-pro-exporter/src/emit.rs +++ b/crates/exporters/go-sms-pro-exporter/src/emit.rs @@ -455,14 +455,21 @@ pub(crate) fn convert_export(args: ConvertExportArgs<'_>) -> Result) -> Result) -> Result) -> Result (record.sender_identity, record.sender_display_name), }; @@ -676,6 +677,7 @@ fn write_conversations( options.emit_log(""); options.emit_log(message_crate_core::CONVERSATION_FILES_PREPARING.line(total as u64)); options.emit_progress(ProgressEvent::Prepare { done: 0, total }); + let backup_taken_at_unix_ms = options.backup_taken_at_unix_ms(); let mut written = 0usize; let mut kept = 0u64; for (chat_identifier, convo) in conversations { @@ -685,7 +687,12 @@ fn write_conversations( continue; } kept += 1; - let doc = pending_to_document(chat_identifier, convo, options.use_caller_id); + let doc = pending_to_document( + chat_identifier, + convo, + options.use_caller_id, + backup_taken_at_unix_ms, + ); let document_id = doc.conversation.chat_identifier.clone(); sink.write_document(doc) .map_err(|e| anyhow!("write {} for {}: {e:#}", format.as_str(), document_id))?; @@ -700,11 +707,14 @@ fn write_conversations( Ok(kept) } -/// Project one accumulated conversation into the shared document shape. +/// Project one accumulated conversation into the shared document shape, +/// read from a backup made at `backup_taken_at_unix_ms` +/// ([`ExportOptions::backup_taken_at_unix_ms`]). fn pending_to_document( chat_identifier: String, convo: PendingConversation, use_caller_id: bool, + backup_taken_at_unix_ms: Option, ) -> ConversationDocument { let export = ExportMeta { source: EXPORT_SOURCE.into(), @@ -714,6 +724,7 @@ fn pending_to_document( owner_display_name: convo .owner_display_name .or_else(|| use_caller_id.then(|| "Me".to_string())), + backup_taken_at_unix_ms, }; // Each message keeps the address it was sent from; the conversation's // owner fills in only where the database recorded none. @@ -749,9 +760,15 @@ fn pending_to_unit( chat_identifier: String, mut convo: PendingConversation, use_caller_id: bool, + backup_taken_at_unix_ms: Option, ) -> ConversationUnit { let loads = std::mem::take(&mut convo.attachment_loads); - let doc = pending_to_document(chat_identifier, convo, use_caller_id); + let doc = pending_to_document( + chat_identifier, + convo, + use_caller_id, + backup_taken_at_unix_ms, + ); let mut loads = loads.into_iter(); ConversationUnit::from_doc(doc, |_, att| attachment_source(loads.next(), att)) } @@ -775,11 +792,19 @@ fn drain_conversations( not_decrypted: &mut NotDecrypted, ) -> Result { let use_caller_id = options.use_caller_id; + let backup_taken_at_unix_ms = options.backup_taken_at_unix_ms(); let units: Vec = collected .conversations .into_iter() .filter(|(_, convo)| !convo.messages.is_empty()) - .map(|(chat_identifier, convo)| pending_to_unit(chat_identifier, convo, use_caller_id)) + .map(|(chat_identifier, convo)| { + pending_to_unit( + chat_identifier, + convo, + use_caller_id, + backup_taken_at_unix_ms, + ) + }) .collect(); let queue = WriteQueueOptions { @@ -1212,7 +1237,7 @@ mod tests { attachment_loads: Vec::new(), }; - let doc = pending_to_document("+15555550122".into(), convo, false); + let doc = pending_to_document("+15555550122".into(), convo, false, None); let senders: Vec<_> = doc .messages @@ -1343,7 +1368,7 @@ mod tests { ], }; - let unit = pending_to_unit("+15555550101".into(), convo, false); + let unit = pending_to_unit("+15555550101".into(), convo, false, None); assert_eq!(unit.attachments.len(), 2); assert_eq!(unit.attachments[0].message_index, 0); diff --git a/crates/exporters/imessage-ir-exporter/src/run.rs b/crates/exporters/imessage-ir-exporter/src/run.rs index 2cbb26750..6972d1817 100644 --- a/crates/exporters/imessage-ir-exporter/src/run.rs +++ b/crates/exporters/imessage-ir-exporter/src/run.rs @@ -134,6 +134,22 @@ impl ExportOptions { pub fn check_cancel(&self) -> Result<()> { message_crate_core::check_cancel(self.cancel.as_ref()).map_err(|e| anyhow!(e)) } + + /// When the Messages data was backed up, in Unix milliseconds: an + /// iPhone backup's `Manifest.plist` date, or a Mac `chat.db`'s + /// modification time, the last time Messages wrote it. `None` when + /// neither can be read. + pub fn backup_taken_at_unix_ms(&self) -> Option { + backup_taken_at_unix_ms(&self.source) + } +} + +/// [`ExportOptions::backup_taken_at_unix_ms`] for `source`. +pub(crate) fn backup_taken_at_unix_ms(source: &Source) -> Option { + match source.platform { + Platform::Ios => ios_backup::ios_backup_date_unix_ms(&source.db_path), + Platform::MacOs => message_crate_core::file_modified_unix_ms(&source.db_path), + } } /// Build options from [`ExporterConfig`], start the `imessage-reader` diff --git a/crates/exporters/imessage-ir-exporter/src/run/tests.rs b/crates/exporters/imessage-ir-exporter/src/run/tests.rs index 7a82ab4b5..ba076f3b8 100644 --- a/crates/exporters/imessage-ir-exporter/src/run/tests.rs +++ b/crates/exporters/imessage-ir-exporter/src/run/tests.rs @@ -959,3 +959,45 @@ fn every_format_refuses_attachments_the_staging_disk_cannot_hold() { ); } } + +/// An iPhone backup is dated by its `Manifest.plist`, which is readable +/// whether or not the backup is encrypted. +#[test] +fn an_iphone_backup_is_dated_by_its_manifest() { + let backup = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/dated-backup"); + let source = Source { + db_path: backup, + platform: Platform::Ios, + backup_password: None, + }; + // 2026-09-30T18:45:12Z + assert_eq!(backup_taken_at_unix_ms(&source), Some(1_790_793_912_000)); +} + +/// A Mac's `chat.db` records no backup date, so it is dated by when +/// Messages last wrote it. +#[test] +fn a_mac_chat_db_is_dated_by_its_modification_time() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("chat.db"); + fs::write(&db, b"not read here").unwrap(); + message_crate_core::testutil::set_modified_unix_ms( + &db, + message_crate_core::testutil::TEST_BACKUP_TAKEN_AT_UNIX_MS, + ); + let source = Source { + db_path: db, + platform: Platform::MacOs, + backup_password: None, + }; + assert_eq!( + backup_taken_at_unix_ms(&source), + Some(message_crate_core::testutil::TEST_BACKUP_TAKEN_AT_UNIX_MS) + ); + let undated = Source { + db_path: dir.path().join("missing.db"), + platform: Platform::Ios, + backup_password: None, + }; + assert_eq!(backup_taken_at_unix_ms(&undated), None); +} diff --git a/crates/exporters/imessage-ir-exporter/tests/fixtures/dated-backup/Manifest.plist b/crates/exporters/imessage-ir-exporter/tests/fixtures/dated-backup/Manifest.plist new file mode 100644 index 000000000..64224a8e8 --- /dev/null +++ b/crates/exporters/imessage-ir-exporter/tests/fixtures/dated-backup/Manifest.plist @@ -0,0 +1,12 @@ + + + + + Date + 2026-09-30T18:45:12Z + IsEncrypted + + Version + 10.0 + + diff --git a/crates/exporters/imessage-ir-exporter/tests/helper_process.rs b/crates/exporters/imessage-ir-exporter/tests/helper_process.rs index 2a929e485..46dc3942c 100644 --- a/crates/exporters/imessage-ir-exporter/tests/helper_process.rs +++ b/crates/exporters/imessage-ir-exporter/tests/helper_process.rs @@ -84,6 +84,30 @@ fn exports_a_mac_chat_db_through_the_helper_process() { assert!(all.contains("\"attachments/"), "{all}"); } +/// Every conversation file of a Mac `chat.db` export says the database's +/// modification time as when the backup was made. +#[test] +fn every_conversation_file_says_when_the_chat_db_was_last_written() { + use message_crate_core::testutil::{ + TEST_BACKUP_TAKEN_AT_UNIX_MS, jsonl_backup_dates, set_modified_unix_ms, + }; + helper_binary(); + let dir = tempfile::tempdir().unwrap(); + let db_path = write_chat_db(dir.path()); + set_modified_unix_ms(&db_path, TEST_BACKUP_TAKEN_AT_UNIX_MS); + let output = dir.path().join("out"); + + imessage_ir_exporter::run(&config(&db_path, &output, None)).unwrap(); + let dates = jsonl_backup_dates(&output); + assert_eq!(dates.len(), 7, "{dates:?}"); + assert!( + dates + .iter() + .all(|d| *d == Some(TEST_BACKUP_TAKEN_AT_UNIX_MS)), + "{dates:?}" + ); +} + /// Every file below `dir`, recursively. fn walk(dir: &Path) -> Vec { let mut out = Vec::new(); diff --git a/crates/exporters/openextract-exporter/src/emit.rs b/crates/exporters/openextract-exporter/src/emit.rs index b6274bf08..60f716805 100644 --- a/crates/exporters/openextract-exporter/src/emit.rs +++ b/crates/exporters/openextract-exporter/src/emit.rs @@ -18,7 +18,7 @@ use phone::Handle; use serde_json::{Map, json}; use sha2::{Digest, Sha256}; use std::collections::{BTreeMap, HashMap, HashSet}; -use std::path::Path; +use std::path::{Path, PathBuf}; const EXPORT_SOURCE: &str = "openextract"; const EXPORT_TOOL: &str = "OpenExtract"; @@ -69,9 +69,10 @@ pub(crate) fn convert_export(args: ConvertExportArgs<'_>) -> Result) -> Result>( .. } = ingest; + // SMS Backup+ writes no backup date of its own, so the backup is as new + // as the newest mail file read. let hooks = SbpProjection { export: message_crate_core::export_meta( EXPORT_SOURCE, @@ -528,6 +530,9 @@ pub(crate) fn convert_export>( EXPORT_TOOL_VERSION, Some(owner_identity), None, + message_crate_core::newest_file_modified_unix_ms( + eml_paths.iter().map(PathBuf::as_path), + ), ), }; let mut documents = Vec::new(); @@ -832,6 +837,7 @@ mod tests { tool_version: String::new(), owner_identity: None, owner_display_name: None, + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "test".into(), @@ -905,6 +911,7 @@ mod tests { EXPORT_TOOL_VERSION, Some("+15555550100".into()), None, + None, ), }; let mut report = ingest.report; diff --git a/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs b/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs index 8a07f13c8..bb7a7a7f7 100644 --- a/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs +++ b/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs @@ -386,3 +386,51 @@ fn a_run_that_copies_no_attachments_still_records_their_size() { assert_eq!(picture.path, None, "nothing was copied"); assert_eq!(picture.size_bytes, Some(22)); } + +/// SMS Backup+ records no backup date, so the conversation file says the +/// newest modification time of the mail files read. +#[test] +fn the_backup_date_is_the_newest_mail_files_modification_time() { + use message_crate_core::testutil::{ + TEST_BACKUP_TAKEN_AT_UNIX_MS, jsonl_backup_dates, set_modified_unix_ms, + }; + let tmp = tempfile::tempdir().expect("tempdir"); + let input = tmp.path().join("in"); + fs::create_dir_all(&input).unwrap(); + for (name, modified) in [ + ("flat_received.eml", TEST_BACKUP_TAKEN_AT_UNIX_MS), + ( + "flat_smssync_276_sam.eml", + TEST_BACKUP_TAKEN_AT_UNIX_MS - 60_000, + ), + ] { + let path = input.join(name); + fs::copy(fixtures().join(name), &path).unwrap(); + set_modified_unix_ms(&path, modified); + } + let output = tmp.path().join("out"); + let cache = tempfile::tempdir().unwrap(); + convert_export(ConvertExportArgs { + inputs: &[input.as_path()], + output_dir: &output, + scratch_dir: cache.path(), + owner_phones: &["+15555550100".into()], + owner_emails: &["owner@example.com".into()], + verbose: false, + transforms: ExportTransforms::none(), + output_format: OutputFormat::Jsonl, + cancel: None, + log: None, + issues: None, + resume: false, + }) + .expect("convert"); + let dates = jsonl_backup_dates(&output); + assert!(!dates.is_empty()); + assert!( + dates + .iter() + .all(|d| *d == Some(TEST_BACKUP_TAKEN_AT_UNIX_MS)), + "{dates:?}" + ); +} diff --git a/crates/exporters/sms-backup-restore-exporter/src/read.rs b/crates/exporters/sms-backup-restore-exporter/src/read.rs index f3d647a34..a40d48bee 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/read.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/read.rs @@ -528,6 +528,7 @@ fn to_document( id: &str, conversation: &PendingConversation, owner_identity: Option<&str>, + backup_taken_at_unix_ms: Option, report: &mut ReadReport, ) -> ConversationDocument { let export = ExportMeta { @@ -536,6 +537,7 @@ fn to_document( tool_version: EXPORT_TOOL_VERSION.into(), owner_identity: owner_identity.map(str::to_string), owner_display_name: None, + backup_taken_at_unix_ms, }; let owner = owner_sender(&export); let messages = conversation @@ -740,6 +742,9 @@ pub fn read_backup( .and_then(OwnerHandleSet::primary_owner_handle); let mut report = ReadReport::default(); let mut conversations = BTreeMap::new(); + // When each file's backup was made: its `backup_date`, or the file's + // modification time when it has none. + let mut backup_dates: HashMap, i64> = HashMap::new(); for path in paths { check_cancel(options.cancel)?; let file: Arc = path.display().to_string().into(); @@ -772,6 +777,12 @@ pub fn read_backup( } }); merge_stats(&mut report, stats); + if let Some(date) = stats + .backup_date_unix_ms + .or_else(|| message_crate_core::file_modified_unix_ms(&path)) + { + backup_dates.insert(file.clone(), date); + } if let Some(error) = spool_error { return Err(error); } @@ -796,10 +807,17 @@ pub fn read_backup( if conversation.messages.is_empty() { continue; } + // A conversation read from two backups is as new as the newer one. + let backup_taken_at_unix_ms = conversation + .messages + .iter() + .filter_map(|message| backup_dates.get(&message.file).copied()) + .max(); documents.push(to_document( &id, &conversation, owner_identity.as_deref(), + backup_taken_at_unix_ms, &mut report, )); report.conversations += 1; diff --git a/crates/exporters/sms-backup-restore-exporter/src/read/tests.rs b/crates/exporters/sms-backup-restore-exporter/src/read/tests.rs index 2a14e7438..aea1ea767 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/read/tests.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/read/tests.rs @@ -556,3 +556,76 @@ fn a_message_kept_with_something_left_out_is_named() { assert_eq!(report.skipped_unreadable_part(), 1); assert_eq!(report.dropped_character_references(), 2); } + +/// The fixture's root says when SMS Backup & Restore made the backup. +fn dated_fixture() -> std::path::PathBuf { + std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/dated_backup.xml") +} + +/// The backup date of each conversation read from `input`, by chat id. +fn backup_dates(input: &Path) -> Vec<(String, Option)> { + let owners = ["+15555550100".to_string()]; + let (docs, _) = read_backup(input, opts(&owners, None, None)).unwrap(); + docs.into_iter() + .map(|doc| { + ( + doc.conversation.chat_identifier, + doc.export.backup_taken_at_unix_ms, + ) + }) + .collect() +} + +/// The root element's `backup_date` decides, not when the file was copied. +#[test] +fn the_backup_date_is_the_files_backup_date_attribute() { + let dir = tempfile::tempdir().unwrap(); + let input = dir.path().join("dated_backup.xml"); + fs::copy(dated_fixture(), &input).unwrap(); + message_crate_core::testutil::set_modified_unix_ms(&input, 1_000_000_000_000); + assert_eq!( + backup_dates(&input), + [ + ("+15555550101".to_string(), Some(1_790_793_912_000)), + ("+15555550102".to_string(), Some(1_790_793_912_000)), + ] + ); +} + +/// A file without `backup_date` is dated by when it was last changed. +#[test] +fn a_file_without_a_backup_date_is_dated_by_its_modification_time() { + let dir = tempfile::tempdir().unwrap(); + let input = dir.path().join("undated.xml"); + fs::write( + &input, + r#""#, + ) + .unwrap(); + message_crate_core::testutil::set_modified_unix_ms(&input, 1_500_000_000_000); + assert_eq!( + backup_dates(&input), + [("+15555550101".to_string(), Some(1_500_000_000_000))] + ); +} + +/// Two backups in one directory: a conversation both hold is as new as the +/// newer, and one only the older holds keeps the older's date. +#[test] +fn a_conversation_in_two_backups_is_as_new_as_the_newer() { + let dir = tempfile::tempdir().unwrap(); + fs::copy(dated_fixture(), dir.path().join("newer.xml")).unwrap(); + fs::write( + dir.path().join("older.xml"), + r#""#, + ) + .unwrap(); + assert_eq!( + backup_dates(dir.path()), + [ + ("+15555550101".to_string(), Some(1_790_793_912_000)), + ("+15555550102".to_string(), Some(1_790_793_912_000)), + ("+15555550103".to_string(), Some(1_780_000_000_000)), + ] + ); +} diff --git a/crates/exporters/sms-backup-restore-exporter/tests/fixtures/dated_backup.xml b/crates/exporters/sms-backup-restore-exporter/tests/fixtures/dated_backup.xml new file mode 100644 index 000000000..405005b23 --- /dev/null +++ b/crates/exporters/sms-backup-restore-exporter/tests/fixtures/dated_backup.xml @@ -0,0 +1,5 @@ + + + + + diff --git a/crates/exporters/whatsapp-exporter/src/emit.rs b/crates/exporters/whatsapp-exporter/src/emit.rs index 5003f2871..a6725b6d7 100644 --- a/crates/exporters/whatsapp-exporter/src/emit.rs +++ b/crates/exporters/whatsapp-exporter/src/emit.rs @@ -42,6 +42,9 @@ pub(crate) struct ConvertRequest<'a> { /// so on every message as the address it was held at. `None` records no /// owner, which leaves the conversations counted toward no identity. pub owner_identity: Option, + /// When the backup was made, in Unix milliseconds, stamped on the export + /// header ([`backup_taken_at_unix_ms`](crate::run::backup_taken_at_unix_ms)). + pub backup_taken_at_unix_ms: Option, pub output_format: OutputFormat, /// Checked between chats (cooperative cancellation). pub cancel: Option<&'a CancelFlag>, @@ -64,6 +67,7 @@ pub(crate) fn convert_json(request: ConvertRequest<'_>) -> Result transforms, media_search_roots, owner_identity, + backup_taken_at_unix_ms, output_format, cancel, resume, @@ -107,6 +111,7 @@ pub(crate) fn convert_json(request: ConvertRequest<'_>) -> Result EXPORT_TOOL_VERSION, owner_identity, None, + backup_taken_at_unix_ms, ), }; let mut documents = Vec::new(); diff --git a/crates/exporters/whatsapp-exporter/src/run.rs b/crates/exporters/whatsapp-exporter/src/run.rs index e89acf4d2..a57bfa46c 100644 --- a/crates/exporters/whatsapp-exporter/src/run.rs +++ b/crates/exporters/whatsapp-exporter/src/run.rs @@ -5,7 +5,8 @@ use crate::emit::{ConvertRequest, convert_json}; use crate::ios_backup::{decrypt_if_encrypted, extract_bytes}; use crate::owner::{owner_from_backup, owner_from_form}; use crate::wtsexporter::{ - Platform, WtsexporterArgs, extracts_ios_backup, resolve_wtsexporter, run_wtsexporter, + Platform, WtsexporterArgs, android_crypt_backup, extracts_ios_backup, resolve_wtsexporter, + run_wtsexporter, }; use anyhow::{Context, Result, bail}; use message_crate_core::{ @@ -58,104 +59,121 @@ pub fn run(config: &ExporterConfig) -> Result { .map(owner_from_form) .transpose()?; - let (json_path, media_roots, owner_identity, _work_keep_alive) = if let Some(json) = - &source.json - { - // Allowed roots are only the backup input and the JSON parent — never - // the process CWD, which would let crafted paths copy arbitrary files. - let mut media_roots = Vec::new(); - if let Some(path) = &input { - media_roots.push(path.clone()); - } - if let Some(parent) = json.parent() { - media_roots.push(parent.to_path_buf()); - } - media_roots.sort(); - media_roots.dedup(); - // A ready-made result.json names no owner; the form's number is all - // there is, and a conversion may leave it empty. - (json.clone(), media_roots, form_owner, None) - } else { - let platform = - platform.ok_or_else(|| anyhow::anyhow!("platform is required unless json is set"))?; - let input = match input { - Some(path) => path, - None => env::current_dir().context("resolve current working directory")?, - }; + let (json_path, media_roots, owner_identity, backup_taken_at_unix_ms, _work_keep_alive) = + if let Some(json) = &source.json { + // Allowed roots are only the backup input and the JSON parent — never + // the process CWD, which would let crafted paths copy arbitrary files. + let mut media_roots = Vec::new(); + if let Some(path) = &input { + media_roots.push(path.clone()); + } + if let Some(parent) = json.parent() { + media_roots.push(parent.to_path_buf()); + } + media_roots.sort(); + media_roots.dedup(); + // A ready-made result.json names no owner; the form's number is all + // there is, and a conversion may leave it empty. It names no backup + // date either, so it is dated by when it was written. + ( + json.clone(), + media_roots, + form_owner, + message_crate_core::file_modified_unix_ms(json), + None, + ) + } else { + let platform = platform + .ok_or_else(|| anyhow::anyhow!("platform is required unless json is set"))?; + let input = match input { + Some(path) => path, + None => env::current_dir().context("resolve current working directory")?, + }; - message_crate_core::check_cancel(config.cancel.as_ref())?; - let bin = resolve_wtsexporter()?; - let work = mark_output_and_make_work_directory(config)?; - let json_out = work.path().join("result.json"); + message_crate_core::check_cancel(config.cancel.as_ref())?; + let bin = resolve_wtsexporter()?; + let work = mark_output_and_make_work_directory(config)?; + let json_out = work.path().join("result.json"); - // Cooperative only: cancel is checked before and after the external process. - // Killing wtsexporter mid-run is not implemented. - message_crate_core::check_cancel(config.cancel.as_ref())?; - let mut args = WtsexporterArgs { - platform, - input: input.clone(), - work_dir: work.path().to_path_buf(), - key: source.key.clone(), - backup: source.backup.clone(), - wa: source.wa.clone(), - media: source.media.clone(), - db: source.db.clone(), - business: source.business, - }; - // wtsexporter cannot be given an iPhone backup password, so an - // encrypted backup's WhatsApp files are decrypted into the work - // directory first and wtsexporter reads those instead of the backup. - // From a backup that is not encrypted, wtsexporter extracts them into - // the work directory itself, so that disk is checked for room first. - if platform == Platform::Ios { - if let Some(decrypted) = decrypt_if_encrypted(source, work.path(), config)? { - args.read_decrypted(decrypted); - } else if extracts_ios_backup(&args)? - && let Some(bytes) = extract_bytes(source)? - { - check_headroom(work.path(), bytes, Disk::Scratch)?; + // Cooperative only: cancel is checked before and after the external process. + // Killing wtsexporter mid-run is not implemented. + message_crate_core::check_cancel(config.cancel.as_ref())?; + let mut args = WtsexporterArgs { + platform, + input: input.clone(), + work_dir: work.path().to_path_buf(), + key: source.key.clone(), + backup: source.backup.clone(), + wa: source.wa.clone(), + media: source.media.clone(), + db: source.db.clone(), + business: source.business, + }; + // Read before an encrypted backup's files are decrypted, which points + // `args` at the decrypted copy, dated the moment it was made. + let backup_taken_at_unix_ms = backup_taken_at_unix_ms(&args); + // wtsexporter cannot be given an iPhone backup password, so an + // encrypted backup's WhatsApp files are decrypted into the work + // directory first and wtsexporter reads those instead of the backup. + // From a backup that is not encrypted, wtsexporter extracts them into + // the work directory itself, so that disk is checked for room first. + if platform == Platform::Ios { + if let Some(decrypted) = decrypt_if_encrypted(source, work.path(), config)? { + args.read_decrypted(decrypted); + } else if extracts_ios_backup(&args)? + && let Some(bytes) = extract_bytes(source)? + { + check_headroom(work.path(), bytes, Disk::Scratch)?; + } } - } - message_crate_core::check_cancel(config.cancel.as_ref())?; - let log = run_wtsexporter(&bin, &args, &json_out)?; - message_crate_core::check_cancel(config.cancel.as_ref())?; + message_crate_core::check_cancel(config.cancel.as_ref())?; + let log = run_wtsexporter(&bin, &args, &json_out)?; + message_crate_core::check_cancel(config.cancel.as_ref())?; - if !log.trim().is_empty() { - let trimmed = log.trim_end_matches('\n'); - messages.push(trimmed.to_string()); - } + if !log.trim().is_empty() { + let trimmed = log.trim_end_matches('\n'); + messages.push(trimmed.to_string()); + } - let kept = config.output.join("wtsexporter_result.json"); - fs::copy(&json_out, &kept).with_context(|| format!("copy JSON to {}", kept.display()))?; - - // Work directory (wtsexporter extract) + backup input. The backup input is the - // process cwd when the config names no input. - let mut media_roots = vec![work.path().to_path_buf(), input]; - media_roots.sort(); - media_roots.dedup(); - - let owner_identity = match platform { - // wtsexporter copies the whole app-group domain into the work - // dir, preferences plist included; a backup someone extracted by - // hand has it under the input directory. The form's number covers a - // backup that carries no owner key (the Business app, a moved key). - Platform::Ios => match owner_from_backup(&media_roots, &mut messages).or(form_owner) { - Some(owner) => owner, - None => bail!( - "the backup does not contain your WhatsApp phone number; \ + let kept = config.output.join("wtsexporter_result.json"); + fs::copy(&json_out, &kept) + .with_context(|| format!("copy JSON to {}", kept.display()))?; + + // Work directory (wtsexporter extract) + backup input. The backup input is the + // process cwd when the config names no input. + let mut media_roots = vec![work.path().to_path_buf(), input]; + media_roots.sort(); + media_roots.dedup(); + + let owner_identity = match platform { + // wtsexporter copies the whole app-group domain into the work + // dir, preferences plist included; a backup someone extracted by + // hand has it under the input directory. The form's number covers a + // backup that carries no owner key (the Business app, a moved key). + Platform::Ios => { + match owner_from_backup(&media_roots, &mut messages).or(form_owner) { + Some(owner) => owner, + None => bail!( + "the backup does not contain your WhatsApp phone number; \ enter it on the import form" - ), - }, - // A crypt backup carries no owner; the form checks the field is - // filled before the run starts, so this only guards a caller - // that skipped the form. - Platform::Android => { - form_owner.ok_or_else(|| anyhow::anyhow!("Owner's WhatsApp number is required."))? - } - }; + ), + } + } + // A crypt backup carries no owner; the form checks the field is + // filled before the run starts, so this only guards a caller + // that skipped the form. + Platform::Android => form_owner + .ok_or_else(|| anyhow::anyhow!("Owner's WhatsApp number is required."))?, + }; - (kept, media_roots, Some(owner_identity), Some(work)) - }; + ( + kept, + media_roots, + Some(owner_identity), + backup_taken_at_unix_ms, + Some(work), + ) + }; if !json_path.is_file() { bail!("JSON not found: {}", json_path.display()); @@ -170,6 +188,7 @@ pub fn run(config: &ExporterConfig) -> Result { transforms, media_search_roots: &media_roots, owner_identity, + backup_taken_at_unix_ms, output_format: config.output_format, cancel: config.cancel.as_ref(), resume: config.resume, @@ -184,6 +203,34 @@ pub fn run(config: &ExporterConfig) -> Result { Ok(result) } +/// When the backup wtsexporter reads was made, in Unix milliseconds: an +/// iPhone backup's `Manifest.plist` date, or for Android the modification +/// time of the WhatsApp database file (`msgstore.db.crypt15` or a decrypted +/// `msgstore.db`), which WhatsApp writes when it backs up. `None` when +/// neither can be read, such as an iPhone backup extracted by hand without +/// its manifest. +pub(crate) fn backup_taken_at_unix_ms(args: &WtsexporterArgs) -> Option { + match args.platform { + Platform::Ios => ::ios_backup::ios_backup_date_unix_ms(&args.input), + Platform::Android => { + let database = args + .backup + .clone() + .or_else(|| args.db.clone()) + .or_else(|| android_crypt_backup(&args.input)) + .unwrap_or_else(|| { + let decrypted = args.input.join("msgstore.db"); + if decrypted.is_file() { + decrypted + } else { + args.input.clone() + } + }); + message_crate_core::file_modified_unix_ms(&database) + } + } +} + /// Mark the output directory as an export directory, then make the work /// directory wtsexporter runs in (its working directory, the extract, and /// `result.json`) under the Scratch Directory's [`WHATSAPP_DIRECTORY`]. The @@ -216,6 +263,7 @@ fn mark_output_and_make_work_directory(config: &ExporterConfig) -> Result WtsexporterArgs { + WtsexporterArgs { + platform, + input: input.to_path_buf(), + work_dir: input.to_path_buf(), + key: None, + backup: None, + wa: None, + media: None, + db: None, + business: false, + } + } + + /// An iPhone backup is dated by its `Manifest.plist`. + #[test] + fn an_iphone_backup_is_dated_by_its_manifest() { + let backup = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/ios-backup"); + let args = args_for(crate::wtsexporter::Platform::Ios, &backup); + // 2026-09-30T18:45:12Z + assert_eq!( + super::backup_taken_at_unix_ms(&args), + Some(1_790_793_912_000) + ); + } + + /// An Android backup records no date inside it, so it is dated by when + /// WhatsApp wrote its database file; an iPhone backup with no manifest + /// has no date. + #[test] + fn an_android_backup_is_dated_by_its_database_file() { + use message_crate_core::testutil::{TEST_BACKUP_TAKEN_AT_UNIX_MS, set_modified_unix_ms}; + let dir = tempfile::tempdir().unwrap(); + let crypt = dir.path().join("msgstore.db.crypt15"); + fs::write(&crypt, b"encrypted").unwrap(); + set_modified_unix_ms(&crypt, TEST_BACKUP_TAKEN_AT_UNIX_MS); + let args = args_for(crate::wtsexporter::Platform::Android, dir.path()); + assert_eq!( + super::backup_taken_at_unix_ms(&args), + Some(TEST_BACKUP_TAKEN_AT_UNIX_MS) + ); + + let named = dir.path().join("elsewhere.crypt15"); + fs::write(&named, b"encrypted").unwrap(); + set_modified_unix_ms(&named, TEST_BACKUP_TAKEN_AT_UNIX_MS - 60_000); + let mut args = args_for(crate::wtsexporter::Platform::Android, dir.path()); + args.backup = Some(named); + assert_eq!( + super::backup_taken_at_unix_ms(&args), + Some(TEST_BACKUP_TAKEN_AT_UNIX_MS - 60_000), + "the file the form names wins" + ); + + let unpacked = tempfile::tempdir().unwrap(); + let args = args_for(crate::wtsexporter::Platform::Ios, unpacked.path()); + assert_eq!(super::backup_taken_at_unix_ms(&args), None); + } } diff --git a/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs b/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs index e9b315a9e..22d181bbf 100644 --- a/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs +++ b/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs @@ -16,6 +16,7 @@ fn convert_fixture_json_individual_and_group() { transforms: ExportTransforms::none(), media_search_roots: &[], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Csv, cancel: None, resume: false, @@ -86,6 +87,7 @@ fn copies_ios_style_media_true_data_paths() { transforms: ExportTransforms::none(), media_search_roots: &[media_root.path().to_path_buf()], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Csv, cancel: None, resume: false, @@ -123,6 +125,7 @@ fn jsonl_drains_the_write_queue_and_a_second_run_resumes_it() { transforms: ExportTransforms::none(), media_search_roots: &[], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Jsonl, cancel: None, resume, @@ -150,6 +153,7 @@ fn convert_to_documents( transforms: ExportTransforms::none(), media_search_roots: &[dir.path().to_path_buf()], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Json, cancel: None, resume: false, @@ -343,6 +347,7 @@ fn a_media_path_in_the_media_field_is_copied() { transforms: ExportTransforms::none(), media_search_roots: &[dir.path().to_path_buf()], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Csv, cancel: None, resume: false, @@ -393,6 +398,7 @@ fn a_media_file_not_found_is_kept_as_file_missing_and_the_guid_does_not_change() transforms: ExportTransforms::none(), media_search_roots: &[dir.path().to_path_buf()], owner_identity: None, + backup_taken_at_unix_ms: None, output_format: OutputFormat::Json, cancel: None, resume: false, diff --git a/crates/exporters/whatsapp-exporter/tests/fixtures/ios-backup/Manifest.plist b/crates/exporters/whatsapp-exporter/tests/fixtures/ios-backup/Manifest.plist new file mode 100644 index 000000000..64224a8e8 --- /dev/null +++ b/crates/exporters/whatsapp-exporter/tests/fixtures/ios-backup/Manifest.plist @@ -0,0 +1,12 @@ + + + + + Date + 2026-09-30T18:45:12Z + IsEncrypted + + Version + 10.0 + + diff --git a/crates/libs/ios-backup/src/backup.rs b/crates/libs/ios-backup/src/backup.rs index 00a900a6c..8713bb97e 100644 --- a/crates/libs/ios-backup/src/backup.rs +++ b/crates/libs/ios-backup/src/backup.rs @@ -26,6 +26,23 @@ pub fn ios_backup_encrypted_flag(backup_root: &Path) -> Option { } } +/// When the iPhone backup at `backup_root` was made, in Unix milliseconds: +/// the `Date` its `Manifest.plist` records. The plist is not encrypted, even +/// in an encrypted backup, so no password is needed. +/// +/// Returns `None` when the file is missing, cannot be parsed, or has no +/// date: the conversation file then says nothing about when the backup was +/// made, and the import falls back to its rules for a file without one. +pub fn ios_backup_date_unix_ms(backup_root: &Path) -> Option { + let file = File::open(backup_root.join("Manifest.plist")).ok()?; + let value = plist::Value::from_reader(file).ok()?; + let date = value.as_dictionary()?.get("Date")?.as_date()?; + let since = std::time::SystemTime::from(date) + .duration_since(std::time::UNIX_EPOCH) + .ok()?; + i64::try_from(since.as_millis()).ok() +} + /// Every regular file `Manifest.db` lists under `domain` in the iPhone /// backup at `backup_root`, which is not encrypted, as its path inside the /// domain and its size where the backup keeps it @@ -77,8 +94,9 @@ pub fn ios_backup_domain_files(backup_root: &Path, domain: &str) -> Result + + + + Date + 2026-09-30T18:45:12Z + IsEncrypted + + Version + 10.0 + + diff --git a/crates/libs/ir-format/src/export_transforms/tests.rs b/crates/libs/ir-format/src/export_transforms/tests.rs index 7e5ae1ce2..44a850d02 100644 --- a/crates/libs/ir-format/src/export_transforms/tests.rs +++ b/crates/libs/ir-format/src/export_transforms/tests.rs @@ -18,6 +18,7 @@ fn doc_with_image_attachment() -> ConversationDocument { tool_version: "0".into(), owner_identity: None, owner_display_name: None, + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), @@ -218,6 +219,7 @@ fn doc_with_a_marker_in_every_field() -> ConversationDocument { tool_version: "10.20".into(), owner_identity: Some("LEAK-01".into()), owner_display_name: Some("LEAK-02".into()), + backup_taken_at_unix_ms: Some(1_790_793_912_000), }, conversation: ConversationMeta { chat_identifier: "LEAK-03".into(), diff --git a/crates/libs/ir-format/src/lib_tests.rs b/crates/libs/ir-format/src/lib_tests.rs index 182ae2cd4..e8f07c436 100644 --- a/crates/libs/ir-format/src/lib_tests.rs +++ b/crates/libs/ir-format/src/lib_tests.rs @@ -820,3 +820,87 @@ fn eml_keeps_each_messages_owner_and_each_attachments_size_and_missing_reason() fn mbox_keeps_each_messages_owner_each_attachments_metadata_and_a_text_attachments_bytes() { assert_hard_fields_survive(OutputFormat::Mbox); } + +/// Write a document whose backup has `backup_taken_at_unix_ms` in `format` +/// and read it back. +fn backup_date_after_round_trip( + format: OutputFormat, + backup_taken_at_unix_ms: Option, +) -> Option { + let mut doc = message_ir::testutil::sample_document("from the backup"); + doc.export.backup_taken_at_unix_ms = backup_taken_at_unix_ms; + let tmp = tempfile::tempdir().unwrap(); + let path = write_format(tmp.path(), format, doc).unwrap(); + let back = match format { + OutputFormat::Json => read_conversation_json(&path), + OutputFormat::Jsonl => read_conversation_jsonl(&path), + OutputFormat::Csv => read_conversation_csv(&path), + OutputFormat::Eml => read_conversation_eml_dir(&path), + OutputFormat::Mbox => read_conversation_mbox(&path), + other => panic!("no round trip for {}", other.as_str()), + } + .unwrap(); + back.export.backup_taken_at_unix_ms +} + +/// Every format keeps when the backup was made, and a file that does not +/// say reads back saying nothing, never a date of the reader's own. +#[test] +fn every_format_keeps_when_the_backup_was_made() { + for format in [ + OutputFormat::Json, + OutputFormat::Jsonl, + OutputFormat::Csv, + OutputFormat::Eml, + OutputFormat::Mbox, + ] { + assert_eq!( + backup_date_after_round_trip(format, Some(1_790_793_912_345)), + Some(1_790_793_912_345), + "{}", + format.as_str() + ); + assert_eq!( + backup_date_after_round_trip(format, None), + None, + "{}", + format.as_str() + ); + } +} + +#[test] +fn csv_refuses_a_backup_date_that_is_not_a_number() { + let doc = message_ir::testutil::sample_document("hello"); + let tmp = tempfile::tempdir().unwrap(); + let path = write_format(tmp.path(), OutputFormat::Csv, doc).unwrap(); + let text = fs::read_to_string(&path).unwrap(); + let mut lines: Vec = text.lines().map(str::to_string).collect(); + let column = CSV_HEADERS + .iter() + .position(|h| *h == "backup_taken_at_unix_ms") + .unwrap(); + let mut row = csv::ReaderBuilder::new() + .has_headers(false) + .from_reader(lines[1].as_bytes()) + .records() + .next() + .unwrap() + .unwrap() + .iter() + .map(str::to_string) + .collect::>(); + row[column] = "yesterday".into(); + let mut out = csv::Writer::from_writer(Vec::new()); + out.write_record(&row).unwrap(); + lines[1] = String::from_utf8(out.into_inner().unwrap()) + .unwrap() + .trim_end() + .to_string(); + fs::write(&path, lines.join("\n")).unwrap(); + let err = read_conversation_csv(&path).unwrap_err(); + assert!( + format!("{err:#}").contains("bad backup_taken_at_unix_ms \"yesterday\""), + "{err:#}" + ); +} diff --git a/crates/libs/ir-format/src/read_csv.rs b/crates/libs/ir-format/src/read_csv.rs index b5b4483f7..bc24e55ff 100644 --- a/crates/libs/ir-format/src/read_csv.rs +++ b/crates/libs/ir-format/src/read_csv.rs @@ -48,7 +48,8 @@ pub fn read_conversation_csv(path: &Path) -> Result { bail!("CSV has no data rows: {}", path.display()); } - let header = header_from_row(&cols, &rows[0]); + let header = header_from_row(&cols, &rows[0]) + .with_context(|| format!("parse CSV row 1 in {}", path.display()))?; let mut messages = Vec::with_capacity(rows.len()); for (i, record) in rows.iter().enumerate() { messages.push( @@ -63,7 +64,15 @@ pub fn read_conversation_csv(path: &Path) -> Result { } /// Rebuild the conversation header from the first CSV row's conversation columns. -fn header_from_row(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> ConversationHeader { +/// +/// # Errors +/// +/// Returns an error when `backup_taken_at_unix_ms` holds anything but a +/// whole number or a blank. +fn header_from_row( + cols: &HashMap<&str, usize>, + row: &csv::StringRecord, +) -> Result { let get = |name: &str| cell(cols, row, name).unwrap_or(""); let participants = parse_participants(get("participants_json")); let group_title = { @@ -74,7 +83,14 @@ fn header_from_row(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> Conv Some(t.to_string()) } }; - ConversationHeader { + let backup_taken_at_unix_ms = match get("backup_taken_at_unix_ms").trim() { + "" => None, + ms => Some( + ms.parse::() + .with_context(|| format!("bad backup_taken_at_unix_ms {ms:?}"))?, + ), + }; + Ok(ConversationHeader { schema_version: SCHEMA_VERSION, export: ExportMeta { source: get("export_source").to_string(), @@ -82,6 +98,7 @@ fn header_from_row(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> Conv tool_version: get("export_tool_version").to_string(), owner_identity: nonempty(get("owner_identity")), owner_display_name: nonempty(get("owner_display_name")), + backup_taken_at_unix_ms, }, conversation: ConversationMeta { chat_identifier: get("chat_identifier").to_string(), @@ -90,7 +107,7 @@ fn header_from_row(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> Conv participants, stats: ConversationStats::default(), }, - } + }) } /// Rebuild one message from a CSV row. diff --git a/crates/libs/ir-format/src/read_mail.rs b/crates/libs/ir-format/src/read_mail.rs index 92d40522b..7be13e041 100644 --- a/crates/libs/ir-format/src/read_mail.rs +++ b/crates/libs/ir-format/src/read_mail.rs @@ -86,6 +86,7 @@ fn document_from_mail_messages(messages: &[MailMessage]) -> Result = first diff --git a/crates/libs/ir-format/src/write.rs b/crates/libs/ir-format/src/write.rs index 1d1a88c61..f7d6fbaae 100644 --- a/crates/libs/ir-format/src/write.rs +++ b/crates/libs/ir-format/src/write.rs @@ -46,6 +46,7 @@ pub const CSV_HEADERS: &[&str] = &[ "export_tool_version", "owner_identity", "owner_display_name", + "backup_taken_at_unix_ms", "message_owner_identity", "android_type", "source_fields_json", @@ -214,6 +215,11 @@ pub(crate) fn write_conversation_csv( }) .collect::>(), ); + let backup_taken_at = doc + .export + .backup_taken_at_unix_ms + .map(|ms| ms.to_string()) + .unwrap_or_default(); message_ir::write_atomic(&path, |out| { let mut wtr = csv::Writer::from_writer(out); @@ -221,8 +227,14 @@ pub(crate) fn write_conversation_csv( .with_context(|| format!("write header {}", path.display()))?; for msg in &doc.messages { let cells = MessageCells::new(msg)?; - wtr.write_record(csv_record(doc, &participants_json, msg, &cells)) - .with_context(|| format!("write row {}", path.display()))?; + wtr.write_record(csv_record( + doc, + &participants_json, + &backup_taken_at, + msg, + &cells, + )) + .with_context(|| format!("write row {}", path.display()))?; } wtr.flush()?; Ok(()) @@ -366,9 +378,10 @@ fn bool_cell(value: bool) -> &'static str { fn csv_record<'a>( doc: &'a ConversationDocument, participants_json: &'a str, + backup_taken_at: &'a str, msg: &'a IrMessage, cells: &'a MessageCells, -) -> [&'a str; 46] { +) -> [&'a str; 47] { let im = &cells.imessage; [ doc.conversation.chat_identifier.as_str(), @@ -403,6 +416,7 @@ fn csv_record<'a>( doc.export.tool_version.as_str(), doc.export.owner_identity.as_deref().unwrap_or(""), doc.export.owner_display_name.as_deref().unwrap_or(""), + backup_taken_at, msg.owner_identity.as_deref().unwrap_or(""), cells.android_type.as_str(), cells.source_fields_json.as_str(), @@ -479,6 +493,7 @@ pub fn document_to_mail_messages( export_source: doc.export.source.clone(), export_tool: doc.export.tool.clone(), export_tool_version: doc.export.tool_version.clone(), + backup_taken_at_unix_ms: doc.export.backup_taken_at_unix_ms, filename_suffix: doc.packaging_stem_suffix.clone(), message: msg.clone(), attachments, diff --git a/crates/libs/ir/src/lib.rs b/crates/libs/ir/src/lib.rs index 44aef8a1c..ba8709bf6 100644 --- a/crates/libs/ir/src/lib.rs +++ b/crates/libs/ir/src/lib.rs @@ -138,7 +138,7 @@ impl std::fmt::Display for UnknownDeletion { impl std::error::Error for UnknownDeletion {} /// Schema version written into every [`ConversationDocument`]. -pub const SCHEMA_VERSION: u32 = 10; +pub const SCHEMA_VERSION: u32 = 11; /// One exported chat: export metadata, conversation roster and stats, and messages. /// @@ -172,6 +172,13 @@ pub struct ExportMeta { pub owner_identity: Option, /// Outgoing display name. Set when known (iMessage caller-id or `"Me"`). pub owner_display_name: Option, + /// When the backup this file was read from was made, in Unix + /// milliseconds: an iPhone backup's `Manifest.plist` date, an SMS Backup + /// & Restore file's `backup_date`, or the backup file's modification + /// time where the source records nothing better. `None` when nothing + /// says. Between two copies of one message from one source, the import + /// lets the copy from the later backup decide its deletion mark and text. + pub backup_taken_at_unix_ms: Option, } /// The shape of a conversation: one-to-one, group, or orphaned. diff --git a/crates/libs/ir/src/projection.rs b/crates/libs/ir/src/projection.rs index 75e1e3fb9..aee12212e 100644 --- a/crates/libs/ir/src/projection.rs +++ b/crates/libs/ir/src/projection.rs @@ -516,6 +516,7 @@ mod tests { tool_version: "0".into(), owner_identity: Some("+15555550100".into()), owner_display_name: None, + backup_taken_at_unix_ms: None, } } diff --git a/crates/libs/ir/src/schema_version.rs b/crates/libs/ir/src/schema_version.rs index 851c94781..3b99d3f3a 100644 --- a/crates/libs/ir/src/schema_version.rs +++ b/crates/libs/ir/src/schema_version.rs @@ -3,12 +3,12 @@ //! Every reader of a [`ConversationDocument`](crate::ConversationDocument) or //! its JSON Lines header — the format reader, the push client, the server's //! import — refuses a version other than [`SCHEMA_VERSION`] with the same -//! words, and refuses it before parsing the rest of the file: a version-9 -//! file is not expected to match version 10 (version 9 kept a reply's link in -//! `imessage.is_reply` and `imessage.in_reply_to_guid`, which version 10 would -//! pass over, so every reply would arrive as a plain message), and the person -//! should read "schema version 9", not a file that imports without its -//! replies. +//! words, and refuses it before parsing the rest of the file: a version-10 +//! file is not expected to match version 11 (version 10 did not say when its +//! backup was made, so an import could not tell which of two backups of one +//! phone is the later one, and a file read as version 11 would claim it has +//! no date when it only never asked), and the person should read "schema +//! version 10", not a file that imports by other rules than its own. use crate::SCHEMA_VERSION; use serde::Deserialize; @@ -95,6 +95,21 @@ mod tests { ); } + /// Version 10 had no `export.backup_taken_at_unix_ms`; version 11 says + /// when the backup was made, and the import lets the later backup decide + /// a message's mark and text. A version-10 file is refused by its + /// version, never read as a file whose backup has no date. + #[test] + fn refuses_a_version_10_file_by_name() { + assert_eq!( + check_schema_version_in_json(r#"{"schema_version":10,"export":{}}"#) + .unwrap_err() + .to_string(), + format!("This file is schema version 10; Message Crate reads version {SCHEMA_VERSION}") + ); + assert_eq!(SCHEMA_VERSION, 11); + } + #[test] fn peeks_at_the_version_before_anything_else() { assert_eq!( diff --git a/crates/libs/ir/src/testutil.rs b/crates/libs/ir/src/testutil.rs index 706de1f62..f394d34a9 100644 --- a/crates/libs/ir/src/testutil.rs +++ b/crates/libs/ir/src/testutil.rs @@ -19,6 +19,7 @@ pub fn sample_document(text: &str) -> ConversationDocument { tool_version: "10.26.003".into(), owner_identity: Some("+15555550100".into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), @@ -95,6 +96,7 @@ pub fn sample_imessage_document() -> ConversationDocument { tool_version: "0.1.0".into(), owner_identity: Some("+15555550100".into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), diff --git a/crates/libs/mail/src/headers.rs b/crates/libs/mail/src/headers.rs index 4e46e1d13..50702809b 100644 --- a/crates/libs/mail/src/headers.rs +++ b/crates/libs/mail/src/headers.rs @@ -23,6 +23,8 @@ pub(crate) const EXPORT_SOURCE: &str = "X-ME-Export-Source"; pub(crate) const EXPORT_TOOL: &str = "X-ME-Export-Tool"; /// Export tool version. pub(crate) const EXPORT_TOOL_VERSION: &str = "X-ME-Export-Tool-Version"; +/// When the backup the export was read from was made, in Unix milliseconds. +pub(crate) const BACKUP_TAKEN_AT_UNIX_MS: &str = "X-ME-Backup-Taken-At-Unix-Ms"; /// Group chat title. pub(crate) const GROUP_TITLE: &str = "X-ME-Group-Title"; /// Conversation roster as JSON. diff --git a/crates/libs/mail/src/lib.rs b/crates/libs/mail/src/lib.rs index 722e6b3bb..7590e3a7b 100644 --- a/crates/libs/mail/src/lib.rs +++ b/crates/libs/mail/src/lib.rs @@ -112,6 +112,9 @@ pub struct MailMessage { pub export_tool: String, /// → `X-ME-Export-Tool-Version`. pub export_tool_version: String, + /// When the backup was made, in Unix milliseconds → + /// `X-ME-Backup-Taken-At-Unix-Ms`; `None` when the export does not say. + pub backup_taken_at_unix_ms: Option, /// Optional stem suffix (e.g. `"__whatsapp"`) for conversation directory / mbox names. pub filename_suffix: Option, /// The message itself (headers read guid, timestamp, direction, service, @@ -911,6 +914,10 @@ fn conversation_headers<'m>( msg.message.sender_display_name.clone(), ), (headers::OWNER_IDENTITY, Some(msg.owner_identity.clone())), + ( + headers::BACKUP_TAKEN_AT_UNIX_MS, + msg.backup_taken_at_unix_ms.map(|ms| ms.to_string()), + ), (headers::OWNER_DISPLAY_NAME, msg.owner_display_name.clone()), ( headers::MESSAGE_OWNER_IDENTITY, diff --git a/crates/libs/mail/src/parse.rs b/crates/libs/mail/src/parse.rs index 5b0c2dc12..17854ba1c 100644 --- a/crates/libs/mail/src/parse.rs +++ b/crates/libs/mail/src/parse.rs @@ -108,6 +108,7 @@ pub fn mail_message_from_eml_bytes(bytes: &[u8]) -> Result { let export_source = optional_header(headers, hn::EXPORT_SOURCE).unwrap_or_default(); let export_tool = optional_header(headers, hn::EXPORT_TOOL).unwrap_or_default(); let export_tool_version = optional_header(headers, hn::EXPORT_TOOL_VERSION).unwrap_or_default(); + let backup_taken_at_unix_ms = parse_backup_taken_at(headers)?; let text = extract_text_body(&mail).unwrap_or_default(); let attachments = merge_attachments(&mail, headers)?; @@ -160,6 +161,7 @@ pub fn mail_message_from_eml_bytes(bytes: &[u8]) -> Result { export_source, export_tool, export_tool_version, + backup_taken_at_unix_ms, filename_suffix: None, message: IrMessage { guid, @@ -323,6 +325,18 @@ fn header_u32(headers: &[MailHeader<'_>], name: &str) -> Option { typed_header(headers, name)?.parse().ok() } +/// When the backup was made, from `X-ME-Backup-Taken-At-Unix-Ms`, or none +/// when the header is absent. A value that is not a whole number is refused +/// rather than read as no date. +fn parse_backup_taken_at(headers: &[MailHeader<'_>]) -> Result> { + let Some(raw) = typed_header(headers, hn::BACKUP_TAKEN_AT_UNIX_MS) else { + return Ok(None); + }; + raw.parse() + .map(Some) + .with_context(|| format!("This mail's {} header {raw:?}", hn::BACKUP_TAKEN_AT_UNIX_MS)) +} + /// The message's mark from `X-ME-Deletion`, or none when the header is /// absent. A value that names neither mark is refused rather than dropped. fn parse_deletion(headers: &[MailHeader<'_>]) -> Result> { @@ -469,6 +483,7 @@ mod tests { export_source: "sms-backup-restore".into(), export_tool: "SMS Backup & Restore".into(), export_tool_version: "10.26.003".into(), + backup_taken_at_unix_ms: None, filename_suffix: None, message: IrMessage { guid: "aabbccddeeff00112233445566778899".into(), @@ -586,6 +601,7 @@ mod tests { export_source: "imessage".into(), export_tool: "imessage-exporter".into(), export_tool_version: "3.1.0".into(), + backup_taken_at_unix_ms: None, filename_suffix: None, message: IrMessage { guid: "AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE".into(), @@ -713,6 +729,7 @@ mod tests { export_source: "imessage".into(), export_tool: "imessage-exporter".into(), export_tool_version: "3.1.0".into(), + backup_taken_at_unix_ms: None, filename_suffix: None, message: IrMessage { guid: "11111111-2222-3333-4444-555555555555".into(), diff --git a/crates/libs/mail/src/tests.rs b/crates/libs/mail/src/tests.rs index 507ba644d..f21893cc9 100644 --- a/crates/libs/mail/src/tests.rs +++ b/crates/libs/mail/src/tests.rs @@ -15,6 +15,7 @@ fn base_sms() -> MailMessage { export_source: "sms-backup-restore".into(), export_tool: "SMS Backup & Restore".into(), export_tool_version: "10.26.003".into(), + backup_taken_at_unix_ms: None, filename_suffix: None, message: IrMessage { guid: "aabbccddeeff00112233445566778899".into(), @@ -912,6 +913,7 @@ fn x_me_values(msg: &MailMessage) -> serde_json::Value { "export_source": msg.export_source, "export_tool": msg.export_tool, "export_tool_version": msg.export_tool_version, + "backup_taken_at_unix_ms": msg.backup_taken_at_unix_ms, "message": msg.message, "attachment_names": msg.attachments.iter().map(|a| &a.meta.original_name).collect::>(), "attachment_types": msg.attachments.iter().map(|a| &a.meta.mime_type).collect::>(), diff --git a/crates/libs/push/src/project.rs b/crates/libs/push/src/project.rs index 48131145e..d7e493e2a 100644 --- a/crates/libs/push/src/project.rs +++ b/crates/libs/push/src/project.rs @@ -138,6 +138,7 @@ mod tests { tool_version: "10.26.003".into(), owner_identity: Some("+15555550100".into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), @@ -154,7 +155,7 @@ mod tests { packaging_stem_suffix: None, }; let header = String::from_utf8(document_header_line(&doc).unwrap()).unwrap(); - assert!(header.contains(r#""schema_version":10"#)); + assert!(header.contains(r#""schema_version":11"#)); assert!(header.contains(r#""sms-backup-restore""#)); assert!(!header.contains(r#""record":"conversation""#)); diff --git a/crates/libs/push/tests/push_mock.rs b/crates/libs/push/tests/push_mock.rs index bee436f8c..93d26f719 100644 --- a/crates/libs/push/tests/push_mock.rs +++ b/crates/libs/push/tests/push_mock.rs @@ -34,6 +34,7 @@ fn sample_doc() -> ConversationDocument { tool_version: "10.26.003".into(), owner_identity: Some("+15555550100".into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: None, }, conversation: ConversationMeta { chat_identifier: "+15555550101".into(), diff --git a/crates/libs/sbr/src/read.rs b/crates/libs/sbr/src/read.rs index de4f5472b..26cdb960a 100644 --- a/crates/libs/sbr/src/read.rs +++ b/crates/libs/sbr/src/read.rs @@ -147,9 +147,14 @@ pub struct Record { pub dropped_character_references: u64, } -/// Counters for seen and skipped messages. +/// Counters for seen and skipped messages, and when the file's backup was +/// made. #[derive(Debug, Default, Clone, Copy)] pub struct ParseStats { + /// The root `` element's `backup_date`: when SMS Backup & Restore + /// made the backup, in Unix milliseconds. `None` when the file has no + /// such attribute or it is not a number. + pub backup_date_unix_ms: Option, /// Number of `` elements encountered. pub sms_seen: u64, /// Number of `` elements encountered. @@ -827,6 +832,7 @@ where loop { match xml.read_event_into(&mut buf) { Ok(Event::Start(e)) => match e.name().as_ref().to_ascii_lowercase().as_str() { + "smses" => stats.backup_date_unix_ms = backup_date(&attrs(&e, &mut 0)), "sms" => { dropped = 0; sms = attrs(&e, &mut dropped); @@ -842,6 +848,7 @@ where _ => {} }, Ok(Event::Empty(e)) => match e.name().as_ref().to_ascii_lowercase().as_str() { + "smses" => stats.backup_date_unix_ms = backup_date(&attrs(&e, &mut 0)), "sms" => { let mut own = 0; let attrs = attrs(&e, &mut own); @@ -888,6 +895,12 @@ where Ok(()) } +/// The root element's `backup_date` in Unix milliseconds, as SMS Backup & +/// Restore writes it, or `None` when it is missing or not a number. +fn backup_date(attrs: &HashMap) -> Option { + get(attrs, "backup_date").trim().parse().ok() +} + #[cfg(test)] fn parse_reader( reader: R, diff --git a/crates/server/demo-seed/src/conversations.rs b/crates/server/demo-seed/src/conversations.rs index 4d8433b8b..c5d810dbb 100644 --- a/crates/server/demo-seed/src/conversations.rs +++ b/crates/server/demo-seed/src/conversations.rs @@ -80,14 +80,17 @@ const PHOTO_CAPTIONS: &[&str] = &[ const EMOJI_ONLY: &[&str] = &["👍", "😂", "❤️", "🎉", "😊"]; /// Export metadata stamped on a conversation header. `owner_identity` is the -/// Demo Account's identity the conversation's messages are held at. -fn export_meta(source: &str, owner_identity: &str) -> ExportMeta { +/// Demo Account's identity the conversation's messages are held at, and +/// `backup_taken_at_unix_ms` when the backup was made: the settings' +/// reference time, which every generated message is before. +fn export_meta(source: &str, owner_identity: &str, backup_taken_at_unix_ms: i64) -> ExportMeta { ExportMeta { source: source.into(), tool: "demo-seed".into(), tool_version: "0.2.0".into(), owner_identity: Some(owner_identity.into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: Some(backup_taken_at_unix_ms), } } @@ -194,6 +197,7 @@ impl Seeder<'_, R> { IrConversationType::Individual, &[], IMESSAGE_SOURCE, + self.cfg.reference_time.timestamp_millis(), )?; self.stats.conversation_files += 1; } @@ -204,6 +208,7 @@ impl Seeder<'_, R> { IrConversationType::Group, &EMPTY_GROUP_MEMBERS, IMESSAGE_SOURCE, + self.cfg.reference_time.timestamp_millis(), )?; self.stats.conversation_files += 1; } @@ -387,7 +392,11 @@ impl Seeder<'_, R> { None, participants, msg_count, - export_meta(source_id(flavor), OWNER_PHONE), + export_meta( + source_id(flavor), + OWNER_PHONE, + self.cfg.reference_time.timestamp_millis(), + ), )?; let timestamps = self.timestamps(msg_count, spec.span_years, sample_direct_day_burst); @@ -479,7 +488,11 @@ impl Seeder<'_, R> { None, individual_participants(chat_id, overlap.display_name.clone()), overlap.msg_count, - export_meta(IMESSAGE_SOURCE, OWNER_PHONE), + export_meta( + IMESSAGE_SOURCE, + OWNER_PHONE, + self.cfg.reference_time.timestamp_millis(), + ), )?; let mut origin_guid: Option = None; for (i, shared) in overlap.shared.iter().enumerate() { @@ -534,7 +547,11 @@ impl Seeder<'_, R> { None, individual_participants(chat_id, overlap.display_name.clone()), android_total, - export_meta(SBR_SOURCE, OWNER_PHONE), + export_meta( + SBR_SOURCE, + OWNER_PHONE, + self.cfg.reference_time.timestamp_millis(), + ), )?; for (i, shared) in overlap.shared.iter().enumerate() { let msg = shared.message( @@ -621,7 +638,11 @@ impl Seeder<'_, R> { None, participants, msg_count, - export_meta(IMESSAGE_SOURCE, owner_identity), + export_meta( + IMESSAGE_SOURCE, + owner_identity, + self.cfg.reference_time.timestamp_millis(), + ), )?; let timestamps = self.timestamps(msg_count, 1.5, sample_direct_day_burst); @@ -683,7 +704,11 @@ impl Seeder<'_, R> { group.title.clone(), participants, header_message_count, - export_meta(IMESSAGE_SOURCE, OWNER_PHONE), + export_meta( + IMESSAGE_SOURCE, + OWNER_PHONE, + self.cfg.reference_time.timestamp_millis(), + ), )?; if let Some(title) = rename_title { @@ -835,7 +860,11 @@ impl Seeder<'_, R> { None, sender.into_iter().collect(), messages.len(), - export_meta(IMESSAGE_SOURCE, OWNER_PHONE), + export_meta( + IMESSAGE_SOURCE, + OWNER_PHONE, + self.cfg.reference_time.timestamp_millis(), + ), )?; for msg in messages { self.emit(&mut file, msg)?; @@ -857,6 +886,7 @@ fn write_header_only( conv_type: IrConversationType, member_phones: &[&str], source: &str, + backup_taken_at_unix_ms: i64, ) -> Result<()> { let path = staging.join(format!("empty-{}.jsonl", sanitize_filename(chat_id))); let mut file = open_jsonl(&path)?; @@ -875,7 +905,7 @@ fn write_header_only( None, participants, 0, - export_meta(source, OWNER_PHONE), + export_meta(source, OWNER_PHONE, backup_taken_at_unix_ms), )?; Ok(()) } diff --git a/src-tauri/src/commands/upload.rs b/src-tauri/src/commands/upload.rs index 405523ae7..85ab7fc06 100644 --- a/src-tauri/src/commands/upload.rs +++ b/src-tauri/src/commands/upload.rs @@ -377,6 +377,7 @@ mod tests { tool_version: "10.26.003".into(), owner_identity: Some("+15555550100".into()), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: None, }, "conversation": ConversationMeta { chat_identifier: "+15555550101".into(), From c11e99070e29524c1500b65445fb3e4aa2f8f4ac Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:47:19 -0400 Subject: [PATCH 13/42] feat(import): the later backup decides a message's mark and text Staging keeps each staged row's backup date (staging_messages .backup_taken_at), and messages keeps the date of the backup that decided the stored copy. Between two copies of one message from one source, when both backups have a date, the copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, whatever the versions' times say; a copy from an earlier or the same backup changes neither. This holds in one import in either file order and across imports in either order, so the duplicate flag, which follows the text, does too. Where either backup has no date, the rules for files without one hold: a mark adds, now also from the second file of one import, and an edit is taken when its newest earlier version is newer. The Import Run records the newest backup date its files name, and the HTTP API answers it as backup_taken_at on ImportRun and ImportRunSummary. A message answers its own backup_taken_at, which an Export Run writes back into the conversation file as the newest date of the conversation's messages. The schema change rebuilds the database; nothing migrates. Part of #1924, #1741, #1804. Co-Authored-By: Claude Opus 5.5 --- crates/libs/api-types/src/lib.rs | 6 + crates/libs/export/src/project.rs | 52 ++ crates/libs/export/src/run.rs | 13 +- crates/server/server/src/accounts_api.rs | 8 +- crates/server/server/src/cli/tests.rs | 1 + .../server/src/db/conversation_messages.rs | 5 +- crates/server/server/src/db/imports.rs | 33 +- crates/server/server/src/db/staging.rs | 217 +++++++- crates/server/server/src/db/staging/tests.rs | 1 + crates/server/server/src/imports_api/mod.rs | 5 + .../server/server/src/imports_api/promote.rs | 11 + .../server/server/src/imports_api/staging.rs | 104 +++- crates/server/server/src/imports_api/tests.rs | 2 + .../src/imports_api/tests/backup_dates.rs | 483 ++++++++++++++++++ crates/server/server/src/models.rs | 32 +- .../server/server/src/test_support/lines.rs | 8 + .../fixtures/apple-messages-deletions.jsonl | 2 +- .../tests/fixtures/apple-messages-edits.jsonl | 2 +- .../fixtures/apple-messages-reactions.jsonl | 2 +- .../contacts-identities-and-messages.md | 29 ++ docs/src/assets/openapi.json | 32 ++ schema/sql/accounts.sql | 6 +- schema/sql/messages.sql | 8 +- schema/sql/staging.sql | 7 +- 24 files changed, 996 insertions(+), 73 deletions(-) create mode 100644 crates/server/server/src/imports_api/tests/backup_dates.rs diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 6cb3ed665..84356d6dd 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -418,6 +418,11 @@ api_shape! { /// each part; `text` is the final version. Empty for a message never /// edited, or from a source that records no edits. pub earlier_versions: Vec, + /// When the backup that gave the message its mark and text was made, + /// RFC 3339 in UTC with a `Z` suffix; `null` when the conversation + /// file did not say. Between two copies of one message from one + /// source, the copy from the later backup decides. + pub backup_taken_at: Option, /// True in a Messages search answer (`GET /v1/messages` with `q`) /// when the message is a hit only because of its earlier versions: /// its final text alone does not match the query, and the versions @@ -653,6 +658,7 @@ mod tests { edited_at: Some("2023-12-31T23:59:00Z".into()), matched: false, }], + backup_taken_at: Some("2024-01-02T08:00:00Z".into()), matched_earlier_version: false, }; diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index 8b4b21140..f94f74282 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -22,6 +22,10 @@ pub fn conversation_key(msg: &Message) -> String { } /// Build one conversation document from a seed message and the mapped rows. +/// +/// The document's backup date is the seed's `backup_taken_at`, which the +/// Export Run sets to the newest of the conversation's messages +/// ([`newest_backup_taken_at`]), or none when no message has one. pub fn build_document( source: &str, seed: &Message, @@ -59,6 +63,10 @@ pub fn build_document( tool_version: env!("CARGO_PKG_VERSION").into(), owner_identity: shared_owner(&messages), owner_display_name: Some("Me".into()), + backup_taken_at_unix_ms: seed + .backup_taken_at + .as_deref() + .and_then(|at| parse_timestamp_unix_ms(at).ok()), }, conversation: ConversationMeta { chat_identifier: seed.conversation.chat_identifier.clone(), @@ -77,6 +85,16 @@ pub fn build_document( } } +/// Keep on `seed` the newer of its own `backup_taken_at` and `msg`'s, so +/// the conversation file says the newest backup any of its messages came +/// from. Every stored time has one fixed RFC 3339 form, so the greater +/// string is the later instant. +pub fn newest_backup_taken_at(seed: &mut Message, msg: &Message) { + if msg.backup_taken_at > seed.backup_taken_at { + seed.backup_taken_at.clone_from(&msg.backup_taken_at); + } +} + /// Map one exported message into the shared conversation message type. /// /// # Errors @@ -617,6 +635,38 @@ mod tests { assert_eq!(doc.conversation.stats.message_count, 2); } + /// The file says the newest backup any of the conversation's messages + /// came from, and nothing when none of them says. + #[test] + fn a_document_says_the_newest_backup_its_messages_came_from() { + let participant = || Participant { + identity: Some("+1".into()), + name: "Sam".into(), + service: None, + contact_id: None, + }; + let mut seed = seed_message_with_participant(participant()); + assert_eq!( + build_document("imessage", &seed, vec![]) + .export + .backup_taken_at_unix_ms, + None + ); + + seed.backup_taken_at = Some("2026-09-01T10:00:00Z".into()); + let mut newer = seed_message_with_participant(participant()); + newer.backup_taken_at = Some("2026-09-30T18:45:12Z".into()); + let undated = seed_message_with_participant(participant()); + newest_backup_taken_at(&mut seed, &newer); + newest_backup_taken_at(&mut seed, &undated); + assert_eq!( + build_document("imessage", &seed, vec![]) + .export + .backup_taken_at_unix_ms, + Some(1_790_793_912_000) + ); + } + /// Whether the conversation is a group comes from the server's /// `is_group`, never from reading `conversation_type` again. #[test] @@ -727,6 +777,7 @@ mod tests { tapbacks: vec![], deletion: None, earlier_versions: Vec::new(), + backup_taken_at: None, matched_earlier_version: false, }; let ir = to_ir_message(&msg, false).unwrap(); @@ -878,6 +929,7 @@ mod tests { tapbacks: vec![], deletion: None, earlier_versions: Vec::new(), + backup_taken_at: None, matched_earlier_version: false, } } diff --git a/crates/libs/export/src/run.rs b/crates/libs/export/src/run.rs index f30feb350..b50687044 100644 --- a/crates/libs/export/src/run.rs +++ b/crates/libs/export/src/run.rs @@ -16,7 +16,10 @@ use serde::Serialize; use crate::http::{CloseAction, ExportMessagesArgs, HttpSession}; use crate::journal::{self, ExportJournalEvent, ExportJournalState, ServerTarget}; use crate::part_file::write_asset; -use crate::project::{ExportPath, build_document, conversation_key, export_path, to_ir_message}; +use crate::project::{ + ExportPath, build_document, conversation_key, export_path, newest_backup_taken_at, + to_ir_message, +}; use message_crate_api_types::{ExportQueryList, ExportRun, ExportScope, Message}; /// Page size for `GET /v1/exports/{id}/messages`; the server's maximum. @@ -496,13 +499,13 @@ impl<'a> Export<'a> { } } let ir = to_ir_message(&msg, cfg.skip_attachments)?; - fetched + let (seed, messages) = fetched .by_conv .entry(conversation_key(&msg)) // Keep first message as seed for conversation metadata. - .or_insert_with(|| (msg.clone(), Vec::new())) - .1 - .push(ir); + .or_insert_with(|| (msg.clone(), Vec::new())); + newest_backup_taken_at(seed, &msg); + messages.push(ir); } match next_offset(offset, limit, page.total) { Some(next) => offset = next, diff --git a/crates/server/server/src/accounts_api.rs b/crates/server/server/src/accounts_api.rs index e9a598df9..93c719fd7 100644 --- a/crates/server/server/src/accounts_api.rs +++ b/crates/server/server/src/accounts_api.rs @@ -1301,9 +1301,9 @@ pub(crate) enum AccountImportRuns { #[serde(untagged)] pub(crate) enum AccountImportRun { /// The account's own run. - Own(ImportRun), + Own(Box), /// Another account's run, as the owner reads it. - Owner(OwnerImportRun), + Owner(Box), } /// An account's Export Runs as its reader may see them: in full for the @@ -1382,10 +1382,10 @@ pub(crate) async fn get_account_import( .await .map_err(ApiError::from)?; let run = crate::imports_api::owner_import_run(&mut conn, row).await?; - return Ok(Json(AccountImportRun::Owner(run))); + return Ok(Json(AccountImportRun::Owner(Box::new(run)))); } let run = crate::imports_api::full_import_run(&mut conn, target, import_id).await?; - Ok(Json(AccountImportRun::Own(run))) + Ok(Json(AccountImportRun::Own(Box::new(run)))) } /// An account's Export Runs as a page, newest first unless `sort` says diff --git a/crates/server/server/src/cli/tests.rs b/crates/server/server/src/cli/tests.rs index 42f8d6447..e6533584d 100644 --- a/crates/server/server/src/cli/tests.rs +++ b/crates/server/server/src/cli/tests.rs @@ -431,6 +431,7 @@ fn imports_discard_prints_the_import_run_or_that_there_was_none() { form_json: None, source_fingerprint: None, source_identities: None, + backup_taken_at: None, }; assert_eq!( format_discarded_import("alice", Some(&row)), diff --git a/crates/server/server/src/db/conversation_messages.rs b/crates/server/server/src/db/conversation_messages.rs index 3926bed7f..f6a0457aa 100644 --- a/crates/server/server/src/db/conversation_messages.rs +++ b/crates/server/server/src/db/conversation_messages.rs @@ -54,6 +54,7 @@ struct RawRow { reply_to: Option, reply_count: i64, deletion: Option, + backup_taken_at: Option, chat_identifier: String, conversation_type: String, group_title: Option, @@ -433,7 +434,7 @@ fn message_page_sql( m.is_announcement, m.is_reply, m.reply_to_guid, m.reply_to_part, ({reply_count}) AS reply_count, hc.raw AS chat_identifier, c.conversation_type, c.group_title, - ho.raw AS owner, {label} AS label, m.deletion + ho.raw AS owner, {label} AS label, m.deletion, m.backup_taken_at {from_sql} WHERE {where_sql} ORDER BY {order_by} LIMIT ? OFFSET ?", @@ -486,6 +487,7 @@ async fn fetch_message_page( owner: row.try_get(19)?, label: row.try_get(20)?, deletion: row.try_get(21)?, + backup_taken_at: row.try_get(22)?, }) }) .collect::, ApiError>>()?; @@ -533,6 +535,7 @@ async fn fetch_message_page( // The column's CHECK admits only the two marks or NULL. deletion: r.deletion.as_deref().and_then(Deletion::parse), earlier_versions: earlier_versions.remove(&r.id).unwrap_or_default(), + backup_taken_at: r.backup_taken_at, matched_earlier_version: false, } }) diff --git a/crates/server/server/src/db/imports.rs b/crates/server/server/src/db/imports.rs index 7714e9d5a..619e71cab 100644 --- a/crates/server/server/src/db/imports.rs +++ b/crates/server/server/src/db/imports.rs @@ -200,6 +200,9 @@ pub struct ImportRow { pub source_fingerprint: Option, /// Addresses the backup's device sent from (JSON array). pub source_identities: Option, + /// When the backup the run read was made: the newest its conversation + /// files name, in the form a message's timestamp takes. + pub backup_taken_at: Option, } /// Outcome fields written when a run completes. @@ -448,7 +451,7 @@ fn is_unique_violation(err: &sqlx::Error) -> bool { const IMPORT_COLUMNS: &str = "id, account_id, source, tool, mode, status, started_at, \ finished_at, message_count, attachment_count, bytes_uploaded, duration_ms, parse_ms, \ attachments_ms, prepare_ms, upload_ms, summary_json, stage, run_dir, device_id, \ - form_json, source_fingerprint, source_identities, dedupe"; + form_json, source_fingerprint, source_identities, dedupe, backup_taken_at"; /// Map one `imports` row by column position. fn import_from_row(row: &SqliteRow) -> Result { @@ -491,9 +494,34 @@ fn import_from_row(row: &SqliteRow) -> Result { source_fingerprint: row.try_get(21)?, source_identities: row.try_get(22)?, dedupe: row.try_get::(23)? != 0, + backup_taken_at: row.try_get(24)?, }) } +/// Record on the run `import_id` that one of its conversation files was read +/// from a backup made at `backup_taken_at`, in the form a message's timestamp +/// takes. The run keeps the newest such date, so a run that read two backups +/// shows the later. +/// +/// # Errors +/// +/// Returns an error when the update fails. +pub async fn note_backup_taken_at( + conn: &mut SqliteConnection, + import_id: i64, + backup_taken_at: &str, +) -> Result<()> { + sqlx::query( + "UPDATE imports SET backup_taken_at = $2 + WHERE id = $1 AND (backup_taken_at IS NULL OR backup_taken_at < $2)", + ) + .bind(import_id) + .bind(backup_taken_at) + .execute(&mut *conn) + .await?; + Ok(()) +} + /// Load an import row owned by `account_id`, or error. pub async fn get_owned_import( conn: &mut SqliteConnection, @@ -1238,7 +1266,8 @@ pub async fn detach_from_account( .await?; sqlx::query( "UPDATE imports SET username = $2, deletion_entry_id = $3, form_json = NULL, run_dir = NULL, - source_fingerprint = NULL, source_identities = NULL, summary_json = NULL + source_fingerprint = NULL, source_identities = NULL, summary_json = NULL, + backup_taken_at = NULL WHERE account_id = $1", ) .bind(account_id) diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index 04f6c97e4..6dffa1876 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -229,6 +229,9 @@ pub struct StagingMessage<'a> { pub sort_order: i64, /// Import run that staged the row. pub import_id: Option, + /// When the backup the row was read from was made, in the form + /// `timestamp` takes; `None` when its file did not say. + pub backup_taken_at: Option<&'a str>, } /// One attachment row as the import stages it: the stored blob's digest, @@ -341,7 +344,7 @@ const TAPBACK_COLUMNS: &[&str] = &[ ]; /// Bind counts, in lockstep with the `INSERT` column lists below. -const MESSAGE_BIND_COLUMNS: usize = 18; +const MESSAGE_BIND_COLUMNS: usize = 19; const ATTACHMENT_BIND_COLUMNS: usize = ATTACHMENT_COLUMNS.len(); const TAPBACK_BIND_COLUMNS: usize = TAPBACK_COLUMNS.len(); const EARLIER_VERSION_BIND_COLUMNS: usize = 4; @@ -368,7 +371,7 @@ pub async fn insert_messages( INSERT INTO staging_messages ( conversation_id, account_id, source, guid, timestamp, is_from_me, sender_handle_id, owner_handle_id, service, subject, body, is_announcement, is_reply, - reply_to_guid, reply_to_part, deletion, sort_order, import_id + reply_to_guid, reply_to_part, deletion, sort_order, import_id, backup_taken_at ) VALUES {} ON CONFLICT DO NOTHING RETURNING id, sort_order @@ -395,7 +398,8 @@ pub async fn insert_messages( .bind(row.reply_to_part) .bind(row.deletion.map(message_ir::Deletion::as_str)) .bind(row.sort_order) - .bind(row.import_id); + .bind(row.import_id) + .bind(row.backup_taken_at); } let returned = q.fetch_all(&mut *conn).await?; let mut by_sort = HashMap::with_capacity(returned.len()); @@ -554,10 +558,104 @@ pub async fn staged_message_id( .await?) } +/// What a second copy of a staged message from the same import decides +/// when its backup and the staged row's both have a date: everything the +/// later backup says about the message's mark and text. +pub struct StagedCopy<'a> { + /// The copy's text. + pub body: Option<&'a str>, + /// The copy's mark, or `None` for none. + pub deletion: Option, + /// The copy's earlier versions, in the order its file listed them. + pub versions: &'a [crate::models::EarlierVersionRecord], + /// When the copy's backup was made. + pub backup_taken_at: &'a str, +} + +/// When the backup the staged message `staged` was read from was made, or +/// `None` when its file did not say. +/// +/// # Errors +/// +/// Returns an error when the query fails. +pub async fn staged_backup_taken_at( + conn: &mut SqliteConnection, + staged: i64, +) -> Result> { + Ok( + sqlx::query_scalar("SELECT backup_taken_at FROM staging_messages WHERE id = $1") + .bind(staged) + .fetch_one(&mut *conn) + .await?, + ) +} + +/// Give the staged message `staged` everything `copy`, another copy of it +/// from a later backup in the same import, says about its mark and text: +/// its text, its earlier versions, its mark or no mark, and its backup's +/// date. The caller has compared the two backups' dates +/// ([`later_backup_sql`]'s rule), so the copy staged first no longer +/// decides whatever its age (#1741, #1804). +/// +/// # Errors +/// +/// Returns an error when a statement fails. +pub async fn take_staged_copy_from_later_backup( + conn: &mut SqliteConnection, + staged: i64, + copy: &StagedCopy<'_>, +) -> Result<()> { + sqlx::query( + "UPDATE staging_messages SET body = $1, deletion = $2, backup_taken_at = $3 WHERE id = $4", + ) + .bind(copy.body) + .bind(copy.deletion.map(message_ir::Deletion::as_str)) + .bind(copy.backup_taken_at) + .bind(staged) + .execute(&mut *conn) + .await?; + sqlx::query("DELETE FROM staging_message_versions WHERE message_id = $1") + .bind(staged) + .execute(&mut *conn) + .await?; + let rows: Vec> = copy + .versions + .iter() + .map(|version| StagingEarlierVersion::from_record(staged, version)) + .collect(); + insert_earlier_versions(conn, &rows).await?; + Ok(()) +} + +/// Give the staged message `staged` the mark `deletion` of another copy of +/// it from the same import, when one of the two backups has no date: a +/// copy that carries a mark adds it, and one with none leaves the staged +/// mark, as [`promote_deletion_marks`] does for a stored message. Returns +/// whether the mark changed. +/// +/// # Errors +/// +/// Returns an error when the update fails. +pub async fn add_staged_copy_mark( + conn: &mut SqliteConnection, + staged: i64, + deletion: message_ir::Deletion, +) -> Result { + let done = sqlx::query( + "UPDATE staging_messages SET deletion = $1 WHERE id = $2 AND deletion IS NOT $1", + ) + .bind(deletion.as_str()) + .bind(staged) + .execute(&mut *conn) + .await?; + Ok(done.rows_affected() == 1) +} + /// Give the staged message `staged` the text `body` and the earlier /// versions `versions` of another copy of it from the same import, when /// that copy records a later edit ([`later_edit_sql`]). Returns whether it -/// did. +/// did. The caller uses it only when one of the two backups has no date; +/// otherwise [`take_staged_copy_from_later_backup`] decides. /// /// Staging keeps one row per guid and skips a second copy, so without this /// the copy staged first counted whatever its age: one import of an @@ -976,12 +1074,13 @@ const INSERT_MESSAGES_FROM_STAGING: &str = r" INSERT INTO messages ( conversation_id, account_id, source, guid, timestamp, is_from_me, sender_handle_id, owner_handle_id, service, subject, body, is_announcement, is_reply, - reply_to_guid, reply_to_part, deletion, sort_order, import_id + reply_to_guid, reply_to_part, deletion, sort_order, import_id, backup_taken_at ) SELECT cm.prod_id, sm.account_id, sm.source, sm.guid, sm.timestamp, sm.is_from_me, sm.sender_handle_id, sm.owner_handle_id, sm.service, sm.subject, sm.body, sm.is_announcement, sm.is_reply, - sm.reply_to_guid, sm.reply_to_part, sm.deletion, sm.sort_order, sm.import_id + sm.reply_to_guid, sm.reply_to_part, sm.deletion, sm.sort_order, sm.import_id, + sm.backup_taken_at FROM staging_messages sm JOIN _promote_conv_map cm ON cm.staging_id = sm.conversation_id WHERE sm.account_id = $1 @@ -1133,31 +1232,76 @@ pub async fn write_message_map( Ok(()) } +/// Whether a staged copy of a message comes from a later backup than the +/// copy held, as an SQL expression over the two backups' dates `staged` and +/// `held`: true when both have a date and the staged one is later, false +/// when both have one and it is not, and NULL when either has none. NULL +/// means the dates cannot decide, and the caller falls back on the rule for +/// files without a date. The one rule for which of two copies of a message +/// from one source is the later backup, for a stored message +/// ([`promote_deletion_marks`], [`write_edit_map`]) and, in Rust, for two +/// copies staged in one import (`imports_api::staging`). +/// +/// Both dates have one fixed whole-second UTC form +/// (`models::conversation_from_ir`), so the text orders as the time. +fn later_backup_sql(staged: &str, held: &str) -> String { + format!("CASE WHEN {staged} IS NOT NULL AND {held} IS NOT NULL THEN {staged} > {held} END") +} + /// Give each stored message the mark its staged row carries, through /// `_promote_msg_map`. An append-mode import skips a message production /// already holds, so a message imported before it was deleted or unsent -/// takes the mark only here. A staged row with no mark leaves the stored -/// mark as it is: a backup that does not say a message was deleted does not -/// say it was restored. Returns how many messages changed. +/// takes the mark only here. +/// +/// When both the staged row's backup and the stored message's have a date +/// ([`later_backup_sql`]), the later backup decides: a staged row from a +/// later backup gives its mark or clears the one held, and one from an +/// earlier or the same backup changes nothing. When either has no date, a +/// staged row with no mark leaves the stored mark as it is: a backup that +/// does not say a message was deleted does not say it was restored, and +/// nothing says which backup is newer. Returns how many messages changed. /// /// # Errors /// /// Returns an error when the update fails. pub async fn promote_deletion_marks(conn: &mut SqliteConnection) -> Result { - Ok(sqlx::query( + let sql = format!( r" UPDATE messages SET deletion = sm.deletion FROM _promote_msg_map mm JOIN staging_messages sm ON sm.id = mm.staging_id WHERE messages.id = mm.prod_id - AND sm.deletion IS NOT NULL AND messages.deletion IS NOT sm.deletion + AND COALESCE({later}, sm.deletion IS NOT NULL) ", - ) - .execute(&mut *conn) - .await? - .rows_affected()) + later = later_backup_sql("sm.backup_taken_at", "messages.backup_taken_at"), + ); + Ok(sqlx::query(&sql).execute(&mut *conn).await?.rows_affected()) +} + +/// Give each stored message the backup date of its staged row when that +/// row comes from a later backup ([`later_backup_sql`]), after its mark +/// and text were taken from that row, so a later import compares with the +/// backup the message now reflects. A stored message from a file with no +/// date keeps none. Returns how many messages changed. +/// +/// # Errors +/// +/// Returns an error when the update fails. +pub async fn promote_backup_dates(conn: &mut SqliteConnection) -> Result { + let sql = format!( + r" + UPDATE messages + SET backup_taken_at = sm.backup_taken_at + FROM _promote_msg_map mm + JOIN staging_messages sm ON sm.id = mm.staging_id + WHERE messages.id = mm.prod_id + AND COALESCE({later}, 0) + ", + later = later_backup_sql("sm.backup_taken_at", "messages.backup_taken_at"), + ); + Ok(sqlx::query(&sql).execute(&mut *conn).await?.rows_affected()) } /// Whether one copy of a message records a later edit than another, as an @@ -1178,7 +1322,8 @@ pub async fn promote_deletion_marks(conn: &mut SqliteConnection) -> Result /// The newest earlier version is the edit before the last one: the time of /// a part's last edit is recorded nowhere. So a later backup that differs /// only by an unsent part, or by one edit after an unsend, can read as not -/// later (#1804). +/// later (#1804). The rule is used only where one of the two backups has no +/// date; where both have one, [`later_backup_sql`] decides instead. /// /// `edited_at` is one fixed whole-second UTC form on both sides /// (`models::earlier_version_from_ir`), so the text orders as the time. @@ -1193,20 +1338,34 @@ fn later_edit_sql(n: &str, newest: &str, held_n: &str, held_newest: &str) -> Str } /// Write `_promote_edit_map`: each message production held before this -/// promotion, those at or below `messages_before`, whose staged row records -/// a later edit than the message holds ([`later_edit_sql`]). Returns how -/// many messages it names. +/// promotion, those at or below `messages_before`, whose staged row gives +/// it a later text. Returns how many messages it names. +/// +/// When both backups have a date ([`later_backup_sql`]), a staged row from +/// a later backup gives its text and earlier versions whatever their times +/// say, when either differs from what the message holds (#1804); one from +/// an earlier or the same backup gives nothing. When either has no date, +/// the staged row gives them when it records a later edit +/// ([`later_edit_sql`]). /// /// An append skips a message production already holds, so a later backup in /// which it was edited again reaches it only here. A message has one staged /// row: staging keeps one row per guid, the later copy when one import -/// carries two ([`take_later_staged_copy`]). +/// carries two ([`take_staged_copy_from_later_backup`], [`take_later_staged_copy`]). /// /// # Errors /// /// Returns an error when a statement fails. pub async fn write_edit_map(conn: &mut SqliteConnection, messages_before: i64) -> Result { reset_id_map(conn, "_promote_edit_map").await?; + // A version list as one value, in the order its rows were written, so + // two lists compare whole. + let versions = |table: &str, id: &str| { + format!( + "(SELECT json_group_array(json_array(v.part_index, v.text, v.edited_at) ORDER BY v.id) \ + FROM {table} v WHERE v.message_id = {id})" + ) + }; let sql = format!( r" INSERT INTO _promote_edit_map (staging_id, prod_id) @@ -1215,6 +1374,10 @@ pub async fn write_edit_map(conn: &mut SqliteConnection, messages_before: i64) - SELECT mm.staging_id, mm.prod_id, + {later_backup} AS later_backup, + sm.body IS NOT m.body + OR {staged_versions} IS NOT {held_versions} AS differs, + sv.message_id IS NOT NULL AS has_versions, sv.n, sv.newest, (SELECT COUNT(*) FROM message_versions v WHERE v.message_id = mm.prod_id) @@ -1222,16 +1385,24 @@ pub async fn write_edit_map(conn: &mut SqliteConnection, messages_before: i64) - (SELECT MAX(v.edited_at) FROM message_versions v WHERE v.message_id = mm.prod_id) AS held_newest FROM _promote_msg_map mm - JOIN ( + JOIN staging_messages sm ON sm.id = mm.staging_id + JOIN messages m ON m.id = mm.prod_id + LEFT JOIN ( SELECT message_id, COUNT(*) AS n, MAX(edited_at) AS newest FROM staging_message_versions GROUP BY message_id ) sv ON sv.message_id = mm.staging_id WHERE mm.prod_id <= $1 ) - WHERE {later} + WHERE CASE + WHEN later_backup IS NOT NULL THEN later_backup AND differs + ELSE has_versions AND {later_edit} + END ", - later = later_edit_sql("n", "newest", "held_n", "held_newest"), + later_backup = later_backup_sql("sm.backup_taken_at", "m.backup_taken_at"), + staged_versions = versions("staging_message_versions", "mm.staging_id"), + held_versions = versions("message_versions", "mm.prod_id"), + later_edit = later_edit_sql("n", "newest", "held_n", "held_newest"), ); Ok(sqlx::query(&sql) .bind(messages_before) diff --git a/crates/server/server/src/db/staging/tests.rs b/crates/server/server/src/db/staging/tests.rs index f038cac7e..3bc608dfb 100644 --- a/crates/server/server/src/db/staging/tests.rs +++ b/crates/server/server/src/db/staging/tests.rs @@ -44,6 +44,7 @@ async fn reset_for_account_leaves_other_accounts() { deletion: None, sort_order: 0, import_id: None, + backup_taken_at: None, }], ) .await diff --git a/crates/server/server/src/imports_api/mod.rs b/crates/server/server/src/imports_api/mod.rs index 444a046c0..5ad274a3d 100644 --- a/crates/server/server/src/imports_api/mod.rs +++ b/crates/server/server/src/imports_api/mod.rs @@ -919,6 +919,10 @@ pub(crate) struct ImportRunSummary { pub(crate) source_fingerprint: serde_json::Value, /// Addresses the backup's device sent from (JSON array), or null. pub(crate) source_identities: serde_json::Value, + /// When the backup the run read was made, UTC: the newest date its + /// conversation files name (`export.backup_taken_at_unix_ms`). Null until + /// a file that names one is imported, and for a run whose files name none. + pub(crate) backup_taken_at: Option, /// What the person approved at the last Review they passed, which /// `PATCH /v1/imports/{id}` writes with its `summary`. A cancelled run /// keeps it. A run that completed or failed holds instead the final @@ -962,6 +966,7 @@ impl From for ImportRunSummary { form: crate::db::imports::json_column(row.form_json), source_fingerprint: crate::db::imports::json_column(row.source_fingerprint), source_identities: crate::db::imports::json_column(row.source_identities), + backup_taken_at: row.backup_taken_at, summary: crate::db::imports::json_column(row.summary_json), issue_count: listed.issue_count, note_count: listed.note_count, diff --git a/crates/server/server/src/imports_api/promote.rs b/crates/server/server/src/imports_api/promote.rs index fd0314e3a..7a7575f8f 100644 --- a/crates/server/server/src/imports_api/promote.rs +++ b/crates/server/server/src/imports_api/promote.rs @@ -276,6 +276,17 @@ impl Promote<'_> { words(as_count(unindexed), "1 search entry", "{n} search entries"), ), ); + + let phase = Self::begin("Recording which backup each changed message came from…"); + let dated = staging::promote_backup_dates(self.tx).await?; + self.done( + phase, + words( + dated, + "1 message now holds a later backup", + "{n} messages now hold a later backup", + ), + ); Ok(messages_before) } diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index 99b0592b1..84bf6d448 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -12,7 +12,7 @@ use crate::db::handles::{ HandleIdCache, handle_type_of, upsert_handle_row, upsert_handle_row_cached, }; use crate::db::staging::{ - self as db_staging, StagingAttachment, StagingConversation, StagingEarlierVersion, + self as db_staging, StagedCopy, StagingAttachment, StagingConversation, StagingEarlierVersion, StagingMessage, StagingMessageKey, StagingTapback, }; use crate::import_media; @@ -367,6 +367,9 @@ struct StagedConversation { group_title: Option, participants: Vec, source: String, + /// When the backup the file was read from was made, in the form a + /// message's timestamp takes; `None` when the file does not say. + backup_taken_at: Option, } impl StagedConversation { @@ -384,6 +387,7 @@ impl StagedConversation { .map(|p| (p.handle, p.name_alias, p.handle_type)) .collect(), source, + backup_taken_at: record.backup_taken_at, } } } @@ -546,6 +550,12 @@ impl FileStaging<'_> { ) .await?; counts.conversations = 1; + if let (Some(import_id), Some(backup_taken_at)) = ( + self.stmts.import_id, + conversation.backup_taken_at.as_deref(), + ) { + crate::db::imports::note_backup_taken_at(self.tx, import_id, backup_taken_at).await?; + } for participant in conversation.participants { insert_participant( @@ -589,8 +599,11 @@ impl FileStaging<'_> { self.tx, self.stmts, &mut counts, - conversation_id, - &conversation.source, + StagedSource { + conversation_id, + source: &conversation.source, + backup_taken_at: conversation.backup_taken_at.as_deref(), + }, self.opts.assets_dir, chunk, ) @@ -821,21 +834,29 @@ struct PendingStagingMessage { sort_order: i64, } +/// Where one conversation's message rows are staged from: its staging +/// conversation, its source, and when its backup was made. +#[derive(Clone, Copy)] +struct StagedSource<'a> { + conversation_id: i64, + source: &'a str, + backup_taken_at: Option<&'a str>, +} + /// Bulk-insert one chunk of message rows, then their attachments, tapbacks /// and earlier versions keyed by the ids returned. async fn flush_staging_message_chunk( tx: &mut SqliteConnection, stmts: &mut StagingInserts, counts: &mut ImportCounts, - conversation_id: i64, - source: &str, + staged_source: StagedSource<'_>, assets_dir: &Path, chunk: &[PendingStagingMessage], ) -> Result<()> { if chunk.is_empty() { return Ok(()); } - let mut by_sort = insert_message_rows(tx, stmts, conversation_id, source, chunk).await?; + let mut by_sort = insert_message_rows(tx, stmts, staged_source, chunk).await?; let mut att_rows = Vec::new(); let mut tap_rows = Vec::new(); @@ -872,47 +893,78 @@ async fn flush_staging_message_chunk( counts.tapbacks += db_staging::insert_tapbacks(tx, &tap_rows).await?; db_staging::insert_earlier_versions(tx, &version_rows).await?; for row in copies { - add_staged_copy(tx, stmts, counts, source, assets_dir, row).await?; + add_staged_copy(tx, stmts, counts, staged_source, assets_dir, row).await?; } Ok(()) } /// Give the message staged under `row`'s guid what `row`, another copy of -/// it from the same import, adds: its text and earlier versions when it -/// records a later edit, and the attachments and reactions the staged -/// message does not hold yet. One import of two backups then stores what -/// two separate imports of them store (#1806, #1837). The copy's deletion -/// mark is not taken (#1741). +/// it from the same import, adds, by the rules a later import of the copy +/// would follow (`db::staging::promote_deletion_marks`, +/// `db::staging::write_edit_map`): the attachments and reactions the staged +/// message does not hold yet, and its mark and text as follows. When both +/// backups have a date, a copy from a later backup gives its text, earlier +/// versions and mark, mark or no mark, and one from an earlier or the same +/// backup gives neither (#1741, #1804). When either has no date, the copy +/// gives its text and earlier versions when it records a later edit, and +/// its mark when it carries one. One import of two backups then stores +/// what two separate imports of them store, in either file order (#1806, +/// #1837). async fn add_staged_copy( tx: &mut SqliteConnection, stmts: &mut StagingInserts, counts: &mut ImportCounts, - source: &str, + staged_source: StagedSource<'_>, assets_dir: &Path, row: &PendingStagingMessage, ) -> Result<()> { if row.msg.earlier_versions.is_empty() && row.attachments.is_empty() && row.msg.tapbacks.is_empty() + && row.msg.deletion.is_none() + && staged_source.backup_taken_at.is_none() { return Ok(()); } let key = StagingMessageKey { account_id: stmts.account_id, - source, + source: staged_source.source, guid: &row.msg.guid, }; let staged = db_staging::staged_message_id(tx, key) .await? .with_context(|| format!("no staged message holds the copy of {}", row.msg.guid))?; - if !row.msg.earlier_versions.is_empty() { - db_staging::take_later_staged_copy( - tx, - staged, - row.body.as_deref(), - &row.msg.earlier_versions, - ) - .await?; + let held_backup = db_staging::staged_backup_taken_at(tx, staged).await?; + match (staged_source.backup_taken_at, held_backup.as_deref()) { + (Some(copy_backup), Some(held_backup)) => { + if copy_backup > held_backup { + db_staging::take_staged_copy_from_later_backup( + tx, + staged, + &StagedCopy { + body: row.body.as_deref(), + deletion: row.msg.deletion, + versions: &row.msg.earlier_versions, + backup_taken_at: copy_backup, + }, + ) + .await?; + } + } + _ => { + if !row.msg.earlier_versions.is_empty() { + db_staging::take_later_staged_copy( + tx, + staged, + row.body.as_deref(), + &row.msg.earlier_versions, + ) + .await?; + } + if let Some(deletion) = row.msg.deletion { + db_staging::add_staged_copy_mark(tx, staged, deletion).await?; + } + } } let att_rows: Vec = row .attachments @@ -933,16 +985,15 @@ async fn add_staged_copy( async fn insert_message_rows( tx: &mut SqliteConnection, stmts: &StagingInserts, - conversation_id: i64, - source: &str, + staged_source: StagedSource<'_>, chunk: &[PendingStagingMessage], ) -> Result> { let rows: Vec> = chunk .iter() .map(|row| StagingMessage { - conversation_id, + conversation_id: staged_source.conversation_id, account_id: stmts.account_id, - source, + source: staged_source.source, guid: &row.msg.guid, timestamp: &row.msg.timestamp, is_from_me: row.msg.is_from_me as i64, @@ -963,6 +1014,7 @@ async fn insert_message_rows( deletion: row.msg.deletion, sort_order: row.sort_order, import_id: stmts.import_id, + backup_taken_at: staged_source.backup_taken_at, }) .collect(); db_staging::insert_messages(tx, &rows).await diff --git a/crates/server/server/src/imports_api/tests.rs b/crates/server/server/src/imports_api/tests.rs index 231fb9a6d..fb124b95f 100644 --- a/crates/server/server/src/imports_api/tests.rs +++ b/crates/server/server/src/imports_api/tests.rs @@ -5808,3 +5808,5 @@ async fn a_page_of_import_runs_is_read_without_a_statement_per_row() { let owner: Page = runs_page(rows); assert_eq!(owner.items[2].issue_count, 3); } + +mod backup_dates; diff --git a/crates/server/server/src/imports_api/tests/backup_dates.rs b/crates/server/server/src/imports_api/tests/backup_dates.rs new file mode 100644 index 000000000..f5bec1f74 --- /dev/null +++ b/crates/server/server/src/imports_api/tests/backup_dates.rs @@ -0,0 +1,483 @@ +//! Two copies of one message from one source, from two backups of one +//! phone: the copy from the later backup decides the message's mark and +//! text, in one import in either file order and across imports in either +//! order (#1924, #1741, #1804). A file without a backup date keeps the +//! rules for files without one: marks add, and edits compare their times. + +use super::*; + +/// The earlier backup of the phone: 2026-09-01T10:00:00Z. +const EARLIER_BACKUP: i64 = 1_788_256_800_000; +/// The later backup of the phone: 2026-09-30T18:45:12Z. +const LATER_BACKUP: i64 = 1_790_793_912_000; + +/// One backup's copy of the message `g-backup`. +struct Copy<'a> { + /// When the backup was made, or `None` for a file that does not say. + backup: Option, + text: &'a str, + versions: &'a [EarlierVersion], + deletion: Option, +} + +/// A conversation file holding `copy` as the one message `g-backup`, +/// written to `name` under `dir`. +fn backup_file(dir: &Path, name: &str, copy: &Copy<'_>) -> PathBuf { + let header = conversation_header("imessage", "+15555550123").participant("+15555550123", None); + let header = match copy.backup { + Some(ms) => header.backup_taken_at(ms), + None => header, + }; + let line = copy.versions.iter().cloned().fold( + message_line("g-backup", copy.text).sender("+15555550123"), + MessageLine::edit, + ); + let line = match copy.deletion { + Some(deletion) => line.deletion(deletion), + None => line, + }; + write_jsonl(dir, name, &format!("{header}\n{line}\n")) +} + +/// What the server holds of `g-backup`: its text, its mark, its earlier +/// versions as `(part_index, text)`, and its backup's date. +#[derive(Debug, PartialEq)] +struct Held { + text: String, + deletion: Option, + versions: Vec<(i64, String)>, + backup_taken_at: Option, +} + +/// The [`Held`] of `g-backup` in the database at `db`. +async fn held(db: &Path) -> Held { + let (_pool, mut conn) = open_verify(db).await; + let (text, deletion, backup_taken_at): (String, Option, Option) = + sqlx::query_as( + "SELECT body, deletion, backup_taken_at FROM messages WHERE guid = 'g-backup'", + ) + .fetch_one(&mut *conn) + .await + .unwrap(); + let versions = sqlx::query_as( + "SELECT v.part_index, v.text FROM message_versions v + JOIN messages m ON m.id = v.message_id + WHERE m.guid = 'g-backup' ORDER BY v.id", + ) + .fetch_all(&mut *conn) + .await + .unwrap(); + Held { + text, + deletion, + versions, + backup_taken_at, + } +} + +/// Import `files` into `db` in one import, appending to what it holds. +async fn import(db: &Path, assets: &Path, root: &Path, files: &[PathBuf]) { + import_jsonl_files(db, files, &edit_options(assets, root, false)) + .await + .unwrap(); +} + +/// What `files` give `g-backup`: imported in one import in the order +/// given, in one import in the other order, one import after another in +/// the order given, and one after another in the other order. The four +/// must agree, so each is returned to compare. +async fn every_order(tmp: &Path, label: &str, files: [&PathBuf; 2]) -> [Held; 4] { + let assets = tmp.join("assets"); + let [a, b] = files; + let mut out = Vec::new(); + for (name, batches) in [ + ("together", vec![vec![a.clone(), b.clone()]]), + ("together-reversed", vec![vec![b.clone(), a.clone()]]), + ("apart", vec![vec![a.clone()], vec![b.clone()]]), + ("apart-reversed", vec![vec![b.clone()], vec![a.clone()]]), + ] { + let db = tmp.join(format!("{label}-{name}.db")); + for batch in batches { + import(&db, &assets, tmp, &batch).await; + } + out.push(held(&db).await); + } + out.try_into().unwrap() +} + +/// The stored date of [`LATER_BACKUP`], in the form a timestamp takes. +const LATER_BACKUP_AT: &str = "2026-09-30T18:45:12Z"; +/// The stored date of [`EARLIER_BACKUP`]. +const EARLIER_BACKUP_AT: &str = "2026-09-01T10:00:00Z"; + +/// Backup A marks the message Deleted in the source app; backup B, made +/// later, after the person recovered it, does not. Whichever is imported +/// first, and whether the two arrive in one import or two, the message +/// is unmarked, and swapping the dates keeps the mark. +#[tokio::test] +async fn the_later_backup_decides_the_deletion_mark_in_every_order() { + let tmp = TempDir::new().unwrap(); + let file = |name: &str, backup: i64, deletion: Option| { + backup_file( + tmp.path(), + name, + &Copy { + backup: Some(backup), + text: "recovered later", + versions: &[], + deletion, + }, + ) + }; + let marked = file( + "marked-earlier.jsonl", + EARLIER_BACKUP, + Some(Deletion::DeletedInSourceApp), + ); + let recovered = file("recovered-later.jsonl", LATER_BACKUP, None); + for held in every_order(tmp.path(), "recovered", [&marked, &recovered]).await { + assert_eq!(held.deletion, None, "{held:?}"); + assert_eq!(held.backup_taken_at.as_deref(), Some(LATER_BACKUP_AT)); + } + + let marked = file( + "marked-later.jsonl", + LATER_BACKUP, + Some(Deletion::DeletedInSourceApp), + ); + let unmarked = file("unmarked-earlier.jsonl", EARLIER_BACKUP, None); + for held in every_order(tmp.path(), "deleted", [&unmarked, &marked]).await { + assert_eq!( + held.deletion.as_deref(), + Some("deleted_in_source_app"), + "{held:?}" + ); + assert_eq!(held.backup_taken_at.as_deref(), Some(LATER_BACKUP_AT)); + } +} + +/// The scenario of #1804: backup A lists part 1's earlier versions +/// [x@t0, y@t100]; backup B, made later, after part 1 was unsent (which +/// drops its versions) and part 0 was edited, lists part 0's [a@t0] only. +/// B's newest version is older than A's, so the version times say A is +/// later; the backups' dates say B, and B's text and versions are kept in +/// every order. With the dates swapped, A's are. +#[tokio::test] +async fn the_later_backup_decides_the_text_whatever_the_version_times_say() { + let tmp = TempDir::new().unwrap(); + let t0 = 1_426_183_462_000; + let a_versions = [edit_version(1, "x", t0), edit_version(1, "y", t0 + 100_000)]; + let b_versions = [edit_version(0, "a", t0)]; + let a = |backup| Copy { + backup: Some(backup), + text: "a z", + versions: &a_versions, + deletion: None, + }; + let b = |backup| Copy { + backup: Some(backup), + text: "b", + versions: &b_versions, + deletion: None, + }; + + let a_earlier = backup_file(tmp.path(), "a-earlier.jsonl", &a(EARLIER_BACKUP)); + let b_later = backup_file(tmp.path(), "b-later.jsonl", &b(LATER_BACKUP)); + for held in every_order(tmp.path(), "b-later", [&a_earlier, &b_later]).await { + assert_eq!( + held, + Held { + text: "b".into(), + deletion: None, + versions: vec![(0, "a".into())], + backup_taken_at: Some(LATER_BACKUP_AT.into()), + } + ); + } + + let a_later = backup_file(tmp.path(), "a-later.jsonl", &a(LATER_BACKUP)); + let b_earlier = backup_file(tmp.path(), "b-earlier.jsonl", &b(EARLIER_BACKUP)); + for held in every_order(tmp.path(), "a-later", [&a_later, &b_earlier]).await { + assert_eq!( + held, + Held { + text: "a z".into(), + deletion: None, + versions: vec![(1, "x".into()), (1, "y".into())], + backup_taken_at: Some(LATER_BACKUP_AT.into()), + } + ); + } +} + +/// An append of the older backup after the newer one changes nothing: +/// not the mark, not the text, not the earlier versions, not the date, +/// even though the older copy carries a mark and lists more versions. +#[tokio::test] +async fn an_append_of_the_older_backup_after_the_newer_changes_nothing() { + let tmp = TempDir::new().unwrap(); + let assets = tmp.path().join("assets"); + let db = tmp.path().join("messagecrate.db"); + let t0 = 1_426_183_462_000; + let newer = backup_file( + tmp.path(), + "newer.jsonl", + &Copy { + backup: Some(LATER_BACKUP), + text: "see you at seven", + versions: &[edit_version(0, "see you at six", t0)], + deletion: None, + }, + ); + let older = backup_file( + tmp.path(), + "older.jsonl", + &Copy { + backup: Some(EARLIER_BACKUP), + text: "see you at eight", + versions: &[ + edit_version(0, "see you at six", t0), + edit_version(0, "see you at seven", t0 + 60_000), + ], + deletion: Some(Deletion::Unsent), + }, + ); + import(&db, &assets, tmp.path(), std::slice::from_ref(&newer)).await; + let before = held(&db).await; + assert_eq!(before.text, "see you at seven"); + import(&db, &assets, tmp.path(), &[older]).await; + assert_eq!(held(&db).await, before); +} + +/// Where either file has no backup date, the rules for files without one +/// hold. In one import, a second file's mark is added and its edit is +/// taken when its versions are newer; across imports, a file without the +/// mark leaves it, and a stored message from an undated file keeps no date +/// when a dated file later gives it nothing. +#[tokio::test] +async fn without_a_backup_date_marks_add_and_edits_compare_their_times() { + let tmp = TempDir::new().unwrap(); + let t0 = 1_426_183_462_000; + let unmarked_undated = backup_file( + tmp.path(), + "unmarked-undated.jsonl", + &Copy { + backup: None, + text: "see you at seven", + versions: &[edit_version(0, "see you at six", t0)], + deletion: None, + }, + ); + let marked_dated = backup_file( + tmp.path(), + "marked-dated.jsonl", + &Copy { + backup: Some(EARLIER_BACKUP), + text: "see you at six", + versions: &[], + deletion: Some(Deletion::Unsent), + }, + ); + // Undated against dated: the mark adds, and the copy with the later edit + // keeps its text, whichever is the dated one. + for held in every_order(tmp.path(), "mixed", [&unmarked_undated, &marked_dated]).await { + assert_eq!(held.deletion.as_deref(), Some("unsent"), "{held:?}"); + assert_eq!(held.text, "see you at seven", "{held:?}"); + assert_eq!(held.versions, vec![(0, "see you at six".into())]); + } + + // Both undated, the mark in the second file of one import: the mark + // adds rather than being lost to the copy staged first. + let marked_undated = backup_file( + tmp.path(), + "marked-undated.jsonl", + &Copy { + backup: None, + text: "see you at seven", + versions: &[edit_version(0, "see you at six", t0)], + deletion: Some(Deletion::DeletedInSourceApp), + }, + ); + for held in every_order(tmp.path(), "undated", [&unmarked_undated, &marked_undated]).await { + assert_eq!( + held.deletion.as_deref(), + Some("deleted_in_source_app"), + "{held:?}" + ); + assert_eq!(held.backup_taken_at, None); + } +} + +/// Two backups in one import give the message the later backup's text, and +/// so the later backup's duplicate flag: with another source holding the +/// later text, the dedupe hides one of the two, in either file order, and +/// with the earlier text winning it would hide neither. The earlier backup's +/// newest earlier version is the newer one, so the version times alone pick +/// the earlier text. +#[tokio::test] +async fn the_later_backup_decides_the_duplicate_flag() { + let tmp = TempDir::new().unwrap(); + let assets = tmp.path().join("assets"); + let t0 = 1_426_183_462_000; + let earlier = backup_file( + tmp.path(), + "earlier.jsonl", + &Copy { + backup: Some(EARLIER_BACKUP), + text: "see you at six", + versions: &[edit_version(1, "bring cake", t0 + 100_000)], + deletion: None, + }, + ); + let later = backup_file( + tmp.path(), + "later.jsonl", + &Copy { + backup: Some(LATER_BACKUP), + text: "see you at seven", + versions: &[edit_version(0, "see you at six", t0)], + deletion: None, + }, + ); + let header = + conversation_header("sms-backup-restore", "+15555550123").participant("+15555550123", None); + let sms = write_jsonl( + tmp.path(), + "sms.jsonl", + &format!( + "{header}\n{}\n", + message_line("g-sms", "see you at seven") + .sender("+15555550123") + .sms() + ), + ); + for (name, files) in [ + ("earlier-first.db", [earlier.clone(), later.clone()]), + ("later-first.db", [later.clone(), earlier.clone()]), + ] { + let db = tmp.path().join(name); + let options = edit_options(&assets, tmp.path(), true); + import_jsonl_files(&db, &files, &options).await.unwrap(); + let sms_options = ImportOptions::fixed(FixedImportArgs { + assets_dir: &assets, + asset_root: tmp.path(), + mode: ImportMode::Append, + source: "sms-backup-restore", + account_id: TEST_ACCOUNT, + fill_content_keys: true, + import_id: None, + }); + import_jsonl_files(&db, std::slice::from_ref(&sms), &sms_options) + .await + .unwrap(); + let (_pool, mut conn) = open_verify(&db).await; + crate::dedupe::dedupe_cross_source(&mut conn, TEST_ACCOUNT, None, 2) + .await + .unwrap(); + let hidden: i64 = + sqlx::query_scalar("SELECT COUNT(*) FROM messages WHERE duplicate_of IS NOT NULL") + .fetch_one(&mut *conn) + .await + .unwrap(); + assert_eq!( + hidden, 1, + "{name}: the two copies of the later text are one" + ); + } +} + +/// The Import Run says when the backup it read was made: the newest date +/// its files name, as `backup_taken_at`, and null for a run whose files +/// name none. +#[tokio::test] +async fn the_import_run_says_when_its_backup_was_made() { + let (state, _fixture, token) = importer().await; + let batch = |backup: Option, guid: &str| { + let header = + conversation_header("imessage", "+15555550123").participant("+15555550123", None); + let header = match backup { + Some(ms) => header.backup_taken_at(ms), + None => header, + }; + format!( + "{header}\n{}\n", + message_line(guid, "hello").sender("+15555550123") + ) + }; + let run = |body: Vec| { + let state = state.clone(); + let token = token.clone(); + async move { + let (_, created): (String, serde_json::Value) = post_created_json( + &state, + "/v1/imports", + &token, + serde_json::json!({ "source": "imessage", "mode": "append" }), + ) + .await; + let id = created["id"].as_i64().unwrap(); + assert_eq!(created["backup_taken_at"], serde_json::Value::Null); + for body in body { + let (status, text) = crate::test_support::post_raw( + &state, + &format!("/v1/imports/{id}/batches"), + &token, + "application/jsonl", + body, + ) + .await; + assert_eq!(status, axum::http::StatusCode::OK, "{text}"); + } + let read: serde_json::Value = + get_json(&state, &format!("/v1/imports/{id}"), &token).await; + let _: serde_json::Value = post_json( + &state, + &format!("/v1/imports/{id}/complete"), + &token, + serde_json::json!({ "status": "completed" }), + ) + .await; + read["backup_taken_at"].clone() + } + }; + assert_eq!( + run(vec![ + batch(Some(LATER_BACKUP), "g-1"), + batch(Some(EARLIER_BACKUP), "g-2"), + batch(None, "g-3"), + ]) + .await, + serde_json::json!(LATER_BACKUP_AT) + ); + assert_eq!( + run(vec![batch(Some(EARLIER_BACKUP), "g-4")]).await, + serde_json::json!(EARLIER_BACKUP_AT) + ); + assert_eq!(run(vec![batch(None, "g-5")]).await, serde_json::Value::Null); +} + +/// A message read back through the API carries the date of the backup that +/// decided it, which an Export Run writes back into the conversation file. +#[tokio::test] +async fn a_message_read_back_carries_its_backup_date() { + let (state, _fixture, token) = importer().await; + let header = conversation_header("imessage", "+15555550123") + .participant("+15555550123", None) + .backup_taken_at(LATER_BACKUP); + import_one_batch( + &state, + &token, + "imessage", + "append", + format!( + "{header}\n{}\n", + message_line("g-read", "hello").sender("+15555550123") + ), + ) + .await; + let page: serde_json::Value = get_json(&state, "/v1/messages", &token).await; + assert_eq!( + page["items"][0]["backup_taken_at"], + serde_json::json!(LATER_BACKUP_AT) + ); +} diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index 927fea3dd..12d28d4e7 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -40,6 +40,9 @@ pub struct ConversationRecord { pub participants: Vec, /// IR `export.source` — used as `messages.source` for directory import. pub export_source: Option, + /// When the backup the file was read from was made, in the form a + /// message's timestamp takes; `None` when the file does not say. + pub backup_taken_at: Option, } impl ConversationRecord { @@ -231,9 +234,12 @@ pub fn parse_ir_lines( line: line_no, detail: format!("the conversation header is not valid: {e}"), })?; - out.push(ExportRecord::Conversation(conversation_from_ir( - &header, line_no, - ))); + let conversation = + conversation_from_ir(&header, line_no).map_err(|e| ImportFailure::Invalid { + line: line_no, + detail: format!("{e:#}"), + })?; + out.push(ExportRecord::Conversation(conversation)); header_owner = header.export.owner_identity.as_deref().and_then(nonempty); saw_header = true; } else { @@ -294,7 +300,12 @@ fn is_ir_header(value: &Value) -> bool { } /// Map a JSON Lines header onto the server's conversation record. -fn conversation_from_ir(header: &ConversationHeader, line: usize) -> ConversationRecord { +/// +/// # Errors +/// +/// Returns an error when the backup date is outside the times a timestamp +/// can hold. +fn conversation_from_ir(header: &ConversationHeader, line: usize) -> Result { let export_source = { let s = header.export.source.trim(); if s.is_empty() { @@ -303,7 +314,15 @@ fn conversation_from_ir(header: &ConversationHeader, line: usize) -> Conversatio Some(s.to_string()) } }; - ConversationRecord { + let backup_taken_at = header + .export + .backup_taken_at_unix_ms + .map(|ms| { + format_utc_timestamp(ms.div_euclid(1000)) + .with_context(|| format!("unrepresentable backup_taken_at_unix_ms {ms}")) + }) + .transpose()?; + Ok(ConversationRecord { line, chat_identifier: header.conversation.chat_identifier.clone(), // Platform identity for handles (phone | whatsapp), not SMS/iMessage/RCS. @@ -326,7 +345,8 @@ fn conversation_from_ir(header: &ConversationHeader, line: usize) -> Conversatio .filter_map(participant_from_ir) .collect(), export_source, - } + backup_taken_at, + }) } /// Map one IR message onto the server's message record. `header_owner` is the diff --git a/crates/server/server/src/test_support/lines.rs b/crates/server/server/src/test_support/lines.rs index 6b4ff3bfb..4a95458b5 100644 --- a/crates/server/server/src/test_support/lines.rs +++ b/crates/server/server/src/test_support/lines.rs @@ -30,6 +30,7 @@ pub fn conversation_header(source: &str, chat_identifier: &str) -> ConversationH tool_version: "0".to_string(), owner_identity: None, owner_display_name: None, + backup_taken_at_unix_ms: None, }, conversation: message_ir::ConversationMeta { chat_identifier: chat_identifier.to_string(), @@ -49,6 +50,13 @@ impl ConversationHeaderLine { self } + /// When the backup the batch was read from was made, in Unix + /// milliseconds. + pub fn backup_taken_at(mut self, unix_ms: i64) -> Self { + self.0.export.backup_taken_at_unix_ms = Some(unix_ms); + self + } + /// A group conversation. pub fn group(mut self) -> Self { self.0.conversation.conversation_type = message_ir::IrConversationType::Group; diff --git a/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl b/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl index 05736d0ca..1e9fb24e9 100644 --- a/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl @@ -1,4 +1,4 @@ -{"schema_version":10,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578308040000,"last_timestamp_unix_ms":1578308160000}}} +{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578308040000,"last_timestamp_unix_ms":1578308160000}}} {"guid":"guid-16","timestamp_unix_ms":1578308040000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Delete me","attachments":[],"deletion":"deleted_in_source_app","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Delete me"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} {"guid":"guid-17","timestamp_unix_ms":1578308100000,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"","attachments":[],"deletion":"unsent","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} {"guid":"guid-18","timestamp_unix_ms":1578308160000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Still here","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Still here"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} diff --git a/crates/server/server/tests/fixtures/apple-messages-edits.jsonl b/crates/server/server/tests/fixtures/apple-messages-edits.jsonl index c9979cd92..2342100ba 100644 --- a/crates/server/server/tests/fixtures/apple-messages-edits.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-edits.jsonl @@ -1,4 +1,4 @@ -{"schema_version":10,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578309000000,"last_timestamp_unix_ms":1578309120000}}} +{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578309000000,"last_timestamp_unix_ms":1578309120000}}} {"guid":"guid-edited-twice","timestamp_unix_ms":1578309000000,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000000},{"part_index":0,"text":"Meet at the museum","edited_at_unix_ms":1578309030000}],"imessage":null,"source":null} {"guid":"guid-edited-final-match","timestamp_unix_ms":1578309060000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"The library opens at nine","attachments":[],"edits":[{"part_index":0,"text":"The library opens at eight","edited_at_unix_ms":1578309060000}],"imessage":null,"source":null} {"guid":"guid-never-edited","timestamp_unix_ms":1578309120000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Nothing changed here","attachments":[],"imessage":null,"source":null} diff --git a/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl b/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl index b5b4d7200..3f2cc0f3d 100644 --- a/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl @@ -1,4 +1,4 @@ -{"schema_version":10,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"chat100","conversation_type":"group","group_title":"Weekend plans","participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"},{"identity":"friend@example.com","display_name":null,"identity_type":"email"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578307860000,"last_timestamp_unix_ms":1578307980000}}} +{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"chat100","conversation_type":"group","group_title":"Weekend plans","participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"},{"identity":"friend@example.com","display_name":null,"identity_type":"email"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578307860000,"last_timestamp_unix_ms":1578307980000}}} {"guid":"00000000-0000-4000-8000-000000000013","timestamp_unix_ms":1578307860000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"friend@example.com","sender_display_name":"friend@example.com","owner_identity":"+15555550106","subject":null,"text":"Pizza?","attachments":[],"reactions":[{"part_index":0,"kind":"loved","is_from_me":false,"reactor_identity":"+15555550107","reactor_display_name":"+15555550107"},{"part_index":0,"kind":"emoji","emoji":"🔥","is_from_me":true,"reactor_display_name":"+15555550106"}],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Pizza?"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} {"guid":"guid-14","timestamp_unix_ms":1578307920000,"direction":"incoming","service":"imessage","message_kind":"tapback","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Loved a message","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"loved","tapback_emoji":null,"tapback_action":"add"},"source":null} {"guid":"guid-15","timestamp_unix_ms":1578307980000,"direction":"outgoing","service":"imessage","message_kind":"tapback","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"🔥 reacted","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"emoji","tapback_emoji":"🔥","tapback_action":"add"},"source":null} diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 53409dbef..e1ed7e37a 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -374,6 +374,35 @@ recorded by every source, and the time an exporter ran says nothing about the backup, so neither decides it ([#1408](https://github.com/messagecrate/message-crate/issues/1408)). +**Between two copies of one message from one source, the copy from the later +backup decides its mark and text.** One message from one source is one row +(`UNIQUE (account_id, source, guid)` on `messages`, and the same on +`staging_messages`), so a second backup of the same phone meets the copy +already there, in the same import or a later one. The conversation file says +when its backup was made (`export.backup_taken_at_unix_ms`), staging keeps +that date on each staged row, and `messages.backup_taken_at` keeps the date of +the backup that decided the stored copy. When both copies have a date, the +copy from the later backup gives the message its deletion mark, mark or no +mark, and its text and earlier versions, whatever the versions' times say; a +copy from an earlier or the same backup changes neither. The duplicate flag +follows the text, because the dedupe compares the text. When either copy has +no date, nothing says which backup is newer, so the rules for files without +one hold: a copy with a mark adds it and one without leaves the mark held, +and a copy takes the text when its newest earlier version is newer +(`later_edit_sql` in `db/staging.rs`). Attachments and reactions add from +either copy, because a backup that lacks one does not say it is gone. The +rule is the same in one import as across several, in any file order +(`later_backup_sql` in `db/staging.rs`, `add_staged_copy` in +`imports_api/staging.rs`). Why: a person recovers a deleted message and +unsends or edits a sent one between two backups, and only the backup's own +date says which state is the newer; the message's own times record when it +was written, not when a part was unsent, so they cannot tell +([#1741](https://github.com/messagecrate/message-crate/issues/1741), +[#1804](https://github.com/messagecrate/message-crate/issues/1804), +[#1924](https://github.com/messagecrate/message-crate/issues/1924)). An append +with dedupe off still leaves a changed message's duplicate flag until the next +dedupe ([#1805](https://github.com/messagecrate/message-crate/issues/1805)). + **A participant's display name has one rule.** The contact's name, else what that backup called them in that conversation, else the identity. One loader applies it for the conversation list, the message pane, and Export. diff --git a/docs/src/assets/openapi.json b/docs/src/assets/openapi.json index 225ef6f4b..0f502b7c5 100644 --- a/docs/src/assets/openapi.json +++ b/docs/src/assets/openapi.json @@ -15308,6 +15308,7 @@ "contacts_new", "contacts_changed", "attachments_ms", + "backup_taken_at", "device_id", "duration_ms", "finished_at", @@ -15332,6 +15333,13 @@ "format": "int64", "description": "Time spent on attachments, when finished." }, + "backup_taken_at": { + "type": [ + "string", + "null" + ], + "description": "When the backup the run read was made, UTC: the newest date its\nconversation files name (`export.backup_taken_at_unix_ms`). Null until\na file that names one is imported, and for a run whose files name none." + }, "bytes_uploaded": { "type": "integer", "format": "int64", @@ -15712,6 +15720,7 @@ "tapbacks", "earlier_versions", "matched_earlier_version", + "backup_taken_at", "deletion", "owner", "reply_to", @@ -15728,6 +15737,13 @@ }, "description": "Attachments on this message." }, + "backup_taken_at": { + "type": [ + "string", + "null" + ], + "description": "When the backup that gave the message its mark and text was made,\nRFC 3339 in UTC with a `Z` suffix; `null` when the conversation\nfile did not say. Between two copies of one message from one\nsource, the copy from the later backup decides." + }, "conversation": { "$ref": "#/components/schemas/MessageConversation", "description": "The conversation this message belongs to." @@ -17459,6 +17475,7 @@ "contacts_new", "contacts_changed", "attachments_ms", + "backup_taken_at", "device_id", "duration_ms", "finished_at", @@ -17483,6 +17500,13 @@ "format": "int64", "description": "Time spent on attachments, when finished." }, + "backup_taken_at": { + "type": [ + "string", + "null" + ], + "description": "When the backup the run read was made, UTC: the newest date its\nconversation files name (`export.backup_taken_at_unix_ms`). Null until\na file that names one is imported, and for a run whose files name none." + }, "bytes_uploaded": { "type": "integer", "format": "int64", @@ -17739,6 +17763,7 @@ "tapbacks", "earlier_versions", "matched_earlier_version", + "backup_taken_at", "deletion", "owner", "reply_to", @@ -17755,6 +17780,13 @@ }, "description": "Attachments on this message." }, + "backup_taken_at": { + "type": [ + "string", + "null" + ], + "description": "When the backup that gave the message its mark and text was made,\nRFC 3339 in UTC with a `Z` suffix; `null` when the conversation\nfile did not say. Between two copies of one message from one\nsource, the copy from the later backup decides." + }, "conversation": { "$ref": "#/components/schemas/MessageConversation", "description": "The conversation this message belongs to." diff --git a/schema/sql/accounts.sql b/schema/sql/accounts.sql index fd98436aa..08a5b6a1c 100644 --- a/schema/sql/accounts.sql +++ b/schema/sql/accounts.sql @@ -219,7 +219,11 @@ CREATE TABLE IF NOT EXISTS imports ( -- Addresses the backup's device sent from (JSON array), read by the -- client before parsing. Lets a resumed Staging Review show the identity -- list without re-reading the backup. - source_identities TEXT + source_identities TEXT, + -- When the backup the run read was made, in the form messages.timestamp + -- holds: the newest its conversation files name. NULL until a file that + -- says is imported, and for a run whose files say nothing. + backup_taken_at TEXT ); CREATE INDEX IF NOT EXISTS ix_imports_account_started diff --git a/schema/sql/messages.sql b/schema/sql/messages.sql index 373e1a671..13c1ac968 100644 --- a/schema/sql/messages.sql +++ b/schema/sql/messages.sql @@ -94,7 +94,13 @@ CREATE TABLE IF NOT EXISTS messages ( -- Points at the kept message when this row is a flagged duplicate. duplicate_of INTEGER REFERENCES messages(id) ON DELETE SET NULL, -- Import run that inserted this row (`imports.id`). - import_id INTEGER REFERENCES imports(id) ON DELETE SET NULL + import_id INTEGER REFERENCES imports(id) ON DELETE SET NULL, + -- When the backup that gave the message its deletion mark and text was + -- made, in the form timestamp holds; NULL when its file did not say. A + -- later import's copy from a later backup replaces both; one from an + -- earlier backup changes neither + -- (docs/architecture/contacts-identities-and-messages.md). + backup_taken_at TEXT ); CREATE INDEX IF NOT EXISTS ix_messages_conversation_timestamp diff --git a/schema/sql/staging.sql b/schema/sql/staging.sql index 84fd99b08..7cf7d6936 100644 --- a/schema/sql/staging.sql +++ b/schema/sql/staging.sql @@ -77,7 +77,12 @@ CREATE TABLE IF NOT EXISTS staging_messages ( -- Stable order within the conversation when timestamps collide. sort_order INTEGER NOT NULL, -- Import run that staged this row (`imports.id`). - import_id INTEGER REFERENCES imports(id) ON DELETE SET NULL + import_id INTEGER REFERENCES imports(id) ON DELETE SET NULL, + -- When the backup this row was read from was made, in the form timestamp + -- holds; NULL when its file did not say. Between two copies of one message + -- from one source, the copy from the later backup decides its deletion mark + -- and text (docs/architecture/contacts-identities-and-messages.md). + backup_taken_at TEXT ); CREATE INDEX IF NOT EXISTS ix_staging_messages_conversation_timestamp From 94debf380cc5bde1b68267fb26516f57c03e0e66 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:47:19 -0400 Subject: [PATCH 14/42] feat(web): Import details show the backup a run read and when it was made MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Settings → Storage → Import history opens a run's details with a Backup line: the backup the desktop app recorded for the run, and when that backup was made, from the run's new backup_taken_at. A backup that records no date says so. The owner's view of another account's run carries neither and shows nothing. Part of #1924. Co-Authored-By: Claude Opus 5.5 --- web/src/lib/serverApi.types.ts | 26 ++++++++++ .../settings/storage/ImportDetailPanel.tsx | 23 +++++++++ .../storage/ImportHistoryTable.test.tsx | 48 +++++++++++++++++++ .../settings/storage/storageUtils.test.ts | 46 ++++++++++++++++++ .../screens/settings/storage/storageUtils.ts | 18 +++++++ web/src/test/apiShapes.ts | 1 + 6 files changed, 162 insertions(+) diff --git a/web/src/lib/serverApi.types.ts b/web/src/lib/serverApi.types.ts index 9bb8d9cea..0c740d40a 100644 --- a/web/src/lib/serverApi.types.ts +++ b/web/src/lib/serverApi.types.ts @@ -3057,6 +3057,12 @@ export interface components { * @description Time spent on attachments, when finished. */ attachments_ms: number | null; + /** + * @description When the backup the run read was made, UTC: the newest date its + * conversation files name (`export.backup_taken_at_unix_ms`). Null until + * a file that names one is imported, and for a run whose files name none. + */ + backup_taken_at: string | null; /** * Format: int64 * @description Bytes uploaded so far. @@ -3288,6 +3294,13 @@ export interface components { Message: { /** @description Attachments on this message. */ attachments: components["schemas"]["Attachment"][]; + /** + * @description When the backup that gave the message its mark and text was made, + * RFC 3339 in UTC with a `Z` suffix; `null` when the conversation + * file did not say. Between two copies of one message from one + * source, the copy from the later backup decides. + */ + backup_taken_at: string | null; /** @description The conversation this message belongs to. */ conversation: components["schemas"]["MessageConversation"]; /** @@ -4204,6 +4217,12 @@ export interface components { * @description Time spent on attachments, when finished. */ attachments_ms: number | null; + /** + * @description When the backup the run read was made, UTC: the newest date its + * conversation files name (`export.backup_taken_at_unix_ms`). Null until + * a file that names one is imported, and for a run whose files name none. + */ + backup_taken_at: string | null; /** * Format: int64 * @description Bytes uploaded so far. @@ -4341,6 +4360,13 @@ export interface components { items: { /** @description Attachments on this message. */ attachments: components["schemas"]["Attachment"][]; + /** + * @description When the backup that gave the message its mark and text was made, + * RFC 3339 in UTC with a `Z` suffix; `null` when the conversation + * file did not say. Between two copies of one message from one + * source, the copy from the later backup decides. + */ + backup_taken_at: string | null; /** @description The conversation this message belongs to. */ conversation: components["schemas"]["MessageConversation"]; /** diff --git a/web/src/screens/settings/storage/ImportDetailPanel.tsx b/web/src/screens/settings/storage/ImportDetailPanel.tsx index c18b41386..4845d9df3 100644 --- a/web/src/screens/settings/storage/ImportDetailPanel.tsx +++ b/web/src/screens/settings/storage/ImportDetailPanel.tsx @@ -7,6 +7,7 @@ import type { AccountImportRun } from "./storageUtils"; import { formatBytes, formatImportDate, + importBackup, importStatusLabel, sectionHint, sectionTitle, @@ -109,6 +110,7 @@ export default function ImportDetailPanel({
Issues
{selectedImport.issue_count.toLocaleString()}
+
@@ -131,3 +133,24 @@ export default function ImportDetailPanel({
); } + +/** + * The backup the run read, beside when that backup was made, so two imports of + * one phone can be told apart. The owner's view of another account's run + * carries neither, and shows nothing here. + */ +function ImportBackupDetail({ run }: { run: AccountImportRun }) { + const backup = importBackup(run); + if (!backup) return null; + return ( +
+
Backup
+
{backup.file ?? "Not recorded"}
+
+ {backup.takenAt + ? `Made ${formatImportDate(backup.takenAt)}` + : "The backup does not say when it was made"} +
+
+ ); +} diff --git a/web/src/screens/settings/storage/ImportHistoryTable.test.tsx b/web/src/screens/settings/storage/ImportHistoryTable.test.tsx index fbe872448..c96620fb0 100644 --- a/web/src/screens/settings/storage/ImportHistoryTable.test.tsx +++ b/web/src/screens/settings/storage/ImportHistoryTable.test.tsx @@ -163,4 +163,52 @@ describe("Import history", () => { expect(await screen.findByRole("heading", { name: "Import Errors" })).toBeInTheDocument(); expect(getAccountImport).toHaveBeenCalledWith(1, expect.anything(), undefined); }); + + it("shows the backup a run read beside when that backup was made", async () => { + const listed = { ...anImport(1) }; + listAccountImports.mockResolvedValue({ items: [listed], total: 1, limit: 50, offset: 0 }); + getAccountImport.mockResolvedValue({ + ...listed, + summary: null, + contacts_new: 0, + contacts_changed: 0, + issues: [], + source_fingerprint: { path: "/backups/iPhone/00008110", size: 12, mtime_ms: 1 }, + backup_taken_at: "2026-09-30T18:45:12Z", + }); + const user = setupUser(); + renderWithProviders(); + const heading = await screen.findByRole("heading", { name: "Import history" }); + const section = within(heading.parentElement as HTMLElement); + await user.click(await section.findByRole("button", { expanded: false })); + + const backup = await screen.findByText("Backup"); + const [file, made] = Array.from(backup.parentElement?.querySelectorAll("dd") ?? []); + expect(file?.textContent).toBe("/backups/iPhone/00008110"); + expect(made?.textContent).toMatch(/^Made .*2026/); + }); + + it("says when a run's backup does not say when it was made", async () => { + const listed = { ...anImport(1) }; + listAccountImports.mockResolvedValue({ items: [listed], total: 1, limit: 50, offset: 0 }); + getAccountImport.mockResolvedValue({ + ...listed, + summary: null, + contacts_new: 0, + contacts_changed: 0, + issues: [], + source_fingerprint: null, + backup_taken_at: null, + }); + const user = setupUser(); + renderWithProviders(); + const heading = await screen.findByRole("heading", { name: "Import history" }); + const section = within(heading.parentElement as HTMLElement); + await user.click(await section.findByRole("button", { expanded: false })); + + const backup = await screen.findByText("Backup"); + const [file, made] = Array.from(backup.parentElement?.querySelectorAll("dd") ?? []); + expect(file?.textContent).toBe("Not recorded"); + expect(made?.textContent).toBe("The backup does not say when it was made"); + }); }); diff --git a/web/src/screens/settings/storage/storageUtils.test.ts b/web/src/screens/settings/storage/storageUtils.test.ts index 1bbe53b10..e3e0b9124 100644 --- a/web/src/screens/settings/storage/storageUtils.test.ts +++ b/web/src/screens/settings/storage/storageUtils.test.ts @@ -5,6 +5,7 @@ import { describeExportScope, formatBytes, formatImportDate, + importBackup, importStatusLabel, toImportSummaryView, } from "./storageUtils"; @@ -35,6 +36,7 @@ function accountImportRun(partial: Partial = {}): AccountImpor form: null, source_fingerprint: null, source_identities: null, + backup_taken_at: null, summary: {}, issue_count: 0, note_count: 0, @@ -217,3 +219,47 @@ describe("toImportSummaryView for the owner", () => { expect(view.issues).toEqual([]); }); }); + +describe("importBackup", () => { + it("reads the backup's file and when it was made from the account's own run", () => { + expect( + importBackup( + accountImportRun({ + source_fingerprint: { path: "/backups/iPhone/00008110", size: 12, mtime_ms: 1 }, + backup_taken_at: "2026-09-30T18:45:12Z", + }), + ), + ).toEqual({ file: "/backups/iPhone/00008110", takenAt: "2026-09-30T18:45:12Z" }); + }); + + it("says nothing it was not told", () => { + expect(importBackup(accountImportRun())).toEqual({ file: null, takenAt: null }); + }); + + it("gives the owner nothing of another account's backup", () => { + expect( + importBackup({ + id: 1, + source: "imessage-ios", + mode: "append", + status: "completed", + tool: null, + started_at: "2026-08-11T12:00:00Z", + finished_at: null, + message_count: 10, + attachment_count: 0, + bytes_uploaded: 0, + duration_ms: null, + parse_ms: null, + attachments_ms: null, + prepare_ms: null, + upload_ms: null, + counts: {}, + issue_count: 0, + note_count: 0, + contacts_new: 0, + contacts_changed: 0, + }), + ).toBeNull(); + }); +}); diff --git a/web/src/screens/settings/storage/storageUtils.ts b/web/src/screens/settings/storage/storageUtils.ts index bbe162293..4cff430de 100644 --- a/web/src/screens/settings/storage/storageUtils.ts +++ b/web/src/screens/settings/storage/storageUtils.ts @@ -92,6 +92,24 @@ export function describeExportScope(scope: Schema["ExportScope"]): string { */ export type AccountImportRun = Schema["AccountImportRun"]; +/** + * The backup an account's own Import Run read: the file the desktop app + * recorded for it, and when the backup was made, each null when the run does + * not say. Null for the owner's view of another account's run, which carries + * neither. + */ +export function importBackup( + run: AccountImportRun, +): { file: string | null; takenAt: string | null } | null { + if (!("backup_taken_at" in run)) return null; + const fingerprint = run.source_fingerprint; + const path = + fingerprint && typeof fingerprint === "object" && "path" in fingerprint + ? fingerprint.path + : null; + return { file: typeof path === "string" && path ? path : null, takenAt: run.backup_taken_at }; +} + /** Human-readable file size (for example "1.2 MB"). */ export function formatBytes(bytes: number): string { if (!Number.isFinite(bytes) || bytes <= 0) return "0 B"; diff --git a/web/src/test/apiShapes.ts b/web/src/test/apiShapes.ts index d5f439376..4e5b321ad 100644 --- a/web/src/test/apiShapes.ts +++ b/web/src/test/apiShapes.ts @@ -59,6 +59,7 @@ export function message(fields: Partial = {}): Schema["Messag tapbacks: [], earlier_versions: [], matched_earlier_version: false, + backup_taken_at: null, conversation: { id: 1, chat_identifier: "x", From b623800c0a4364bc1de99b46493e507fbf3920eb Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:47:19 -0400 Subject: [PATCH 15/42] docs: describe when a backup was made and how the import uses it The common message page documents export.backup_taken_at_unix_ms, where each exporter reads it from, and the rule the import applies; the CSV and mail references list the new column and header; the import and storage pages say which backup wins and where the date shows. The CHANGELOG entry for 0.11.0 says files at schema version 10 are refused. Closes #1924. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 18 +++++++++++ .../developer/architecture/common-message.md | 31 ++++++++++++++++--- .../docs/developer/formats/mail-archive.md | 2 ++ .../docs/docs/developer/message-transfer.md | 2 +- .../docs/developer/reference/csv-columns.md | 1 + .../docs/user/features/messages/import.md | 9 ++++-- .../docs/user/features/settings/storage.md | 8 ++++- 7 files changed, 62 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 80dd40217..2fdd5ff01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,19 @@ released versions carry their date on the heading. ### Features +- 2026-10-05: **The newer backup decides when a message changed between two + backups of one phone.** Every message file now says when its backup was + made: an iPhone backup's own date, the date an SMS Backup & Restore file + records, an iMazing export's date, or when the backup's files were last + written for a source that records none. Importing two backups of the same + phone, in one import or two and in either order, gives each message the + newer backup's text, earlier versions and Deleted in the source app or + Unsent mark, so a message recovered after it was deleted loses its mark, + and a message unsent after the older backup reads as unsent. An older + backup imported after a newer one changes nothing. Import details under + Settings → Storage show the backup each import read and when it was made. + A message file at the previous schema version, 10, is refused, and the + backup must be exported again with this build. - 2026-10-05: **A WhatsApp reply now names the message it quotes.** When the quoted message is in the same chat of the same backup, the reply is linked to it, as Apple Messages replies already were: a mail export threads @@ -109,6 +122,11 @@ released versions carry their date on the heading. in place of `num_replies`. - A Saved Search that uses `deleted:yes` finds fewer messages than before: it leaves unsent messages out. Add `or unsent:yes` to it to find both. +- Message files at schema version 10, exported before they said when their + backup was made, are refused when you import or convert them. Export the + backup again with this build. A program that reads the HTTP API finds the + backup's date in a message's `backup_taken_at` and an Import Run's + `backup_taken_at`. ## [0.10.1] - 2026-10-05 diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index b58a8ad77..90f32d5af 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -30,19 +30,20 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. - **Common-message path** (`ConversationDocument` → `message_ir_format::FormatSink`, one of json/jsonl/csv/eml/mbox/xml): all exporters, including iMessage (`imessage-ir-exporter`). Per-chat formats also accept `write_format`; XML uses a single `smses.xml` via the sink. - **Media + obfuscate** run inside `FormatSink::finish` for every format (`message_crate_core::ExportTransforms`: none / copy / convert / compress, plus optional obfuscate). When obfuscate is on, exporters skip staging real attachment bytes and convert/compress is not run — only placeholder files are written. Exporters pass transforms from `ExporterConfig.media` / `.obfuscate`; there is no CSV-only post-step. EML / MBOX / XML embed media and drop the staged `attachments/` directory afterward. -- **Schema version 10 only** (breaking). Version 10 keeps the message a reply quotes in the message's own `reply_to`, for every source (see [Replies](#replies)), where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`, and a reply count in `imessage.num_replies`. Version 9 had given orphaned messages conversations of type `orphaned` (see [Orphaned messages](#orphaned-messages)), where version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, for every source, where version 7 kept the Apple Messages edit history as a JSON value in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, for every source, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, one shape for every source, where version 5 kept Apple Messages reactions as a JSON value in `imessage.tapbacks`. Version 5 had named every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. Version 9 and older are refused, never upgraded. Typed enums/bags, filled outgoing identity, conversation stats, stable null/`[]` keys. Older common-message JSON is not read — regenerate exports after schema changes. +- **Schema version 11 only** (breaking). Version 11 says when the backup was made, in `export.backup_taken_at_unix_ms` (see [When the backup was made](#when-the-backup-was-made)), which version 10 did not, so an import could not tell which of two backups of one phone is the later one. Version 10 had kept the message a reply quotes in the message's own `reply_to`, for every source (see [Replies](#replies)), where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`, and a reply count in `imessage.num_replies`. Version 9 had given orphaned messages conversations of type `orphaned` (see [Orphaned messages](#orphaned-messages)), where version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, for every source, where version 7 kept the Apple Messages edit history as a JSON value in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, for every source, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, one shape for every source, where version 5 kept Apple Messages reactions as a JSON value in `imessage.tapbacks`. Version 5 had named every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. Version 10 and older are refused, never upgraded. Typed enums/bags, filled outgoing identity, conversation stats, stable null/`[]` keys. Older common-message JSON is not read — regenerate exports after schema changes. -## Document schema (`schema_version: 10`) +## Document schema (`schema_version: 11`) ```json { - "schema_version": 10, + "schema_version": 11, "export": { "source": "sms-backup-restore", "tool": "SMS Backup & Restore", "tool_version": "10.26.003", "owner_identity": "+15555550100", - "owner_display_name": "Me" + "owner_display_name": "Me", + "backup_taken_at_unix_ms": 1400800000000 }, "conversation": { "chat_identifier": "+15555550101", @@ -95,6 +96,26 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. - `guid` is Apple's own id for Apple Messages. Every other source's `guid` is a `MessageGuid`: SHA-256 of the chat id, the direction, the sender of an incoming message, the UTC instant in milliseconds, the text with whitespace collapsed, the sorted attachment digests, and the source's own key where it has one (WhatsApp's `key_id`). It reads no time zone and no display format, so one backup gives the same ids on any computer. The server refuses a message whose `guid` is empty. - Two records a backup cannot tell apart are one message, and the exporter keeps one (`message_ir::one_copy_per_message`). The server's content key, which matches one message across sources, is the same identity at whole seconds. +### When the backup was made + +`export.backup_taken_at_unix_ms` is when the backup the file was read from was made, in Unix milliseconds, or `null` when nothing says. Each exporter reads it from its source: + +| Source | Date | +|--------|------| +| Apple Messages from an iPhone backup | The `Date` in the backup's `Manifest.plist`, which is readable whether or not the backup is encrypted | +| Apple Messages from a Mac `chat.db` | The database file's modification time, when Messages last wrote it | +| WhatsApp from an iPhone backup | The `Date` in the backup's `Manifest.plist` | +| WhatsApp from Android | The modification time of the database file, `msgstore.db.crypt15` or a decrypted `msgstore.db` | +| WhatsApp from a ready-made `result.json` | The JSON file's modification time | +| SMS Backup & Restore | The root element's `backup_date` attribute; a file without one is dated by its modification time. A conversation read from two files is as new as the newer | +| iMazing | The export date: the newest modification time of the CSV files iMazing wrote | +| OpenExtract, GO SMS Pro, SMS Backup+ | The newest modification time of the files read, because none of them records a date of its own | +| An Export Run of the server | The newest backup date of the conversation's messages | + +The import reads it to decide between two copies of one message from one source. The copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, in one import in either file order and across imports in either order. A copy from an earlier backup changes neither. A file that says nothing keeps the rules for files without a date: a mark adds and is never cleared, and an edit counts as later when its newest earlier version is newer. Attachments and reactions add from either copy. + +CSV carries it in the `backup_taken_at_unix_ms` column of every row, and EML and MBOX in the `X-ME-Backup-Taken-At-Unix-Ms` header of every mail, blank or absent when the file says nothing. A value that is not a whole number is refused rather than read as no date. + ### Reactions `reactions` is the list of reactions that stand on a message, the same shape for every source. Each one is a `Reaction`: @@ -183,7 +204,7 @@ Attachment **bytes** are never stored in JSON/JSONL (`#[serde(skip)]`). Paths + ## JSONL layout ```text -{"schema_version":10,"export":{…},"conversation":{…}} +{"schema_version":11,"export":{…},"conversation":{…}} {"guid":"…","timestamp_unix_ms":…, …} … ``` diff --git a/docs/src/content/docs/docs/developer/formats/mail-archive.md b/docs/src/content/docs/docs/developer/formats/mail-archive.md index b75275248..50229f3aa 100644 --- a/docs/src/content/docs/docs/developer/formats/mail-archive.md +++ b/docs/src/content/docs/docs/developer/formats/mail-archive.md @@ -149,6 +149,7 @@ A mail an earlier Message Crate wrote names its addresses with `X-ME-Sender-Hand | `X-ME-Export-Source` | string | e.g. `sms-backup-restore` | | `X-ME-Export-Tool` | string | | | `X-ME-Export-Tool-Version` | string | | +| `X-ME-Backup-Taken-At-Unix-Ms` | integer string | When the backup was made, in Unix milliseconds; omitted when the export does not say. A value that is not a whole number is refused | | `X-ME-Android-Type` | integer string | Optional; SMS `type` / MMS `msg_box` | | `X-ME-Source-Fields` | JSON | Optional full-fidelity bag (CSV `source_fields_json` / PDU extras) | | `X-ME-Attachment-Meta` | JSON array | Parallel to MIME attachment parts (see Attachments) | @@ -350,6 +351,7 @@ Normal sticker sends: image MIME part + `X-ME-Attachment-Meta` (`is_sticker`, `s | `source_fields_json` / PDU extras | `X-ME-Source-Fields` | | `export_*` | `X-ME-Export-*` | | `owner_identity` / `owner_display_name` | `X-ME-Owner-*` | +| `backup_taken_at_unix_ms` | `X-ME-Backup-Taken-At-Unix-Ms` | | `message_owner_identity` | `X-ME-Message-Owner-Identity` | | `participants_json` (iMessage) | `X-ME-Participants` | | `reactions_json` | `X-ME-Reactions` | diff --git a/docs/src/content/docs/docs/developer/message-transfer.md b/docs/src/content/docs/docs/developer/message-transfer.md index 78965d276..ea18e9124 100644 --- a/docs/src/content/docs/docs/developer/message-transfer.md +++ b/docs/src/content/docs/docs/developer/message-transfer.md @@ -41,7 +41,7 @@ Each conversation is one text file whose name ends in `.jsonl`. JSON Lines means Pictures and other media sit next to those files in `attachments/`. ```jsonl title="One conversation file" -{"schema_version":10,"export":{"source":"sms-backup-restore","tool":"SMS Backup & Restore","owner_identity":"+15555550100","owner_display_name":"Me"},"conversation":{"chat_identifier":"+15555550101","conversation_type":"individual","participants":[{"identity":"+15555550101","display_name":"Sam"}]}} +{"schema_version":11,"export":{"source":"sms-backup-restore","tool":"SMS Backup & Restore","owner_identity":"+15555550100","owner_display_name":"Me","backup_taken_at_unix_ms":1400800000000},"conversation":{"chat_identifier":"+15555550101","conversation_type":"individual","participants":[{"identity":"+15555550101","display_name":"Sam"}]}} {"guid":"msg-1","timestamp_unix_ms":1400773261000,"direction":"outgoing","service":"sms","text":"Hello"} ``` diff --git a/docs/src/content/docs/docs/developer/reference/csv-columns.md b/docs/src/content/docs/docs/developer/reference/csv-columns.md index a98184fee..5d4d77615 100644 --- a/docs/src/content/docs/docs/developer/reference/csv-columns.md +++ b/docs/src/content/docs/docs/developer/reference/csv-columns.md @@ -48,6 +48,7 @@ CSV output contains one row per message. Conversation and export identity are re | `export_tool_version` | Source version recorded by the importer. | | `owner_identity` | Phone number or email for the person whose backup was exported. | | `owner_display_name` | Display name for the owner. | +| `backup_taken_at_unix_ms` | When the backup the export was read from was made, in Unix milliseconds; blank when the export does not say. | | `message_owner_identity` | The owner's own address on this message: the one it was sent from or received at. Apple Messages records it per message, so one conversation can hold rows from a phone number and an Apple ID. Empty when the source records no owner per message, and `owner_identity` then stands for the row. | | `android_type` | Original Android SMS type or MMS box number, or empty for other sources. | | `source_fields_json` | Compact JSON containing source-specific fields that do not have shared columns. | diff --git a/docs/src/content/docs/docs/user/features/messages/import.md b/docs/src/content/docs/docs/user/features/messages/import.md index 9924be4c9..025f5e030 100644 --- a/docs/src/content/docs/docs/user/features/messages/import.md +++ b/docs/src/content/docs/docs/user/features/messages/import.md @@ -381,7 +381,12 @@ The record of the run stays under **Import history** in [**Settings → Storage* A second import of the same backup creates no duplicates. The Message Crate recognises the messages it already holds and skips them, so a newer backup of the same phone adds only what is new. -A message edited again since the first import is the exception: it takes the newer backup's text and earlier versions ([Edited messages](/docs/user/features/messages/browse/#edited-messages)). +A message that changed since the first import is the exception: it takes the newer backup's text, earlier versions, and mark ([Edited messages](/docs/user/features/messages/browse/#edited-messages)). +A message recovered after it was deleted loses its **Deleted in the source app** mark, and a message unsent since is marked **Unsent**. An older backup imported after a newer one leaves the message as it is. -One import that carries an older and a newer backup of the same phone gives an edited message the newer backup's text and earlier versions, whichever order the files are in. +One import that carries an older and a newer backup of the same phone gives each message the newer backup's text, earlier versions, and mark, whichever order the files are in. + +Which backup is newer comes from the backup's own date: the date an iPhone backup records, the date an SMS Backup & Restore file records, or the date its files were last written where the backup records none. +**Settings → Storage → Import history** shows it beside the backup each run read. +A backup that records no date at all keeps a mark once given, and takes a newer text only when its edits are newer. It also keeps the attachments and reactions each backup holds of a message, each one once, so one import of two backups stores what two separate imports of them store. diff --git a/docs/src/content/docs/docs/user/features/settings/storage.md b/docs/src/content/docs/docs/user/features/settings/storage.md index 2d612d44f..dfc380bb1 100644 --- a/docs/src/content/docs/docs/user/features/settings/storage.md +++ b/docs/src/content/docs/docs/user/features/settings/storage.md @@ -39,7 +39,13 @@ Selecting the row again, or the **×** button, closes it. Three labels sit at the top: **Type**, the source, **Mode**, `Append` or `Replace`, and **Status**. -Five figures follow: **Started**, **Finished**, **Messages**, **Attachments**, and **Bytes uploaded**. +Six figures follow: **Started**, **Finished**, **Messages**, **Attachments**, **Bytes uploaded**, and **Issues**. + +**Backup** names the backup the run read, as the desktop app recorded it, and when that backup was made, so two imports of one phone can be told apart. +The date comes from the backup itself: an iPhone backup's own date, the date an SMS Backup & Restore file records, or the date its files were last written where the backup records none. +When the run read two backups, it shows the later. +"The backup does not say when it was made" appears for a backup that records no date. +The owner, reading another account's runs, does not see this line. A list of four steps comes next, each with the time it took: **Parse backup**, **Attachments**, **Preparing messages**, and **Upload to Message Crate**. From c337dd5d1dc0e7b8633858caa8b57b07500960ec Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Mon, 5 Oct 2026 23:58:19 -0400 Subject: [PATCH 16/42] test(sms-backup-restore): a JSON export is at schema version 11 Co-Authored-By: Claude Opus 5.5 --- .../sms-backup-restore-exporter/tests/convert_smoke.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs b/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs index 748265f38..3392361e3 100644 --- a/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs +++ b/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs @@ -344,7 +344,7 @@ fn convert_export_json_and_jsonl_use_pristine_v4() { .expect("expected .json"); let raw = fs::read_to_string(&json_path).unwrap(); let doc: serde_json::Value = serde_json::from_str(&raw).unwrap(); - assert_eq!(doc["schema_version"], 10); + assert_eq!(doc["schema_version"], 11); assert!( doc["conversation"]["stats"]["message_count"] .as_u64() @@ -389,7 +389,7 @@ fn convert_export_json_and_jsonl_use_pristine_v4() { let body = fs::read_to_string(&jsonl_path).unwrap(); let mut lines = body.lines(); let header: serde_json::Value = serde_json::from_str(lines.next().unwrap()).unwrap(); - assert_eq!(header["schema_version"], 10); + assert_eq!(header["schema_version"], 11); assert!(header.get("messages").is_none()); assert!( header["conversation"]["stats"]["message_count"] From 10a09465e052ac59b2490cb6569d6249d6263e81 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:10:08 -0400 Subject: [PATCH 17/42] fix(search): a later backup that clears an Unsent mark gives the index row back #1957 reindexes a stored message only when a staged row carries a mark, because a promotion never cleared one. With #1924 a later backup can clear an Unsent mark, and that message stayed out of the search index. The promotion now names every message whose mark it changes, set or cleared, in _promote_mark_map, and the reindex set reads that table. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/db/schema.rs | 13 +++--- crates/server/server/src/db/schema/tests.rs | 8 ++++ crates/server/server/src/db/staging.rs | 41 +++++++++++++------ .../src/imports_api/tests/backup_dates.rs | 40 ++++++++++++++++++ 4 files changed, 83 insertions(+), 19 deletions(-) diff --git a/crates/server/server/src/db/schema.rs b/crates/server/server/src/db/schema.rs index 605b2f97f..e399b52e4 100644 --- a/crates/server/server/src/db/schema.rs +++ b/crates/server/server/src/db/schema.rs @@ -375,7 +375,8 @@ pub(crate) async fn create_messages_secondary_indexes(conn: &mut SqliteConnectio /// The stored messages, those at or below `$1` (`min_new_message_id`), whose /// index row a promotion removes and writes again: each one that gained an /// attachment above `$2` (`min_new_attachment_id`), took a later edit, or -/// is marked by a staged row. One set for the delete and the insert in +/// whose mark `staging::promote_deletion_marks` changed, set or cleared +/// (`_promote_mark_map`). One set for the delete and the insert in /// [`index_messages_fts_from_promote_map`], so a row removed is always a row /// considered for writing again. const STORED_MESSAGES_TO_REINDEX: &str = " @@ -384,9 +385,7 @@ const STORED_MESSAGES_TO_REINDEX: &str = " UNION SELECT prod_id FROM _promote_edit_map UNION - SELECT pm.prod_id FROM _promote_msg_map pm - JOIN staging_messages sm ON sm.id = pm.staging_id - WHERE sm.deletion IS NOT NULL AND pm.prod_id <= $1"; + SELECT prod_id FROM _promote_mark_map WHERE prod_id <= $1"; /// Bulk-index promoted messages (joined via temp `_promote_msg_map`). /// Call after attachment rows exist so `attachment_text` is complete. @@ -407,14 +406,14 @@ const STORED_MESSAGES_TO_REINDEX: &str = " /// attachment above it was inserted by this promotion. The same goes for an /// existing message that took a later edit, which `_promote_edit_map` /// names: its index row holds the text the edit replaced, and for an -/// existing message a staged row marks, whose mark -/// `staging::promote_deletion_marks` may have changed. +/// existing message whose mark `staging::promote_deletion_marks` changed, +/// which `_promote_mark_map` names. /// /// An Unsent message is given no index row, as the sync triggers give it /// none (`fts_triggers_create.sql`): it shows none of its text, so a word of /// that text must not find it (#1758). A stored message this promotion marks /// Unsent loses its row here, and one whose Unsent mark becomes Deleted in -/// the source app is written again. +/// the source app, or is cleared by a later backup, is written again. pub(crate) async fn index_messages_fts_from_promote_map( conn: &mut SqliteConnection, min_new_message_id: i64, diff --git a/crates/server/server/src/db/schema/tests.rs b/crates/server/server/src/db/schema/tests.rs index 4194b6443..59f6565ee 100644 --- a/crates/server/server/src/db/schema/tests.rs +++ b/crates/server/server/src/db/schema/tests.rs @@ -98,6 +98,10 @@ async fn promote_fts_indexing_covers_only_rows_inserted_by_this_promotion() { staging_id INTEGER PRIMARY KEY, prod_id INTEGER NOT NULL ); + CREATE TEMP TABLE _promote_mark_map ( + staging_id INTEGER PRIMARY KEY, + prod_id INTEGER NOT NULL + ); INSERT INTO _promote_msg_map (staging_id, prod_id) VALUES (1, 10), (2, 11), (3, 11); ", ) @@ -173,6 +177,10 @@ async fn promote_fts_indexing_reindexes_an_existing_message_that_gained_an_attac staging_id INTEGER PRIMARY KEY, prod_id INTEGER NOT NULL ); + CREATE TEMP TABLE _promote_mark_map ( + staging_id INTEGER PRIMARY KEY, + prod_id INTEGER NOT NULL + ); INSERT INTO _promote_msg_map (staging_id, prod_id) VALUES (1, 10), (2, 12); INSERT INTO attachments (message_id, original_name) VALUES (10, 'lateinvoice.pdf'); ", diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index e9b1bbeb0..818bd9eef 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -3,8 +3,8 @@ //! `staging_conversations`, `staging_participants`, `staging_messages`, //! `staging_attachments`, `staging_tapbacks` and `staging_message_versions`, //! and the temp id maps -//! (`_promote_conv_map`, `_promote_msg_map`, `_promote_edit_map`) the -//! promotion joins through. +//! (`_promote_conv_map`, `_promote_msg_map`, `_promote_mark_map`, +//! `_promote_edit_map`) the promotion joins through. //! //! `imports_api::staging` fills the tables and `imports_api::promote` runs //! the promotion. Those stages sequence the statements, log them and keep @@ -1259,28 +1259,45 @@ fn later_backup_sql(staged: &str, held: &str) -> String { /// earlier or the same backup changes nothing. When either has no date, a /// staged row with no mark leaves the stored mark as it is: a backup that /// does not say a message was deleted does not say it was restored, and -/// nothing says which backup is newer. The search index follows the mark -/// when the promotion indexes (`schema::index_messages_fts_from_promote_map`): -/// a message marked Unsent loses its index row, because it shows none of its -/// text (#1758). Returns how many messages changed. +/// nothing says which backup is newer. +/// +/// Each message whose mark changes is named in `_promote_mark_map`, so the +/// search index follows the mark when the promotion indexes +/// (`schema::index_messages_fts_from_promote_map`): a message marked Unsent +/// loses its index row, because it shows none of its text (#1758), and one +/// whose Unsent mark a later backup clears or turns into Deleted in the +/// source app has it written again. Returns how many messages changed. /// /// # Errors /// /// Returns an error when the update fails. pub async fn promote_deletion_marks(conn: &mut SqliteConnection) -> Result { + reset_id_map(conn, "_promote_mark_map").await?; let sql = format!( r" - UPDATE messages - SET deletion = sm.deletion + INSERT INTO _promote_mark_map (staging_id, prod_id) + SELECT mm.staging_id, mm.prod_id FROM _promote_msg_map mm JOIN staging_messages sm ON sm.id = mm.staging_id - WHERE messages.id = mm.prod_id - AND messages.deletion IS NOT sm.deletion + JOIN messages m ON m.id = mm.prod_id + WHERE m.deletion IS NOT sm.deletion AND COALESCE({later}, sm.deletion IS NOT NULL) ", - later = later_backup_sql("sm.backup_taken_at", "messages.backup_taken_at"), + later = later_backup_sql("sm.backup_taken_at", "m.backup_taken_at"), ); - Ok(sqlx::query(&sql).execute(&mut *conn).await?.rows_affected()) + sqlx::query(&sql).execute(&mut *conn).await?; + Ok(sqlx::query( + r" + UPDATE messages + SET deletion = sm.deletion + FROM _promote_mark_map pm + JOIN staging_messages sm ON sm.id = pm.staging_id + WHERE messages.id = pm.prod_id + ", + ) + .execute(&mut *conn) + .await? + .rows_affected()) } /// Give each stored message the backup date of its staged row when that diff --git a/crates/server/server/src/imports_api/tests/backup_dates.rs b/crates/server/server/src/imports_api/tests/backup_dates.rs index f5bec1f74..01c78a363 100644 --- a/crates/server/server/src/imports_api/tests/backup_dates.rs +++ b/crates/server/server/src/imports_api/tests/backup_dates.rs @@ -156,6 +156,46 @@ async fn the_later_backup_decides_the_deletion_mark_in_every_order() { } } +/// Backup A marks the message Unsent, so it has no search index row +/// (#1758); backup B, made later, carries it unmarked with the same text. +/// In every order the message is unmarked and a word of its text finds it: +/// a later backup that clears the mark gives the index row back. +#[tokio::test] +async fn a_later_backup_that_clears_an_unsent_mark_makes_the_text_searchable() { + let tmp = TempDir::new().unwrap(); + let file = |name: &str, backup: i64, deletion: Option| { + backup_file( + tmp.path(), + name, + &Copy { + backup: Some(backup), + text: "zqlighthouse", + versions: &[], + deletion, + }, + ) + }; + let unsent = file( + "unsent-earlier.jsonl", + EARLIER_BACKUP, + Some(Deletion::Unsent), + ); + let shown = file("shown-later.jsonl", LATER_BACKUP, None); + for held in every_order(tmp.path(), "unsent", [&unsent, &shown]).await { + assert_eq!(held.deletion, None, "{held:?}"); + } + for name in ["together", "together-reversed", "apart", "apart-reversed"] { + let (_pool, mut conn) = open_verify(&tmp.path().join(format!("unsent-{name}.db"))).await; + let hits: i64 = sqlx::query_scalar( + "SELECT COUNT(*) FROM messages_fts WHERE messages_fts MATCH 'zqlighthouse'", + ) + .fetch_one(&mut *conn) + .await + .unwrap(); + assert_eq!(hits, 1, "{name}: the text is searchable again"); + } +} + /// The scenario of #1804: backup A lists part 1's earlier versions /// [x@t0, y@t100]; backup B, made later, after part 1 was unsent (which /// drops its versions) and part 0 was edited, lists part 0's [a@t0] only. From 8597d7c0caa8097a57be1f9c793279644732413d Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:14:58 -0400 Subject: [PATCH 18/42] fix(import): keep a backup's date to the millisecond, as a message time The merge of #1924 with #1096 left the backup date going through the millisecond time writer as seconds, so every backup read as 1970. It now passes milliseconds, and the stored date takes the three-digit form every stored time takes. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/imports_api/tests/backup_dates.rs | 4 ++-- crates/server/server/src/models.rs | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/crates/server/server/src/imports_api/tests/backup_dates.rs b/crates/server/server/src/imports_api/tests/backup_dates.rs index f5bec1f74..af7c1e43f 100644 --- a/crates/server/server/src/imports_api/tests/backup_dates.rs +++ b/crates/server/server/src/imports_api/tests/backup_dates.rs @@ -106,9 +106,9 @@ async fn every_order(tmp: &Path, label: &str, files: [&PathBuf; 2]) -> [Held; 4] } /// The stored date of [`LATER_BACKUP`], in the form a timestamp takes. -const LATER_BACKUP_AT: &str = "2026-09-30T18:45:12Z"; +const LATER_BACKUP_AT: &str = "2026-09-30T18:45:12.000Z"; /// The stored date of [`EARLIER_BACKUP`]. -const EARLIER_BACKUP_AT: &str = "2026-09-01T10:00:00Z"; +const EARLIER_BACKUP_AT: &str = "2026-09-01T10:00:00.000Z"; /// Backup A marks the message Deleted in the source app; backup B, made /// later, after the person recovered it, does not. Whichever is imported diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index 5d4d5f763..9c76eb3b3 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -323,7 +323,7 @@ fn conversation_from_ir(header: &ConversationHeader, line: usize) -> Result Date: Tue, 6 Oct 2026 00:22:02 -0400 Subject: [PATCH 19/42] fix(import): equal backup dates fall back to the rules for files without one Two reads of one Mac's chat.db within one second carry the same date, and the later-backup rule read them as the same backup, so an unsend or an edit between them was lost. Equal dates now cannot decide, in SQL (later_backup_sql) and in Rust (the new later_backup helper, which the in-import copy rule uses instead of restating the comparison), and the undated rules hold: they change nothing for the same backup read again. add_staged_copy_mark no longer returns a flag its caller drops. The architecture doc says the date is kept to the second. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/db/staging.rs | 70 ++++++++++++------- .../server/server/src/imports_api/staging.rs | 14 ++-- .../src/imports_api/tests/backup_dates.rs | 27 +++++++ .../contacts-identities-and-messages.md | 9 +-- 4 files changed, 84 insertions(+), 36 deletions(-) diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index 818bd9eef..a28f5e6c3 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -628,10 +628,10 @@ pub async fn take_staged_copy_from_later_backup( } /// Give the staged message `staged` the mark `deletion` of another copy of -/// it from the same import, when one of the two backups has no date: a +/// it from the same import, when the two backups' dates cannot decide +/// ([`later_backup`]): a /// copy that carries a mark adds it, and one with none leaves the staged -/// mark, as [`promote_deletion_marks`] does for a stored message. Returns -/// whether the mark changed. +/// mark, as [`promote_deletion_marks`] does for a stored message. /// /// # Errors /// @@ -640,15 +640,13 @@ pub async fn add_staged_copy_mark( conn: &mut SqliteConnection, staged: i64, deletion: message_ir::Deletion, -) -> Result { - let done = sqlx::query( - "UPDATE staging_messages SET deletion = $1 WHERE id = $2 AND deletion IS NOT $1", - ) - .bind(deletion.as_str()) - .bind(staged) - .execute(&mut *conn) - .await?; - Ok(done.rows_affected() == 1) +) -> Result<()> { + sqlx::query("UPDATE staging_messages SET deletion = $1 WHERE id = $2") + .bind(deletion.as_str()) + .bind(staged) + .execute(&mut *conn) + .await?; + Ok(()) } /// Give the staged message `staged` the text `body` and the earlier @@ -1235,17 +1233,37 @@ pub async fn write_message_map( /// Whether a staged copy of a message comes from a later backup than the /// copy held, as an SQL expression over the two backups' dates `staged` and /// `held`: true when both have a date and the staged one is later, false -/// when both have one and it is not, and NULL when either has none. NULL -/// means the dates cannot decide, and the caller falls back on the rule for -/// files without a date. The one rule for which of two copies of a message -/// from one source is the later backup, for a stored message -/// ([`promote_deletion_marks`], [`write_edit_map`]) and, in Rust, for two -/// copies staged in one import (`imports_api::staging`). +/// when both have one and it is earlier, and NULL when either has none or +/// the two are equal. NULL means the dates cannot decide, and the caller +/// falls back on the rule for files without a date. Equal dates are the +/// same backup read again, where that rule changes nothing because the two +/// copies agree, or two reads of one Mac's `chat.db` within one second, +/// where it adds a mark and takes a later edit as it would with no dates. +/// The one rule for which of two copies of a message from one source is +/// the later backup, for a stored message ([`promote_deletion_marks`], +/// [`write_edit_map`]) and, in Rust ([`later_backup`]), for two copies +/// staged in one import (`imports_api::staging`). /// /// Both dates have one fixed whole-second UTC form -/// (`models::conversation_from_ir`), so the text orders as the time. +/// (`models::conversation_from_ir`), so the text orders as the time, and +/// two backups are told apart to the second. fn later_backup_sql(staged: &str, held: &str) -> String { - format!("CASE WHEN {staged} IS NOT NULL AND {held} IS NOT NULL THEN {staged} > {held} END") + format!( + "CASE WHEN {staged} IS NOT NULL AND {held} IS NOT NULL AND {staged} <> {held} \ + THEN {staged} > {held} END" + ) +} + +/// [`later_backup_sql`]'s rule in Rust, for two copies of a message staged +/// in one import: `Some(true)` when the copy `staged` comes from a later +/// backup than the copy `held`, `Some(false)` when from an earlier one, and +/// `None` when either has no date or the two are equal. +#[must_use] +pub fn later_backup(staged: Option<&str>, held: Option<&str>) -> Option { + match (staged, held) { + (Some(staged), Some(held)) if staged != held => Some(staged > held), + _ => None, + } } /// Give each stored message the mark its staged row carries, through @@ -1256,10 +1274,10 @@ fn later_backup_sql(staged: &str, held: &str) -> String { /// When both the staged row's backup and the stored message's have a date /// ([`later_backup_sql`]), the later backup decides: a staged row from a /// later backup gives its mark or clears the one held, and one from an -/// earlier or the same backup changes nothing. When either has no date, a -/// staged row with no mark leaves the stored mark as it is: a backup that -/// does not say a message was deleted does not say it was restored, and -/// nothing says which backup is newer. +/// earlier backup changes nothing. When either has no date, or the two are +/// equal, a staged row with no mark leaves the stored mark as it is: a +/// backup that does not say a message was deleted does not say it was +/// restored, and nothing says which backup is newer. /// /// Each message whose mark changes is named in `_promote_mark_map`, so the /// search index follows the mark when the promotion indexes @@ -1364,8 +1382,8 @@ fn later_edit_sql(n: &str, newest: &str, held_n: &str, held_newest: &str) -> Str /// When both backups have a date ([`later_backup_sql`]), a staged row from /// a later backup gives its text and earlier versions whatever their times /// say, when either differs from what the message holds (#1804); one from -/// an earlier or the same backup gives nothing. When either has no date, -/// the staged row gives them when it records a later edit +/// an earlier backup gives nothing. When either has no date, or the two +/// are equal, the staged row gives them when it records a later edit /// ([`later_edit_sql`]). /// /// An append skips a message production already holds, so a later backup in diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index c266543e9..bcf758171 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -909,8 +909,9 @@ async fn flush_staging_message_chunk( /// `db::staging::write_edit_map`): the attachments and reactions the staged /// message does not hold yet, and its mark and text as follows. When both /// backups have a date, a copy from a later backup gives its text, earlier -/// versions and mark, mark or no mark, and one from an earlier or the same -/// backup gives neither (#1741, #1804). When either has no date, the copy +/// versions and mark, mark or no mark, and one from an earlier backup gives +/// neither (#1741, #1804). When either has no date, or the two dates are +/// equal ([`db_staging::later_backup`]), the copy /// gives its text and earlier versions when it records a later edit, and /// its mark when it carries one. One import of two backups then stores /// what two separate imports of them store, in either file order (#1806, @@ -940,9 +941,9 @@ async fn add_staged_copy( .await? .with_context(|| format!("no staged message holds the copy of {}", row.msg.guid))?; let held_backup = db_staging::staged_backup_taken_at(tx, staged).await?; - match (staged_source.backup_taken_at, held_backup.as_deref()) { - (Some(copy_backup), Some(held_backup)) => { - if copy_backup > held_backup { + match db_staging::later_backup(staged_source.backup_taken_at, held_backup.as_deref()) { + Some(true) => { + if let Some(copy_backup) = staged_source.backup_taken_at { db_staging::take_staged_copy_from_later_backup( tx, staged, @@ -956,7 +957,8 @@ async fn add_staged_copy( .await?; } } - _ => { + Some(false) => {} + None => { if !row.msg.earlier_versions.is_empty() { db_staging::take_later_staged_copy( tx, diff --git a/crates/server/server/src/imports_api/tests/backup_dates.rs b/crates/server/server/src/imports_api/tests/backup_dates.rs index 01c78a363..cdb55d0f7 100644 --- a/crates/server/server/src/imports_api/tests/backup_dates.rs +++ b/crates/server/server/src/imports_api/tests/backup_dates.rs @@ -196,6 +196,33 @@ async fn a_later_backup_that_clears_an_unsent_mark_makes_the_text_searchable() { } } +/// Two files with the same backup date, as two reads of one Mac's +/// `chat.db` within one second give: the date cannot say which is newer, +/// so the rules for files without one hold, and the mark one of them +/// carries is added in every order rather than lost to the other. +#[tokio::test] +async fn equal_backup_dates_fall_back_to_the_rules_for_files_without_one() { + let tmp = TempDir::new().unwrap(); + let file = |name: &str, deletion: Option| { + backup_file( + tmp.path(), + name, + &Copy { + backup: Some(LATER_BACKUP), + text: "read twice", + versions: &[], + deletion, + }, + ) + }; + let unmarked = file("unmarked-same.jsonl", None); + let marked = file("marked-same.jsonl", Some(Deletion::Unsent)); + for held in every_order(tmp.path(), "same-date", [&unmarked, &marked]).await { + assert_eq!(held.deletion.as_deref(), Some("unsent"), "{held:?}"); + assert_eq!(held.backup_taken_at.as_deref(), Some(LATER_BACKUP_AT)); + } +} + /// The scenario of #1804: backup A lists part 1's earlier versions /// [x@t0, y@t100]; backup B, made later, after part 1 was unsent (which /// drops its versions) and part 0 was edited, lists part 0's [a@t0] only. diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 205f34e33..6a9641575 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -420,10 +420,11 @@ that date on each staged row, and `messages.backup_taken_at` keeps the date of the backup that decided the stored copy. When both copies have a date, the copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, whatever the versions' times say; a -copy from an earlier or the same backup changes neither. The duplicate flag -follows the text, because the dedupe compares the text. When either copy has -no date, nothing says which backup is newer, so the rules for files without -one hold: a copy with a mark adds it and one without leaves the mark held, +copy from an earlier backup changes neither. The duplicate flag follows the +text, because the dedupe compares the text. The date is kept to the second, +the form every stored time takes. When either copy has no date, or the two +dates are equal, nothing says which backup is newer, so the rules for files +without one hold: a copy with a mark adds it and one without leaves the mark held, and a copy takes the text when its newest earlier version is newer (`later_edit_sql` in `db/staging.rs`). Attachments and reactions add from either copy, because a backup that lacks one does not say it is gone. The From 5de3d2ea3b85bd947da0f7744ef3192909b0bcdc Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:22:38 -0400 Subject: [PATCH 20/42] fix(imessage): a Mac read is dated by chat.db and its write-ahead log Messages keeps chat.db in write-ahead mode, so a row written since the last checkpoint moves only chat.db-wal's time, and two reads of one Mac could carry the same date or an iPhone backup could read as later than a deletion the Mac already held. The date is now the newest modification time of chat.db, chat.db-wal and chat.db-shm. The format doc also says equal dates fall back to the rules for files without one. Co-Authored-By: Claude Opus 5.5 --- .../exporters/imessage-ir-exporter/src/run.rs | 22 +++++++++++++++---- .../imessage-ir-exporter/src/run/tests.rs | 14 +++++++++++- .../developer/architecture/common-message.md | 4 ++-- 3 files changed, 33 insertions(+), 7 deletions(-) diff --git a/crates/exporters/imessage-ir-exporter/src/run.rs b/crates/exporters/imessage-ir-exporter/src/run.rs index 6972d1817..75cfa4a03 100644 --- a/crates/exporters/imessage-ir-exporter/src/run.rs +++ b/crates/exporters/imessage-ir-exporter/src/run.rs @@ -136,9 +136,11 @@ impl ExportOptions { } /// When the Messages data was backed up, in Unix milliseconds: an - /// iPhone backup's `Manifest.plist` date, or a Mac `chat.db`'s - /// modification time, the last time Messages wrote it. `None` when - /// neither can be read. + /// iPhone backup's `Manifest.plist` date, or the last time Messages + /// wrote a Mac's `chat.db`: the newest modification time of `chat.db` + /// and its `-wal` and `-shm` files, because Messages keeps the database + /// in write-ahead mode and new rows reach `chat.db` itself only at a + /// checkpoint. `None` when neither can be read. pub fn backup_taken_at_unix_ms(&self) -> Option { backup_taken_at_unix_ms(&self.source) } @@ -148,7 +150,19 @@ impl ExportOptions { pub(crate) fn backup_taken_at_unix_ms(source: &Source) -> Option { match source.platform { Platform::Ios => ios_backup::ios_backup_date_unix_ms(&source.db_path), - Platform::MacOs => message_crate_core::file_modified_unix_ms(&source.db_path), + Platform::MacOs => { + let sidecar = |suffix: &str| { + let mut name = source.db_path.clone().into_os_string(); + name.push(suffix); + std::path::PathBuf::from(name) + }; + let (wal, shm) = (sidecar("-wal"), sidecar("-shm")); + message_crate_core::newest_file_modified_unix_ms([ + source.db_path.as_path(), + wal.as_path(), + shm.as_path(), + ]) + } } } diff --git a/crates/exporters/imessage-ir-exporter/src/run/tests.rs b/crates/exporters/imessage-ir-exporter/src/run/tests.rs index 1ac31ed82..a72649396 100644 --- a/crates/exporters/imessage-ir-exporter/src/run/tests.rs +++ b/crates/exporters/imessage-ir-exporter/src/run/tests.rs @@ -984,7 +984,7 @@ fn an_iphone_backup_is_dated_by_its_manifest() { } /// A Mac's `chat.db` records no backup date, so it is dated by when -/// Messages last wrote it. +/// Messages last wrote it, to the database or to its write-ahead log. #[test] fn a_mac_chat_db_is_dated_by_its_modification_time() { let dir = tempfile::tempdir().unwrap(); @@ -1003,6 +1003,18 @@ fn a_mac_chat_db_is_dated_by_its_modification_time() { backup_taken_at_unix_ms(&source), Some(message_crate_core::testutil::TEST_BACKUP_TAKEN_AT_UNIX_MS) ); + // Messages keeps chat.db in write-ahead mode: a row written since the + // last checkpoint moves only chat.db-wal's time. + let wal = dir.path().join("chat.db-wal"); + fs::write(&wal, b"not read here").unwrap(); + message_crate_core::testutil::set_modified_unix_ms( + &wal, + message_crate_core::testutil::TEST_BACKUP_TAKEN_AT_UNIX_MS + 3_600_000, + ); + assert_eq!( + backup_taken_at_unix_ms(&source), + Some(message_crate_core::testutil::TEST_BACKUP_TAKEN_AT_UNIX_MS + 3_600_000) + ); let undated = Source { db_path: dir.path().join("missing.db"), platform: Platform::Ios, diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index 90f32d5af..2ac887954 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -103,7 +103,7 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. | Source | Date | |--------|------| | Apple Messages from an iPhone backup | The `Date` in the backup's `Manifest.plist`, which is readable whether or not the backup is encrypted | -| Apple Messages from a Mac `chat.db` | The database file's modification time, when Messages last wrote it | +| Apple Messages from a Mac `chat.db` | When Messages last wrote the database: the newest modification time of `chat.db` and its `chat.db-wal` and `chat.db-shm` files | | WhatsApp from an iPhone backup | The `Date` in the backup's `Manifest.plist` | | WhatsApp from Android | The modification time of the database file, `msgstore.db.crypt15` or a decrypted `msgstore.db` | | WhatsApp from a ready-made `result.json` | The JSON file's modification time | @@ -112,7 +112,7 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. | OpenExtract, GO SMS Pro, SMS Backup+ | The newest modification time of the files read, because none of them records a date of its own | | An Export Run of the server | The newest backup date of the conversation's messages | -The import reads it to decide between two copies of one message from one source. The copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, in one import in either file order and across imports in either order. A copy from an earlier backup changes neither. A file that says nothing keeps the rules for files without a date: a mark adds and is never cleared, and an edit counts as later when its newest earlier version is newer. Attachments and reactions add from either copy. +The import reads it to decide between two copies of one message from one source. The copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, in one import in either file order and across imports in either order. A copy from an earlier backup changes neither. The server keeps the date to the second. A file that says nothing, or two copies whose dates are the same second, keep the rules for files without a date: a mark adds and is never cleared, and an edit counts as later when its newest earlier version is newer. Attachments and reactions add from either copy. CSV carries it in the `backup_taken_at_unix_ms` column of every row, and EML and MBOX in the `X-ME-Backup-Taken-At-Unix-Ms` header of every mail, blank or absent when the file says nothing. A value that is not a whole number is refused rather than read as no date. From 9cc6ce15c5ff8a7be6e45f7b32dfa9df6e763b03 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:23:13 -0400 Subject: [PATCH 21/42] fix(export): a conversation file names a backup only when all its messages share it An Export Run stamped every message of a conversation file with the newest backup date of any of them. Imported elsewhere, a message decided by an older backup then claimed the newest date and ignored a backup made between the two, and a message with no date gained one. The file carries one date for all its messages, so it now names one only when every message has the same, and none otherwise, which keeps the rules for files without a date. Co-Authored-By: Claude Opus 5.5 --- crates/libs/export/src/project.rs | 72 +++++++++++-------- crates/libs/export/src/run.rs | 4 +- .../developer/architecture/common-message.md | 2 +- 3 files changed, 47 insertions(+), 31 deletions(-) diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index 1fbc26ad4..c646e4329 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -24,8 +24,8 @@ pub fn conversation_key(msg: &Message) -> String { /// Build one conversation document from a seed message and the mapped rows. /// /// The document's backup date is the seed's `backup_taken_at`, which the -/// Export Run sets to the newest of the conversation's messages -/// ([`newest_backup_taken_at`]), or none when no message has one. +/// Export Run keeps only while every message of the conversation has the +/// same one ([`common_backup_taken_at`]), and none otherwise. pub fn build_document( source: &str, seed: &Message, @@ -85,13 +85,17 @@ pub fn build_document( } } -/// Keep on `seed` the newer of its own `backup_taken_at` and `msg`'s, so -/// the conversation file says the newest backup any of its messages came -/// from. Every stored time has one fixed RFC 3339 form, so the greater -/// string is the later instant. -pub fn newest_backup_taken_at(seed: &mut Message, msg: &Message) { - if msg.backup_taken_at > seed.backup_taken_at { - seed.backup_taken_at.clone_from(&msg.backup_taken_at); +/// Clear `seed`'s `backup_taken_at` when `msg`'s differs from it, so the +/// conversation file names a backup date only when every message of the +/// conversation came from that one backup. The file carries one date for +/// all its messages, and any one date would be wrong for some of them when +/// they differ: the newest would make a message decided by an older backup +/// win over a backup made between the two, and would date a message that +/// had none. With no date, an import of the file keeps the rules for files +/// without one. +pub fn common_backup_taken_at(seed: &mut Message, msg: &Message) { + if msg.backup_taken_at != seed.backup_taken_at { + seed.backup_taken_at = None; } } @@ -635,35 +639,47 @@ mod tests { assert_eq!(doc.conversation.stats.message_count, 2); } - /// The file says the newest backup any of the conversation's messages - /// came from, and nothing when none of them says. + /// The file says the backup its messages came from when they all came + /// from one, and nothing when any two differ or none says. #[test] - fn a_document_says_the_newest_backup_its_messages_came_from() { + fn a_document_says_its_backup_only_when_every_message_came_from_it() { let participant = || Participant { identity: Some("+1".into()), name: "Sam".into(), service: None, contact_id: None, }; - let mut seed = seed_message_with_participant(participant()); - assert_eq!( - build_document("imessage", &seed, vec![]) + let backup_of = |seed: &Message| { + build_document("imessage", seed, vec![]) .export - .backup_taken_at_unix_ms, - None - ); + .backup_taken_at_unix_ms + }; + let dated = |at: Option<&str>| { + let mut msg = seed_message_with_participant(participant()); + msg.backup_taken_at = at.map(str::to_string); + msg + }; + let later = Some("2026-09-30T18:45:12Z"); + let earlier = Some("2026-09-01T10:00:00Z"); - seed.backup_taken_at = Some("2026-09-01T10:00:00Z".into()); - let mut newer = seed_message_with_participant(participant()); - newer.backup_taken_at = Some("2026-09-30T18:45:12Z".into()); - let undated = seed_message_with_participant(participant()); - newest_backup_taken_at(&mut seed, &newer); - newest_backup_taken_at(&mut seed, &undated); + assert_eq!(backup_of(&dated(None)), None); + + let mut seed = dated(later); + common_backup_taken_at(&mut seed, &dated(later)); + assert_eq!(backup_of(&seed), Some(1_790_793_912_000)); + + let mut seed = dated(earlier); + common_backup_taken_at(&mut seed, &dated(later)); + common_backup_taken_at(&mut seed, &dated(None)); + assert_eq!(backup_of(&seed), None, "two backups: no one date is right"); + + let mut seed = dated(later); + common_backup_taken_at(&mut seed, &dated(None)); + common_backup_taken_at(&mut seed, &dated(later)); assert_eq!( - build_document("imessage", &seed, vec![]) - .export - .backup_taken_at_unix_ms, - Some(1_790_793_912_000) + backup_of(&seed), + None, + "a message without a date stays without one" ); } diff --git a/crates/libs/export/src/run.rs b/crates/libs/export/src/run.rs index c17c9ae24..7fd81a9b4 100644 --- a/crates/libs/export/src/run.rs +++ b/crates/libs/export/src/run.rs @@ -17,7 +17,7 @@ use crate::http::{CloseAction, ExportMessagesArgs, HttpSession}; use crate::journal::{self, ExportJournalEvent, ExportJournalState, ServerTarget}; use crate::part_file::write_asset; use crate::project::{ - ExportPath, build_document, conversation_key, export_path, newest_backup_taken_at, + ExportPath, build_document, common_backup_taken_at, conversation_key, export_path, to_ir_message, }; use message_crate_api_types::{ExportQueryList, ExportRun, ExportScope, Message}; @@ -504,7 +504,7 @@ impl<'a> Export<'a> { .entry(conversation_key(&msg)) // Keep first message as seed for conversation metadata. .or_insert_with(|| (msg.clone(), Vec::new())); - newest_backup_taken_at(seed, &msg); + common_backup_taken_at(seed, &msg); messages.push(ir); } match next_offset(offset, limit, page.total) { diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index 2ac887954..a7905a71d 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -110,7 +110,7 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. | SMS Backup & Restore | The root element's `backup_date` attribute; a file without one is dated by its modification time. A conversation read from two files is as new as the newer | | iMazing | The export date: the newest modification time of the CSV files iMazing wrote | | OpenExtract, GO SMS Pro, SMS Backup+ | The newest modification time of the files read, because none of them records a date of its own | -| An Export Run of the server | The newest backup date of the conversation's messages | +| An Export Run of the server | The backup date of the conversation's messages when they all have the same one, else `null`, because one date for messages from two backups would be wrong for some of them | The import reads it to decide between two copies of one message from one source. The copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, in one import in either file order and across imports in either order. A copy from an earlier backup changes neither. The server keeps the date to the second. A file that says nothing, or two copies whose dates are the same second, keep the rules for files without a date: a mark adds and is never cleared, and an edit counts as later when its newest earlier version is newer. Attachments and reactions add from either copy. From 973edb5017ff863e88bc1589e5b690031adbb7e1 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:23:52 -0400 Subject: [PATCH 22/42] fix(sbr): an XML export writes the backup date back The SMS Backup & Restore writer left backup_date off the root element, so an Export or conversion to XML read back in was dated by when Message Crate wrote the file and read as the newest backup. The writer now writes the newest backup date of the conversations it holds, the rule the reader follows for a conversation read from two files, and leaves the attribute out when none has one. Co-Authored-By: Claude Opus 5.5 --- .../sms-backup-restore-exporter/src/write.rs | 5 ++- crates/libs/sbr/src/lib.rs | 23 ++++++++++- crates/libs/sbr/src/read.rs | 39 +++++++++++++++++++ .../developer/architecture/common-message.md | 2 +- 4 files changed, 65 insertions(+), 4 deletions(-) diff --git a/crates/exporters/sms-backup-restore-exporter/src/write.rs b/crates/exporters/sms-backup-restore-exporter/src/write.rs index 9450c1bcf..85320de3e 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/write.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/write.rs @@ -54,10 +54,13 @@ impl SbrBackupSession { } /// Write the SMS and MMS of one conversation as SBR `` or `` - /// elements, and count every other message as left out. A conversation + /// elements, note its backup date for the file's `backup_date`, and + /// count every other message as left out. A conversation /// with no SMS or MMS writes nothing. pub fn append_document(&mut self, doc: &ConversationDocument) -> Result<()> { self.not_sms_or_mms += doc.messages.iter().filter(|m| !m.is_sms_or_mms()).count() as u64; + self.writer + .note_backup_date(doc.export.backup_taken_at_unix_ms); for msg in document_to_sbr_messages(doc, &self.output_dir)? { self.writer.write_message(&msg)?; } diff --git a/crates/libs/sbr/src/lib.rs b/crates/libs/sbr/src/lib.rs index bb0227c19..279de1877 100644 --- a/crates/libs/sbr/src/lib.rs +++ b/crates/libs/sbr/src/lib.rs @@ -67,6 +67,7 @@ pub struct SbrBackupWriter { body: BufWriter, count: u64, characters_left_out: u64, + backup_date_unix_ms: Option, } impl SbrBackupWriter { @@ -94,6 +95,7 @@ impl SbrBackupWriter { body, count: 0, characters_left_out: 0, + backup_date_unix_ms: None, }) } @@ -107,6 +109,15 @@ impl SbrBackupWriter { self.characters_left_out } + /// Note when the backup of messages written to this file was made, in + /// Unix milliseconds. The file's root `backup_date` says the newest date + /// noted, as the reader dates a conversation read from two files by the + /// newer, and is left out when none was noted. Without it, the reader + /// would date the file by when it was written. + pub fn note_backup_date(&mut self, unix_ms: Option) { + self.backup_date_unix_ms = self.backup_date_unix_ms.max(unix_ms); + } + /// Serialize one SMS/MMS element into the sidecar body file and increment /// the count. /// @@ -130,7 +141,8 @@ impl SbrBackupWriter { Ok(()) } - /// Finalize `count`, close ``, and replace `path`. + /// Finalize `count` and `backup_date`, close ``, and replace + /// `path`. /// /// # Errors /// @@ -154,7 +166,14 @@ impl SbrBackupWriter { out, r"" )?; - writeln!(out, r#""#, self.count)?; + match self.backup_date_unix_ms { + Some(date) => writeln!( + out, + r#""#, + self.count + )?, + None => writeln!(out, r#""#, self.count)?, + } // Every element written to the body ends with a line break, so // the closing tag starts a line of its own. io::copy(&mut body, &mut out) diff --git a/crates/libs/sbr/src/read.rs b/crates/libs/sbr/src/read.rs index 26cdb960a..a03470fb3 100644 --- a/crates/libs/sbr/src/read.rs +++ b/crates/libs/sbr/src/read.rs @@ -1334,6 +1334,45 @@ mod tests { assert_eq!(records[0].text, "line1\nline2\ttab"); } + /// The newest backup date noted is written as the root `backup_date`, + /// which the reader reads back, and a file with none noted has none. + #[test] + fn the_backup_date_noted_survives_a_round_trip() { + let dir = tempfile::tempdir().unwrap(); + let written = |name: &str, dates: &[Option]| { + let mut writer = crate::SbrBackupWriter::create(&dir.path().join(name)).unwrap(); + for &date in dates { + writer.note_backup_date(date); + } + let attrs: BTreeMap = [ + ("protocol", "0"), + ("address", "+15555550101"), + ("date", "1"), + ("type", "1"), + ("body", "hi"), + ] + .into_iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + writer + .write_message(&crate::SbrMessage::sms(attrs)) + .unwrap(); + let path = writer.finish().unwrap(); + parse_reader(std::fs::read(&path).unwrap().as_slice(), None) + .unwrap() + .1 + .backup_date_unix_ms + }; + assert_eq!( + written( + "dated.xml", + &[Some(1_788_256_800_000), None, Some(1_790_793_912_000)] + ), + Some(1_790_793_912_000) + ); + assert_eq!(written("undated.xml", &[None]), None); + } + #[test] fn a_literal_line_break_in_an_attribute_is_kept() { let xml = b""; diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index a7905a71d..c3f939e75 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -107,7 +107,7 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. | WhatsApp from an iPhone backup | The `Date` in the backup's `Manifest.plist` | | WhatsApp from Android | The modification time of the database file, `msgstore.db.crypt15` or a decrypted `msgstore.db` | | WhatsApp from a ready-made `result.json` | The JSON file's modification time | -| SMS Backup & Restore | The root element's `backup_date` attribute; a file without one is dated by its modification time. A conversation read from two files is as new as the newer | +| SMS Backup & Restore | The root element's `backup_date` attribute; a file without one is dated by its modification time. A conversation read from two files is as new as the newer. Message Crate's own XML export writes `backup_date` back, the newest date of the conversations it holds | | iMazing | The export date: the newest modification time of the CSV files iMazing wrote | | OpenExtract, GO SMS Pro, SMS Backup+ | The newest modification time of the files read, because none of them records a date of its own | | An Export Run of the server | The backup date of the conversation's messages when they all have the same one, else `null`, because one date for messages from two backups would be wrong for some of them | From edeada42c6aa181d50e5d594018f5bd254c50c36 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:24:21 -0400 Subject: [PATCH 23/42] docs: the changelog names no schema version, and two reasons catch up The 0.11.0 entries said "schema version 10", which the changelog rule leaves out; they now say which files are refused in the reader's words. The group title rule's reason no longer says the backup's date is not recorded: it says why the date the file now carries still does not decide a title. http-api.md says the owner's view of an Import Run carries the run's own times and nothing of the backup it read. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 13 ++++++------- .../contacts-identities-and-messages.md | 8 +++++--- docs/architecture/http-api.md | 5 +++++ 3 files changed, 16 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c344c6bbf..daac131f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,8 +32,8 @@ released versions carry their date on the heading. and a message unsent after the older backup reads as unsent. An older backup imported after a newer one changes nothing. Import details under Settings → Storage show the backup each import read and when it was made. - A message file at the previous schema version, 10, is refused, and the - backup must be exported again with this build. + Message files exported before they said when their backup was made are + refused, and the backup must be exported again with this build. - 2026-10-05: **A WhatsApp reply now names the message it quotes.** When the quoted message is in the same chat of the same backup, the reply is linked to it, as Apple Messages replies already were: a mail export threads @@ -154,11 +154,10 @@ released versions carry their date on the heading. in place of `num_replies`. - A Saved Search that uses `deleted:yes` finds fewer messages than before: it leaves unsent messages out. Add `or unsent:yes` to it to find both. -- Message files at schema version 10, exported before they said when their - backup was made, are refused when you import or convert them. Export the - backup again with this build. A program that reads the HTTP API finds the - backup's date in a message's `backup_taken_at` and an Import Run's - `backup_taken_at`. +- Message files exported before they said when their backup was made are + refused when you import or convert them. Export the backup again with + this build. A program that reads the HTTP API finds the backup's date in + a message's `backup_taken_at` and an Import Run's `backup_taken_at`. ## [0.10.1] - 2026-10-05 diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 6a9641575..8de14dfb7 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -405,9 +405,11 @@ otherwise keep an older title against a newer one. A title of only spaces counts as no title. The rule is the same in one batch as across several, in any order. Why: a group is renamed over time, so the copy whose messages run later carries the name the group has now. An old backup uploaded after a newer one can't bring the old -name back, because its messages stop earlier. The time a backup was made is not -recorded by every source, and the time an exporter ran says nothing about the -backup, so neither decides it +name back, because its messages stop earlier. The backup date a conversation +file carries (`export.backup_taken_at_unix_ms`) is missing where a source records +none, and for several sources is a file's modification time, which copying the +file can change; the time an exporter ran says nothing about the backup; so +neither decides the title ([#1408](https://github.com/messagecrate/message-crate/issues/1408)). **Between two copies of one message from one source, the copy from the later diff --git a/docs/architecture/http-api.md b/docs/architecture/http-api.md index 4532cb629..42174496b 100644 --- a/docs/architecture/http-api.md +++ b/docs/architecture/http-api.md @@ -585,6 +585,11 @@ What each reaches: `OwnerImportRun` or `OwnerExportRun`: the source, mode, tool, times, outcome and counts, with the counts an import's summary reported and how many issues it recorded, and for an export only which form its scope took. + The times are the run's own, when it started and ended; the owner reads + nothing of the backup an import read, neither the file the desktop app + recorded (`source_fingerprint`) nor when the backup was made + (`backup_taken_at`), because both say when and from what the account's + phone was backed up. Why: a staging summary lists the addresses of everyone in the backup, an issue names its conversation's file, a note names a file or an address, and an export's query is a search over From c43183331c788fb2371fcc93d9f77510d1b7db36 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:24:48 -0400 Subject: [PATCH 24/42] refactor(whatsapp): the JSON a run converts is a named struct, not a 5-tuple Adding the backup date made the result of reading the source a 5-tuple and reindented the whole branch. ReadJson names each value, and the branch is back at its old indentation. Co-Authored-By: Claude Opus 5.5 --- crates/exporters/whatsapp-exporter/src/run.rs | 242 ++++++++++-------- 1 file changed, 131 insertions(+), 111 deletions(-) diff --git a/crates/exporters/whatsapp-exporter/src/run.rs b/crates/exporters/whatsapp-exporter/src/run.rs index a57bfa46c..01b765d20 100644 --- a/crates/exporters/whatsapp-exporter/src/run.rs +++ b/crates/exporters/whatsapp-exporter/src/run.rs @@ -59,122 +59,126 @@ pub fn run(config: &ExporterConfig) -> Result { .map(owner_from_form) .transpose()?; - let (json_path, media_roots, owner_identity, backup_taken_at_unix_ms, _work_keep_alive) = - if let Some(json) = &source.json { - // Allowed roots are only the backup input and the JSON parent — never - // the process CWD, which would let crafted paths copy arbitrary files. - let mut media_roots = Vec::new(); - if let Some(path) = &input { - media_roots.push(path.clone()); - } - if let Some(parent) = json.parent() { - media_roots.push(parent.to_path_buf()); - } - media_roots.sort(); - media_roots.dedup(); - // A ready-made result.json names no owner; the form's number is all - // there is, and a conversion may leave it empty. It names no backup - // date either, so it is dated by when it was written. - ( - json.clone(), - media_roots, - form_owner, - message_crate_core::file_modified_unix_ms(json), - None, - ) - } else { - let platform = platform - .ok_or_else(|| anyhow::anyhow!("platform is required unless json is set"))?; - let input = match input { - Some(path) => path, - None => env::current_dir().context("resolve current working directory")?, - }; - - message_crate_core::check_cancel(config.cancel.as_ref())?; - let bin = resolve_wtsexporter()?; - let work = mark_output_and_make_work_directory(config)?; - let json_out = work.path().join("result.json"); - - // Cooperative only: cancel is checked before and after the external process. - // Killing wtsexporter mid-run is not implemented. - message_crate_core::check_cancel(config.cancel.as_ref())?; - let mut args = WtsexporterArgs { - platform, - input: input.clone(), - work_dir: work.path().to_path_buf(), - key: source.key.clone(), - backup: source.backup.clone(), - wa: source.wa.clone(), - media: source.media.clone(), - db: source.db.clone(), - business: source.business, - }; - // Read before an encrypted backup's files are decrypted, which points - // `args` at the decrypted copy, dated the moment it was made. - let backup_taken_at_unix_ms = backup_taken_at_unix_ms(&args); - // wtsexporter cannot be given an iPhone backup password, so an - // encrypted backup's WhatsApp files are decrypted into the work - // directory first and wtsexporter reads those instead of the backup. - // From a backup that is not encrypted, wtsexporter extracts them into - // the work directory itself, so that disk is checked for room first. - if platform == Platform::Ios { - if let Some(decrypted) = decrypt_if_encrypted(source, work.path(), config)? { - args.read_decrypted(decrypted); - } else if extracts_ios_backup(&args)? - && let Some(bytes) = extract_bytes(source)? - { - check_headroom(work.path(), bytes, Disk::Scratch)?; - } - } - message_crate_core::check_cancel(config.cancel.as_ref())?; - let log = run_wtsexporter(&bin, &args, &json_out)?; - message_crate_core::check_cancel(config.cancel.as_ref())?; + let read = if let Some(json) = &source.json { + // Allowed roots are only the backup input and the JSON parent — never + // the process CWD, which would let crafted paths copy arbitrary files. + let mut media_roots = Vec::new(); + if let Some(path) = &input { + media_roots.push(path.clone()); + } + if let Some(parent) = json.parent() { + media_roots.push(parent.to_path_buf()); + } + media_roots.sort(); + media_roots.dedup(); + // A ready-made result.json names no owner; the form's number is all + // there is, and a conversion may leave it empty. It names no backup + // date either, so it is dated by when it was written. + ReadJson { + json_path: json.clone(), + media_roots, + owner_identity: form_owner, + backup_taken_at_unix_ms: message_crate_core::file_modified_unix_ms(json), + work: None, + } + } else { + let platform = + platform.ok_or_else(|| anyhow::anyhow!("platform is required unless json is set"))?; + let input = match input { + Some(path) => path, + None => env::current_dir().context("resolve current working directory")?, + }; + + message_crate_core::check_cancel(config.cancel.as_ref())?; + let bin = resolve_wtsexporter()?; + let work = mark_output_and_make_work_directory(config)?; + let json_out = work.path().join("result.json"); - if !log.trim().is_empty() { - let trimmed = log.trim_end_matches('\n'); - messages.push(trimmed.to_string()); + // Cooperative only: cancel is checked before and after the external process. + // Killing wtsexporter mid-run is not implemented. + message_crate_core::check_cancel(config.cancel.as_ref())?; + let mut args = WtsexporterArgs { + platform, + input: input.clone(), + work_dir: work.path().to_path_buf(), + key: source.key.clone(), + backup: source.backup.clone(), + wa: source.wa.clone(), + media: source.media.clone(), + db: source.db.clone(), + business: source.business, + }; + // Read before an encrypted backup's files are decrypted, which points + // `args` at the decrypted copy, dated the moment it was made. + let backup_taken_at_unix_ms = backup_taken_at_unix_ms(&args); + // wtsexporter cannot be given an iPhone backup password, so an + // encrypted backup's WhatsApp files are decrypted into the work + // directory first and wtsexporter reads those instead of the backup. + // From a backup that is not encrypted, wtsexporter extracts them into + // the work directory itself, so that disk is checked for room first. + if platform == Platform::Ios { + if let Some(decrypted) = decrypt_if_encrypted(source, work.path(), config)? { + args.read_decrypted(decrypted); + } else if extracts_ios_backup(&args)? + && let Some(bytes) = extract_bytes(source)? + { + check_headroom(work.path(), bytes, Disk::Scratch)?; } + } + message_crate_core::check_cancel(config.cancel.as_ref())?; + let log = run_wtsexporter(&bin, &args, &json_out)?; + message_crate_core::check_cancel(config.cancel.as_ref())?; - let kept = config.output.join("wtsexporter_result.json"); - fs::copy(&json_out, &kept) - .with_context(|| format!("copy JSON to {}", kept.display()))?; - - // Work directory (wtsexporter extract) + backup input. The backup input is the - // process cwd when the config names no input. - let mut media_roots = vec![work.path().to_path_buf(), input]; - media_roots.sort(); - media_roots.dedup(); - - let owner_identity = match platform { - // wtsexporter copies the whole app-group domain into the work - // dir, preferences plist included; a backup someone extracted by - // hand has it under the input directory. The form's number covers a - // backup that carries no owner key (the Business app, a moved key). - Platform::Ios => { - match owner_from_backup(&media_roots, &mut messages).or(form_owner) { - Some(owner) => owner, - None => bail!( - "the backup does not contain your WhatsApp phone number; \ + if !log.trim().is_empty() { + let trimmed = log.trim_end_matches('\n'); + messages.push(trimmed.to_string()); + } + + let kept = config.output.join("wtsexporter_result.json"); + fs::copy(&json_out, &kept).with_context(|| format!("copy JSON to {}", kept.display()))?; + + // Work directory (wtsexporter extract) + backup input. The backup input is the + // process cwd when the config names no input. + let mut media_roots = vec![work.path().to_path_buf(), input]; + media_roots.sort(); + media_roots.dedup(); + + let owner_identity = match platform { + // wtsexporter copies the whole app-group domain into the work + // dir, preferences plist included; a backup someone extracted by + // hand has it under the input directory. The form's number covers a + // backup that carries no owner key (the Business app, a moved key). + Platform::Ios => match owner_from_backup(&media_roots, &mut messages).or(form_owner) { + Some(owner) => owner, + None => bail!( + "the backup does not contain your WhatsApp phone number; \ enter it on the import form" - ), - } - } - // A crypt backup carries no owner; the form checks the field is - // filled before the run starts, so this only guards a caller - // that skipped the form. - Platform::Android => form_owner - .ok_or_else(|| anyhow::anyhow!("Owner's WhatsApp number is required."))?, - }; - - ( - kept, - media_roots, - Some(owner_identity), - backup_taken_at_unix_ms, - Some(work), - ) + ), + }, + // A crypt backup carries no owner; the form checks the field is + // filled before the run starts, so this only guards a caller + // that skipped the form. + Platform::Android => { + form_owner.ok_or_else(|| anyhow::anyhow!("Owner's WhatsApp number is required."))? + } }; + ReadJson { + json_path: kept, + media_roots, + owner_identity: Some(owner_identity), + backup_taken_at_unix_ms, + work: Some(work), + } + }; + + let ReadJson { + json_path, + media_roots, + owner_identity, + backup_taken_at_unix_ms, + work, + } = read; if !json_path.is_file() { bail!("JSON not found: {}", json_path.display()); } @@ -195,7 +199,7 @@ pub fn run(config: &ExporterConfig) -> Result { issues: config.issues.as_ref(), })?; // The work directory goes once the conversion has copied the media. - drop(_work_keep_alive); + drop(work); let mut result = message_crate_core::finish_run(config, &report, needs_media_tools)?; messages.append(&mut result.messages); @@ -203,6 +207,22 @@ pub fn run(config: &ExporterConfig) -> Result { Ok(result) } +/// The `result.json` a run converts, and what the conversion needs to know +/// about the backup it came from. +struct ReadJson { + /// The JSON to convert. + json_path: std::path::PathBuf, + /// Where the conversion may look for media. + media_roots: Vec, + /// The owner's WhatsApp number, when known. + owner_identity: Option, + /// When the backup was made, in Unix milliseconds. + backup_taken_at_unix_ms: Option, + /// The work directory wtsexporter wrote into, kept until the media is + /// copied; `None` for a ready-made `result.json`. + work: Option, +} + /// When the backup wtsexporter reads was made, in Unix milliseconds: an /// iPhone backup's `Manifest.plist` date, or for Android the modification /// time of the WhatsApp database file (`msgstore.db.crypt15` or a decrypted From f9bac6840d436b4dceb4c15505640d2c075df984 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:28:14 -0400 Subject: [PATCH 25/42] refactor: name the backup order, and date an XML file by what it writes later_backup returns a BackupOrder (Later with the date, Earlier, Undecided) rather than an Option its caller had to unpack again. The SBR writer writes its root element once, and notes a conversation's backup date only when the conversation writes an SMS or MMS, so a WhatsApp or iMessage conversation left out does not move the file's date. The WhatsApp run's ConversionInput is destructured where it is built. Two doc comments are rewrapped. Co-Authored-By: Claude Opus 5.5 --- .../sms-backup-restore-exporter/src/write.rs | 13 +++--- .../src/write/tests.rs | 19 +++++++++ crates/exporters/whatsapp-exporter/src/run.rs | 21 +++++----- crates/libs/sbr/src/lib.rs | 13 +++--- crates/server/server/src/db/staging.rs | 31 +++++++++----- .../server/server/src/imports_api/staging.rs | 40 +++++++++---------- 6 files changed, 83 insertions(+), 54 deletions(-) diff --git a/crates/exporters/sms-backup-restore-exporter/src/write.rs b/crates/exporters/sms-backup-restore-exporter/src/write.rs index 85320de3e..67110ed5f 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/write.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/write.rs @@ -54,14 +54,17 @@ impl SbrBackupSession { } /// Write the SMS and MMS of one conversation as SBR `` or `` - /// elements, note its backup date for the file's `backup_date`, and - /// count every other message as left out. A conversation + /// elements, note its backup date for the file's `backup_date` when it + /// writes any, and count every other message as left out. A conversation /// with no SMS or MMS writes nothing. pub fn append_document(&mut self, doc: &ConversationDocument) -> Result<()> { self.not_sms_or_mms += doc.messages.iter().filter(|m| !m.is_sms_or_mms()).count() as u64; - self.writer - .note_backup_date(doc.export.backup_taken_at_unix_ms); - for msg in document_to_sbr_messages(doc, &self.output_dir)? { + let messages = document_to_sbr_messages(doc, &self.output_dir)?; + if !messages.is_empty() { + self.writer + .note_backup_date(doc.export.backup_taken_at_unix_ms); + } + for msg in messages { self.writer.write_message(&msg)?; } Ok(()) diff --git a/crates/exporters/sms-backup-restore-exporter/src/write/tests.rs b/crates/exporters/sms-backup-restore-exporter/src/write/tests.rs index 64e2d8166..53e4097b8 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/write/tests.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/write/tests.rs @@ -596,3 +596,22 @@ fn the_archive_counts_the_characters_xml_cannot_carry() { let text = fs::read_to_string(&path).unwrap(); assert!(text.contains(r#"body="bell and escape""#), "{text}"); } + +/// The file's `backup_date` is the newest date of the conversations it +/// writes messages for: a WhatsApp conversation, which writes none, does +/// not move it. +#[test] +fn the_backup_date_comes_from_the_conversations_written() { + let mut sms = message_ir::testutil::sample_document("hello ir"); + sms.export.backup_taken_at_unix_ms = Some(1_788_256_800_000); + let mut whatsapp = message_ir::testutil::sample_whatsapp_document("hello whatsapp"); + whatsapp.export.backup_taken_at_unix_ms = Some(1_790_793_912_000); + + let tmp = tempfile::tempdir().unwrap(); + let mut report = message_crate_core::ExportReport::default(); + let path = SbrArchive + .write(tmp.path(), &[sms, whatsapp], &mut report) + .unwrap(); + let text = fs::read_to_string(&path).unwrap(); + assert!(text.contains(r#"backup_date="1788256800000""#), "{text}"); +} diff --git a/crates/exporters/whatsapp-exporter/src/run.rs b/crates/exporters/whatsapp-exporter/src/run.rs index 01b765d20..828c21585 100644 --- a/crates/exporters/whatsapp-exporter/src/run.rs +++ b/crates/exporters/whatsapp-exporter/src/run.rs @@ -59,7 +59,13 @@ pub fn run(config: &ExporterConfig) -> Result { .map(owner_from_form) .transpose()?; - let read = if let Some(json) = &source.json { + let ConversionInput { + json_path, + media_roots, + owner_identity, + backup_taken_at_unix_ms, + work, + } = if let Some(json) = &source.json { // Allowed roots are only the backup input and the JSON parent — never // the process CWD, which would let crafted paths copy arbitrary files. let mut media_roots = Vec::new(); @@ -74,7 +80,7 @@ pub fn run(config: &ExporterConfig) -> Result { // A ready-made result.json names no owner; the form's number is all // there is, and a conversion may leave it empty. It names no backup // date either, so it is dated by when it was written. - ReadJson { + ConversionInput { json_path: json.clone(), media_roots, owner_identity: form_owner, @@ -163,7 +169,7 @@ pub fn run(config: &ExporterConfig) -> Result { } }; - ReadJson { + ConversionInput { json_path: kept, media_roots, owner_identity: Some(owner_identity), @@ -172,13 +178,6 @@ pub fn run(config: &ExporterConfig) -> Result { } }; - let ReadJson { - json_path, - media_roots, - owner_identity, - backup_taken_at_unix_ms, - work, - } = read; if !json_path.is_file() { bail!("JSON not found: {}", json_path.display()); } @@ -209,7 +208,7 @@ pub fn run(config: &ExporterConfig) -> Result { /// The `result.json` a run converts, and what the conversion needs to know /// about the backup it came from. -struct ReadJson { +struct ConversionInput { /// The JSON to convert. json_path: std::path::PathBuf, /// Where the conversion may look for media. diff --git a/crates/libs/sbr/src/lib.rs b/crates/libs/sbr/src/lib.rs index 279de1877..550d96845 100644 --- a/crates/libs/sbr/src/lib.rs +++ b/crates/libs/sbr/src/lib.rs @@ -166,14 +166,11 @@ impl SbrBackupWriter { out, r"" )?; - match self.backup_date_unix_ms { - Some(date) => writeln!( - out, - r#""#, - self.count - )?, - None => writeln!(out, r#""#, self.count)?, - } + let backup_date = self + .backup_date_unix_ms + .map(|date| format!(r#" backup_date="{date}""#)) + .unwrap_or_default(); + writeln!(out, r#""#, self.count)?; // Every element written to the body ends with a line break, so // the closing tag starts a line of its own. io::copy(&mut body, &mut out) diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index a28f5e6c3..b85be5de3 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -629,9 +629,9 @@ pub async fn take_staged_copy_from_later_backup( /// Give the staged message `staged` the mark `deletion` of another copy of /// it from the same import, when the two backups' dates cannot decide -/// ([`later_backup`]): a -/// copy that carries a mark adds it, and one with none leaves the staged -/// mark, as [`promote_deletion_marks`] does for a stored message. +/// ([`later_backup`]): a copy that carries a mark adds it, and one with +/// none leaves the staged mark, as [`promote_deletion_marks`] does for a +/// stored message. /// /// # Errors /// @@ -1254,15 +1254,28 @@ fn later_backup_sql(staged: &str, held: &str) -> String { ) } +/// Which of two copies of a message comes from the later backup, by +/// [`later_backup`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BackupOrder<'a> { + /// The copy comes from a later backup, made at this date. + Later(&'a str), + /// The copy comes from an earlier backup. + Earlier, + /// Either copy has no date, or the two dates are equal: the dates + /// cannot decide, and the rules for files without one hold. + Undecided, +} + /// [`later_backup_sql`]'s rule in Rust, for two copies of a message staged -/// in one import: `Some(true)` when the copy `staged` comes from a later -/// backup than the copy `held`, `Some(false)` when from an earlier one, and -/// `None` when either has no date or the two are equal. +/// in one import: whether the copy from the backup made at `staged` is +/// later than the copy held from the backup made at `held`. #[must_use] -pub fn later_backup(staged: Option<&str>, held: Option<&str>) -> Option { +pub fn later_backup<'a>(staged: Option<&'a str>, held: Option<&str>) -> BackupOrder<'a> { match (staged, held) { - (Some(staged), Some(held)) if staged != held => Some(staged > held), - _ => None, + (Some(staged), Some(held)) if staged > held => BackupOrder::Later(staged), + (Some(staged), Some(held)) if staged < held => BackupOrder::Earlier, + _ => BackupOrder::Undecided, } } diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index bcf758171..e89054f9d 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -12,8 +12,8 @@ use crate::db::handles::{ HandleIdCache, handle_type_on, upsert_handle_row, upsert_handle_row_cached, }; use crate::db::staging::{ - self as db_staging, StagedCopy, StagingAttachment, StagingConversation, StagingEarlierVersion, - StagingMessage, StagingMessageKey, StagingTapback, + self as db_staging, BackupOrder, StagedCopy, StagingAttachment, StagingConversation, + StagingEarlierVersion, StagingMessage, StagingMessageKey, StagingTapback, }; use crate::import_media; use crate::jsonl::{self, ReadRecordsError}; @@ -911,9 +911,9 @@ async fn flush_staging_message_chunk( /// backups have a date, a copy from a later backup gives its text, earlier /// versions and mark, mark or no mark, and one from an earlier backup gives /// neither (#1741, #1804). When either has no date, or the two dates are -/// equal ([`db_staging::later_backup`]), the copy -/// gives its text and earlier versions when it records a later edit, and -/// its mark when it carries one. One import of two backups then stores +/// equal ([`db_staging::later_backup`]), the copy gives its text and +/// earlier versions when it records a later edit, and its mark when it +/// carries one. One import of two backups then stores /// what two separate imports of them store, in either file order (#1806, /// #1837). async fn add_staged_copy( @@ -942,23 +942,21 @@ async fn add_staged_copy( .with_context(|| format!("no staged message holds the copy of {}", row.msg.guid))?; let held_backup = db_staging::staged_backup_taken_at(tx, staged).await?; match db_staging::later_backup(staged_source.backup_taken_at, held_backup.as_deref()) { - Some(true) => { - if let Some(copy_backup) = staged_source.backup_taken_at { - db_staging::take_staged_copy_from_later_backup( - tx, - staged, - &StagedCopy { - body: row.body.as_deref(), - deletion: row.msg.deletion, - versions: &row.msg.earlier_versions, - backup_taken_at: copy_backup, - }, - ) - .await?; - } + BackupOrder::Later(copy_backup) => { + db_staging::take_staged_copy_from_later_backup( + tx, + staged, + &StagedCopy { + body: row.body.as_deref(), + deletion: row.msg.deletion, + versions: &row.msg.earlier_versions, + backup_taken_at: copy_backup, + }, + ) + .await?; } - Some(false) => {} - None => { + BackupOrder::Earlier => {} + BackupOrder::Undecided => { if !row.msg.earlier_versions.is_empty() { db_staging::take_later_staged_copy( tx, From dcc9d245db17fa1ab2722814ae70dd73dcbe7a9f Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:38:47 -0400 Subject: [PATCH 26/42] feat(ir): each message says whether its time has milliseconds A message in the conversation file gains a required time_precision, seconds or milliseconds, and schema_version goes from 11 to 12; a version-11 file is refused by its version. The flag, never the time, says whether a time has milliseconds, since a millisecond time can end in .000. Every exporter writes the precision it already knew: iMazing, OpenExtract, GO SMS Pro's PDU files and SMS Backup+ without X-smssync-date write seconds; Apple Messages, WhatsApp, SMS Backup & Restore, GO SMS Pro's XML and SMS Backup+ with X-smssync-date write milliseconds. When an exporter keeps one copy of a message, a whole-second copy that takes a millisecond copy's time takes its precision too. The demo seed writes milliseconds. ir-format carries the field in every format: JSON and JSON Lines through serde, CSV in a time_precision column, EML and MBOX in an X-ME-Time-Precision header. A blank or unknown value is refused rather than guessed. Part of #1923. Co-Authored-By: Claude Opus 5.5 --- .../src/attachment_jobs/tests.rs | 2 + .../tests/convert_smoke.rs | 5 + .../imazing-exporter/tests/convert_smoke.rs | 17 +++- .../imessage-ir-exporter/src/convert.rs | 6 +- .../tests/helper_process.rs | 16 ++- .../tests/convert_smoke.rs | 3 + .../sms-backup-plus-exporter/src/emit.rs | 1 + .../tests/convert_smoke.rs | 39 ++++++++ .../sms-backup-restore-exporter/src/read.rs | 9 +- .../tests/convert_smoke.rs | 7 +- .../whatsapp-exporter/tests/convert_smoke.rs | 12 +++ .../ir-format/src/export_transforms/tests.rs | 3 + crates/libs/ir-format/src/lib_tests.rs | 98 +++++++++++++++++++ crates/libs/ir-format/src/read_csv.rs | 9 +- crates/libs/ir-format/src/write.rs | 4 +- crates/libs/ir/src/identity.rs | 74 ++++++++++---- crates/libs/ir/src/lib.rs | 8 +- crates/libs/ir/src/projection.rs | 3 +- crates/libs/ir/src/schema_version.rs | 29 ++++-- crates/libs/ir/src/testutil.rs | 3 + crates/libs/mail/src/headers.rs | 3 + crates/libs/mail/src/lib.rs | 1 + crates/libs/mail/src/parse.rs | 22 ++++- crates/libs/mail/src/tests.rs | 19 +++- crates/libs/push/src/project.rs | 4 +- crates/libs/push/tests/push_mock.rs | 1 + crates/server/demo-seed/src/conversations.rs | 6 +- .../server/src/imports_api/staging/tests.rs | 2 +- crates/server/server/src/imports_api/tests.rs | 10 +- .../server/server/src/test_support/lines.rs | 1 + .../fixtures/apple-messages-deletions.jsonl | 8 +- .../tests/fixtures/apple-messages-edits.jsonl | 8 +- .../fixtures/apple-messages-reactions.jsonl | 8 +- .../apple-messages-sub-second-times.jsonl | 6 +- src-tauri/src/commands/upload.rs | 1 + 35 files changed, 383 insertions(+), 65 deletions(-) diff --git a/crates/core/message-crate-core/src/attachment_jobs/tests.rs b/crates/core/message-crate-core/src/attachment_jobs/tests.rs index 67b69de56..46868cc0c 100644 --- a/crates/core/message-crate-core/src/attachment_jobs/tests.rs +++ b/crates/core/message-crate-core/src/attachment_jobs/tests.rs @@ -480,6 +480,7 @@ fn staging_a_conversation_writes_the_files_counts_them_and_frees_the_bytes() { messages: vec![IrMessage { guid: "guid-1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Mms, @@ -723,6 +724,7 @@ fn staging_frees_the_bytes_the_documents_were_carrying() { messages: vec![IrMessage { guid: "guid-1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Mms, diff --git a/crates/exporters/go-sms-pro-exporter/tests/convert_smoke.rs b/crates/exporters/go-sms-pro-exporter/tests/convert_smoke.rs index cacad45b2..910a2dc3f 100644 --- a/crates/exporters/go-sms-pro-exporter/tests/convert_smoke.rs +++ b/crates/exporters/go-sms-pro-exporter/tests/convert_smoke.rs @@ -60,6 +60,8 @@ fn convert_smoke_writes_csv_not_json() { ("direction", "incoming"), ("sender_identity", "+14075550107"), ("timestamp_unix_ms", "1609459200000"), + // The XML records milliseconds, though these end in `.000`. + ("time_precision", "milliseconds"), ("chat_identifier", "+14075550107"), ], ); @@ -73,6 +75,7 @@ fn convert_smoke_writes_csv_not_json() { ("text", "smoke reply"), ("direction", "outgoing"), ("timestamp_unix_ms", "1609459260000"), + ("time_precision", "milliseconds"), ], ); // The PDU beside the XML lands in the same conversation. @@ -82,6 +85,8 @@ fn convert_smoke_writes_csv_not_json() { ("text", "Hello one to one"), ("direction", "incoming"), ("sender_identity", "+14075550107"), + // A PDU file's name records whole seconds only. + ("time_precision", "seconds"), ], ); } diff --git a/crates/exporters/imazing-exporter/tests/convert_smoke.rs b/crates/exporters/imazing-exporter/tests/convert_smoke.rs index 9feb6c825..ba4a0fe43 100644 --- a/crates/exporters/imazing-exporter/tests/convert_smoke.rs +++ b/crates/exporters/imazing-exporter/tests/convert_smoke.rs @@ -49,9 +49,18 @@ fn convert_messages_keys_the_chat_by_its_number() { ("direction", "incoming"), ("service", "sms"), ("sender_display_name", "Bob Sample"), + // iMazing writes whole seconds only. + ("time_precision", "seconds"), + ], + ); + assert_csv_row( + &out, + &[ + ("text", "Hi Bob"), + ("direction", "outgoing"), + ("time_precision", "seconds"), ], ); - assert_csv_row(&out, &[("text", "Hi Bob"), ("direction", "outgoing")]); // The third row is an iMessage carrying an attachment, so it proves both // that the service column follows the source and that the attachment file // name reached the row rather than only the directory. @@ -91,7 +100,11 @@ fn convert_whatsapp_csv_direct() { // parse that lost the column or put the flag on the wrong message fails. assert_csv_row( &out, - &[("text", "Hello on WhatsApp"), ("direction", "incoming")], + &[ + ("text", "Hello on WhatsApp"), + ("direction", "incoming"), + ("time_precision", "seconds"), + ], ); assert_csv_row( &out, diff --git a/crates/exporters/imessage-ir-exporter/src/convert.rs b/crates/exporters/imessage-ir-exporter/src/convert.rs index 6fbf5a7a2..ef316c9df 100644 --- a/crates/exporters/imessage-ir-exporter/src/convert.rs +++ b/crates/exporters/imessage-ir-exporter/src/convert.rs @@ -28,7 +28,7 @@ use message_crate_core::{ use message_ir::{ ConversationDocument, ConversationMeta, ExportMeta, HandleType, IrAttachment, IrConversationType, IrDirection, IrImessage, IrMessage, IrMessageKind, IrParticipant, - IrService, SCHEMA_VERSION, nonempty, owner_sender, + IrService, SCHEMA_VERSION, TimePrecision, nonempty, owner_sender, }; use message_ir_format::FormatSink; use message_staging::{ @@ -410,6 +410,9 @@ fn message_to_ir( let message = IrMessage { guid: record.guid, timestamp_unix_ms: record.timestamp_unix_ms, + // `chat.db` records a message's time in nanoseconds (since macOS + // 10.13 and iOS 11). + time_precision: TimePrecision::Milliseconds, direction, service: IrService::parse(&record.service), message_kind: IrMessageKind::parse(&record.message_kind), @@ -1317,6 +1320,7 @@ mod tests { IrMessage { guid: format!("guid-{ts}"), timestamp_unix_ms: ts, + time_precision: TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, diff --git a/crates/exporters/imessage-ir-exporter/tests/helper_process.rs b/crates/exporters/imessage-ir-exporter/tests/helper_process.rs index 46dc3942c..1c58b262b 100644 --- a/crates/exporters/imessage-ir-exporter/tests/helper_process.rs +++ b/crates/exporters/imessage-ir-exporter/tests/helper_process.rs @@ -22,7 +22,7 @@ use chat_db_fixture::{ }; use common::{config, helper_binary}; use message_crate_core::{ExporterConfig, OutputFormat}; -use message_ir::{ConversationDocument, Deletion, IrDirection, IrMessage}; +use message_ir::{ConversationDocument, Deletion, IrDirection, IrMessage, TimePrecision}; use message_ir_format::{ read_conversation_csv, read_conversation_eml_dir, read_conversation_jsonl, read_conversation_mbox, @@ -82,6 +82,20 @@ fn exports_a_mac_chat_db_through_the_helper_process() { assert_eq!(staged.len(), 1, "{staged:?}"); assert_eq!(fs::read(&staged[0]).unwrap(), PHOTO_BYTES); assert!(all.contains("\"attachments/"), "{all}"); + + // `chat.db` records times below the second, so every message says + // milliseconds. + for path in &files { + let doc = read_conversation_jsonl(path).unwrap(); + for msg in &doc.messages { + assert_eq!( + msg.time_precision, + TimePrecision::Milliseconds, + "{}", + msg.guid + ); + } + } } /// Every conversation file of a Mac `chat.db` export says the database's diff --git a/crates/exporters/openextract-exporter/tests/convert_smoke.rs b/crates/exporters/openextract-exporter/tests/convert_smoke.rs index c29a10f8b..7feea154f 100644 --- a/crates/exporters/openextract-exporter/tests/convert_smoke.rs +++ b/crates/exporters/openextract-exporter/tests/convert_smoke.rs @@ -68,6 +68,8 @@ fn convert_all_conversations_keys_the_chat_by_its_number() { ("sender_identity", "+15555550122"), // 2020-01-01T17:00:00+00:00 in the source. ("timestamp_unix_ms", "1577898000000"), + // OpenExtract writes whole seconds only. + ("time_precision", "seconds"), ], ); // "Is From Me" is True on this row, and the direction column is where that @@ -78,6 +80,7 @@ fn convert_all_conversations_keys_the_chat_by_its_number() { ("text", "Hi Sam"), ("direction", "outgoing"), ("timestamp_unix_ms", "1577898060000"), + ("time_precision", "seconds"), ], ); } diff --git a/crates/exporters/sms-backup-plus-exporter/src/emit.rs b/crates/exporters/sms-backup-plus-exporter/src/emit.rs index 41d56152d..a2965c7ea 100644 --- a/crates/exporters/sms-backup-plus-exporter/src/emit.rs +++ b/crates/exporters/sms-backup-plus-exporter/src/emit.rs @@ -851,6 +851,7 @@ mod tests { messages: vec![IrMessage { guid: "g".into(), timestamp_unix_ms: 0, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Mms, diff --git a/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs b/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs index bb7a7a7f7..fe0c6555d 100644 --- a/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs +++ b/crates/exporters/sms-backup-plus-exporter/tests/convert_smoke.rs @@ -86,6 +86,9 @@ fn convert_smoke_writes_csv_not_json() { ("text", "Hello from Alice"), ("direction", "incoming"), ("timestamp_unix_ms", "1609459200000"), + // `X-smssync-date` records milliseconds, though these end in + // `.000`. + ("time_precision", "milliseconds"), ("chat_identifier", "+14075550107"), ], ); @@ -316,6 +319,7 @@ fn two_messages_sharing_an_smssync_id_both_survive() { ("text", "Hello from Alex"), ("direction", "outgoing"), ("timestamp_unix_ms", "1609459200313"), + ("time_precision", "milliseconds"), ], ); assert_csv_row( @@ -323,6 +327,7 @@ fn two_messages_sharing_an_smssync_id_both_survive() { &[ ("text", "Hello from Sam"), ("timestamp_unix_ms", "1609459300000"), + ("time_precision", "milliseconds"), ], ); // The same-chat pair: both messages, in one conversation file. @@ -434,3 +439,37 @@ fn the_backup_date_is_the_newest_mail_files_modification_time() { "{dates:?}" ); } + +/// A message with no `X-smssync-date` takes its time from the mail's `Date`, +/// which records whole seconds, and says so. +#[test] +fn a_message_timed_by_its_date_header_has_whole_seconds() { + let tmp = tempfile::tempdir().expect("tempdir"); + let input = tmp.path().join("in"); + let out = tmp.path().join("out"); + fs::create_dir_all(&input).expect("input dir"); + fs::write( + input.join("dated.eml"), + "From: dana@unknown.email\n\ + To: me@example.com\n\ + Subject: SMS with Dana\n\ + Date: Fri, 01 Jan 2021 00:00:05 +0000\n\ + X-smssync-type: 1\n\ + X-smssync-address: +15555550133\n\ + Content-Type: text/plain; charset=utf-8\n\ + \n\ + Timed by Date\n", + ) + .expect("write fixture"); + + convert(&[input.as_path()], &out).expect("convert"); + + assert_csv_row( + &out.join("+15555550133.csv"), + &[ + ("text", "Timed by Date"), + ("timestamp_unix_ms", "1609459205000"), + ("time_precision", "seconds"), + ], + ); +} diff --git a/crates/exporters/sms-backup-restore-exporter/src/read.rs b/crates/exporters/sms-backup-restore-exporter/src/read.rs index 9c4f8c715..8b95ceaa9 100644 --- a/crates/exporters/sms-backup-restore-exporter/src/read.rs +++ b/crates/exporters/sms-backup-restore-exporter/src/read.rs @@ -493,8 +493,10 @@ fn dedupe(messages: &mut Vec) -> u64 { let before = messages.len(); let mut kept = kept.into_iter(); messages.retain_mut(|m| match kept.next().flatten() { - Some(ms) => { - if m.time().0 != ms { + Some((ms, precision)) => { + // A whole-second copy that keeps a millisecond copy's time keeps + // its precision too, even when that time ends in `.000`. + if m.time() != (ms, precision) { m.date_ms = ms.to_string(); } true @@ -624,7 +626,7 @@ fn ir_message( message: &PendingMessage, owner: &(Option, Option), ) -> IrMessage { - let (timestamp_unix_ms, _) = message.time(); + let (timestamp_unix_ms, time_precision) = message.time(); let digests = message.attachment_digests(); let (sender_identity, sender_display_name) = if message.is_from_me { owner.clone() @@ -643,6 +645,7 @@ fn ir_message( }) .into_string(), timestamp_unix_ms, + time_precision, direction: if message.is_from_me { IrDirection::Outgoing } else { diff --git a/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs b/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs index 3392361e3..3af8a95e9 100644 --- a/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs +++ b/crates/exporters/sms-backup-restore-exporter/tests/convert_smoke.rs @@ -70,6 +70,7 @@ fn convert_export_smoke_on_sample_fixture() { ("text", "hello"), ("direction", "incoming"), ("timestamp_unix_ms", "1400773261000"), + ("time_precision", "milliseconds"), ("chat_identifier", "+15555550101"), ], ); @@ -83,6 +84,7 @@ fn convert_export_smoke_on_sample_fixture() { ("text", "hey"), ("direction", "outgoing"), ("timestamp_unix_ms", "1400773321000"), + ("time_precision", "milliseconds"), ], ); // The `` is a different parse: its text lives in a `text/plain` part @@ -95,6 +97,7 @@ fn convert_export_smoke_on_sample_fixture() { ("direction", "incoming"), ("message_kind", "mms"), ("timestamp_unix_ms", "1400773400000"), + ("time_precision", "milliseconds"), ], ); @@ -344,7 +347,7 @@ fn convert_export_json_and_jsonl_use_pristine_v4() { .expect("expected .json"); let raw = fs::read_to_string(&json_path).unwrap(); let doc: serde_json::Value = serde_json::from_str(&raw).unwrap(); - assert_eq!(doc["schema_version"], 11); + assert_eq!(doc["schema_version"], 12); assert!( doc["conversation"]["stats"]["message_count"] .as_u64() @@ -389,7 +392,7 @@ fn convert_export_json_and_jsonl_use_pristine_v4() { let body = fs::read_to_string(&jsonl_path).unwrap(); let mut lines = body.lines(); let header: serde_json::Value = serde_json::from_str(lines.next().unwrap()).unwrap(); - assert_eq!(header["schema_version"], 11); + assert_eq!(header["schema_version"], 12); assert!(header.get("messages").is_none()); assert!( header["conversation"]["stats"]["message_count"] diff --git a/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs b/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs index 22d181bbf..4daa16736 100644 --- a/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs +++ b/crates/exporters/whatsapp-exporter/tests/convert_smoke.rs @@ -230,6 +230,18 @@ fn messages_keep_their_time_and_their_identity() { (9_999_999_999_000, "last second"), ] ); + // WhatsApp stores milliseconds, and the seconds wtsexporter writes carry + // them as a fraction, so every time has milliseconds, whole or not. + assert!( + doc.messages + .iter() + .all(|m| m.time_precision == message_ir::TimePrecision::Milliseconds), + "{:?}", + doc.messages + .iter() + .map(|m| m.time_precision) + .collect::>() + ); assert_ne!( doc.messages[1].guid, doc.messages[2].guid, "the same text in the same second is two messages" diff --git a/crates/libs/ir-format/src/export_transforms/tests.rs b/crates/libs/ir-format/src/export_transforms/tests.rs index 44a850d02..9f9d0bfbe 100644 --- a/crates/libs/ir-format/src/export_transforms/tests.rs +++ b/crates/libs/ir-format/src/export_transforms/tests.rs @@ -34,6 +34,7 @@ fn doc_with_image_attachment() -> ConversationDocument { messages: vec![IrMessage { guid: "guid-1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, @@ -240,6 +241,7 @@ fn doc_with_a_marker_in_every_field() -> ConversationDocument { messages: vec![IrMessage { guid: "LEAK-27".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, @@ -313,6 +315,7 @@ const KEPT_AS_IS: &[(&str, &str)] = &[ ), ("conversation.conversation_type", "enum value"), ("conversation.participants[].identity_type", "enum value"), + ("messages[].time_precision", "enum value"), ("messages[].direction", "enum value"), ("messages[].service", "enum value"), ("messages[].message_kind", "enum value"), diff --git a/crates/libs/ir-format/src/lib_tests.rs b/crates/libs/ir-format/src/lib_tests.rs index e8f07c436..efb13e591 100644 --- a/crates/libs/ir-format/src/lib_tests.rs +++ b/crates/libs/ir-format/src/lib_tests.rs @@ -4,6 +4,7 @@ use super::*; use message_crate_core::OutputFormat; use message_ir::{ ConversationDocument, IrDirection, IrImessage, IrMessage, IrMessageKind, IrService, + TimePrecision, }; use serde_json::{Value, json}; use std::fs; @@ -234,6 +235,7 @@ fn csv_omits_trivial_parts_json_keeps_rich_parts() { doc.messages.push(IrMessage { guid: "MULTI-PART-GUID".into(), timestamp_unix_ms: 1_400_773_263_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, @@ -904,3 +906,99 @@ fn csv_refuses_a_backup_date_that_is_not_a_number() { "{err:#}" ); } + +/// Write a document of two messages, one whose source recorded whole +/// seconds and one whose source recorded milliseconds that end in `.000`, +/// in `format`, and read back each message's precision in time order. +fn precisions_after_round_trip(format: OutputFormat) -> Vec<(i64, TimePrecision)> { + let mut doc = message_ir::testutil::sample_document("whole second"); + let mut whole = doc.messages[0].clone(); + whole.timestamp_unix_ms = 1_400_773_261_000; + whole.time_precision = TimePrecision::Seconds; + let mut exact = whole.clone(); + exact.guid = "bbccddeeff00112233445566778899aa".into(); + exact.text = "milliseconds that end in .000".into(); + exact.timestamp_unix_ms = 1_400_773_262_000; + exact.time_precision = TimePrecision::Milliseconds; + doc.messages = vec![whole, exact]; + let tmp = tempfile::tempdir().unwrap(); + let path = write_format(tmp.path(), format, doc).unwrap(); + let back = match format { + OutputFormat::Json => read_conversation_json(&path), + OutputFormat::Jsonl => read_conversation_jsonl(&path), + OutputFormat::Csv => read_conversation_csv(&path), + OutputFormat::Eml => read_conversation_eml_dir(&path), + OutputFormat::Mbox => read_conversation_mbox(&path), + other => panic!("no round trip for {}", other.as_str()), + } + .unwrap(); + let mut out: Vec<(i64, TimePrecision)> = back + .messages + .iter() + .map(|m| (m.timestamp_unix_ms, m.time_precision)) + .collect(); + out.sort_unstable(); + out +} + +/// Every format keeps whether a message's time has milliseconds, and a +/// millisecond time that ends in `.000` reads back as milliseconds, never +/// as whole seconds. +#[test] +fn every_format_keeps_whether_a_time_has_milliseconds() { + for format in [ + OutputFormat::Json, + OutputFormat::Jsonl, + OutputFormat::Csv, + OutputFormat::Eml, + OutputFormat::Mbox, + ] { + assert_eq!( + precisions_after_round_trip(format), + [ + (1_400_773_261_000, TimePrecision::Seconds), + (1_400_773_262_000, TimePrecision::Milliseconds), + ], + "{}", + format.as_str() + ); + } +} + +/// A CSV row whose `time_precision` is blank is refused, never read as +/// either precision. +#[test] +fn csv_refuses_a_row_without_a_time_precision() { + let doc = message_ir::testutil::sample_document("hello"); + let tmp = tempfile::tempdir().unwrap(); + let path = write_format(tmp.path(), OutputFormat::Csv, doc).unwrap(); + let text = fs::read_to_string(&path).unwrap(); + let mut lines: Vec = text.lines().map(str::to_string).collect(); + let column = CSV_HEADERS + .iter() + .position(|h| *h == "time_precision") + .unwrap(); + let mut row = csv::ReaderBuilder::new() + .has_headers(false) + .from_reader(lines[1].as_bytes()) + .records() + .next() + .unwrap() + .unwrap() + .iter() + .map(str::to_string) + .collect::>(); + row[column] = String::new(); + let mut out = csv::Writer::from_writer(Vec::new()); + out.write_record(&row).unwrap(); + lines[1] = String::from_utf8(out.into_inner().unwrap()) + .unwrap() + .trim_end() + .to_string(); + fs::write(&path, lines.join("\n")).unwrap(); + let err = read_conversation_csv(&path).unwrap_err(); + assert!( + format!("{err:#}").contains("bad time_precision \"\""), + "{err:#}" + ); +} diff --git a/crates/libs/ir-format/src/read_csv.rs b/crates/libs/ir-format/src/read_csv.rs index bc24e55ff..a57d0dc6f 100644 --- a/crates/libs/ir-format/src/read_csv.rs +++ b/crates/libs/ir-format/src/read_csv.rs @@ -7,8 +7,8 @@ use message_csv::{AttachmentCell, ParticipantCell}; use message_ir::{ ConversationDocument, ConversationHeader, ConversationMeta, ConversationStats, EarlierVersion, ExportMeta, IrAttachment, IrConversationType, IrDirection, IrImessage, IrMessage, - IrMessageKind, IrParticipant, IrService, Reaction, ReplyTo, SCHEMA_VERSION, nonempty, - parse_android_type, + IrMessageKind, IrParticipant, IrService, Reaction, ReplyTo, SCHEMA_VERSION, TimePrecision, + nonempty, parse_android_type, }; use serde_json::Value; use std::collections::HashMap; @@ -116,6 +116,10 @@ fn message_from_record(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> let timestamp_unix_ms = get("timestamp_unix_ms") .parse::() .with_context(|| format!("bad timestamp_unix_ms {:?}", get("timestamp_unix_ms")))?; + // The flag, never the time, says whether a time has milliseconds, so a + // blank or unknown precision is refused rather than guessed. + let time_precision = TimePrecision::parse(get("time_precision")) + .with_context(|| format!("bad time_precision {:?}", get("time_precision")))?; let direction = match get("direction").to_ascii_lowercase().as_str() { "outgoing" => IrDirection::Outgoing, _ => IrDirection::Incoming, @@ -158,6 +162,7 @@ fn message_from_record(cols: &HashMap<&str, usize>, row: &csv::StringRecord) -> Ok(IrMessage { guid: get("guid").to_string(), timestamp_unix_ms, + time_precision, direction, service: IrService::parse(get("service")), message_kind: IrMessageKind::parse(get("message_kind")), diff --git a/crates/libs/ir-format/src/write.rs b/crates/libs/ir-format/src/write.rs index f7d6fbaae..3928e486b 100644 --- a/crates/libs/ir-format/src/write.rs +++ b/crates/libs/ir-format/src/write.rs @@ -26,6 +26,7 @@ pub const CSV_HEADERS: &[&str] = &[ "timestamp_utc", "timestamp_display", "timestamp_unix_ms", + "time_precision", "direction", "service", "sender_identity", @@ -381,7 +382,7 @@ fn csv_record<'a>( backup_taken_at: &'a str, msg: &'a IrMessage, cells: &'a MessageCells, -) -> [&'a str; 47] { +) -> [&'a str; 48] { let im = &cells.imessage; [ doc.conversation.chat_identifier.as_str(), @@ -393,6 +394,7 @@ fn csv_record<'a>( cells.ts_utc.as_str(), cells.ts_display.as_str(), cells.timestamp_unix_ms.as_str(), + msg.time_precision.as_str(), msg.direction.as_str(), msg.service.as_str(), msg.sender_identity.as_deref().unwrap_or(""), diff --git a/crates/libs/ir/src/identity.rs b/crates/libs/ir/src/identity.rs index 6f72fa799..8784ff813 100644 --- a/crates/libs/ir/src/identity.rs +++ b/crates/libs/ir/src/identity.rs @@ -11,11 +11,17 @@ //! so the id carries no counter of occurrences: a counter would depend on the //! order the copies are read in. +use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::collections::HashMap; -/// How finely a source recorded a message's time. -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +/// How finely a source recorded a message's time: a message's +/// `time_precision` in the conversation file. +/// +/// The flag, never the value, says whether a time has milliseconds: a +/// millisecond time can end in `.000`, and a whole-second one always does. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)] +#[serde(rename_all = "lowercase")] pub enum TimePrecision { /// Whole seconds: the milliseconds are zero because the source has none. Seconds, @@ -23,6 +29,26 @@ pub enum TimePrecision { Milliseconds, } +impl TimePrecision { + /// The name the conversation file, the database and the HTTP API use: + /// `seconds` or `milliseconds`. + pub fn as_str(self) -> &'static str { + match self { + Self::Seconds => "seconds", + Self::Milliseconds => "milliseconds", + } + } + + /// The precision [`Self::as_str`] names, or `None` for any other text. + pub fn parse(s: &str) -> Option { + match s { + "seconds" => Some(Self::Seconds), + "milliseconds" => Some(Self::Milliseconds), + _ => None, + } + } +} + /// What a message's identity is made from. /// /// The chat is the caller's: an exporter passes the conversation's chat id, @@ -164,10 +190,10 @@ pub struct MessageCopy<'a> { /// which copy was read first. Two millisecond copies with different times /// are two messages. /// -/// Returns one entry per copy, in order: `Some(time)` for a copy that is -/// kept, with the time in milliseconds it keeps, and `None` for a copy that -/// repeats a kept one. -pub fn one_copy_per_message(copies: &[MessageCopy<'_>]) -> Vec> { +/// Returns one entry per copy, in order: `Some((time, precision))` for a +/// copy that is kept, with the time in milliseconds it keeps and how finely +/// that time was recorded, and `None` for a copy that repeats a kept one. +pub fn one_copy_per_message(copies: &[MessageCopy<'_>]) -> Vec> { struct Kept { index: usize, timestamp_unix_ms: i64, @@ -244,7 +270,12 @@ pub fn one_copy_per_message(copies: &[MessageCopy<'_>]) -> Vec> { } } for k in kept { - fate[k.index] = Some(k.timestamp_unix_ms); + let precision = if k.exact { + TimePrecision::Milliseconds + } else { + TimePrecision::Seconds + }; + fate[k.index] = Some((k.timestamp_unix_ms, precision)); } } fate @@ -353,7 +384,10 @@ mod tests { ]; assert_eq!( one_copy_per_message(&copies), - [Some(1_609_459_200_000), Some(1_609_459_200_000)] + [ + Some((1_609_459_200_000, SECS)), + Some((1_609_459_200_000, SECS)) + ] ); } @@ -365,7 +399,7 @@ mod tests { ]; assert_eq!( one_copy_per_message(&copies), - [Some(1_609_459_200_300), None] + [Some((1_609_459_200_300, MS)), None] ); } @@ -377,7 +411,7 @@ mod tests { ]; assert_eq!( one_copy_per_message(&copies), - [Some(1_609_459_200_100), Some(1_609_459_200_400)] + [Some((1_609_459_200_100, MS)), Some((1_609_459_200_400, MS))] ); } @@ -387,11 +421,11 @@ mod tests { let exact = copy("+15555550122", 1_609_459_200_876, MS, &[]); assert_eq!( one_copy_per_message(&[whole, exact]), - [None, Some(1_609_459_200_876)] + [None, Some((1_609_459_200_876, MS))] ); assert_eq!( one_copy_per_message(&[exact, whole]), - [Some(1_609_459_200_876), None] + [Some((1_609_459_200_876, MS)), None] ); } @@ -402,11 +436,11 @@ mod tests { let xml = copy("+15555550122", 1_609_459_200_250, MS, &[]); assert_eq!( one_copy_per_message(&[xml, pdu]), - [None, Some(1_609_459_200_250)] + [None, Some((1_609_459_200_250, MS))] ); assert_eq!( one_copy_per_message(&[pdu, xml]), - [Some(1_609_459_200_250), None] + [Some((1_609_459_200_250, MS)), None] ); } @@ -420,7 +454,10 @@ mod tests { ]; assert_eq!( one_copy_per_message(&copies), - [Some(1_609_459_200_000), Some(1_609_459_200_000)] + [ + Some((1_609_459_200_000, SECS)), + Some((1_609_459_200_000, SECS)) + ] ); } @@ -432,7 +469,7 @@ mod tests { b.vendor_key = Some("B2"); assert_eq!( one_copy_per_message(&[a, b]), - [Some(1_609_459_200_000), Some(1_609_459_200_000)] + [Some((1_609_459_200_000, MS)), Some((1_609_459_200_000, MS))] ); } @@ -444,7 +481,10 @@ mod tests { ]; assert_eq!( one_copy_per_message(&copies), - [Some(1_609_459_200_000), Some(1_609_459_201_000)] + [ + Some((1_609_459_200_000, SECS)), + Some((1_609_459_201_000, SECS)) + ] ); } } diff --git a/crates/libs/ir/src/lib.rs b/crates/libs/ir/src/lib.rs index ba8709bf6..9dd139cef 100644 --- a/crates/libs/ir/src/lib.rs +++ b/crates/libs/ir/src/lib.rs @@ -138,7 +138,7 @@ impl std::fmt::Display for UnknownDeletion { impl std::error::Error for UnknownDeletion {} /// Schema version written into every [`ConversationDocument`]. -pub const SCHEMA_VERSION: u32 = 11; +pub const SCHEMA_VERSION: u32 = 12; /// One exported chat: export metadata, conversation roster and stats, and messages. /// @@ -473,6 +473,12 @@ pub struct IrMessage { pub guid: String, /// Unix milliseconds; the chronological sort key. pub timestamp_unix_ms: i64, + /// Whether the source recorded [`Self::timestamp_unix_ms`] to the + /// millisecond or in whole seconds. Required: the flag, never the + /// value, says whether a time has milliseconds, since a millisecond + /// time can end in `.000`. The import shows a whole-second message once + /// when its source also holds it with milliseconds in the same second. + pub time_precision: TimePrecision, /// Incoming or outgoing. pub direction: IrDirection, /// Transport the message arrived on. diff --git a/crates/libs/ir/src/projection.rs b/crates/libs/ir/src/projection.rs index aee12212e..36f8cdc09 100644 --- a/crates/libs/ir/src/projection.rs +++ b/crates/libs/ir/src/projection.rs @@ -274,7 +274,7 @@ pub fn pending_to_document( let mut replies: Vec<(usize, PendingReply)> = Vec::new(); let mut guid_by_reply_key: HashMap> = HashMap::new(); for ((msg, p), kept) in convo.messages.iter().zip(&prepared).zip(kept) { - let Some(timestamp_unix_ms) = kept else { + let Some((timestamp_unix_ms, time_precision)) = kept else { tally.duplicates += 1; continue; }; @@ -322,6 +322,7 @@ pub fn pending_to_document( messages.push(IrMessage { guid, timestamp_unix_ms, + time_precision, direction: if outgoing { IrDirection::Outgoing } else { diff --git a/crates/libs/ir/src/schema_version.rs b/crates/libs/ir/src/schema_version.rs index 3b99d3f3a..d7905fea9 100644 --- a/crates/libs/ir/src/schema_version.rs +++ b/crates/libs/ir/src/schema_version.rs @@ -3,12 +3,12 @@ //! Every reader of a [`ConversationDocument`](crate::ConversationDocument) or //! its JSON Lines header — the format reader, the push client, the server's //! import — refuses a version other than [`SCHEMA_VERSION`] with the same -//! words, and refuses it before parsing the rest of the file: a version-10 -//! file is not expected to match version 11 (version 10 did not say when its -//! backup was made, so an import could not tell which of two backups of one -//! phone is the later one, and a file read as version 11 would claim it has -//! no date when it only never asked), and the person should read "schema -//! version 10", not a file that imports by other rules than its own. +//! words, and refuses it before parsing the rest of the file: a version-11 +//! file is not expected to match version 12 (version 11 did not say whether +//! a message's time has milliseconds, so a time ending in `.000` could be +//! either, and the import could not tell a whole-second copy of a message +//! from a millisecond one), and the person should read "schema version 11", +//! not a file that imports by other rules than its own. use crate::SCHEMA_VERSION; use serde::Deserialize; @@ -107,7 +107,22 @@ mod tests { .to_string(), format!("This file is schema version 10; Message Crate reads version {SCHEMA_VERSION}") ); - assert_eq!(SCHEMA_VERSION, 11); + } + + /// Version 11 had no `time_precision` on a message; version 12 says + /// whether each time has milliseconds, and the import shows a + /// whole-second message once when its source also holds it with + /// milliseconds. A version-11 file is refused by its version, never read + /// with its messages missing the field. + #[test] + fn refuses_a_version_11_file_by_name() { + assert_eq!( + check_schema_version_in_json(r#"{"schema_version":11,"export":{}}"#) + .unwrap_err() + .to_string(), + format!("This file is schema version 11; Message Crate reads version {SCHEMA_VERSION}") + ); + assert_eq!(SCHEMA_VERSION, 12); } #[test] diff --git a/crates/libs/ir/src/testutil.rs b/crates/libs/ir/src/testutil.rs index f394d34a9..5f7921f11 100644 --- a/crates/libs/ir/src/testutil.rs +++ b/crates/libs/ir/src/testutil.rs @@ -35,6 +35,7 @@ pub fn sample_document(text: &str) -> ConversationDocument { messages: vec![IrMessage { guid: "aabbccddeeff00112233445566778899".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: crate::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, @@ -113,6 +114,7 @@ pub fn sample_imessage_document() -> ConversationDocument { IrMessage { guid: "AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: crate::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, @@ -157,6 +159,7 @@ pub fn sample_imessage_document() -> ConversationDocument { IrMessage { guid: "TAPBACK-GUID-0001".into(), timestamp_unix_ms: 1_400_773_262_000, + time_precision: crate::TimePrecision::Milliseconds, direction: IrDirection::Outgoing, service: IrService::IMessage, message_kind: IrMessageKind::Tapback, diff --git a/crates/libs/mail/src/headers.rs b/crates/libs/mail/src/headers.rs index 50702809b..478a66016 100644 --- a/crates/libs/mail/src/headers.rs +++ b/crates/libs/mail/src/headers.rs @@ -15,6 +15,9 @@ pub(crate) const SERVICE: &str = "X-ME-Service"; pub(crate) const MESSAGE_KIND: &str = "X-ME-Message-Kind"; /// Message timestamp in Unix milliseconds. pub(crate) const TIMESTAMP_UNIX_MS: &str = "X-ME-Timestamp-Unix-Ms"; +/// Whether the timestamp has milliseconds: `seconds` or `milliseconds`, as +/// `message_ir::TimePrecision` names them. +pub(crate) const TIME_PRECISION: &str = "X-ME-Time-Precision"; /// Message guid. pub(crate) const GUID: &str = "X-ME-Guid"; /// Export source id. diff --git a/crates/libs/mail/src/lib.rs b/crates/libs/mail/src/lib.rs index 7590e3a7b..5438e3aab 100644 --- a/crates/libs/mail/src/lib.rs +++ b/crates/libs/mail/src/lib.rs @@ -885,6 +885,7 @@ fn conversation_headers<'m>( (headers::SERVICE, msg.message.service.as_str()), (headers::MESSAGE_KIND, msg.message.message_kind.as_str()), (headers::TIMESTAMP_UNIX_MS, ×tamp), + (headers::TIME_PRECISION, msg.message.time_precision.as_str()), (headers::GUID, msg.message.guid.as_str()), (headers::EXPORT_SOURCE, msg.export_source.as_str()), (headers::EXPORT_TOOL, msg.export_tool.as_str()), diff --git a/crates/libs/mail/src/parse.rs b/crates/libs/mail/src/parse.rs index 17854ba1c..8317f8bbe 100644 --- a/crates/libs/mail/src/parse.rs +++ b/crates/libs/mail/src/parse.rs @@ -6,7 +6,7 @@ use anyhow::{Context, Result, bail}; use mailparse::{MailHeader, MailHeaderMap, ParsedMail}; use message_ir::{ Deletion, EarlierVersion, IrDirection, IrImessage, IrMessage, IrMessageKind, IrService, - IrSource, Reaction, ReplyTo, + IrSource, Reaction, ReplyTo, TimePrecision, }; use serde::Deserialize; use serde::de::DeserializeOwned; @@ -87,6 +87,7 @@ pub fn mail_message_from_eml_bytes(bytes: &[u8]) -> Result { .with_context(|| format!("missing required header {}", hn::TIMESTAMP_UNIX_MS))? .parse::() .context("parse X-ME-Timestamp-Unix-Ms")?; + let time_precision = parse_time_precision(headers)?; let direction = match typed_header(headers, hn::DIRECTION) .unwrap_or_default() .to_ascii_lowercase() @@ -166,6 +167,7 @@ pub fn mail_message_from_eml_bytes(bytes: &[u8]) -> Result { message: IrMessage { guid, timestamp_unix_ms, + time_precision, direction, service, message_kind, @@ -337,6 +339,21 @@ fn parse_backup_taken_at(headers: &[MailHeader<'_>]) -> Result> { .with_context(|| format!("This mail's {} header {raw:?}", hn::BACKUP_TAKEN_AT_UNIX_MS)) } +/// Whether the message's time has milliseconds, from the required +/// `X-ME-Time-Precision`. A missing header or a value other than `seconds` +/// or `milliseconds` is refused: the flag, never the time, says whether a +/// time ending in `.000` has milliseconds. +fn parse_time_precision(headers: &[MailHeader<'_>]) -> Result { + let raw = typed_header(headers, hn::TIME_PRECISION) + .with_context(|| format!("missing required header {}", hn::TIME_PRECISION))?; + TimePrecision::parse(&raw).with_context(|| { + format!( + "This mail's {} header {raw:?} is neither seconds nor milliseconds", + hn::TIME_PRECISION + ) + }) +} + /// The message's mark from `X-ME-Deletion`, or none when the header is /// absent. A value that names neither mark is refused rather than dropped. fn parse_deletion(headers: &[MailHeader<'_>]) -> Result> { @@ -488,6 +505,7 @@ mod tests { message: IrMessage { guid: "aabbccddeeff00112233445566778899".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Outgoing, service: IrService::Sms, message_kind: IrMessageKind::Sms, @@ -606,6 +624,7 @@ mod tests { message: IrMessage { guid: "AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, @@ -734,6 +753,7 @@ mod tests { message: IrMessage { guid: "11111111-2222-3333-4444-555555555555".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::IMessage, message_kind: IrMessageKind::IMessage, diff --git a/crates/libs/mail/src/tests.rs b/crates/libs/mail/src/tests.rs index f21893cc9..1624b3e92 100644 --- a/crates/libs/mail/src/tests.rs +++ b/crates/libs/mail/src/tests.rs @@ -20,6 +20,7 @@ fn base_sms() -> MailMessage { message: IrMessage { guid: "aabbccddeeff00112233445566778899".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: message_ir::IrService::Sms, message_kind: message_ir::IrMessageKind::Sms, @@ -489,7 +490,7 @@ fn writes_conversation_mboxrd() { a.message.timestamp_unix_ms = 1_400_773_261_000; let mut b = base_sms(); b.message.guid = "bbccddeeff00112233445566778899aa".into(); - b.message.text = "second".into(); + b.message.text = "the next one".into(); b.message.timestamp_unix_ms = 1_400_773_361_000; let tmp = tempfile::tempdir().unwrap(); @@ -501,7 +502,7 @@ fn writes_conversation_mboxrd() { assert!(text.contains(">From spoofed")); // Chronological: first then second let first_pos = text.find("first").unwrap(); - let second_pos = text.find("second").unwrap(); + let second_pos = text.find("the next one").unwrap(); assert!(first_pos < second_pos); assert_eq!(text.matches("\nFrom ").count(), 1); // one additional From_ between records assert!(text.contains("X-ME-Guid: aabbccddeeff00112233445566778899")); @@ -511,7 +512,7 @@ fn writes_conversation_mboxrd() { let parsed = mail_messages_from_mbox(&path).unwrap(); assert_eq!(parsed.len(), 2); assert_eq!(parsed[0].message.text, "From spoofed\nfirst\nlast"); - assert_eq!(parsed[1].message.text, "second"); + assert_eq!(parsed[1].message.text, "the next one"); } #[test] @@ -565,6 +566,7 @@ fn a_mail_that_names_addresses_handles_is_refused() { "X-ME-Chat-Identifier: sam@example.com\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Participants: [{\"handle\":\"sam@example.com\"}]\r\n", "X-ME-Sender-Handle: sam@example.com\r\n", "\r\n", @@ -614,6 +616,7 @@ fn a_mail_that_keeps_reactions_in_x_me_tapbacks_is_refused() { "X-ME-Chat-Identifier: +15555550101\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Tapbacks: [{\"part_index\":0,\"kind\":\"loved\",\"is_from_me\":false,\"reactor_identity\":\"+15555550101\"}]\r\n", "\r\n", "hello\r\n", @@ -635,6 +638,7 @@ fn a_mail_that_keeps_a_reply_link_in_x_me_thread_originator_guid_is_refused() { "X-ME-Chat-Identifier: +15555550101\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Is-Reply: true\r\n", "X-ME-Thread-Originator-Guid: parent-guid\r\n", "\r\n", @@ -713,6 +717,7 @@ fn a_mail_that_keeps_the_deleted_mark_in_x_me_is_deleted_is_refused() { "X-ME-Chat-Identifier: +15555550101\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Is-Deleted: true\r\n", "\r\n", "hello\r\n", @@ -734,6 +739,7 @@ fn a_mail_that_keeps_the_edit_history_in_x_me_edits_is_refused() { "X-ME-Chat-Identifier: +15555550101\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Edits: [{\"part_index\":0,\"status\":\"edited\",\"text\":\"helo\"}]\r\n", "\r\n", "hello\r\n", @@ -767,6 +773,7 @@ fn assert_json_header_is_refused(header: &str, value: &str, what: &str) { "X-ME-Chat-Identifier: +15555550101\r\n\ X-ME-Guid: g1\r\n\ X-ME-Timestamp-Unix-Ms: 1400773261000\r\n\ + X-ME-Time-Precision: milliseconds\r\n\ X-ME-Service: imessage\r\n\ {header}: {value}\r\n\ \r\n\ @@ -817,6 +824,7 @@ fn a_mail_whose_deletion_names_no_mark_is_refused() { "X-ME-Chat-Identifier: +15555550101\r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000\r\n", + "X-ME-Time-Precision: milliseconds\r\n", "X-ME-Deletion: trashed\r\n", "\r\n", "hello\r\n", @@ -1019,6 +1027,7 @@ fn a_typed_header_that_ends_in_a_space_reads_as_its_value() { "X-ME-Conversation-Type: group \r\n", "X-ME-Guid: g1\r\n", "X-ME-Timestamp-Unix-Ms: 1400773261000 \r\n", + "X-ME-Time-Precision: seconds \r\n", "X-ME-Direction: outgoing \r\n", "X-ME-Service: imessage \r\n", "X-ME-Message-Kind: imessage \r\n", @@ -1036,6 +1045,10 @@ fn a_typed_header_that_ends_in_a_space_reads_as_its_value() { let msg = crate::mail_message_from_eml_bytes(eml.as_bytes()).unwrap(); assert_eq!(msg.conversation_type, "group"); assert_eq!(msg.message.timestamp_unix_ms, 1_400_773_261_000); + assert_eq!( + msg.message.time_precision, + message_ir::TimePrecision::Seconds + ); assert_eq!(msg.message.direction, IrDirection::Outgoing); assert_eq!(msg.message.service, message_ir::IrService::IMessage); assert_eq!( diff --git a/crates/libs/push/src/project.rs b/crates/libs/push/src/project.rs index d7e493e2a..cfbd2be7a 100644 --- a/crates/libs/push/src/project.rs +++ b/crates/libs/push/src/project.rs @@ -155,13 +155,14 @@ mod tests { packaging_stem_suffix: None, }; let header = String::from_utf8(document_header_line(&doc).unwrap()).unwrap(); - assert!(header.contains(r#""schema_version":11"#)); + assert!(header.contains(r#""schema_version":12"#)); assert!(header.contains(r#""sms-backup-restore""#)); assert!(!header.contains(r#""record":"conversation""#)); let msg = IrMessage { guid: "g1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, @@ -190,6 +191,7 @@ mod tests { let msg = IrMessage { guid: "g1".into(), timestamp_unix_ms: 1, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, diff --git a/crates/libs/push/tests/push_mock.rs b/crates/libs/push/tests/push_mock.rs index 93d26f719..858636e12 100644 --- a/crates/libs/push/tests/push_mock.rs +++ b/crates/libs/push/tests/push_mock.rs @@ -50,6 +50,7 @@ fn sample_doc() -> ConversationDocument { messages: vec![IrMessage { guid: "guid-1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, diff --git a/crates/server/demo-seed/src/conversations.rs b/crates/server/demo-seed/src/conversations.rs index c5d810dbb..f9ab94971 100644 --- a/crates/server/demo-seed/src/conversations.rs +++ b/crates/server/demo-seed/src/conversations.rs @@ -13,7 +13,7 @@ use chrono::{Duration, Utc}; use message_ir::{ ConversationHeader, ConversationMeta, ConversationStats, Deletion, EarlierVersion, ExportMeta, IrAttachment, IrConversationType, IrDirection, IrImessage, IrMessage, IrMessageKind, - IrParticipant, IrService, Reaction, ReplyTo, SCHEMA_VERSION, orphaned_chat_id, + IrParticipant, IrService, Reaction, ReplyTo, SCHEMA_VERSION, TimePrecision, orphaned_chat_id, }; use rand::Rng; use rand::RngExt; @@ -317,6 +317,8 @@ impl SharedMessage { IrMessage { guid, timestamp_unix_ms: self.timestamp, + // Every source the demo imitates records milliseconds. + time_precision: TimePrecision::Milliseconds, direction: if self.from_me { IrDirection::Outgoing } else { @@ -1127,6 +1129,8 @@ impl Seeder<'_, R> { IrMessage { guid: guid.into(), timestamp_unix_ms, + // Every source the demo imitates records milliseconds. + time_precision: TimePrecision::Milliseconds, direction: if from_me { IrDirection::Outgoing } else { diff --git a/crates/server/server/src/imports_api/staging/tests.rs b/crates/server/server/src/imports_api/staging/tests.rs index 0e0c10ea6..4f0da36a3 100644 --- a/crates/server/server/src/imports_api/staging/tests.rs +++ b/crates/server/server/src/imports_api/staging/tests.rs @@ -188,7 +188,7 @@ async fn an_attachment_staging_refuses_is_a_rejection_naming_its_file() { let tmp = TempDir::new().unwrap(); let header = one_to_one_header(); // The path is the input under test, so the line stays written out. - let message = r#"{"guid":"g-escape","timestamp_unix_ms":1426183462000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550154","sender_display_name":null,"subject":null,"text":"hi","attachments":[{"path":"../escape.txt","original_name":null,"mime_type":null,"is_sticker":false,"transcription":null,"sticker_effect":null}],"imessage":null,"source":null} + let message = r#"{"guid":"g-escape","timestamp_unix_ms":1426183462000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550154","sender_display_name":null,"subject":null,"text":"hi","attachments":[{"path":"../escape.txt","original_name":null,"mime_type":null,"is_sticker":false,"transcription":null,"sticker_effect":null}],"imessage":null,"source":null} "#; let path = tmp.path().join("+15555550154.jsonl"); std::fs::write(&path, format!("{header}{message}")).unwrap(); diff --git a/crates/server/server/src/imports_api/tests.rs b/crates/server/server/src/imports_api/tests.rs index fb124b95f..c8e43e88c 100644 --- a/crates/server/server/src/imports_api/tests.rs +++ b/crates/server/server/src/imports_api/tests.rs @@ -676,8 +676,8 @@ async fn staging_keeps_both_rows_when_guids_differ_only_by_whitespace() { let assets = tmp.path().join("assets"); let header = conversation_header("imessage", "+15555550123").participant("+15555550123", None); // The guids are the input under test, so these lines stay written out. - let first = r#"{"guid":"g-space","timestamp_unix_ms":1426183462000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"trimmed","attachments":[{"path":"attachments/trim.bin","original_name":"trim.bin","mime_type":"application/octet-stream","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":"not_found"}],"imessage":null,"source":null}"#; - let second = r#"{"guid":" g-space","timestamp_unix_ms":1426183463000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"padded","attachments":[{"path":"attachments/pad.bin","original_name":"pad.bin","mime_type":"application/octet-stream","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":"not_found"}],"imessage":null,"source":null}"#; + let first = r#"{"guid":"g-space","timestamp_unix_ms":1426183462000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"trimmed","attachments":[{"path":"attachments/trim.bin","original_name":"trim.bin","mime_type":"application/octet-stream","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":"not_found"}],"imessage":null,"source":null}"#; + let second = r#"{"guid":" g-space","timestamp_unix_ms":1426183463000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"padded","attachments":[{"path":"attachments/pad.bin","original_name":"pad.bin","mime_type":"application/octet-stream","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":"not_found"}],"imessage":null,"source":null}"#; let path = write_jsonl( tmp.path(), "guid-whitespace.jsonl", @@ -2357,7 +2357,7 @@ async fn rejects_attachment_path_traversal() { conversation_header("sms-backup-restore", "+15555550123") .participant("+15555550123", None), // The path is the input under test, so the line stays written out. - r#"{"guid":"g-trav","timestamp_unix_ms":1426183462000,"direction":"incoming","service":"sms","message_kind":"mms","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"x","attachments":[{"path":"../secret.txt","original_name":"secret.txt","mime_type":"text/plain","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":null}],"imessage":null,"source":null} + r#"{"guid":"g-trav","timestamp_unix_ms":1426183462000,"time_precision":"milliseconds","direction":"incoming","service":"sms","message_kind":"mms","sender_identity":"+15555550123","sender_display_name":null,"subject":null,"text":"x","attachments":[{"path":"../secret.txt","original_name":"secret.txt","mime_type":"text/plain","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":12,"missing_reason":null}],"imessage":null,"source":null} "# ), ); @@ -2428,7 +2428,7 @@ async fn failed_replace_keeps_existing_messages() { conversation_header("sms-backup-restore", "+14075550107") .participant("+14075550107", None), // The path is the input under test, so the line stays written out. - r#"{"guid":"g-bad","timestamp_unix_ms":1426183462000,"direction":"incoming","service":"sms","message_kind":"mms","sender_identity":"+14075550107","sender_display_name":null,"subject":null,"text":"nope","attachments":[{"path":"../secret.txt","original_name":"secret.txt","mime_type":"text/plain","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":1,"missing_reason":null}],"imessage":null,"source":null} + r#"{"guid":"g-bad","timestamp_unix_ms":1426183462000,"time_precision":"milliseconds","direction":"incoming","service":"sms","message_kind":"mms","sender_identity":"+14075550107","sender_display_name":null,"subject":null,"text":"nope","attachments":[{"path":"../secret.txt","original_name":"secret.txt","mime_type":"text/plain","digest_sha256":null,"is_sticker":false,"transcription":null,"sticker_effect":null,"size_bytes":1,"missing_reason":null}],"imessage":null,"source":null} "# ), ); @@ -2789,7 +2789,7 @@ fn one_attachment_batch(path: &str, sha: Option<&str>) -> String { let digest = sha.map_or("null".to_string(), |sha| format!(r#""{sha}""#)); format!( r#"{header} -{{"guid":"g-att","timestamp_unix_ms":1700000000000,"direction":"incoming","service":"whatsapp","message_kind":"sms","sender_identity":"+15555550151","sender_display_name":null,"subject":null,"text":"x","attachments":[{{"path":"{path}","original_name":"a.bin","mime_type":"application/octet-stream","digest_sha256":{digest},"is_sticker":false,"transcription":null,"sticker_effect":null}}],"imessage":null,"source":null}} +{{"guid":"g-att","timestamp_unix_ms":1700000000000,"time_precision":"milliseconds","direction":"incoming","service":"whatsapp","message_kind":"sms","sender_identity":"+15555550151","sender_display_name":null,"subject":null,"text":"x","attachments":[{{"path":"{path}","original_name":"a.bin","mime_type":"application/octet-stream","digest_sha256":{digest},"is_sticker":false,"transcription":null,"sticker_effect":null}}],"imessage":null,"source":null}} "# ) } diff --git a/crates/server/server/src/test_support/lines.rs b/crates/server/server/src/test_support/lines.rs index 4a95458b5..2bedc7a71 100644 --- a/crates/server/server/src/test_support/lines.rs +++ b/crates/server/server/src/test_support/lines.rs @@ -149,6 +149,7 @@ pub fn message_line(guid: &str, text: &str) -> MessageLine { MessageLine(message_ir::IrMessage { guid: guid.to_string(), timestamp_unix_ms: 1_426_183_462_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: message_ir::IrDirection::Incoming, service: message_ir::IrService::IMessage, message_kind: message_ir::IrMessageKind::IMessage, diff --git a/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl b/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl index 1e9fb24e9..0d76945a4 100644 --- a/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-deletions.jsonl @@ -1,4 +1,4 @@ -{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578308040000,"last_timestamp_unix_ms":1578308160000}}} -{"guid":"guid-16","timestamp_unix_ms":1578308040000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Delete me","attachments":[],"deletion":"deleted_in_source_app","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Delete me"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} -{"guid":"guid-17","timestamp_unix_ms":1578308100000,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"","attachments":[],"deletion":"unsent","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} -{"guid":"guid-18","timestamp_unix_ms":1578308160000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Still here","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Still here"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} +{"schema_version":12,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578308040000,"last_timestamp_unix_ms":1578308160000}}} +{"guid":"guid-16","timestamp_unix_ms":1578308040000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Delete me","attachments":[],"deletion":"deleted_in_source_app","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Delete me"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} +{"guid":"guid-17","timestamp_unix_ms":1578308100000,"time_precision":"milliseconds","direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"","attachments":[],"deletion":"unsent","imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} +{"guid":"guid-18","timestamp_unix_ms":1578308160000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Still here","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Still here"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} diff --git a/crates/server/server/tests/fixtures/apple-messages-edits.jsonl b/crates/server/server/tests/fixtures/apple-messages-edits.jsonl index 2342100ba..70311c1dc 100644 --- a/crates/server/server/tests/fixtures/apple-messages-edits.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-edits.jsonl @@ -1,4 +1,4 @@ -{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578309000000,"last_timestamp_unix_ms":1578309120000}}} -{"guid":"guid-edited-twice","timestamp_unix_ms":1578309000000,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000000},{"part_index":0,"text":"Meet at the museum","edited_at_unix_ms":1578309030000}],"imessage":null,"source":null} -{"guid":"guid-edited-final-match","timestamp_unix_ms":1578309060000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"The library opens at nine","attachments":[],"edits":[{"part_index":0,"text":"The library opens at eight","edited_at_unix_ms":1578309060000}],"imessage":null,"source":null} -{"guid":"guid-never-edited","timestamp_unix_ms":1578309120000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Nothing changed here","attachments":[],"imessage":null,"source":null} +{"schema_version":12,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578309000000,"last_timestamp_unix_ms":1578309120000}}} +{"guid":"guid-edited-twice","timestamp_unix_ms":1578309000000,"time_precision":"milliseconds","direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000000},{"part_index":0,"text":"Meet at the museum","edited_at_unix_ms":1578309030000}],"imessage":null,"source":null} +{"guid":"guid-edited-final-match","timestamp_unix_ms":1578309060000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"The library opens at nine","attachments":[],"edits":[{"part_index":0,"text":"The library opens at eight","edited_at_unix_ms":1578309060000}],"imessage":null,"source":null} +{"guid":"guid-never-edited","timestamp_unix_ms":1578309120000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Nothing changed here","attachments":[],"imessage":null,"source":null} diff --git a/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl b/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl index 3f2cc0f3d..7c0219e75 100644 --- a/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-reactions.jsonl @@ -1,4 +1,4 @@ -{"schema_version":11,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"chat100","conversation_type":"group","group_title":"Weekend plans","participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"},{"identity":"friend@example.com","display_name":null,"identity_type":"email"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578307860000,"last_timestamp_unix_ms":1578307980000}}} -{"guid":"00000000-0000-4000-8000-000000000013","timestamp_unix_ms":1578307860000,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"friend@example.com","sender_display_name":"friend@example.com","owner_identity":"+15555550106","subject":null,"text":"Pizza?","attachments":[],"reactions":[{"part_index":0,"kind":"loved","is_from_me":false,"reactor_identity":"+15555550107","reactor_display_name":"+15555550107"},{"part_index":0,"kind":"emoji","emoji":"🔥","is_from_me":true,"reactor_display_name":"+15555550106"}],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Pizza?"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} -{"guid":"guid-14","timestamp_unix_ms":1578307920000,"direction":"incoming","service":"imessage","message_kind":"tapback","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Loved a message","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"loved","tapback_emoji":null,"tapback_action":"add"},"source":null} -{"guid":"guid-15","timestamp_unix_ms":1578307980000,"direction":"outgoing","service":"imessage","message_kind":"tapback","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"🔥 reacted","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"emoji","tapback_emoji":"🔥","tapback_action":"add"},"source":null} +{"schema_version":12,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"chat100","conversation_type":"group","group_title":"Weekend plans","participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"},{"identity":"friend@example.com","display_name":null,"identity_type":"email"}],"stats":{"message_count":3,"attachment_count":0,"first_timestamp_unix_ms":1578307860000,"last_timestamp_unix_ms":1578307980000}}} +{"guid":"00000000-0000-4000-8000-000000000013","timestamp_unix_ms":1578307860000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"friend@example.com","sender_display_name":"friend@example.com","owner_identity":"+15555550106","subject":null,"text":"Pizza?","attachments":[],"reactions":[{"part_index":0,"kind":"loved","is_from_me":false,"reactor_identity":"+15555550107","reactor_display_name":"+15555550107"},{"part_index":0,"kind":"emoji","emoji":"🔥","is_from_me":true,"reactor_display_name":"+15555550106"}],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":[{"index":0,"kind":"run","text":"Pizza?"}],"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":null,"associated_part":null,"tapback_kind":null,"tapback_emoji":null,"tapback_action":null},"source":null} +{"guid":"guid-14","timestamp_unix_ms":1578307920000,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"tapback","sender_identity":"+15555550107","sender_display_name":"+15555550107","owner_identity":"+15555550106","subject":null,"text":"Loved a message","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"loved","tapback_emoji":null,"tapback_action":"add"},"source":null} +{"guid":"guid-15","timestamp_unix_ms":1578307980000,"time_precision":"milliseconds","direction":"outgoing","service":"imessage","message_kind":"tapback","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"🔥 reacted","attachments":[],"imessage":{"send_effect":null,"shared_location":null,"announcement":null,"read_receipt_rfc3339":null,"parts":null,"app":null,"balloon_bundle_id":null,"balloon_kind":null,"associated_guid":"00000000-0000-4000-8000-000000000013","associated_part":0,"tapback_kind":"emoji","tapback_emoji":"🔥","tapback_action":"add"},"source":null} diff --git a/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl b/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl index 04a008e8d..e1f2db387 100644 --- a/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl +++ b/crates/server/server/tests/fixtures/apple-messages-sub-second-times.jsonl @@ -1,3 +1,3 @@ -{"schema_version":10,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":2,"attachment_count":0,"first_timestamp_unix_ms":1578309000250,"last_timestamp_unix_ms":1578309000550}}} -{"guid":"guid-300-ms-later","timestamp_unix_ms":1578309000550,"direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"On my way","attachments":[],"imessage":null,"source":null} -{"guid":"guid-first","timestamp_unix_ms":1578309000250,"direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000125}],"imessage":null,"source":null} +{"schema_version":12,"export":{"source":"imessage","tool":"imessage-ir-exporter","tool_version":"0.1.0","owner_identity":"+15555550106","owner_display_name":null},"conversation":{"chat_identifier":"+15555550107","conversation_type":"individual","group_title":null,"participants":[{"identity":"+15555550107","display_name":null,"identity_type":"phone"}],"stats":{"message_count":2,"attachment_count":0,"first_timestamp_unix_ms":1578309000250,"last_timestamp_unix_ms":1578309000550}}} +{"guid":"guid-300-ms-later","timestamp_unix_ms":1578309000550,"time_precision":"milliseconds","direction":"incoming","service":"imessage","message_kind":"imessage","sender_identity":"+15555550107","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"On my way","attachments":[],"imessage":null,"source":null} +{"guid":"guid-first","timestamp_unix_ms":1578309000250,"time_precision":"milliseconds","direction":"outgoing","service":"imessage","message_kind":"imessage","sender_identity":"+15555550106","sender_display_name":null,"owner_identity":"+15555550106","subject":null,"text":"Meet at the bakery","attachments":[],"edits":[{"part_index":0,"text":"Meet at the library","edited_at_unix_ms":1578309000125}],"imessage":null,"source":null} diff --git a/src-tauri/src/commands/upload.rs b/src-tauri/src/commands/upload.rs index 85ab7fc06..1641e0bf8 100644 --- a/src-tauri/src/commands/upload.rs +++ b/src-tauri/src/commands/upload.rs @@ -394,6 +394,7 @@ mod tests { let message = json!(IrMessage { guid: "guid-1".into(), timestamp_unix_ms: 1_400_773_261_000, + time_precision: message_ir::TimePrecision::Milliseconds, direction: IrDirection::Incoming, service: IrService::Sms, message_kind: IrMessageKind::Sms, From fea14d22863233d04040a598b9bae74366239268 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:41:00 -0400 Subject: [PATCH 27/42] style: rewrap two paragraphs, and fixtures carry the stored date form Two paragraphs the merge of #1963 edited ran past the wrap width. Test fixtures that stand for a stored backup date now use the three-digit millisecond form the server writes. Co-Authored-By: Claude Opus 5.5 --- crates/libs/api-types/src/lib.rs | 2 +- crates/libs/export/src/project.rs | 4 ++-- crates/server/server/src/db/staging.rs | 3 ++- docs/architecture/contacts-identities-and-messages.md | 9 +++++---- .../screens/settings/storage/ImportHistoryTable.test.tsx | 2 +- web/src/screens/settings/storage/storageUtils.test.ts | 4 ++-- 6 files changed, 13 insertions(+), 11 deletions(-) diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 86d4efa93..ad4e4479f 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -661,7 +661,7 @@ mod tests { edited_at: Some("2023-12-31T23:59:00Z".into()), matched: false, }], - backup_taken_at: Some("2024-01-02T08:00:00Z".into()), + backup_taken_at: Some("2024-01-02T08:00:00.000Z".into()), matched_earlier_version: false, }; diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index c646e4329..0b5f6a134 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -659,8 +659,8 @@ mod tests { msg.backup_taken_at = at.map(str::to_string); msg }; - let later = Some("2026-09-30T18:45:12Z"); - let earlier = Some("2026-09-01T10:00:00Z"); + let later = Some("2026-09-30T18:45:12.000Z"); + let earlier = Some("2026-09-01T10:00:00.000Z"); assert_eq!(backup_of(&dated(None)), None); diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index b8af9f905..424a6f9e9 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -1238,7 +1238,8 @@ pub async fn write_message_map( /// falls back on the rule for files without a date. Equal dates are the /// same backup read again, where that rule changes nothing because the two /// copies agree, or two reads of one Mac's `chat.db` that Messages did -/// not write between, where it adds a mark and takes a later edit as it would with no dates. +/// not write between, where it adds a mark and takes a later edit as it +/// would with no dates. /// The one rule for which of two copies of a message from one source is /// the later backup, for a stored message ([`promote_deletion_marks`], /// [`write_edit_map`]) and, in Rust ([`later_backup`]), for two copies diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index abfd90377..bf002ae3a 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -424,10 +424,11 @@ copy from the later backup gives the message its deletion mark, mark or no mark, and its text and earlier versions, whatever the versions' times say; a copy from an earlier backup changes neither. The duplicate flag follows the text, because the dedupe compares the text. The date is kept to the -millisecond, the form every stored time takes. When either copy has no date, or the two -dates are equal, nothing says which backup is newer, so the rules for files -without one hold: a copy with a mark adds it and one without leaves the mark held, -and a copy takes the text when its newest earlier version is newer +millisecond, the form every stored time takes. When either copy has no date, +or the two dates are equal, nothing says which backup is newer, so the rules +for files without one hold: a copy with a mark adds it and one without leaves +the mark held, and a copy takes the text when its newest earlier version is +newer (`later_edit_sql` in `db/staging.rs`). Attachments and reactions add from either copy, because a backup that lacks one does not say it is gone. The rule is the same in one import as across several, in any file order diff --git a/web/src/screens/settings/storage/ImportHistoryTable.test.tsx b/web/src/screens/settings/storage/ImportHistoryTable.test.tsx index c96620fb0..60b1c40c3 100644 --- a/web/src/screens/settings/storage/ImportHistoryTable.test.tsx +++ b/web/src/screens/settings/storage/ImportHistoryTable.test.tsx @@ -174,7 +174,7 @@ describe("Import history", () => { contacts_changed: 0, issues: [], source_fingerprint: { path: "/backups/iPhone/00008110", size: 12, mtime_ms: 1 }, - backup_taken_at: "2026-09-30T18:45:12Z", + backup_taken_at: "2026-09-30T18:45:12.000Z", }); const user = setupUser(); renderWithProviders(); diff --git a/web/src/screens/settings/storage/storageUtils.test.ts b/web/src/screens/settings/storage/storageUtils.test.ts index e3e0b9124..4a3cea555 100644 --- a/web/src/screens/settings/storage/storageUtils.test.ts +++ b/web/src/screens/settings/storage/storageUtils.test.ts @@ -226,10 +226,10 @@ describe("importBackup", () => { importBackup( accountImportRun({ source_fingerprint: { path: "/backups/iPhone/00008110", size: 12, mtime_ms: 1 }, - backup_taken_at: "2026-09-30T18:45:12Z", + backup_taken_at: "2026-09-30T18:45:12.000Z", }), ), - ).toEqual({ file: "/backups/iPhone/00008110", takenAt: "2026-09-30T18:45:12Z" }); + ).toEqual({ file: "/backups/iPhone/00008110", takenAt: "2026-09-30T18:45:12.000Z" }); }); it("says nothing it was not told", () => { From 5905982ce14cc4b6f22906c793613a70737745c2 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:43:00 -0400 Subject: [PATCH 28/42] feat(server): store a message's time precision and answer it staging_messages and messages gain time_precision, seconds or milliseconds, from the conversation file, and the HTTP API answers it as time_precision on a Message. An Export Run writes the stored value back into the conversation file. The schema change rebuilds the database; nothing migrates. Part of #1923. Co-Authored-By: Claude Opus 5.5 --- crates/libs/api-types/src/lib.rs | 36 ++++++++++++++++ crates/libs/export/src/project.rs | 43 +++++++++++++++++++ crates/libs/export/src/run.rs | 1 + crates/libs/export/tests/export_mock.rs | 1 + .../server/src/db/conversation_messages.rs | 12 +++++- crates/server/server/src/db/staging.rs | 12 ++++-- crates/server/server/src/db/staging/tests.rs | 1 + crates/server/server/src/db/trash/tests.rs | 7 ++- .../server/server/src/imports_api/staging.rs | 1 + crates/server/server/src/models.rs | 10 +++-- crates/server/server/src/test_support.rs | 9 +++- .../server/server/src/test_support/lines.rs | 7 +++ schema/sql/messages.sql | 7 +++ schema/sql/staging.sql | 3 ++ 14 files changed, 138 insertions(+), 12 deletions(-) diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 86d4efa93..97000adc7 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -380,6 +380,10 @@ api_shape! { /// (`Account.time_zone`); the database stores nothing /// about where the phone was. pub timestamp: String, + /// Whether the source recorded `timestamp` to the millisecond or in + /// whole seconds. A `timestamp` ending in `.000` is a whole second + /// only when this says `seconds`. + pub time_precision: TimePrecision, /// Ordering key within the conversation. pub sort_order: i64, /// True for messages sent by the account owner. @@ -470,6 +474,35 @@ api_shape! { } } +/// How finely the source recorded a message's time. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[cfg_attr(feature = "schema", derive(utoipa::ToSchema))] +#[serde(rename_all = "lowercase")] +pub enum TimePrecision { + /// Whole seconds: the source records no milliseconds, and the time's + /// milliseconds are `.000`. + Seconds, + /// Milliseconds, as the phone stored them; they can be `.000` too. + Milliseconds, +} + +impl TimePrecision { + /// The precision as the wire and the database spell it. + pub const fn as_str(self) -> &'static str { + match self { + Self::Seconds => "seconds", + Self::Milliseconds => "milliseconds", + } + } + + /// Read a sent or stored value; anything else names no precision. + pub fn parse(value: &str) -> Option { + [Self::Seconds, Self::Milliseconds] + .into_iter() + .find(|p| p.as_str() == value) + } +} + /// Why a message's content is gone in the app it came from. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[cfg_attr(feature = "schema", derive(utoipa::ToSchema))] @@ -612,6 +645,7 @@ mod tests { service: None, guid: "g1".into(), timestamp: "2024-01-01T00:00:00.000Z".into(), + time_precision: TimePrecision::Seconds, sort_order: 0, is_from_me: false, sender: None, @@ -725,6 +759,8 @@ mod tests { assert_eq!(read.attachments.len(), 1); assert_eq!(read.tapbacks[0].kind, "loved"); assert_eq!(written["deletion"], "deleted_in_source_app"); + assert_eq!(written["time_precision"], "seconds"); + assert_eq!(read.time_precision, TimePrecision::Seconds); assert_eq!(read.deletion, Some(Deletion::DeletedInSourceApp)); assert_eq!(read.earlier_versions[0].text, "helo"); assert_eq!( diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index 1fbc26ad4..ce409a1cb 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -11,6 +11,7 @@ use message_ir::{ ConversationDocument, ConversationMeta, ConversationStats, Deletion, EarlierVersion, ExportMeta, IrAttachment, IrConversationType, IrDirection, IrImessage, IrMessage, IrMessageKind, IrParticipant, IrService, IrSource, Reaction, ReplyTo, SCHEMA_VERSION, + TimePrecision, }; use serde_json::json; @@ -143,6 +144,7 @@ pub fn to_ir_message(msg: &Message, skip_attachments: bool) -> Result Ok(IrMessage { guid: msg.guid.clone(), timestamp_unix_ms, + time_precision: time_precision_from_api(msg.time_precision), direction, service, message_kind, @@ -188,6 +190,14 @@ fn earlier_version_from_api( } /// The mark a server message carries, as the conversation file writes it. +/// The precision the server stored, as the conversation file writes it. +fn time_precision_from_api(precision: message_crate_api_types::TimePrecision) -> TimePrecision { + match precision { + message_crate_api_types::TimePrecision::Seconds => TimePrecision::Seconds, + message_crate_api_types::TimePrecision::Milliseconds => TimePrecision::Milliseconds, + } +} + fn deletion_from_api(deletion: message_crate_api_types::Deletion) -> Deletion { match deletion { message_crate_api_types::Deletion::DeletedInSourceApp => Deletion::DeletedInSourceApp, @@ -399,6 +409,7 @@ mod tests { "service": "iMessage", "guid": "3A9E-0001", "timestamp": "2015-03-12T18:05:22Z", + "time_precision": "milliseconds", "sort_order": 0, "is_from_me": false, "sender": "+15555550100", @@ -742,6 +753,36 @@ mod tests { assert!(ir.imessage.is_none()); } + /// The conversation file says the precision the server stored, so a + /// whole-second message is written back as whole seconds and a + /// millisecond time that ends in `.000` as milliseconds. + #[test] + fn a_message_keeps_the_precision_the_server_stored() { + let participant = Participant { + identity: Some("+1".into()), + name: "Sam".into(), + service: None, + contact_id: None, + }; + let mut msg = seed_message_with_participant(participant); + msg.timestamp = "2015-03-12T18:05:22.000Z".into(); + for (stored, written) in [ + ( + message_crate_api_types::TimePrecision::Seconds, + TimePrecision::Seconds, + ), + ( + message_crate_api_types::TimePrecision::Milliseconds, + TimePrecision::Milliseconds, + ), + ] { + msg.time_precision = stored; + let ir = to_ir_message(&msg, false).unwrap(); + assert_eq!(ir.timestamp_unix_ms, 1_426_183_522_000); + assert_eq!(ir.time_precision, written); + } + } + #[test] fn maps_basic_message() { let msg = Message { @@ -750,6 +791,7 @@ mod tests { service: Some("iMessage".into()), guid: "g1".into(), timestamp: "2015-03-12T18:05:22Z".into(), + time_precision: message_crate_api_types::TimePrecision::Milliseconds, is_from_me: false, sender: Some("+1".into()), owner: None, @@ -907,6 +949,7 @@ mod tests { service: Some("iMessage".into()), guid: "g1".into(), timestamp: "2015-03-12T18:05:22Z".into(), + time_precision: message_crate_api_types::TimePrecision::Milliseconds, is_from_me: false, sender: Some("+1".into()), owner: None, diff --git a/crates/libs/export/src/run.rs b/crates/libs/export/src/run.rs index c17c9ae24..cddf1510b 100644 --- a/crates/libs/export/src/run.rs +++ b/crates/libs/export/src/run.rs @@ -980,6 +980,7 @@ mod asset_ref_tests { "source": source, "guid": "g1", "timestamp": "2015-03-12T18:05:22Z", + "time_precision": "milliseconds", "sort_order": 0, "is_from_me": false, "is_announcement": false, diff --git a/crates/libs/export/tests/export_mock.rs b/crates/libs/export/tests/export_mock.rs index 6c35abc6c..b8bb350bd 100644 --- a/crates/libs/export/tests/export_mock.rs +++ b/crates/libs/export/tests/export_mock.rs @@ -77,6 +77,7 @@ fn message( "service": "sms", "guid": guid, "timestamp": timestamp, + "time_precision": "milliseconds", "sort_order": id, "is_from_me": false, "sender": "+15555550101", diff --git a/crates/server/server/src/db/conversation_messages.rs b/crates/server/server/src/db/conversation_messages.rs index f6a0457aa..49ba4f97f 100644 --- a/crates/server/server/src/db/conversation_messages.rs +++ b/crates/server/server/src/db/conversation_messages.rs @@ -19,6 +19,7 @@ use sqlx::{Executor, Row}; pub use message_crate_api_types::{ Attachment, Deletion, EarlierVersion, Message, MessageConversation, ReplyTo, Tapback, + TimePrecision, }; use crate::db::conversations::is_group_type; @@ -55,6 +56,7 @@ struct RawRow { reply_count: i64, deletion: Option, backup_taken_at: Option, + time_precision: TimePrecision, chat_identifier: String, conversation_type: String, group_title: Option, @@ -434,7 +436,8 @@ fn message_page_sql( m.is_announcement, m.is_reply, m.reply_to_guid, m.reply_to_part, ({reply_count}) AS reply_count, hc.raw AS chat_identifier, c.conversation_type, c.group_title, - ho.raw AS owner, {label} AS label, m.deletion, m.backup_taken_at + ho.raw AS owner, {label} AS label, m.deletion, m.backup_taken_at, + m.time_precision {from_sql} WHERE {where_sql} ORDER BY {order_by} LIMIT ? OFFSET ?", @@ -488,6 +491,12 @@ async fn fetch_message_page( label: row.try_get(20)?, deletion: row.try_get(21)?, backup_taken_at: row.try_get(22)?, + time_precision: { + let stored: String = row.try_get(23)?; + TimePrecision::parse(&stored).ok_or_else(|| { + sqlx::Error::Decode(format!("time_precision {stored:?}").into()) + })? + }, }) }) .collect::, ApiError>>()?; @@ -512,6 +521,7 @@ async fn fetch_message_page( service: r.service, guid: r.guid, timestamp: r.timestamp, + time_precision: r.time_precision, sort_order: r.sort_order, is_from_me: r.is_from_me, sender: r.sender, diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index e9b1bbeb0..b6719625c 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -203,6 +203,8 @@ pub struct StagingMessage<'a> { pub guid: &'a str, /// RFC 3339 UTC instant the message was sent. pub timestamp: &'a str, + /// Whether the source recorded `timestamp` to the millisecond. + pub time_precision: message_ir::TimePrecision, /// 1 when the account holder sent it. pub is_from_me: i64, /// Sender's handle id; `None` when unknown. @@ -344,7 +346,7 @@ const TAPBACK_COLUMNS: &[&str] = &[ ]; /// Bind counts, in lockstep with the `INSERT` column lists below. -const MESSAGE_BIND_COLUMNS: usize = 19; +const MESSAGE_BIND_COLUMNS: usize = 20; const ATTACHMENT_BIND_COLUMNS: usize = ATTACHMENT_COLUMNS.len(); const TAPBACK_BIND_COLUMNS: usize = TAPBACK_COLUMNS.len(); const EARLIER_VERSION_BIND_COLUMNS: usize = 4; @@ -369,7 +371,7 @@ pub async fn insert_messages( let sql = format!( r" INSERT INTO staging_messages ( - conversation_id, account_id, source, guid, timestamp, is_from_me, + conversation_id, account_id, source, guid, timestamp, time_precision, is_from_me, sender_handle_id, owner_handle_id, service, subject, body, is_announcement, is_reply, reply_to_guid, reply_to_part, deletion, sort_order, import_id, backup_taken_at ) VALUES {} @@ -386,6 +388,7 @@ pub async fn insert_messages( .bind(row.source) .bind(row.guid) .bind(row.timestamp) + .bind(row.time_precision.as_str()) .bind(row.is_from_me) .bind(row.sender_handle_id) .bind(row.owner_handle_id) @@ -1072,12 +1075,13 @@ pub async fn staged_message_id_bounds( /// the production ids follow it, which the id-map zip relies on. const INSERT_MESSAGES_FROM_STAGING: &str = r" INSERT INTO messages ( - conversation_id, account_id, source, guid, timestamp, is_from_me, + conversation_id, account_id, source, guid, timestamp, time_precision, is_from_me, sender_handle_id, owner_handle_id, service, subject, body, is_announcement, is_reply, reply_to_guid, reply_to_part, deletion, sort_order, import_id, backup_taken_at ) SELECT - cm.prod_id, sm.account_id, sm.source, sm.guid, sm.timestamp, sm.is_from_me, + cm.prod_id, sm.account_id, sm.source, sm.guid, sm.timestamp, sm.time_precision, + sm.is_from_me, sm.sender_handle_id, sm.owner_handle_id, sm.service, sm.subject, sm.body, sm.is_announcement, sm.is_reply, sm.reply_to_guid, sm.reply_to_part, sm.deletion, sm.sort_order, sm.import_id, sm.backup_taken_at diff --git a/crates/server/server/src/db/staging/tests.rs b/crates/server/server/src/db/staging/tests.rs index 90522030c..7e8a54ed2 100644 --- a/crates/server/server/src/db/staging/tests.rs +++ b/crates/server/server/src/db/staging/tests.rs @@ -31,6 +31,7 @@ async fn reset_for_account_leaves_other_accounts() { source: "sms", guid: "g1", timestamp: "2020-01-01T00:00:00.000Z", + time_precision: message_ir::TimePrecision::Milliseconds, is_from_me: 0, sender_handle_id: None, owner_handle_id: None, diff --git a/crates/server/server/src/db/trash/tests.rs b/crates/server/server/src/db/trash/tests.rs index 99227f4ce..da87e8970 100644 --- a/crates/server/server/src/db/trash/tests.rs +++ b/crates/server/server/src/db/trash/tests.rs @@ -607,8 +607,11 @@ async fn delete_reports_only_the_files_no_remaining_message_uses() { .unwrap(); let staging_message: i64 = sqlx::query_scalar( "INSERT INTO staging_messages ( - conversation_id, account_id, source, guid, timestamp, is_from_me, sort_order - ) VALUES ($1, $2, 'imessage', 'g-staged', '2020-01-01T00:00:00Z', 1, 0) RETURNING id", + conversation_id, account_id, source, guid, timestamp, time_precision, is_from_me, + sort_order + ) VALUES ( + $1, $2, 'imessage', 'g-staged', '2020-01-01T00:00:00.000Z', 'milliseconds', 1, 0 + ) RETURNING id", ) .bind(staging_conversation) .bind(ACCOUNT_A) diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index c266543e9..f3a97ab49 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -1001,6 +1001,7 @@ async fn insert_message_rows( source: staged_source.source, guid: &row.msg.guid, timestamp: &row.msg.timestamp, + time_precision: row.msg.time_precision, is_from_me: row.msg.is_from_me as i64, sender_handle_id: row.sender_handle_id, owner_handle_id: row.owner_handle_id, diff --git a/crates/server/server/src/models.rs b/crates/server/server/src/models.rs index 9c76eb3b3..404871244 100644 --- a/crates/server/server/src/models.rs +++ b/crates/server/server/src/models.rs @@ -6,7 +6,7 @@ use anyhow::{Context, Result}; use chrono::{DateTime, TimeZone, Utc}; use message_ir::{ ConversationHeader, Deletion, EarlierVersion, HandleService, HandleType, IrAttachment, - IrDirection, IrMessage, IrMessageKind, IrParticipant, Reaction, ReplyTo, + IrDirection, IrMessage, IrMessageKind, IrParticipant, Reaction, ReplyTo, TimePrecision, check_schema_version_in_json, nonempty, trimmed, }; use phone::Handle; @@ -99,6 +99,9 @@ pub struct MessageRecord { /// with three fractional digits and a `Z` suffix /// (`2015-03-12T18:04:22.250Z`). pub timestamp: String, + /// Whether the source recorded `timestamp` to the millisecond or in + /// whole seconds, as the conversation file says. + pub time_precision: TimePrecision, /// True for messages sent by the account owner. pub is_from_me: bool, /// Sender handle for incoming messages: the address, or the name when the @@ -396,6 +399,7 @@ fn message_from_ir( guid: msg.guid.clone(), line, timestamp, + time_precision: msg.time_precision, is_from_me, sender: sender.as_ref().map(|(value, _)| value.clone()), sender_handle_type: sender.and_then(|(_, kind)| kind), @@ -826,7 +830,7 @@ mod tests { .participant("+15555550101", Some("Sam")) .to_string(); // The timestamp is the input under test, so the line stays written out. - let msg = r#"{"guid":"g1","timestamp_unix_ms":9223372036854775807,"direction":"incoming","service":"sms","message_kind":"sms","sender_identity":"+15555550101","sender_display_name":"Sam","subject":null,"text":"hello","attachments":[],"imessage":null,"source":null}"#; + let msg = r#"{"guid":"g1","timestamp_unix_ms":9223372036854775807,"time_precision":"milliseconds","direction":"incoming","service":"sms","message_kind":"sms","sender_identity":"+15555550101","sender_display_name":"Sam","subject":null,"text":"hello","attachments":[],"imessage":null,"source":null}"#; let failure = parse_ir_lines([header, msg.to_string()]).unwrap_err(); match failure { ImportFailure::Invalid { line, .. } => assert_eq!(line, 2), @@ -845,7 +849,7 @@ mod tests { // The guids are the input under test, so the lines stay written out. let msg = |guid: &str| { format!( - r#"{{"guid":"{guid}","timestamp_unix_ms":1400773261000,"direction":"incoming","service":"sms","message_kind":"sms","sender_identity":"+15555550101","sender_display_name":"Sam","subject":null,"text":"hello","attachments":[],"imessage":null,"source":null}}"# + r#"{{"guid":"{guid}","timestamp_unix_ms":1400773261000,"time_precision":"milliseconds","direction":"incoming","service":"sms","message_kind":"sms","sender_identity":"+15555550101","sender_display_name":"Sam","subject":null,"text":"hello","attachments":[],"imessage":null,"source":null}}"# ) }; let lines = [header.to_string(), msg("g1"), msg(""), msg(" ")]; diff --git a/crates/server/server/src/test_support.rs b/crates/server/server/src/test_support.rs index cb84904d0..f3c37fb71 100644 --- a/crates/server/server/src/test_support.rs +++ b/crates/server/server/src/test_support.rs @@ -837,6 +837,8 @@ pub struct MessageRow<'a> { pub guid: Option, /// RFC 3339 in UTC to the millisecond, as the importer writes it. pub timestamp: &'a str, + /// `messages.time_precision`. + pub time_precision: message_ir::TimePrecision, /// Whether the account sent it. pub is_from_me: bool, /// `messages.sender_handle_id`. @@ -880,6 +882,7 @@ impl MessageRow<'_> { source: "imessage", guid: Some(unique_guid()), timestamp: "2020-01-01T00:00:00.000Z", + time_precision: message_ir::TimePrecision::Milliseconds, is_from_me: false, sender_handle_id: None, owner_handle_id: None, @@ -925,10 +928,11 @@ impl MessageRow<'_> { id, conversation_id, account_id, source, guid, timestamp, is_from_me, sender_handle_id, owner_handle_id, service, subject, body, is_announcement, is_reply, reply_to_guid, reply_to_part, - deletion, sort_order, content_key, duplicate_of, import_id + deletion, sort_order, content_key, duplicate_of, import_id, + time_precision ) VALUES ( $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, - $13, $14, $15, $16, $17, $18, $19, $20, $21 + $13, $14, $15, $16, $17, $18, $19, $20, $21, $22 ) RETURNING id", ) .bind(self.id) @@ -952,6 +956,7 @@ impl MessageRow<'_> { .bind(self.content_key) .bind(self.duplicate_of) .bind(self.import_id) + .bind(self.time_precision.as_str()) .fetch_one(&mut **tx) .await } diff --git a/crates/server/server/src/test_support/lines.rs b/crates/server/server/src/test_support/lines.rs index 2bedc7a71..0c025d99e 100644 --- a/crates/server/server/src/test_support/lines.rs +++ b/crates/server/server/src/test_support/lines.rs @@ -175,6 +175,13 @@ impl MessageLine { self } + /// From a source that records whole seconds: `time_precision` is + /// `seconds`. The time stays as it was. + pub fn whole_seconds(mut self) -> Self { + self.0.time_precision = message_ir::TimePrecision::Seconds; + self + } + /// Sent by the account holder. pub fn outgoing(mut self) -> Self { self.0.direction = message_ir::IrDirection::Outgoing; diff --git a/schema/sql/messages.sql b/schema/sql/messages.sql index 063323782..250564af6 100644 --- a/schema/sql/messages.sql +++ b/schema/sql/messages.sql @@ -62,6 +62,13 @@ CREATE TABLE IF NOT EXISTS messages ( -- text sorts in time order and lists order by it. Shown, searched and -- filed by day and year in the account's time zone (accounts.time_zone). timestamp TEXT NOT NULL, + -- Whether the source recorded timestamp to the millisecond or in whole + -- seconds, as the conversation file's time_precision says. The flag, never + -- the time, decides: a millisecond time can end in .000. Within one + -- source, a 'seconds' message is the duplicate of one that matches it in + -- everything else with 'milliseconds' in the same second + -- (docs/architecture/contacts-identities-and-messages.md). + time_precision TEXT NOT NULL CHECK (time_precision IN ('seconds', 'milliseconds')), -- 1 = sent by the account holder; 0 = received from someone else. is_from_me INTEGER NOT NULL, -- Sender identity (`handles.id`); NULL when unknown. diff --git a/schema/sql/staging.sql b/schema/sql/staging.sql index c7246c47f..b78a4e1f9 100644 --- a/schema/sql/staging.sql +++ b/schema/sql/staging.sql @@ -49,6 +49,9 @@ CREATE TABLE IF NOT EXISTS staging_messages ( -- The instant the message was sent, in the form messages.timestamp holds: -- RFC 3339 in UTC to the millisecond (2015-03-12T18:04:22.250Z). timestamp TEXT NOT NULL, + -- Whether the source recorded timestamp to the millisecond or in whole + -- seconds, as messages.time_precision. + time_precision TEXT NOT NULL CHECK (time_precision IN ('seconds', 'milliseconds')), -- 1 = sent by the account holder; 0 = received from someone else. is_from_me INTEGER NOT NULL, -- Sender identity handle id; NULL when unknown. From 5b3b8891fa684d45019671b6b8d1428968f2da23 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:43:00 -0400 Subject: [PATCH 29/42] feat(dedupe): a whole-second copy of a message is the duplicate of its millisecond twin Within one source, a message whose time is in whole seconds is hidden as the duplicate of a message that matches it in everything else and has milliseconds in the same second, so an SMS Backup+ message imported once from a mail timed by its Date header and once from one timed by X-smssync-date is shown once, with its milliseconds. Before, the exact pass hid a duplicate only when another source held it. The content key stays at whole seconds, and the stored flag decides, never the time: a millisecond time that ends in .000 is not taken for a whole second. Part of #1923. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 89 ++++++++--- crates/server/server/src/dedupe/tests.rs | 129 ++++++++++++++++ crates/server/server/src/imports_api/tests.rs | 1 + .../src/imports_api/tests/time_precision.rs | 144 ++++++++++++++++++ 4 files changed, 344 insertions(+), 19 deletions(-) create mode 100644 crates/server/server/src/imports_api/tests/time_precision.rs diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 3a64ba25b..411e9f530 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -142,10 +142,12 @@ pub struct DedupeStats { /// Content keys written: those missing and those whose inputs changed /// (not a duplicate count). pub keys_filled: u64, - /// Groups of messages sharing one content key. + /// Groups of messages sharing one content key in which a message was + /// hidden. pub exact_groups: u64, /// Messages hidden as exact duplicates: all but as many per group as one - /// source holds. + /// source holds, and each whole-second message whose own source holds + /// it with milliseconds too. pub exact_flagged: u64, /// Messages flagged as near duplicates. pub near_flagged: u64, @@ -175,7 +177,8 @@ pub async fn source_priority_from_db( } /// Refresh the content keys, clear prior flags, then soft-hide cross-source -/// duplicates, all in one transaction. +/// duplicates and the whole-second twins of a message one source holds with +/// milliseconds too, all in one transaction. /// /// Survivor preference: most attachments, then the source imported first (min /// message id, then source name), then the lowest message id. Optional @@ -536,9 +539,17 @@ struct Cand { att_count: i64, } +/// One message of a content-key group in the exact pass: the candidate, +/// and whether its source recorded its time in whole seconds. +struct KeyedCand { + cand: Cand, + whole_seconds: bool, +} + /// Hide the messages that share a fingerprint with a preferred-source twin, -/// keeping as many as one source holds (see [`exact_group_flags`]). Returns -/// (groups, hidden). +/// keeping as many as one source holds (see [`exact_group_flags`]), and +/// each whole-second message whose own source holds it with milliseconds +/// too (see [`content_key_group_flags`]). Returns (groups, hidden). async fn flag_exact_content_key_dupes( tx: &mut WriteTx<'_>, account_id: i64, @@ -547,9 +558,9 @@ async fn flag_exact_content_key_dupes( let conn: &mut SqliteConnection = tx; // One scan of messages + one aggregated attachment pass, then group in Rust. // Avoids N round-trips (one SELECT + several UPDATEs per duplicate key). - let rows: Vec<(i64, String, String, i64)> = sqlx::query_as( + let rows: Vec<(i64, String, String, i64, String)> = sqlx::query_as( r" - SELECT m.id, m.source, m.content_key, COALESCE(ac.n, 0) + SELECT m.id, m.source, m.content_key, COALESCE(ac.n, 0), m.time_precision FROM messages m JOIN conversations c ON c.id = m.conversation_id LEFT JOIN ( @@ -569,24 +580,28 @@ async fn flag_exact_content_key_dupes( .fetch_all(&mut *conn) .await?; - let mut by_key: HashMap> = HashMap::new(); - for (id, source, content_key, att_count) in rows { - by_key.entry(content_key).or_default().push(Cand { - id, - source, - att_count, + let mut by_key: HashMap> = HashMap::new(); + for (id, source, content_key, att_count, time_precision) in rows { + by_key.entry(content_key).or_default().push(KeyedCand { + cand: Cand { + id, + source, + att_count, + }, + // The flag decides, never the time: a millisecond time can end + // in `.000`. + whole_seconds: time_precision == message_ir::TimePrecision::Seconds.as_str(), }); } let mut flags: Vec<(i64, i64)> = Vec::new(); // (loser_id, winner_id) let mut groups = 0u64; - for cands in by_key.values() { - let sources: HashSet<&str> = cands.iter().map(|c| c.source.as_str()).collect(); - if sources.len() < 2 { - continue; + for cands in by_key.into_values() { + let group_flags = content_key_group_flags(cands, prio); + if !group_flags.is_empty() { + groups += 1; } - groups += 1; - flags.extend(exact_group_flags(cands, prio)); + flags.extend(group_flags); } let flagged = flags.len() as u64; @@ -599,6 +614,42 @@ async fn flag_exact_content_key_dupes( Ok((groups, flagged)) } +/// The `(loser, winner)` pairs of the messages that share one content key. +/// +/// A whole-second message whose own source also holds a message of the same +/// key with milliseconds is first set aside as that message's twin: the +/// key is taken at whole seconds, so the two match in everything else and +/// fall in the same second, and the source recorded the message twice, once +/// without its milliseconds (an SMS Backup+ mail timed by its `Date` header +/// beside one timed by `X-smssync-date`). The rest are flagged by +/// [`exact_group_flags`] when two or more sources hold them, and each twin +/// is hidden under the rest's winner, which is always shown, so the message +/// is shown once and with its milliseconds. A source that holds the +/// message only in whole seconds keeps every copy, as one that holds it +/// only with milliseconds does. +fn content_key_group_flags(cands: Vec, prio: &HashMap<&str, usize>) -> Vec<(i64, i64)> { + let with_milliseconds: HashSet<&str> = cands + .iter() + .filter(|c| !c.whole_seconds) + .map(|c| c.cand.source.as_str()) + .collect(); + let (twins, rest): (Vec<&KeyedCand>, Vec<&KeyedCand>) = cands + .iter() + .partition(|c| c.whole_seconds && with_milliseconds.contains(c.cand.source.as_str())); + let rest: Vec = rest.into_iter().map(|c| c.cand.clone()).collect(); + let sources: HashSet<&str> = rest.iter().map(|c| c.source.as_str()).collect(); + let mut flags = if sources.len() < 2 { + Vec::new() + } else { + exact_group_flags(&rest, prio) + }; + if !twins.is_empty() { + let winner = pick_winner(&rest, prio); + flags.extend(twins.into_iter().map(|t| (t.cand.id, winner))); + } + flags +} + /// The `(loser, winner)` pairs of one group of copies that two or more /// sources hold: the messages of one content key in the exact pass, or one /// cluster of the near-time pass ([`cluster_near_dupes`]). diff --git a/crates/server/server/src/dedupe/tests.rs b/crates/server/server/src/dedupe/tests.rs index 7fe715a6a..84ccd9965 100644 --- a/crates/server/server/src/dedupe/tests.rs +++ b/crates/server/server/src/dedupe/tests.rs @@ -675,6 +675,135 @@ async fn identical_rows_from_one_source_are_both_kept() { } } +/// Insert one received SMS Backup+ message "On my way" at `timestamp`, +/// recorded with `precision`, and answer its id. +async fn sms_backup_plus_row( + conn: &mut SqliteConnection, + guid: &str, + timestamp: &'static str, + precision: message_ir::TimePrecision, +) -> i64 { + MessageRow { + source: "sms-backup-plus", + guid: Some(guid.into()), + timestamp, + time_precision: precision, + is_from_me: false, + body: Some("On my way"), + sort_order: 0, + ..MessageRow::new(TEST_ACCOUNT_ID, 1) + } + .insert(conn) + .await +} + +/// One source that holds a message once in whole seconds and once with +/// milliseconds in the same second shows it once, with the milliseconds: +/// the whole-second copy is the duplicate (#1923). +#[tokio::test] +async fn a_whole_second_message_is_the_duplicate_of_its_millisecond_twin_in_one_source() { + let (pool, _dir) = engine::test_pool().await; + let mut conn = pool.acquire().await.unwrap(); + setup_db(&mut conn).await; + let whole = sms_backup_plus_row( + &mut conn, + "g-whole", + "2015-03-12T18:04:22.000Z", + message_ir::TimePrecision::Seconds, + ) + .await; + let exact = sms_backup_plus_row( + &mut conn, + "g-exact", + "2015-03-12T18:04:22.250Z", + message_ir::TimePrecision::Milliseconds, + ) + .await; + + let stats = dedupe_cross_source(&mut conn, TEST_ACCOUNT_ID, None, 2) + .await + .unwrap(); + + assert_eq!((stats.exact_groups, stats.exact_flagged), (1, 1)); + assert_eq!(duplicate_of(&mut conn, whole).await, Some(exact)); + assert_eq!(duplicate_of(&mut conn, exact).await, None); +} + +/// A time whose source recorded milliseconds is never taken for a whole +/// second because it ends in `.000`: the flag decides, never the time. One +/// source holds the message at `.000` with milliseconds, at `.250` with +/// milliseconds, and once in whole seconds. The two millisecond copies are +/// two messages and both stay shown; only the whole-second copy is hidden. +#[tokio::test] +async fn a_millisecond_time_ending_in_000_is_not_whole_seconds() { + let (pool, _dir) = engine::test_pool().await; + let mut conn = pool.acquire().await.unwrap(); + setup_db(&mut conn).await; + let on_the_second = sms_backup_plus_row( + &mut conn, + "g-000", + "2015-03-12T18:04:22.000Z", + message_ir::TimePrecision::Milliseconds, + ) + .await; + let later = sms_backup_plus_row( + &mut conn, + "g-250", + "2015-03-12T18:04:22.250Z", + message_ir::TimePrecision::Milliseconds, + ) + .await; + let whole = sms_backup_plus_row( + &mut conn, + "g-whole", + "2015-03-12T18:04:22.000Z", + message_ir::TimePrecision::Seconds, + ) + .await; + + dedupe_cross_source(&mut conn, TEST_ACCOUNT_ID, None, 2) + .await + .unwrap(); + + assert_eq!(duplicate_of(&mut conn, on_the_second).await, None); + assert_eq!(duplicate_of(&mut conn, later).await, None); + assert!( + [Some(on_the_second), Some(later)].contains(&duplicate_of(&mut conn, whole).await), + "the whole-second copy is hidden under a millisecond copy" + ); +} + +/// A source that holds a message only in whole seconds keeps every copy, +/// as it did before: two whole-second messages in one second are two +/// messages. +#[tokio::test] +async fn two_whole_second_messages_from_one_source_are_both_kept() { + let (pool, _dir) = engine::test_pool().await; + let mut conn = pool.acquire().await.unwrap(); + setup_db(&mut conn).await; + let mut ids = Vec::new(); + for guid in ["g1", "g2"] { + ids.push( + sms_backup_plus_row( + &mut conn, + guid, + "2015-03-12T18:04:22.000Z", + message_ir::TimePrecision::Seconds, + ) + .await, + ); + } + + let stats = dedupe_cross_source(&mut conn, TEST_ACCOUNT_ID, None, 2) + .await + .unwrap(); + + assert_eq!((stats.exact_groups, stats.exact_flagged), (0, 0)); + for id in ids { + assert_eq!(duplicate_of(&mut conn, id).await, None); + } +} + /// One message held by three sources is one exact group with one survivor: /// the first-listed source's copy, with the other two pointing at it. #[tokio::test] diff --git a/crates/server/server/src/imports_api/tests.rs b/crates/server/server/src/imports_api/tests.rs index c8e43e88c..55a998362 100644 --- a/crates/server/server/src/imports_api/tests.rs +++ b/crates/server/server/src/imports_api/tests.rs @@ -5810,3 +5810,4 @@ async fn a_page_of_import_runs_is_read_without_a_statement_per_row() { } mod backup_dates; +mod time_precision; diff --git a/crates/server/server/src/imports_api/tests/time_precision.rs b/crates/server/server/src/imports_api/tests/time_precision.rs new file mode 100644 index 000000000..5962ac3b9 --- /dev/null +++ b/crates/server/server/src/imports_api/tests/time_precision.rs @@ -0,0 +1,144 @@ +//! A message's `time_precision` through import, the API and an Export Run, +//! and the whole-second twin of a message one source holds with +//! milliseconds too (#1923). + +use super::*; + +/// 2015-03-12T18:04:22Z, the second every message here falls in. +const SECOND: i64 = 1_426_183_462_000; + +/// Create an Import Run for `source` with dedupe on, post `body` as its one +/// batch, and complete it. +async fn import_with_dedupe( + state: &crate::server::AppState, + token: &str, + source: &str, + body: String, +) { + let (_, created): (String, serde_json::Value) = post_created_json( + state, + "/v1/imports", + token, + serde_json::json!({ "source": source, "mode": "append", "dedupe": true }), + ) + .await; + let id = created["id"].as_i64().unwrap(); + let (status, text) = crate::test_support::post_raw( + state, + &format!("/v1/imports/{id}/batches"), + token, + "application/jsonl", + body, + ) + .await; + assert_eq!(status, axum::http::StatusCode::OK, "{text}"); + let _: serde_json::Value = post_json( + state, + &format!("/v1/imports/{id}/complete"), + token, + serde_json::json!({ "status": "completed" }), + ) + .await; +} + +/// An SMS Backup+ file holding `line` as its one message. +fn sms_backup_plus_file(line: MessageLine) -> String { + let header = + conversation_header("sms-backup-plus", "+15555550123").participant("+15555550123", None); + format!("{header}\n{}\n", line.sms().sender("+15555550123")) +} + +/// The same SMS Backup+ message imported once from a file timed in whole +/// seconds and once from a file timed in milliseconds, in either order, is +/// shown once, with the millisecond time. +#[tokio::test] +async fn a_whole_second_copy_and_a_millisecond_copy_are_shown_once_with_the_milliseconds() { + let whole = || { + sms_backup_plus_file( + message_line("g-whole", "On my way") + .at(SECOND) + .whole_seconds(), + ) + }; + let exact = || sms_backup_plus_file(message_line("g-exact", "On my way").at(SECOND + 250)); + for (label, files) in [ + ("whole second first", [whole(), exact()]), + ("milliseconds first", [exact(), whole()]), + ] { + let (state, _fixture, token) = importer().await; + for file in files { + import_with_dedupe(&state, &token, "sms-backup-plus", file).await; + } + let page: serde_json::Value = get_json(&state, "/v1/messages", &token).await; + let items = page["items"].as_array().unwrap(); + assert_eq!(items.len(), 1, "{label}: {page}"); + assert_eq!(items[0]["timestamp"], "2015-03-12T18:04:22.250Z", "{label}"); + assert_eq!(items[0]["time_precision"], "milliseconds", "{label}"); + } +} + +/// A message whose source recorded whole seconds and one whose source +/// recorded milliseconds that end in `.000` keep their precision through +/// the import, the API and an Export Run, and neither hides the other: they +/// are two messages, and the flag, never the time, says which has +/// milliseconds. +#[tokio::test] +async fn each_precision_is_kept_through_import_the_api_and_an_export_run() { + let (state, _fixture, token) = importer().await; + let header = + conversation_header("sms-backup-plus", "+15555550123").participant("+15555550123", None); + let whole = message_line("g-whole", "whole second") + .at(SECOND) + .whole_seconds() + .sms() + .sender("+15555550123"); + let exact = message_line("g-exact", "milliseconds that end in .000") + .at(SECOND + 1000) + .sms() + .sender("+15555550123"); + import_with_dedupe( + &state, + &token, + "sms-backup-plus", + format!("{header}\n{whole}\n{exact}\n"), + ) + .await; + + let precisions = |page: &serde_json::Value| -> Vec<(String, String)> { + let mut out: Vec<(String, String)> = page["items"] + .as_array() + .unwrap() + .iter() + .map(|m| { + ( + m["guid"].as_str().unwrap().to_string(), + m["time_precision"].as_str().unwrap().to_string(), + ) + }) + .collect(); + out.sort(); + out + }; + let expected = vec![ + ("g-exact".to_string(), "milliseconds".to_string()), + ("g-whole".to_string(), "seconds".to_string()), + ]; + + let listed: serde_json::Value = get_json(&state, "/v1/messages", &token).await; + assert_eq!(precisions(&listed), expected, "{listed}"); + + let (_, run): (String, serde_json::Value) = post_created_json( + &state, + "/v1/exports", + &token, + serde_json::json!({ "scope": { "kind": "everything" } }), + ) + .await; + let exported: serde_json::Value = get_json( + &state, + &format!("/v1/exports/{}/messages", run["id"]), + &token, + ) + .await; + assert_eq!(precisions(&exported), expected, "{exported}"); +} From d33c404634cc6061b05a6ace885bebff54345b4d Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:43:39 -0400 Subject: [PATCH 30/42] chore(api): regenerate the OpenAPI document and web types for time_precision Co-Authored-By: Claude Opus 5.5 --- docs/src/assets/openapi.json | 18 ++++++++++++++++++ web/src/lib/serverApi.types.ts | 17 +++++++++++++++++ web/src/test/apiShapes.ts | 1 + 3 files changed, 36 insertions(+) diff --git a/docs/src/assets/openapi.json b/docs/src/assets/openapi.json index 1cb2bd79f..19072d54e 100644 --- a/docs/src/assets/openapi.json +++ b/docs/src/assets/openapi.json @@ -15711,6 +15711,7 @@ "source", "guid", "timestamp", + "time_precision", "sort_order", "is_from_me", "is_announcement", @@ -15854,6 +15855,10 @@ ], "description": "Body text, when present." }, + "time_precision": { + "$ref": "#/components/schemas/TimePrecision", + "description": "Whether the source recorded `timestamp` to the millisecond or in\nwhole seconds. A `timestamp` ending in `.000` is a whole second\nonly when this says `seconds`." + }, "timestamp": { "type": "string", "description": "The instant the message was sent, to the millisecond: RFC 3339\nin UTC with three fractional digits and a `Z` suffix\n(`2015-03-12T18:04:22.250Z`; `.000` when the source records\nwhole seconds). Messages are listed in the order of this time. A\ncaller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." @@ -17754,6 +17759,7 @@ "source", "guid", "timestamp", + "time_precision", "sort_order", "is_from_me", "is_announcement", @@ -17897,6 +17903,10 @@ ], "description": "Body text, when present." }, + "time_precision": { + "$ref": "#/components/schemas/TimePrecision", + "description": "Whether the source recorded `timestamp` to the millisecond or in\nwhole seconds. A `timestamp` ending in `.000` is a whole second\nonly when this says `seconds`." + }, "timestamp": { "type": "string", "description": "The instant the message was sent, to the millisecond: RFC 3339\nin UTC with three fractional digits and a `Z` suffix\n(`2015-03-12T18:04:22.250Z`; `.000` when the source records\nwhole seconds). Messages are listed in the order of this time. A\ncaller shows it in the account's time zone\n(`Account.time_zone`); the database stores nothing\nabout where the phone was." @@ -18892,6 +18902,14 @@ } } }, + "TimePrecision": { + "type": "string", + "description": "How finely the source recorded a message's time.", + "enum": [ + "seconds", + "milliseconds" + ] + }, "TopAttachment": { "type": "object", "description": "One of an account's largest attachments by byte size.", diff --git a/web/src/lib/serverApi.types.ts b/web/src/lib/serverApi.types.ts index ed23ee677..72e76a04c 100644 --- a/web/src/lib/serverApi.types.ts +++ b/web/src/lib/serverApi.types.ts @@ -3379,6 +3379,12 @@ export interface components { tapbacks: components["schemas"]["Tapback"][]; /** @description Body text, when present. */ text: string | null; + /** + * @description Whether the source recorded `timestamp` to the millisecond or in + * whole seconds. A `timestamp` ending in `.000` is a whole second + * only when this says `seconds`. + */ + time_precision: components["schemas"]["TimePrecision"]; /** * @description The instant the message was sent, to the millisecond: RFC 3339 * in UTC with three fractional digits and a `Z` suffix @@ -4448,6 +4454,12 @@ export interface components { tapbacks: components["schemas"]["Tapback"][]; /** @description Body text, when present. */ text: string | null; + /** + * @description Whether the source recorded `timestamp` to the millisecond or in + * whole seconds. A `timestamp` ending in `.000` is a whole second + * only when this says `seconds`. + */ + time_precision: components["schemas"]["TimePrecision"]; /** * @description The instant the message was sent, to the millisecond: RFC 3339 * in UTC with three fractional digits and a `Z` suffix @@ -5041,6 +5053,11 @@ export interface components { /** @description The identity that reacted, for incoming reactions. */ sender: string | null; }; + /** + * @description How finely the source recorded a message's time. + * @enum {string} + */ + TimePrecision: "seconds" | "milliseconds"; /** @description One of an account's largest attachments by byte size. */ TopAttachment: { /** @description Raw text of the identity that keys the conversation. */ diff --git a/web/src/test/apiShapes.ts b/web/src/test/apiShapes.ts index 4e5b321ad..e4f04d856 100644 --- a/web/src/test/apiShapes.ts +++ b/web/src/test/apiShapes.ts @@ -45,6 +45,7 @@ export function message(fields: Partial = {}): Schema["Messag service: null, guid: "g1", timestamp: "2026-08-11T15:04:00Z", + time_precision: "milliseconds", sort_order: 0, is_from_me: false, is_announcement: false, From 9fd576e662f3ac0b410ec770d757931b0546b21d Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:45:29 -0400 Subject: [PATCH 31/42] docs: describe a message's time precision and the whole-second twin rule The common message page, the export structure, the CSV columns, the mail archive headers and the message transfer page describe time_precision at schema version 12. The architecture note gains the rule that, within one source, a whole-second message is the duplicate of its millisecond twin, with its reason. The changelog says version-11 files are refused. Part of #1923. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 19 +++++++++++++ crates/server/server/src/cli.rs | 3 +- .../contacts-identities-and-messages.md | 20 +++++++++++++ .../developer/architecture/common-message.md | 28 +++++++++++++++++-- .../docs/developer/formats/mail-archive.md | 2 ++ .../docs/docs/developer/message-transfer.md | 6 ++-- .../docs/developer/reference/csv-columns.md | 1 + .../docs/docs/developer/reference/database.md | 3 +- .../developer/reference/export-structure.md | 10 ++++--- .../docs/developer/reference/server-cli.md | 4 +-- .../docs/user/features/messages/import.md | 3 ++ 11 files changed, 85 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fe1a4ffdf..092807142 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,20 @@ released versions carry their date on the heading. ### Features +- 2026-10-06: **A message recorded once to the second and once to the + millisecond shows once.** Every message file now says whether each + message's time has milliseconds or only whole seconds, as its backup app + recorded it. iMazing, OpenExtract, GO SMS Pro's PDU files and SMS Backup+ + mails without an `X-smssync-date` record whole seconds; the other sources + record milliseconds. When one backup app holds a message twice, once to + the second and once to the millisecond, such as an SMS Backup+ mail timed + by its `Date` header and one timed by `X-smssync-date`, hiding duplicates + hides the whole-second copy and shows the message at its time to the + millisecond. A time to the millisecond that ends in `.000` counts as + milliseconds. CSV exports carry it in a `time_precision` column, and mail + exports in an `X-ME-Time-Precision` header. A message file at the + previous schema version, 11, is refused, and the backup must be exported + again with this build. - 2026-10-05: **The newer backup decides when a message changed between two backups of one phone.** Every message file now says when its backup was made: an iPhone backup's own date, the date an SMS Backup & Restore file @@ -172,6 +186,11 @@ released versions carry their date on the heading. - If you have a program that reads messages from the HTTP API, a message's `timestamp` and an earlier version's `edited_at` now carry three digits of milliseconds, such as `2015-03-12T18:04:22.250Z`, where they had none. +- Message files at schema version 11, exported before each message said + whether its time has milliseconds, are refused when you import or convert + them. Export the backup again with this build. A program that reads the + HTTP API finds it in a message's `time_precision`, `seconds` or + `milliseconds`. ## [0.10.1] - 2026-10-05 diff --git a/crates/server/server/src/cli.rs b/crates/server/server/src/cli.rs index 7d98db14e..e991907fc 100644 --- a/crates/server/server/src/cli.rs +++ b/crates/server/server/src/cli.rs @@ -39,7 +39,8 @@ pub enum Commands { /// Work on an account's Import Runs (`discard` clears a stranded one) Imports(ImportsArgs), - /// Soft-hide the same SMS when it appears under more than one import source + /// Soft-hide the same SMS when it appears under more than one import source, + /// or once in whole seconds beside its millisecond copy in one source DedupeCrossSource(DedupeArgs), /// Rebuild the Demo Account: generate Demo Data, clear the account, diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 9630a1af2..bc2a501c6 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -439,6 +439,26 @@ was written, not when a part was unsent, so they cannot tell with dedupe off still leaves a changed message's duplicate flag until the next dedupe ([#1805](https://github.com/messagecrate/message-crate/issues/1805)). +**Within one source, a whole-second message is the duplicate of its +millisecond twin.** Each message says whether its source recorded its time to +the millisecond or in whole seconds (`time_precision` in the conversation +file, `messages.time_precision`). The dedupe hides a whole-second message as +the duplicate of a message from the same source that matches it in +everything else and has milliseconds in the same second, so the message is +shown once, with its milliseconds. The content key stays at whole seconds, +so the two share a key, and the exact pass sets the whole-second copy aside +before it compares sources (`content_key_group_flags` in `dedupe.rs`). A +source that holds a message only in whole seconds, or only with +milliseconds, keeps every copy, as before. The flag decides, never the time: +a millisecond time that ends in `.000` is not a whole second. Why: one +source can record one message twice, once without its milliseconds (an SMS +Backup+ mail timed by its `Date` header beside one timed by +`X-smssync-date`), and the two copies have different guids, which are made +at milliseconds, so both reach the database; without the flag, a copy that +landed on `.000` could not be told from a copy that never had milliseconds +([#1096](https://github.com/messagecrate/message-crate/issues/1096), +[#1923](https://github.com/messagecrate/message-crate/issues/1923)). + **A participant's display name has one rule.** The contact's name, else what that backup called them in that conversation, else the identity. One loader applies it for the conversation list, the message pane, and Export. diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index 90f32d5af..91d5222f3 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -30,13 +30,13 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. - **Common-message path** (`ConversationDocument` → `message_ir_format::FormatSink`, one of json/jsonl/csv/eml/mbox/xml): all exporters, including iMessage (`imessage-ir-exporter`). Per-chat formats also accept `write_format`; XML uses a single `smses.xml` via the sink. - **Media + obfuscate** run inside `FormatSink::finish` for every format (`message_crate_core::ExportTransforms`: none / copy / convert / compress, plus optional obfuscate). When obfuscate is on, exporters skip staging real attachment bytes and convert/compress is not run — only placeholder files are written. Exporters pass transforms from `ExporterConfig.media` / `.obfuscate`; there is no CSV-only post-step. EML / MBOX / XML embed media and drop the staged `attachments/` directory afterward. -- **Schema version 11 only** (breaking). Version 11 says when the backup was made, in `export.backup_taken_at_unix_ms` (see [When the backup was made](#when-the-backup-was-made)), which version 10 did not, so an import could not tell which of two backups of one phone is the later one. Version 10 had kept the message a reply quotes in the message's own `reply_to`, for every source (see [Replies](#replies)), where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`, and a reply count in `imessage.num_replies`. Version 9 had given orphaned messages conversations of type `orphaned` (see [Orphaned messages](#orphaned-messages)), where version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, for every source, where version 7 kept the Apple Messages edit history as a JSON value in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, for every source, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, one shape for every source, where version 5 kept Apple Messages reactions as a JSON value in `imessage.tapbacks`. Version 5 had named every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. Version 10 and older are refused, never upgraded. Typed enums/bags, filled outgoing identity, conversation stats, stable null/`[]` keys. Older common-message JSON is not read — regenerate exports after schema changes. +- **Schema version 12 only** (breaking). Version 12 says whether each message's time has milliseconds, in its required `time_precision` (see [Time precision](#time-precision)), which version 11 did not, so a time ending in `.000` could be a whole second or a millisecond time, and the import could not tell a whole-second copy of a message from a millisecond one. Version 11 had said when the backup was made, in `export.backup_taken_at_unix_ms` (see [When the backup was made](#when-the-backup-was-made)), which version 10 did not, so an import could not tell which of two backups of one phone is the later one. Version 10 had kept the message a reply quotes in the message's own `reply_to`, for every source (see [Replies](#replies)), where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`, and a reply count in `imessage.num_replies`. Version 9 had given orphaned messages conversations of type `orphaned` (see [Orphaned messages](#orphaned-messages)), where version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, for every source, where version 7 kept the Apple Messages edit history as a JSON value in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, for every source, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, one shape for every source, where version 5 kept Apple Messages reactions as a JSON value in `imessage.tapbacks`. Version 5 had named every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. Version 11 and older are refused, never upgraded. Typed enums/bags, filled outgoing identity, conversation stats, stable null/`[]` keys. Older common-message JSON is not read — regenerate exports after schema changes. -## Document schema (`schema_version: 11`) +## Document schema (`schema_version: 12`) ```json { - "schema_version": 11, + "schema_version": 12, "export": { "source": "sms-backup-restore", "tool": "SMS Backup & Restore", @@ -63,6 +63,7 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. { "guid": "…", "timestamp_unix_ms": 1400773261000, + "time_precision": "milliseconds", "direction": "outgoing", "service": "sms", "message_kind": "sms", @@ -96,6 +97,27 @@ Pipeline: `backup → common message → FormatSink → user-picked format`. - `guid` is Apple's own id for Apple Messages. Every other source's `guid` is a `MessageGuid`: SHA-256 of the chat id, the direction, the sender of an incoming message, the UTC instant in milliseconds, the text with whitespace collapsed, the sorted attachment digests, and the source's own key where it has one (WhatsApp's `key_id`). It reads no time zone and no display format, so one backup gives the same ids on any computer. The server refuses a message whose `guid` is empty. - Two records a backup cannot tell apart are one message, and the exporter keeps one (`message_ir::one_copy_per_message`). The server's content key, which matches one message across sources, is the same identity at whole seconds. +### Time precision + +Every message has a required `time_precision`: `milliseconds` when the source recorded the time below the second, `seconds` when it recorded whole seconds and `timestamp_unix_ms` ends in `000` because the source has nothing finer. The flag, never the time, says which: a millisecond time can end in `000` too, and it is still `milliseconds`. Each exporter writes what its source records: + +| Source | Precision | +|--------|-----------| +| Apple Messages | `milliseconds` | +| WhatsApp | `milliseconds` | +| SMS Backup & Restore | `milliseconds`, from the `date` attribute | +| GO SMS Pro | `milliseconds` for a message from the XML backup; `seconds` for one read only from a PDU file, whose name records the second | +| SMS Backup+ | `milliseconds` from `X-smssync-date`; `seconds` for a mail without it, timed by its `Date` header | +| iMazing | `seconds` | +| OpenExtract | `seconds` | +| An Export Run of the server | The precision the server stored | + +When an exporter keeps one copy of a message that its source recorded twice ([Identity](#identity)), a whole-second copy that takes a millisecond copy's time takes its precision too. + +The server keeps the flag and answers it as `time_precision` on a message. Within one source, it shows a whole-second message once when the source also holds it with milliseconds in the same second: the whole-second copy is hidden as the duplicate, and the message is shown with its milliseconds. An SMS Backup+ message imported once from a mail timed by `Date` and once from one timed by `X-smssync-date` is one message. + +CSV carries it in the `time_precision` column, and EML and MBOX in the `X-ME-Time-Precision` header. A blank or unknown value is refused rather than guessed. + ### When the backup was made `export.backup_taken_at_unix_ms` is when the backup the file was read from was made, in Unix milliseconds, or `null` when nothing says. Each exporter reads it from its source: diff --git a/docs/src/content/docs/docs/developer/formats/mail-archive.md b/docs/src/content/docs/docs/developer/formats/mail-archive.md index 50229f3aa..c184f51e7 100644 --- a/docs/src/content/docs/docs/developer/formats/mail-archive.md +++ b/docs/src/content/docs/docs/developer/formats/mail-archive.md @@ -144,6 +144,7 @@ A mail an earlier Message Crate wrote names its addresses with `X-ME-Sender-Hand | `X-ME-Service` | lowercase common-message vocabulary preferred (`sms` / `imessage` / …) | Older exports may use `SMS` / `iMessage` | | `X-ME-Message-Kind` | see taxonomy below | | | `X-ME-Timestamp-Unix-Ms` | integer string | Authoritative epoch ms (UTC) | +| `X-ME-Time-Precision` | `seconds` / `milliseconds` | Whether the source recorded the time below the second; required. The header, never the time, decides: a millisecond time can end in `000`. A missing header or another value is refused | | `X-ME-Subject` | string | When distinct from mail `Subject` | | `X-ME-Guid` | hex / guid string | Matches CSV `guid` when possible | | `X-ME-Export-Source` | string | e.g. `sms-backup-restore` | @@ -340,6 +341,7 @@ Normal sticker sends: image MIME part + `X-ME-Attachment-Meta` (`is_sticker`, `s | `group_title` | `X-ME-Group-Title` | | `guid` | `X-ME-Guid` + `Message-ID` | | `timestamp` / `timestamp_utc` / `timestamp_unix_ms` | `Date` + `X-ME-Timestamp-Unix-Ms` | +| `time_precision` | `X-ME-Time-Precision` (`seconds`/`milliseconds`; required) | | `direction` | `X-ME-Direction` | | `service` | `X-ME-Service` | | `sender_identity` / `sender_display_name` | headers + `From` phrase | diff --git a/docs/src/content/docs/docs/developer/message-transfer.md b/docs/src/content/docs/docs/developer/message-transfer.md index ea18e9124..09a6c6191 100644 --- a/docs/src/content/docs/docs/developer/message-transfer.md +++ b/docs/src/content/docs/docs/developer/message-transfer.md @@ -41,11 +41,11 @@ Each conversation is one text file whose name ends in `.jsonl`. JSON Lines means Pictures and other media sit next to those files in `attachments/`. ```jsonl title="One conversation file" -{"schema_version":11,"export":{"source":"sms-backup-restore","tool":"SMS Backup & Restore","owner_identity":"+15555550100","owner_display_name":"Me","backup_taken_at_unix_ms":1400800000000},"conversation":{"chat_identifier":"+15555550101","conversation_type":"individual","participants":[{"identity":"+15555550101","display_name":"Sam"}]}} -{"guid":"msg-1","timestamp_unix_ms":1400773261000,"direction":"outgoing","service":"sms","text":"Hello"} +{"schema_version":12,"export":{"source":"sms-backup-restore","tool":"SMS Backup & Restore","owner_identity":"+15555550100","owner_display_name":"Me","backup_taken_at_unix_ms":1400800000000},"conversation":{"chat_identifier":"+15555550101","conversation_type":"individual","participants":[{"identity":"+15555550101","display_name":"Sam"}]}} +{"guid":"msg-1","timestamp_unix_ms":1400773261000,"time_precision":"milliseconds","direction":"outgoing","service":"sms","text":"Hello"} ``` -The server only reads this current layout (schema version 10). A version-9 file is refused by name, never upgraded. The full field list is on [Export structure](/docs/developer/reference/export-structure/). +The server only reads this current layout (schema version 12). A version-11 file is refused by name, never upgraded. The full field list is on [Export structure](/docs/developer/reference/export-structure/). ## Converters for full backups diff --git a/docs/src/content/docs/docs/developer/reference/csv-columns.md b/docs/src/content/docs/docs/developer/reference/csv-columns.md index 5d4d77615..a316a0d7e 100644 --- a/docs/src/content/docs/docs/developer/reference/csv-columns.md +++ b/docs/src/content/docs/docs/developer/reference/csv-columns.md @@ -24,6 +24,7 @@ CSV output contains one row per message. Conversation and export identity are re | `timestamp_utc` | UTC RFC 3339 time. | | `timestamp_display` | Human-readable time. | | `timestamp_unix_ms` | Unix time in milliseconds. | +| `time_precision` | `milliseconds` when the source recorded the time below the second, `seconds` when it recorded whole seconds. A millisecond time can end in `000` and is still `milliseconds`. A file with a blank or any other value is refused. | | `direction` | `incoming` or `outgoing`. | | `service` | `sms`, `imessage`, `whatsapp`, `rcs`, `discord`, `signal`, `telegram`, `slack`, or `unknown`. | | `sender_identity` | Sender phone number, email, or other identity. Outgoing rows use the export owner when known. | diff --git a/docs/src/content/docs/docs/developer/reference/database.md b/docs/src/content/docs/docs/developer/reference/database.md index e81c4abb0..1386607a0 100644 --- a/docs/src/content/docs/docs/developer/reference/database.md +++ b/docs/src/content/docs/docs/developer/reference/database.md @@ -54,7 +54,8 @@ handle is on in `contact_handles`. ### `messages` -One row = one message (`source`, `guid`, timestamps, `is_from_me`, optional +One row = one message (`source`, `guid`, timestamps, `time_precision`, +`is_from_me`, optional `service` for per-message transport such as `sms` / `imessage` / `rcs` / `whatsapp`, `body`, `content_key`, optional `sender_handle_id` → `handles`, optional `duplicate_of`). diff --git a/docs/src/content/docs/docs/developer/reference/export-structure.md b/docs/src/content/docs/docs/developer/reference/export-structure.md index 2b1e64d5a..57ef70797 100644 --- a/docs/src/content/docs/docs/developer/reference/export-structure.md +++ b/docs/src/content/docs/docs/developer/reference/export-structure.md @@ -1,9 +1,9 @@ --- title: Export structure -description: The JSONL format Message Crate imports — schema version 10, one file per conversation. +description: The JSONL format Message Crate imports — schema version 12, one file per conversation. --- -Message Crate imports JSONL (JSON Lines) exports at schema version 10. Version 9 and older are refused, never upgraded. This page describes the format for CLI users and tool authors. +Message Crate imports JSONL (JSON Lines) exports at schema version 12. Version 11 and older are refused, never upgraded. This page describes the format for CLI users and tool authors. ## Happy path @@ -16,7 +16,9 @@ The JSONL files are plain text — one JSON object per line. The format is the s One `*.jsonl` per conversation, plus media files under `attachments/`: 1. **Line 1** — Conversation header (`schema_version`, `export`, `conversation`) -2. **Following lines** — One message per line (`timestamp_unix_ms`, `direction`, `service`, `text`, `attachments`, and optional fields) +2. **Following lines** — One message per line (`timestamp_unix_ms`, `time_precision`, `direction`, `service`, `text`, `attachments`, and optional fields) + +`time_precision` is required: `milliseconds` when the source recorded the time below the second, `seconds` when it recorded whole seconds. A millisecond time can end in `000` and is still `milliseconds`. Within one source, the server shows a whole-second message once when the source also holds it with milliseconds in the same second, with the milliseconds. `service` is the channel (`sms`, `imessage`, `rcs`, …). A message's reactions are its `reactions` list, one shape for every source: each names the part reacted to (`part_index`), what the reaction is (`kind`, and `emoji` for an emoji reaction), whether the owner reacted (`is_from_me`), and who did (`reactor_identity`, `reactor_display_name`). The server stores each reaction under the person who reacted. A message deleted in the source app before the backup carries `"deletion": "deleted_in_source_app"`, and one its sender unsent carries `"deletion": "unsent"`; a message with neither leaves `deletion` out. The server stores the mark and imports the message like any other. An edited message carries its earlier versions in `edits`, oldest first within each part, each with its `part_index`, its `text` and the time it was written (`edited_at_unix_ms`); `text` is the final version, and a message never edited leaves `edits` out. The server stores each version, and search finds the message by any of them. A reply carries `reply_to`, naming the quoted message's `guid` when the source names it and the part replied to (`part_index`) when the source records one; a reply whose source names no quoted message has a `null` `guid`, and a message that is not a reply leaves `reply_to` out. A named message can still be missing from the export, such as an Apple Messages thread's first message deleted before the backup was made, which the backup does not hold. The server counts each message's replies when it reads it. Apple-specific fields such as message effects live under an optional `imessage` object. @@ -29,7 +31,7 @@ Attachment records may include `digest_sha256` so clients can upload by hash (`P ## Schema compatibility -The server reads one schema version, currently 10. Version 10 keeps the message a reply quotes in the message's own `reply_to`, for every source, where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`. Version 9 had given orphaned messages, ones the backup holds without recording which conversation they were said in, conversations of type `orphaned`: one for each sender, keyed `orphaned:` and the sender's address, and one with no participants, keyed `orphaned:`, for the ones the account holder sent; version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, where version 7 kept the Apple Messages edit history in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, where version 5 kept Apple Messages reactions in `imessage.tapbacks`. Version 5 had called every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. A file written at any other version is refused, with an error naming both the file's version and the version the server expects. To import an older export, re-export it with the current desktop app. +The server reads one schema version, currently 12. Version 12 says whether each message's time has milliseconds, in its `time_precision`, where version 11 did not. Version 11 had said when the backup was made, in `export.backup_taken_at_unix_ms`, where version 10 did not. Version 10 had kept the message a reply quotes in the message's own `reply_to`, for every source, where version 9 kept the Apple Messages reply link in `imessage.is_reply` and `imessage.in_reply_to_guid`. Version 9 had given orphaned messages, ones the backup holds without recording which conversation they were said in, conversations of type `orphaned`: one for each sender, keyed `orphaned:` and the sender's address, and one with no participants, keyed `orphaned:`, for the ones the account holder sent; version 8 put them all in one `individual` conversation named `orphaned`. Version 8 had kept an edited message's earlier versions in its own `edits`, where version 7 kept the Apple Messages edit history in `imessage.edits`. Version 7 had moved a message's mark, Deleted in the source app or Unsent, in its own `deletion`, where version 6 kept the Apple Messages deleted mark in `imessage.is_deleted`. Version 6 had moved a message's reactions into its own `reactions` list, where version 5 kept Apple Messages reactions in `imessage.tapbacks`. Version 5 had called every address an identity (`identity`, `identity_type`, `owner_identity`, `sender_identity`, `reactor_identity`) where version 4 said `handle`. A file written at any other version is refused, with an error naming both the file's version and the version the server expects. To import an older export, re-export it with the current desktop app. ## Related diff --git a/docs/src/content/docs/docs/developer/reference/server-cli.md b/docs/src/content/docs/docs/developer/reference/server-cli.md index e36411e45..2dd4634af 100644 --- a/docs/src/content/docs/docs/developer/reference/server-cli.md +++ b/docs/src/content/docs/docs/developer/reference/server-cli.md @@ -37,7 +37,7 @@ Import and view messages in SQLite * `import` — Import a message-ir JSONL directory, one Import Run per source (source from export.source unless --source) * `imports` — Work on an account's Import Runs (`discard` clears a stranded one) -* `dedupe-cross-source` — Soft-hide the same SMS when it appears under more than one import source +* `dedupe-cross-source` — Soft-hide the same SMS when it appears under more than one import source, or once in whole seconds beside its millisecond copy in one source * `reset-demo` — Rebuild the Demo Account: generate Demo Data, clear the account, import, and process assets. Adds the account when it is not there * `create-database` — Create an empty database, with no Demo Account. `serve` adds the Demo Account only to a database that does not exist yet, so this is how a Message Crate starts empty * `serve` — Run the HTTP API. A database that does not exist yet is created with the Demo Account before the server listens @@ -109,7 +109,7 @@ Discard the account's running Import Run, if it has one. A killed `import` leave ## `message-crate-server dedupe-cross-source` -Soft-hide the same SMS when it appears under more than one import source +Soft-hide the same SMS when it appears under more than one import source, or once in whole seconds beside its millisecond copy in one source **Usage:** `message-crate-server dedupe-cross-source [OPTIONS] --account ` diff --git a/docs/src/content/docs/docs/user/features/messages/import.md b/docs/src/content/docs/docs/user/features/messages/import.md index 025f5e030..eea06b86a 100644 --- a/docs/src/content/docs/docs/user/features/messages/import.md +++ b/docs/src/content/docs/docs/user/features/messages/import.md @@ -390,3 +390,6 @@ Which backup is newer comes from the backup's own date: the date an iPhone backu **Settings → Storage → Import history** shows it beside the backup each run read. A backup that records no date at all keeps a mark once given, and takes a newer text only when its edits are newer. It also keeps the attachments and reactions each backup holds of a message, each one once, so one import of two backups stores what two separate imports of them store. + +Some backups record a message's time to the second and others to the millisecond, and an SMS Backup+ backup can hold both for one message. +When the Message Crate hides duplicates (the server's `import` and `dedupe-cross-source` commands do), a message one backup app holds once to the second and once to the millisecond is shown once, at its time to the millisecond. From 5fe0a144701935bb3f2685ecfd29f9deefa26887 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 00:47:09 -0400 Subject: [PATCH 32/42] docs: name schema version 12 where the docs still named an older one Co-Authored-By: Claude Opus 5.5 --- .../content/docs/docs/developer/architecture/common-message.md | 2 +- docs/src/content/docs/docs/developer/formats/index.md | 2 +- docs/src/content/docs/docs/developer/release.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index 88873db1f..b60450df1 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -226,7 +226,7 @@ Attachment **bytes** are never stored in JSON/JSONL (`#[serde(skip)]`). Paths + ## JSONL layout ```text -{"schema_version":11,"export":{…},"conversation":{…}} +{"schema_version":12,"export":{…},"conversation":{…}} {"guid":"…","timestamp_unix_ms":…, …} … ``` diff --git a/docs/src/content/docs/docs/developer/formats/index.md b/docs/src/content/docs/docs/developer/formats/index.md index f439d6d44..2820a89c7 100644 --- a/docs/src/content/docs/docs/developer/formats/index.md +++ b/docs/src/content/docs/docs/developer/formats/index.md @@ -9,7 +9,7 @@ What each converter writes (and where it falls short). Marks: **yes** / **partia ## Shared model -All converters build a **common message** per conversation (`ConversationDocument`, schema version 10 in [`message-ir`](https://github.com/messagecrate/message-crate/tree/main/crates/libs/ir)), then project the user-picked format via `FormatSink` in [`message-ir-format`](https://github.com/messagecrate/message-crate/tree/main/crates/libs/ir-format) (default **JSON**). When packaging is CSV, columns follow [`CSV_HEADERS`](https://github.com/messagecrate/message-crate/blob/main/crates/libs/ir-format/src/write.rs). Across the board: +All converters build a **common message** per conversation (`ConversationDocument`, schema version 12 in [`message-ir`](https://github.com/messagecrate/message-crate/tree/main/crates/libs/ir)), then project the user-picked format via `FormatSink` in [`message-ir-format`](https://github.com/messagecrate/message-crate/tree/main/crates/libs/ir-format) (default **JSON**). When packaging is CSV, columns follow [`CSV_HEADERS`](https://github.com/messagecrate/message-crate/blob/main/crates/libs/ir-format/src/write.rs). Across the board: - The peer is `chat_identifier` — there is **no** dedicated receiver-phone column - Every participant is a **typed identity**: `identity_type` (`phone` / `email` / `username` / `other`) on each JSON/JSONL participant and inside `participants_json`; the CSV `identity_type` column carries the sender's type, inferred from the identity when the source doesn't supply it diff --git a/docs/src/content/docs/docs/developer/release.md b/docs/src/content/docs/docs/developer/release.md index 4ce3c4459..a7345f0df 100644 --- a/docs/src/content/docs/docs/developer/release.md +++ b/docs/src/content/docs/docs/developer/release.md @@ -12,7 +12,7 @@ The same tag publishes this documentation site to messagecrate.app, so the site Nothing is published to npm or PyPI. Pushing the git tag `v` is what runs the release jobs. A merge to `main` does not ship. -The JSONL schema version 10 is independent of the product version. Version 9 and older are refused, never upgraded. Leave other `Cargo.toml` files at `0.1.0`, and don't bump `web-next/` for a product release. +The JSONL schema version 12 is independent of the product version. Version 11 and older are refused, never upgraded. Leave other `Cargo.toml` files at `0.1.0`, and don't bump `web-next/` for a product release. ## Before tagging From 99660f358ae14f65037de4e6aea3c82e6fe7d6d5 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:05:41 -0400 Subject: [PATCH 33/42] docs(export): each projection function keeps its own doc comment Co-Authored-By: Claude Opus 5.5 --- crates/libs/export/src/project.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/libs/export/src/project.rs b/crates/libs/export/src/project.rs index 8e08da1ba..97948ffc8 100644 --- a/crates/libs/export/src/project.rs +++ b/crates/libs/export/src/project.rs @@ -193,7 +193,6 @@ fn earlier_version_from_api( }) } -/// The mark a server message carries, as the conversation file writes it. /// The precision the server stored, as the conversation file writes it. fn time_precision_from_api(precision: message_crate_api_types::TimePrecision) -> TimePrecision { match precision { @@ -202,6 +201,7 @@ fn time_precision_from_api(precision: message_crate_api_types::TimePrecision) -> } } +/// The mark a server message carries, as the conversation file writes it. fn deletion_from_api(deletion: message_crate_api_types::Deletion) -> Deletion { match deletion { message_crate_api_types::Deletion::DeletedInSourceApp => Deletion::DeletedInSourceApp, From 0353434fa55754d70e115ce8fa9b8dd4777c64ab Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:05:50 -0400 Subject: [PATCH 34/42] docs(changelog): the time precision entry names no mail headers Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c6653fbe2..bfca7489c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,12 +25,12 @@ released versions carry their date on the heading. millisecond shows once.** Every message file now says whether each message's time has milliseconds or only whole seconds, as its backup app recorded it. iMazing, OpenExtract, GO SMS Pro's PDU files and SMS Backup+ - mails without an `X-smssync-date` record whole seconds; the other sources + mails that record no milliseconds record whole seconds; the other sources record milliseconds. When one backup app holds a message twice, once to - the second and once to the millisecond, such as an SMS Backup+ mail timed - by its `Date` header and one timed by `X-smssync-date`, hiding duplicates - hides the whole-second copy and shows the message at its time to the - millisecond. A time to the millisecond that ends in `.000` counts as + the second and once to the millisecond, such as two SMS Backup+ mails of + one message, one timed to the second and one to the millisecond, hiding + duplicates hides the whole-second copy and shows the message at its time + to the millisecond. A time to the millisecond that ends in `.000` counts as milliseconds. CSV exports carry it in a `time_precision` column, and mail exports in an `X-ME-Time-Precision` header. Message files exported before they said whether each time has milliseconds are refused, and the From 58595522f2173e5eea18021d0717e70da24aeb79 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:06:04 -0400 Subject: [PATCH 35/42] refactor(ir, api-types): both TimePrecision enums parse the same way Each gains ALL and parses by finding the variant whose name matches, as Deletion does. Co-Authored-By: Claude Opus 5.5 --- crates/libs/api-types/src/lib.rs | 7 ++++--- crates/libs/ir/src/identity.rs | 11 +++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/crates/libs/api-types/src/lib.rs b/crates/libs/api-types/src/lib.rs index 24953d7f3..b4da4b702 100644 --- a/crates/libs/api-types/src/lib.rs +++ b/crates/libs/api-types/src/lib.rs @@ -487,6 +487,9 @@ pub enum TimePrecision { } impl TimePrecision { + /// Both precisions. + pub const ALL: [Self; 2] = [Self::Seconds, Self::Milliseconds]; + /// The precision as the wire and the database spell it. pub const fn as_str(self) -> &'static str { match self { @@ -497,9 +500,7 @@ impl TimePrecision { /// Read a sent or stored value; anything else names no precision. pub fn parse(value: &str) -> Option { - [Self::Seconds, Self::Milliseconds] - .into_iter() - .find(|p| p.as_str() == value) + Self::ALL.into_iter().find(|p| p.as_str() == value) } } diff --git a/crates/libs/ir/src/identity.rs b/crates/libs/ir/src/identity.rs index 8784ff813..259b73c95 100644 --- a/crates/libs/ir/src/identity.rs +++ b/crates/libs/ir/src/identity.rs @@ -30,6 +30,9 @@ pub enum TimePrecision { } impl TimePrecision { + /// Both precisions. + pub const ALL: [Self; 2] = [Self::Seconds, Self::Milliseconds]; + /// The name the conversation file, the database and the HTTP API use: /// `seconds` or `milliseconds`. pub fn as_str(self) -> &'static str { @@ -40,12 +43,8 @@ impl TimePrecision { } /// The precision [`Self::as_str`] names, or `None` for any other text. - pub fn parse(s: &str) -> Option { - match s { - "seconds" => Some(Self::Seconds), - "milliseconds" => Some(Self::Milliseconds), - _ => None, - } + pub fn parse(value: &str) -> Option { + Self::ALL.into_iter().find(|p| p.as_str() == value) } } From 7093bcb712a0ace9731d856c6bc0476e2e3131c8 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:06:43 -0400 Subject: [PATCH 36/42] fix(dedupe): a stored time_precision the dedupe cannot read fails the pass It was compared as text, so an unknown value counted as milliseconds, where the message read path refuses it. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 411e9f530..8347d499f 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -590,7 +590,9 @@ async fn flag_exact_content_key_dupes( }, // The flag decides, never the time: a millisecond time can end // in `.000`. - whole_seconds: time_precision == message_ir::TimePrecision::Seconds.as_str(), + whole_seconds: message_ir::TimePrecision::parse(&time_precision) + .with_context(|| format!("message {id}: time_precision {time_precision:?}"))? + == message_ir::TimePrecision::Seconds, }); } From 1b5c35d94bcceeadd165359631e55e30d380548c Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:07:00 -0400 Subject: [PATCH 37/42] refactor(dedupe): the twin split moves each message instead of cloning it Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 8347d499f..442743691 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -630,15 +630,15 @@ async fn flag_exact_content_key_dupes( /// message only in whole seconds keeps every copy, as one that holds it /// only with milliseconds does. fn content_key_group_flags(cands: Vec, prio: &HashMap<&str, usize>) -> Vec<(i64, i64)> { - let with_milliseconds: HashSet<&str> = cands + let with_milliseconds: HashSet = cands .iter() .filter(|c| !c.whole_seconds) - .map(|c| c.cand.source.as_str()) + .map(|c| c.cand.source.clone()) .collect(); - let (twins, rest): (Vec<&KeyedCand>, Vec<&KeyedCand>) = cands - .iter() + let (twins, rest): (Vec, Vec) = cands + .into_iter() .partition(|c| c.whole_seconds && with_milliseconds.contains(c.cand.source.as_str())); - let rest: Vec = rest.into_iter().map(|c| c.cand.clone()).collect(); + let rest: Vec = rest.into_iter().map(|c| c.cand).collect(); let sources: HashSet<&str> = rest.iter().map(|c| c.source.as_str()).collect(); let mut flags = if sources.len() < 2 { Vec::new() From 04273c3dc27fc2b96b142bccdcdabf6a377ca583 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:07:14 -0400 Subject: [PATCH 38/42] docs(dedupe): a twin is shown with milliseconds unless another source's copy wins The twin is hidden under the cross-source winner, which pick_winner ranks by attachments and import order, not precision. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/dedupe.rs | 4 +++- docs/architecture/contacts-identities-and-messages.md | 4 +++- .../docs/docs/developer/architecture/common-message.md | 2 +- .../content/docs/docs/developer/reference/export-structure.md | 2 +- docs/src/content/docs/docs/user/features/messages/import.md | 2 +- 5 files changed, 9 insertions(+), 5 deletions(-) diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 442743691..61aec8bde 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -626,7 +626,9 @@ async fn flag_exact_content_key_dupes( /// beside one timed by `X-smssync-date`). The rest are flagged by /// [`exact_group_flags`] when two or more sources hold them, and each twin /// is hidden under the rest's winner, which is always shown, so the message -/// is shown once and with its milliseconds. A source that holds the +/// is shown once. It keeps its milliseconds unless another source's copy +/// wins the cross-source comparison, as one whole-second source imported +/// first does. A source that holds the /// message only in whole seconds keeps every copy, as one that holds it /// only with milliseconds does. fn content_key_group_flags(cands: Vec, prio: &HashMap<&str, usize>) -> Vec<(i64, i64)> { diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 66d189bd3..17a01c224 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -449,7 +449,9 @@ the millisecond or in whole seconds (`time_precision` in the conversation file, `messages.time_precision`). The dedupe hides a whole-second message as the duplicate of a message from the same source that matches it in everything else and has milliseconds in the same second, so the message is -shown once, with its milliseconds. The content key stays at whole seconds, +shown once. It is shown with its milliseconds unless another source holds it +too and that source's copy wins the cross-source comparison (`pick_winner`, +which ranks by attachments and then import order, not precision). The content key stays at whole seconds, so the two share a key, and the exact pass sets the whole-second copy aside before it compares sources (`content_key_group_flags` in `dedupe.rs`). A source that holds a message only in whole seconds, or only with diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index b60450df1..9c773640a 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -114,7 +114,7 @@ Every message has a required `time_precision`: `milliseconds` when the source re When an exporter keeps one copy of a message that its source recorded twice ([Identity](#identity)), a whole-second copy that takes a millisecond copy's time takes its precision too. -The server keeps the flag and answers it as `time_precision` on a message. Within one source, it shows a whole-second message once when the source also holds it with milliseconds in the same second: the whole-second copy is hidden as the duplicate, and the message is shown with its milliseconds. An SMS Backup+ message imported once from a mail timed by `Date` and once from one timed by `X-smssync-date` is one message. +The server keeps the flag and answers it as `time_precision` on a message. Within one source, it shows a whole-second message once when the source also holds it with milliseconds in the same second: the whole-second copy is hidden as the duplicate, and the message is shown with its milliseconds, unless another source holds it too and that source's copy is the one shown. An SMS Backup+ message imported once from a mail timed by `Date` and once from one timed by `X-smssync-date` is one message. CSV carries it in the `time_precision` column, and EML and MBOX in the `X-ME-Time-Precision` header. A blank or unknown value is refused rather than guessed. diff --git a/docs/src/content/docs/docs/developer/reference/export-structure.md b/docs/src/content/docs/docs/developer/reference/export-structure.md index 57ef70797..1e9619e48 100644 --- a/docs/src/content/docs/docs/developer/reference/export-structure.md +++ b/docs/src/content/docs/docs/developer/reference/export-structure.md @@ -18,7 +18,7 @@ One `*.jsonl` per conversation, plus media files under `attachments/`: 1. **Line 1** — Conversation header (`schema_version`, `export`, `conversation`) 2. **Following lines** — One message per line (`timestamp_unix_ms`, `time_precision`, `direction`, `service`, `text`, `attachments`, and optional fields) -`time_precision` is required: `milliseconds` when the source recorded the time below the second, `seconds` when it recorded whole seconds. A millisecond time can end in `000` and is still `milliseconds`. Within one source, the server shows a whole-second message once when the source also holds it with milliseconds in the same second, with the milliseconds. +`time_precision` is required: `milliseconds` when the source recorded the time below the second, `seconds` when it recorded whole seconds. A millisecond time can end in `000` and is still `milliseconds`. Within one source, the server shows a whole-second message once when the source also holds it with milliseconds in the same second, with the milliseconds, unless another source holds it too and that source's copy is the one shown. `service` is the channel (`sms`, `imessage`, `rcs`, …). A message's reactions are its `reactions` list, one shape for every source: each names the part reacted to (`part_index`), what the reaction is (`kind`, and `emoji` for an emoji reaction), whether the owner reacted (`is_from_me`), and who did (`reactor_identity`, `reactor_display_name`). The server stores each reaction under the person who reacted. A message deleted in the source app before the backup carries `"deletion": "deleted_in_source_app"`, and one its sender unsent carries `"deletion": "unsent"`; a message with neither leaves `deletion` out. The server stores the mark and imports the message like any other. An edited message carries its earlier versions in `edits`, oldest first within each part, each with its `part_index`, its `text` and the time it was written (`edited_at_unix_ms`); `text` is the final version, and a message never edited leaves `edits` out. The server stores each version, and search finds the message by any of them. A reply carries `reply_to`, naming the quoted message's `guid` when the source names it and the part replied to (`part_index`) when the source records one; a reply whose source names no quoted message has a `null` `guid`, and a message that is not a reply leaves `reply_to` out. A named message can still be missing from the export, such as an Apple Messages thread's first message deleted before the backup was made, which the backup does not hold. The server counts each message's replies when it reads it. Apple-specific fields such as message effects live under an optional `imessage` object. diff --git a/docs/src/content/docs/docs/user/features/messages/import.md b/docs/src/content/docs/docs/user/features/messages/import.md index eea06b86a..ef6e7115c 100644 --- a/docs/src/content/docs/docs/user/features/messages/import.md +++ b/docs/src/content/docs/docs/user/features/messages/import.md @@ -392,4 +392,4 @@ A backup that records no date at all keeps a mark once given, and takes a newer It also keeps the attachments and reactions each backup holds of a message, each one once, so one import of two backups stores what two separate imports of them store. Some backups record a message's time to the second and others to the millisecond, and an SMS Backup+ backup can hold both for one message. -When the Message Crate hides duplicates (the server's `import` and `dedupe-cross-source` commands do), a message one backup app holds once to the second and once to the millisecond is shown once, at its time to the millisecond. +When the Message Crate hides duplicates (the server's `import` and `dedupe-cross-source` commands do), a message one backup app holds once to the second and once to the millisecond is shown once, at its time to the millisecond. When another backup app holds the same message too, the copy shown follows the rule for messages two backup apps hold, and that copy may have only whole seconds. From e2096473db1822d242141421bfbbdb05072a5926 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:07:25 -0400 Subject: [PATCH 39/42] docs(common-message): SMS Backup & Restore XML cannot carry the time precision Co-Authored-By: Claude Opus 5.5 --- .../content/docs/docs/developer/architecture/common-message.md | 2 +- docs/src/content/docs/docs/user/features/messages/import.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/src/content/docs/docs/developer/architecture/common-message.md b/docs/src/content/docs/docs/developer/architecture/common-message.md index 9c773640a..ebda41ed6 100644 --- a/docs/src/content/docs/docs/developer/architecture/common-message.md +++ b/docs/src/content/docs/docs/developer/architecture/common-message.md @@ -116,7 +116,7 @@ When an exporter keeps one copy of a message that its source recorded twice ([Id The server keeps the flag and answers it as `time_precision` on a message. Within one source, it shows a whole-second message once when the source also holds it with milliseconds in the same second: the whole-second copy is hidden as the duplicate, and the message is shown with its milliseconds, unless another source holds it too and that source's copy is the one shown. An SMS Backup+ message imported once from a mail timed by `Date` and once from one timed by `X-smssync-date` is one message. -CSV carries it in the `time_precision` column, and EML and MBOX in the `X-ME-Time-Precision` header. A blank or unknown value is refused rather than guessed. +CSV carries it in the `time_precision` column, and EML and MBOX in the `X-ME-Time-Precision` header. A blank or unknown value is refused rather than guessed. SMS Backup & Restore XML has no place for it: an Export Run that writes XML leaves it out, and the XML reads back as `milliseconds`, the precision its `date` attribute holds, whatever the server stored. ### When the backup was made diff --git a/docs/src/content/docs/docs/user/features/messages/import.md b/docs/src/content/docs/docs/user/features/messages/import.md index ef6e7115c..c4bdbc869 100644 --- a/docs/src/content/docs/docs/user/features/messages/import.md +++ b/docs/src/content/docs/docs/user/features/messages/import.md @@ -392,4 +392,4 @@ A backup that records no date at all keeps a mark once given, and takes a newer It also keeps the attachments and reactions each backup holds of a message, each one once, so one import of two backups stores what two separate imports of them store. Some backups record a message's time to the second and others to the millisecond, and an SMS Backup+ backup can hold both for one message. -When the Message Crate hides duplicates (the server's `import` and `dedupe-cross-source` commands do), a message one backup app holds once to the second and once to the millisecond is shown once, at its time to the millisecond. When another backup app holds the same message too, the copy shown follows the rule for messages two backup apps hold, and that copy may have only whole seconds. +When the Message Crate hides duplicates (the server's `import` and `dedupe-cross-source` commands do), a message one backup app holds once to the second and once to the millisecond is shown once, at its time to the millisecond. When another backup app holds the same message too, the Message Crate may show that app's copy instead, which can have only whole seconds. From 6bfdb84cd97b811dcd40b3f1661d9c9d08fa458a Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:09:57 -0400 Subject: [PATCH 40/42] fix(import): one message held at whole seconds and at .000 milliseconds says milliseconds The two copies share a guid, so the copy stored first kept its flag and a whole-second copy stored first stayed seconds for good. A staged or promoted copy that says milliseconds now raises the stored flag. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/db/staging.rs | 44 +++++++++++++++++++ .../server/server/src/imports_api/promote.rs | 1 + .../server/server/src/imports_api/staging.rs | 7 ++- .../src/imports_api/tests/time_precision.rs | 32 ++++++++++++++ .../contacts-identities-and-messages.md | 6 ++- 5 files changed, 88 insertions(+), 2 deletions(-) diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index 3e2f3a4e9..aac539ab8 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -1359,6 +1359,50 @@ pub async fn promote_backup_dates(conn: &mut SqliteConnection) -> Result { Ok(sqlx::query(&sql).execute(&mut *conn).await?.rows_affected()) } +/// Mark each stored message `milliseconds` when its staged row is: a +/// whole-second copy and a millisecond copy whose time ends in `.000` have +/// one guid, so they are one message, and the source did record its time +/// to the millisecond, whichever copy was stored first (#1923). A staged +/// `seconds` row leaves the stored flag as it is. Returns how many messages +/// changed. +/// +/// # Errors +/// +/// Returns an error when the update fails. +pub async fn promote_time_precision(conn: &mut SqliteConnection) -> Result { + Ok(sqlx::query( + r" + UPDATE messages + SET time_precision = sm.time_precision + FROM _promote_msg_map mm + JOIN staging_messages sm ON sm.id = mm.staging_id + WHERE messages.id = mm.prod_id + AND sm.time_precision = $1 + AND messages.time_precision != $1 + ", + ) + .bind(message_ir::TimePrecision::Milliseconds.as_str()) + .execute(&mut *conn) + .await? + .rows_affected()) +} + +/// Mark the staged message `staged` `milliseconds`, for another copy of it +/// from the same import that is: the staged-row form of +/// [`promote_time_precision`]. +/// +/// # Errors +/// +/// Returns an error when the update fails. +pub async fn add_staged_copy_milliseconds(conn: &mut SqliteConnection, staged: i64) -> Result<()> { + sqlx::query("UPDATE staging_messages SET time_precision = $1 WHERE id = $2") + .bind(message_ir::TimePrecision::Milliseconds.as_str()) + .bind(staged) + .execute(&mut *conn) + .await?; + Ok(()) +} + /// Whether one copy of a message records a later edit than another, as an /// SQL expression over four SQL values: the copy's earlier-version count /// `n` and newest `edited_at` `newest`, and the other copy's `held_n` and diff --git a/crates/server/server/src/imports_api/promote.rs b/crates/server/server/src/imports_api/promote.rs index 7a7575f8f..3283797f8 100644 --- a/crates/server/server/src/imports_api/promote.rs +++ b/crates/server/server/src/imports_api/promote.rs @@ -279,6 +279,7 @@ impl Promote<'_> { let phase = Self::begin("Recording which backup each changed message came from…"); let dated = staging::promote_backup_dates(self.tx).await?; + staging::promote_time_precision(self.tx).await?; self.done( phase, words( diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index 931e7445d..7a02ea0f6 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -913,7 +913,8 @@ async fn flush_staging_message_chunk( /// neither (#1741, #1804). When either has no date, or the two dates are /// equal ([`db_staging::later_backup`]), the copy gives its text and /// earlier versions when it records a later edit, and its mark when it -/// carries one. One import of two backups then stores +/// carries one. A copy whose time has milliseconds marks the staged message +/// `milliseconds` ([`db_staging::add_staged_copy_milliseconds`]). One import of two backups then stores /// what two separate imports of them store, in either file order (#1806, /// #1837). async fn add_staged_copy( @@ -929,6 +930,7 @@ async fn add_staged_copy( && row.msg.tapbacks.is_empty() && row.msg.deletion.is_none() && staged_source.backup_taken_at.is_none() + && row.msg.time_precision == message_ir::TimePrecision::Seconds { return Ok(()); } @@ -940,6 +942,9 @@ async fn add_staged_copy( let staged = db_staging::staged_message_id(tx, key) .await? .with_context(|| format!("no staged message holds the copy of {}", row.msg.guid))?; + if row.msg.time_precision == message_ir::TimePrecision::Milliseconds { + db_staging::add_staged_copy_milliseconds(tx, staged).await?; + } let held_backup = db_staging::staged_backup_taken_at(tx, staged).await?; match db_staging::later_backup(staged_source.backup_taken_at, held_backup.as_deref()) { BackupOrder::Later(copy_backup) => { diff --git a/crates/server/server/src/imports_api/tests/time_precision.rs b/crates/server/server/src/imports_api/tests/time_precision.rs index 5962ac3b9..73775fb37 100644 --- a/crates/server/server/src/imports_api/tests/time_precision.rs +++ b/crates/server/server/src/imports_api/tests/time_precision.rs @@ -142,3 +142,35 @@ async fn each_precision_is_kept_through_import_the_api_and_an_export_run() { .await; assert_eq!(precisions(&exported), expected, "{exported}"); } + +/// A whole-second copy and a millisecond copy that ends in `.000` have one +/// guid, so they are one stored message. It says `milliseconds` whichever +/// copy came first, in two imports or in one: the source did record the +/// time to the millisecond. +#[tokio::test] +async fn one_message_held_at_whole_seconds_and_at_000_milliseconds_says_milliseconds() { + let line = |whole: bool| { + let line = message_line("g-same", "On my way").at(SECOND); + let line = if whole { line.whole_seconds() } else { line }; + line.sms().sender("+15555550123") + }; + let header = + conversation_header("sms-backup-plus", "+15555550123").participant("+15555550123", None); + let file = |whole: bool| format!("{header}\n{}\n", line(whole)); + let both = |first: bool| format!("{header}\n{}\n{}\n", line(first), line(!first)); + for (label, imports) in [ + ("whole second first, two imports", vec![file(true), file(false)]), + ("milliseconds first, two imports", vec![file(false), file(true)]), + ("whole second first, one import", vec![both(true)]), + ("milliseconds first, one import", vec![both(false)]), + ] { + let (state, _fixture, token) = importer().await; + for body in imports { + import_with_dedupe(&state, &token, "sms-backup-plus", body).await; + } + let page: serde_json::Value = get_json(&state, "/v1/messages", &token).await; + let items = page["items"].as_array().unwrap(); + assert_eq!(items.len(), 1, "{label}: {page}"); + assert_eq!(items[0]["time_precision"], "milliseconds", "{label}"); + } +} diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 17a01c224..36f3122bc 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -456,7 +456,11 @@ so the two share a key, and the exact pass sets the whole-second copy aside before it compares sources (`content_key_group_flags` in `dedupe.rs`). A source that holds a message only in whole seconds, or only with milliseconds, keeps every copy, as before. The flag decides, never the time: -a millisecond time that ends in `.000` is not a whole second. Why: one +a millisecond time that ends in `.000` is not a whole second. A whole-second +copy and a millisecond copy whose time ends in `.000` have one guid, so they +are one stored message, and it says `milliseconds` whichever copy came first, +in one import or across several (`promote_time_precision` and +`add_staged_copy_milliseconds` in `db/staging.rs`). Why: one source can record one message twice, once without its milliseconds (an SMS Backup+ mail timed by its `Date` header beside one timed by `X-smssync-date`), and the two copies have different guids, which are made From d6f19fb12c28c791d9d97d2472cac1f243967b63 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:12:57 -0400 Subject: [PATCH 41/42] fix(import): a millisecond copy raises the flag only at the stored message's time A copy cut to the second elsewhere that kept the guid of a message with other milliseconds was marked milliseconds at a .000 time it never had. promote_time_precision returns nothing, and its phase names what it does. Co-Authored-By: Claude Opus 5.5 --- crates/server/server/src/db/staging.rs | 36 ++++++++------ .../server/server/src/imports_api/promote.rs | 4 +- .../server/server/src/imports_api/staging.rs | 8 ++-- .../src/imports_api/tests/time_precision.rs | 48 ++++++++++++++++++- 4 files changed, 76 insertions(+), 20 deletions(-) diff --git a/crates/server/server/src/db/staging.rs b/crates/server/server/src/db/staging.rs index aac539ab8..c2229174a 100644 --- a/crates/server/server/src/db/staging.rs +++ b/crates/server/server/src/db/staging.rs @@ -1359,18 +1359,19 @@ pub async fn promote_backup_dates(conn: &mut SqliteConnection) -> Result { Ok(sqlx::query(&sql).execute(&mut *conn).await?.rows_affected()) } -/// Mark each stored message `milliseconds` when its staged row is: a -/// whole-second copy and a millisecond copy whose time ends in `.000` have -/// one guid, so they are one message, and the source did record its time -/// to the millisecond, whichever copy was stored first (#1923). A staged -/// `seconds` row leaves the stored flag as it is. Returns how many messages -/// changed. +/// Mark each stored message `milliseconds` when its staged row is and has +/// the same time: a whole-second copy and a millisecond copy whose time +/// ends in `.000` have one guid, so they are one message, and the source +/// did record its time to the millisecond, whichever copy was stored first +/// (#1923). A staged `seconds` row, or one at another time (a copy cut to +/// the second elsewhere that kept the guid), leaves the stored flag as it +/// is. /// /// # Errors /// /// Returns an error when the update fails. -pub async fn promote_time_precision(conn: &mut SqliteConnection) -> Result { - Ok(sqlx::query( +pub async fn promote_time_precision(conn: &mut SqliteConnection) -> Result<()> { + sqlx::query( r" UPDATE messages SET time_precision = sm.time_precision @@ -1379,25 +1380,32 @@ pub async fn promote_time_precision(conn: &mut SqliteConnection) -> Result WHERE messages.id = mm.prod_id AND sm.time_precision = $1 AND messages.time_precision != $1 + AND sm.timestamp = messages.timestamp ", ) .bind(message_ir::TimePrecision::Milliseconds.as_str()) .execute(&mut *conn) - .await? - .rows_affected()) + .await?; + Ok(()) } -/// Mark the staged message `staged` `milliseconds`, for another copy of it -/// from the same import that is: the staged-row form of +/// Mark the staged message `staged` `milliseconds` when it is at +/// `timestamp`, for another copy of it from the same import that is +/// `milliseconds` at that time: the staged-row form of /// [`promote_time_precision`]. /// /// # Errors /// /// Returns an error when the update fails. -pub async fn add_staged_copy_milliseconds(conn: &mut SqliteConnection, staged: i64) -> Result<()> { - sqlx::query("UPDATE staging_messages SET time_precision = $1 WHERE id = $2") +pub async fn add_staged_copy_milliseconds( + conn: &mut SqliteConnection, + staged: i64, + timestamp: &str, +) -> Result<()> { + sqlx::query("UPDATE staging_messages SET time_precision = $1 WHERE id = $2 AND timestamp = $3") .bind(message_ir::TimePrecision::Milliseconds.as_str()) .bind(staged) + .bind(timestamp) .execute(&mut *conn) .await?; Ok(()) diff --git a/crates/server/server/src/imports_api/promote.rs b/crates/server/server/src/imports_api/promote.rs index 3283797f8..7b7c52d0d 100644 --- a/crates/server/server/src/imports_api/promote.rs +++ b/crates/server/server/src/imports_api/promote.rs @@ -277,7 +277,9 @@ impl Promote<'_> { ), ); - let phase = Self::begin("Recording which backup each changed message came from…"); + let phase = Self::begin( + "Recording which backup each changed message came from, and whether its time has milliseconds…", + ); let dated = staging::promote_backup_dates(self.tx).await?; staging::promote_time_precision(self.tx).await?; self.done( diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index 7a02ea0f6..f16b87a24 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -913,8 +913,10 @@ async fn flush_staging_message_chunk( /// neither (#1741, #1804). When either has no date, or the two dates are /// equal ([`db_staging::later_backup`]), the copy gives its text and /// earlier versions when it records a later edit, and its mark when it -/// carries one. A copy whose time has milliseconds marks the staged message -/// `milliseconds` ([`db_staging::add_staged_copy_milliseconds`]). One import of two backups then stores +/// carries one. A copy at the staged message's time that has milliseconds +/// marks it `milliseconds` +/// ([`db_staging::add_staged_copy_milliseconds`]). One import of two +/// backups then stores /// what two separate imports of them store, in either file order (#1806, /// #1837). async fn add_staged_copy( @@ -943,7 +945,7 @@ async fn add_staged_copy( .await? .with_context(|| format!("no staged message holds the copy of {}", row.msg.guid))?; if row.msg.time_precision == message_ir::TimePrecision::Milliseconds { - db_staging::add_staged_copy_milliseconds(tx, staged).await?; + db_staging::add_staged_copy_milliseconds(tx, staged, &row.msg.timestamp).await?; } let held_backup = db_staging::staged_backup_taken_at(tx, staged).await?; match db_staging::later_backup(staged_source.backup_taken_at, held_backup.as_deref()) { diff --git a/crates/server/server/src/imports_api/tests/time_precision.rs b/crates/server/server/src/imports_api/tests/time_precision.rs index 73775fb37..813222828 100644 --- a/crates/server/server/src/imports_api/tests/time_precision.rs +++ b/crates/server/server/src/imports_api/tests/time_precision.rs @@ -77,6 +77,44 @@ async fn a_whole_second_copy_and_a_millisecond_copy_are_shown_once_with_the_mill } } +/// A copy cut to the second elsewhere that kept the guid of a message whose +/// time has other milliseconds stays `seconds`: its time is not the +/// millisecond copy's, so nothing says the source recorded it. +#[tokio::test] +async fn a_whole_second_copy_at_another_time_keeps_seconds() { + let header = + conversation_header("sms-backup-plus", "+15555550123").participant("+15555550123", None); + let whole = message_line("g-same", "On my way") + .at(SECOND) + .whole_seconds() + .sms() + .sender("+15555550123"); + let exact = message_line("g-same", "On my way") + .at(SECOND + 678) + .sms() + .sender("+15555550123"); + for (label, imports) in [ + ( + "two imports", + vec![ + format!("{header}\n{whole}\n"), + format!("{header}\n{exact}\n"), + ], + ), + ("one import", vec![format!("{header}\n{whole}\n{exact}\n")]), + ] { + let (state, _fixture, token) = importer().await; + for body in imports { + import_with_dedupe(&state, &token, "sms-backup-plus", body).await; + } + let page: serde_json::Value = get_json(&state, "/v1/messages", &token).await; + let items = page["items"].as_array().unwrap(); + assert_eq!(items.len(), 1, "{label}: {page}"); + assert_eq!(items[0]["timestamp"], "2015-03-12T18:04:22.000Z", "{label}"); + assert_eq!(items[0]["time_precision"], "seconds", "{label}"); + } +} + /// A message whose source recorded whole seconds and one whose source /// recorded milliseconds that end in `.000` keep their precision through /// the import, the API and an Export Run, and neither hides the other: they @@ -159,8 +197,14 @@ async fn one_message_held_at_whole_seconds_and_at_000_milliseconds_says_millisec let file = |whole: bool| format!("{header}\n{}\n", line(whole)); let both = |first: bool| format!("{header}\n{}\n{}\n", line(first), line(!first)); for (label, imports) in [ - ("whole second first, two imports", vec![file(true), file(false)]), - ("milliseconds first, two imports", vec![file(false), file(true)]), + ( + "whole second first, two imports", + vec![file(true), file(false)], + ), + ( + "milliseconds first, two imports", + vec![file(false), file(true)], + ), ("whole second first, one import", vec![both(true)]), ("milliseconds first, one import", vec![both(false)]), ] { From f833bc80e79618f8aa56471855eec1057748d4f5 Mon Sep 17 00:00:00 2001 From: Matt Beisser <225018+mbeisser1@users.noreply.github.com> Date: Tue, 6 Oct 2026 01:13:19 -0400 Subject: [PATCH 42/42] docs: the twin rule's caveat reaches the CHANGELOG, and its passages are rewrapped Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 15 ++++---- crates/server/server/src/dedupe.rs | 11 +++--- .../server/server/src/imports_api/staging.rs | 20 +++++----- .../contacts-identities-and-messages.md | 38 +++++++++---------- 4 files changed, 41 insertions(+), 43 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bfca7489c..911dac6f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,16 +25,17 @@ released versions carry their date on the heading. millisecond shows once.** Every message file now says whether each message's time has milliseconds or only whole seconds, as its backup app recorded it. iMazing, OpenExtract, GO SMS Pro's PDU files and SMS Backup+ - mails that record no milliseconds record whole seconds; the other sources + mails timed only to the second record whole seconds; the other sources record milliseconds. When one backup app holds a message twice, once to the second and once to the millisecond, such as two SMS Backup+ mails of - one message, one timed to the second and one to the millisecond, hiding - duplicates hides the whole-second copy and shows the message at its time - to the millisecond. A time to the millisecond that ends in `.000` counts as + one message, hiding duplicates hides the whole-second copy and shows the + message at its time to the millisecond. When another backup app holds the + message too, its copy may be the one shown, and it can have whole seconds + only. A time to the millisecond that ends in `.000` counts as milliseconds. CSV exports carry it in a `time_precision` column, and mail - exports in an `X-ME-Time-Precision` header. Message files exported - before they said whether each time has milliseconds are refused, and the - backup must be exported again with this build. + exports in an `X-ME-Time-Precision` header. Message files exported before + they said whether each time has milliseconds are refused, and the backup + must be exported again with this build. - 2026-10-05: **The newer backup decides when a message changed between two backups of one phone.** Every message file now says when its backup was made: an iPhone backup's own date, the date an SMS Backup & Restore file diff --git a/crates/server/server/src/dedupe.rs b/crates/server/server/src/dedupe.rs index 61aec8bde..87d499234 100644 --- a/crates/server/server/src/dedupe.rs +++ b/crates/server/server/src/dedupe.rs @@ -619,18 +619,17 @@ async fn flag_exact_content_key_dupes( /// The `(loser, winner)` pairs of the messages that share one content key. /// /// A whole-second message whose own source also holds a message of the same -/// key with milliseconds is first set aside as that message's twin: the -/// key is taken at whole seconds, so the two match in everything else and -/// fall in the same second, and the source recorded the message twice, once +/// key with milliseconds is first set aside as that message's twin: the key +/// is taken at whole seconds, so the two match in everything else and fall +/// in the same second, and the source recorded the message twice, once /// without its milliseconds (an SMS Backup+ mail timed by its `Date` header /// beside one timed by `X-smssync-date`). The rest are flagged by /// [`exact_group_flags`] when two or more sources hold them, and each twin /// is hidden under the rest's winner, which is always shown, so the message /// is shown once. It keeps its milliseconds unless another source's copy /// wins the cross-source comparison, as one whole-second source imported -/// first does. A source that holds the -/// message only in whole seconds keeps every copy, as one that holds it -/// only with milliseconds does. +/// first does. A source that holds the message only in whole seconds keeps +/// every copy, as one that holds it only with milliseconds does. fn content_key_group_flags(cands: Vec, prio: &HashMap<&str, usize>) -> Vec<(i64, i64)> { let with_milliseconds: HashSet = cands .iter() diff --git a/crates/server/server/src/imports_api/staging.rs b/crates/server/server/src/imports_api/staging.rs index f16b87a24..e46d14d44 100644 --- a/crates/server/server/src/imports_api/staging.rs +++ b/crates/server/server/src/imports_api/staging.rs @@ -903,22 +903,20 @@ async fn flush_staging_message_chunk( Ok(()) } -/// Give the message staged under `row`'s guid what `row`, another copy of -/// it from the same import, adds, by the rules a later import of the copy -/// would follow (`db::staging::promote_deletion_marks`, +/// Give the message staged under `row`'s guid what `row`, another copy of it +/// from the same import, adds, by the rules a later import of the copy would +/// follow (`db::staging::promote_deletion_marks`, /// `db::staging::write_edit_map`): the attachments and reactions the staged /// message does not hold yet, and its mark and text as follows. When both /// backups have a date, a copy from a later backup gives its text, earlier /// versions and mark, mark or no mark, and one from an earlier backup gives /// neither (#1741, #1804). When either has no date, or the two dates are -/// equal ([`db_staging::later_backup`]), the copy gives its text and -/// earlier versions when it records a later edit, and its mark when it -/// carries one. A copy at the staged message's time that has milliseconds -/// marks it `milliseconds` -/// ([`db_staging::add_staged_copy_milliseconds`]). One import of two -/// backups then stores -/// what two separate imports of them store, in either file order (#1806, -/// #1837). +/// equal ([`db_staging::later_backup`]), the copy gives its text and earlier +/// versions when it records a later edit, and its mark when it carries one. +/// A copy at the staged message's time that has milliseconds marks it +/// `milliseconds` ([`db_staging::add_staged_copy_milliseconds`]). One import +/// of two backups then stores what two separate imports of them store, in +/// either file order (#1806, #1837). async fn add_staged_copy( tx: &mut SqliteConnection, stmts: &mut StagingInserts, diff --git a/docs/architecture/contacts-identities-and-messages.md b/docs/architecture/contacts-identities-and-messages.md index 36f3122bc..0bcae4e7c 100644 --- a/docs/architecture/contacts-identities-and-messages.md +++ b/docs/architecture/contacts-identities-and-messages.md @@ -447,25 +447,25 @@ dedupe ([#1805](https://github.com/messagecrate/message-crate/issues/1805)). millisecond twin.** Each message says whether its source recorded its time to the millisecond or in whole seconds (`time_precision` in the conversation file, `messages.time_precision`). The dedupe hides a whole-second message as -the duplicate of a message from the same source that matches it in -everything else and has milliseconds in the same second, so the message is -shown once. It is shown with its milliseconds unless another source holds it -too and that source's copy wins the cross-source comparison (`pick_winner`, -which ranks by attachments and then import order, not precision). The content key stays at whole seconds, -so the two share a key, and the exact pass sets the whole-second copy aside -before it compares sources (`content_key_group_flags` in `dedupe.rs`). A -source that holds a message only in whole seconds, or only with -milliseconds, keeps every copy, as before. The flag decides, never the time: -a millisecond time that ends in `.000` is not a whole second. A whole-second -copy and a millisecond copy whose time ends in `.000` have one guid, so they -are one stored message, and it says `milliseconds` whichever copy came first, -in one import or across several (`promote_time_precision` and -`add_staged_copy_milliseconds` in `db/staging.rs`). Why: one -source can record one message twice, once without its milliseconds (an SMS -Backup+ mail timed by its `Date` header beside one timed by -`X-smssync-date`), and the two copies have different guids, which are made -at milliseconds, so both reach the database; without the flag, a copy that -landed on `.000` could not be told from a copy that never had milliseconds +the duplicate of a message from the same source that matches it in everything +else and has milliseconds in the same second, so the message is shown once. It +is shown with its milliseconds unless another source holds it too and that +source's copy wins the cross-source comparison (`pick_winner`, which ranks by +attachments and then import order, not precision). The content key stays at +whole seconds, so the two share a key, and the exact pass sets the +whole-second copy aside before it compares sources (`content_key_group_flags` +in `dedupe.rs`). A source that holds a message only in whole seconds, or only +with milliseconds, keeps every copy, as before. The flag decides, never the +time: a millisecond time that ends in `.000` is not a whole second. A +whole-second copy and a millisecond copy whose time ends in `.000` have one +guid, so they are one stored message, and it says `milliseconds` whichever +copy came first, in one import or across several (`promote_time_precision` and +`add_staged_copy_milliseconds` in `db/staging.rs`). Why: one source can record +one message twice, once without its milliseconds (an SMS Backup+ mail timed by +its `Date` header beside one timed by `X-smssync-date`), and the two copies +have different guids, which are made at milliseconds, so both reach the +database; without the flag, a copy that landed on `.000` could not be told +from a copy that never had milliseconds ([#1096](https://github.com/messagecrate/message-crate/issues/1096), [#1923](https://github.com/messagecrate/message-crate/issues/1923)).