feat: the conversation file says when its backup was made, and the later backup decides a message's mark and text

[mbeisser1]

Base note: this branch is built on feat/1906-export-crate-name (#1961, the message-crate-pull → message-crate-export rename), which had not merged into main when this was opened. Until #1961 merges, this diff includes the rename's commits; after it merges, only the #1924 commits remain. The rename's latest head is merged in (a7ea0650b).

Feature Description

Every conversation file now says when the backup it was read from was made, and the import uses that date to decide between two copies of one message from one source. The copy from the later backup gives the message its deletion mark (mark or no mark), its text and its earlier versions, in one import in either file order and across imports in either order. An older backup appended after a newer one changes nothing. A file that does not say keeps today's rules: marks add, edits compare their version times. Import details under Settings → Storage show the backup a run read and when it was made.

This is the decision of 2026-10-05 on #1741 (option 3) and #1804 (option 1).

User Story

As a person importing more than one backup of the same phone, I want the newer backup to decide a message that changed between them (recovered after deletion, unsent, edited) so that what Message Crate shows matches my phone now, whichever backup I import first.

Implementation Details

Conversation file (schema 10 → 11). ExportMeta gains backup_taken_at_unix_ms: Option<i64>. SCHEMA_VERSION is 11, and check_schema_version refuses a version-10 file by name, with a test (refuses_a_version_10_file_by_name). Nothing is upgraded.

Where each exporter reads the date.

Source	Date
Apple Messages, iPhone backup	Manifest.plist Date (new ios_backup::ios_backup_date_unix_ms; the plist is readable in an encrypted backup, so the Apple Messages Reader protocol is unchanged)
Apple Messages, Mac chat.db	the file's modification time
WhatsApp, iPhone backup	Manifest.plist Date
WhatsApp, Android	modification time of msgstore.db.crypt15 / msgstore.db (the form's file wins)
WhatsApp, ready-made result.json	the JSON's modification time
SMS Backup & Restore	root backup_date attribute, else the file's modification time; a conversation read from two files takes the newer
iMazing	newest modification time of the CSVs (the export date)
OpenExtract, GO SMS Pro, SMS Backup+	newest modification time of the files read
Export Run (message-crate-export)	newest backup_taken_at of the conversation's messages
demo seed	the settings' reference time

Formats. JSON/JSONL through serde; CSV gains a backup_taken_at_unix_ms column; EML and MBOX gain an X-ME-Backup-Taken-At-Unix-Ms header (the formats' existing X-ME-* prefix). A value that is not a whole number is refused.

Import. Staging stores the date on each staged row (staging_messages.backup_taken_at), and messages.backup_taken_at keeps the date of the backup that decided the stored copy. One SQL rule, later_backup_sql in db/staging.rs, says whether a staged copy is from a later backup (NULL when either side has no date):

• promote_deletion_marks: a later-backup row sets its mark or clears the held one; an earlier/same-backup row changes nothing; undated falls back to "a mark adds".
• write_edit_map: a later-backup row goes into the edit map when its text or version list differs (compared whole with json_group_array(... ORDER BY id)), whatever the version times say; undated falls back to later_edit_sql.
• promote_backup_dates (new phase after the edits) moves the stored date forward.
• In one import, add_staged_copy applies the same rule to the second copy of a staged message (take_staged_copy_from_later_backup); undated keeps take_later_staged_copy and now also adds the copy's mark rather than dropping it to ON CONFLICT DO NOTHING.

The duplicate flag follows the text, because the dedupe compares the text; a test covers it in one import with dedupe on. #1805 (an append with dedupe off leaves a changed message's flag) is not part of this change.

The Import Run records the newest date its files name (db::imports::note_backup_taken_at, per staged conversation).

Key Files Changed

• crates/libs/ir/src/lib.rs, schema_version.rs - the field, version 11, the refusal
• crates/libs/ir-format/src/{write,read_csv,read_mail}.rs, crates/libs/mail/src/* - CSV column and mail header
• crates/libs/ios-backup/src/backup.rs - ios_backup_date_unix_ms
• crates/core/message-crate-core/src/pipeline.rs - export_meta takes the date; file_modified_unix_ms, newest_file_modified_unix_ms
• crates/exporters/* and crates/libs/sbr/src/read.rs - each exporter fills the date
• crates/libs/export/src/{project,run}.rs - an Export Run writes the stored date back
• crates/server/server/src/db/staging.rs, imports_api/{staging,promote}.rs - the later-backup rule
• crates/server/server/src/imports_api/tests/backup_dates.rs - promotion and API tests
• web/src/screens/settings/storage/ImportDetailPanel.tsx, storageUtils.ts - the Backup line
• docs/architecture/contacts-identities-and-messages.md - the rule and its reason

HTTP API Changes

• Message.backup_taken_at (string, RFC 3339 UTC, nullable) on every route that answers a message (GET /v1/messages, conversation messages, and the export read path): when the backup that decided the message's mark and text was made.
• ImportRunSummary.backup_taken_at / ImportRun.backup_taken_at (string, nullable) on GET /v1/imports, GET /v1/imports/{id}, GET /v1/accounts/{id}/imports[/{id}] and the run answers of PATCH/complete: the newest backup date the run's files name. Not on OwnerImportRun: the owner's view of another account's run carries nothing of its backup.
• AccountImportRun's two variants are boxed in Rust (Clippy large_enum_variant); the JSON is unchanged.
• OpenAPI regenerated (docs/src/assets/openapi.json) and web types regenerated (web/src/lib/serverApi.types.ts).

Database Changes

• No schema changes
• schema/sql/*.sql changed (every database rebuilds empty and re-imports; the fingerprint is computed, nothing to bump)

staging_messages.backup_taken_at, messages.backup_taken_at, imports.backup_taken_at (all TEXT, the form timestamp holds). The conversation file format changes too: schema version 11, and version 10 is refused.

How to Test

1. cargo test -p message-crate-server --lib backup_dates — the two-backup cases: the deletion mark and the text (the An append keeps the older text when a later backup only unsent a part #1804 scenario) in one import in both file orders and across two imports in both orders, the dates swapped, an append of the older backup changing nothing, the duplicate flag, the no-date fallback, the Import Run's and a message's backup_taken_at.
2. cargo test -p message-ir-format backup — every format (JSON, JSONL, CSV, EML, MBOX) keeps the date, and CSV refuses a non-number; cargo test -p message-ir schema_version — version 10 refused.
3. One test per exporter that the date is read from its source: ios-backup (the_backup_date_is_the_manifest_s_date, fixture tests/fixtures/dated-backup/Manifest.plist), imessage-ir-exporter (an_iphone_backup_is_dated_by_its_manifest, a_mac_chat_db_is_dated_by_its_modification_time, and through the real reader every_conversation_file_says_when_the_chat_db_was_last_written), whatsapp-exporter (an_iphone_backup_is_dated_by_its_manifest, an_android_backup_is_dated_by_its_database_file), sms-backup-restore-exporter (the_backup_date_is_the_files_backup_date_attribute with fixture tests/fixtures/dated_backup.xml, the modification-time fallback, two backups in one directory), imazing-exporter, openextract-exporter, go-sms-pro-exporter, sms-backup-plus-exporter (modification time), message-crate-export (a_document_says_the_newest_backup_its_messages_came_from).
4. cd web && npx vitest run src/screens/settings/storage — the Backup line in Import details and importBackup.
5. In the app: import two backups of one phone, then open Settings → Storage → Import history → a run; the Backup line names the backup and when it was made.

Each new test fails without the fix, checked by hand: with later_backup_sql forced to NULL and the in-import copy rule and additive copy mark switched off, 5 of the 7 promotion tests fail (the other two cover the new API fields, which do not exist without the change); with the date reads in file_modified_unix_ms, sbr::read::backup_date and ios_backup_date_unix_ms switched off, every exporter test above fails; with the Backup line removed from ImportDetailPanel, both new Vitest cases fail.

Checklist

• Code follows the project's style guidelines
• Self-reviewed the code
• Added unit tests for new functionality
• Added integration tests where applicable
• Existing tests pass locally
• Updated documentation
• ./scripts/check-pr.sh passes and formatter rewrites are committed

Run locally after merging the rename's head: ./scripts/check-pr.sh, cargo test --workspace --no-fail-fast, cargo test --manifest-path src-tauri/Cargo.toml, cd web && npm test && npm run build, ./scripts/check-generated-api-types.sh, cd docs && npm run check && npm run build.

Related Issues

Closes #1924

Unblocks #1741 and #1804, whose decisions this implements; their own acceptance tests and closing are left to them. Related: #1805 (an append with dedupe off still leaves a changed message's duplicate flag), #1923 (the next schema bump), #1906 / #1961 (the base).

Screenshots / Demo

Not captured. Import details gains one line under the figures: Backup, the path the desktop app recorded, and "Made " or "The backup does not say when it was made".

Deployment Notes

The database rebuilds empty on first start (schema change). Conversation files at schema version 10 are refused; export the backup again with this build. CHANGELOG entry under 0.11.0 (Features and Upgrading).

The SMS Backup & Restore XML writer does not write backup_date back yet, so an Export to XML read back in is dated by the file's modification time.

🤖 Generated with Claude Code

[mbeisser1]

Review of 10a0946 (origin/main merged in, so the diff against main holds only the #1924 work). Findings by axis: Merge review (the merge of main, with #1957's search index change) 2, Spec 3, Standards 9, Correctness 3. Three findings have no line in the diff and follow in a top-level comment.

[mbeisser1]

Findings with no line in the diff (review of 10a0946).

1. Spec · spec — "message-crate-pull writes the stored value back out." The export writes the date into every ir-format format, but the SMS Backup & Restore XML writer does not write backup_date. An Export to XML, read back in, is dated by the file's modification time instead of the stored date, which can reverse which backup counts as later. Fix: write the stored date as the root backup_date attribute, with a round-trip test, or file an issue.

2. Standards · bug — crates/libs/sbr/src/lib.rs, line 157: the writer emits <smses count="N"> with no backup_date. The reader then falls back to the modification time, which is when Message Crate wrote the file, so an old backup converted or exported to XML and imported later reads as the newest backup, and its marks and text overwrite those of a backup that is really later. Fix: write backup_date from export.backup_taken_at_unix_ms.

3. Standards · judgement — docs/architecture/contacts-identities-and-messages.md, lines 408–410: the title rule's reason still says "The time a backup was made is not recorded by every source". Every file now carries a date field, so the reason reads stale. Fix: rewrite the reason, or file an issue to move titles onto the backup date.

[mbeisser1]

Re-review of the fix commits (47aff7c, 5de3d2e, 9cc6ce1, 973edb5, edeada4, c431833), and the merge review of the second merge of main (f4f5a07, which brought #1963's millisecond times and #1964). Standards 4, Correctness 1, Merge review 1. Each is answered in its thread.

[mbeisser1]

Answers to the top-level findings:

1. Spec · spec — "message-crate-pull writes the stored value back out." … the SMS Backup & Restore XML writer does not write backup_date.

2. Standards · bug — crates/libs/sbr/src/lib.rs, line 157: the writer emits <smses count="N"> with no backup_date.

Fixed in 973edb5 and f9bac68: SbrBackupWriter::note_backup_date collects the dates and finish writes the newest as the root backup_date, the rule the reader uses for a conversation read from two files; a conversation is noted only when it writes an SMS or MMS. Tests the_backup_date_noted_survives_a_round_trip (sbr) and the_backup_date_comes_from_the_conversations_written (exporter). The one-date-per-file limit is deferred to #1967.

3. Standards · judgement — docs/architecture/contacts-identities-and-messages.md, lines 408–410: the title rule's reason still says "The time a backup was made is not recorded by every source".

Fixed in edeada4: the reason now says the file's backup date is missing where a source records none and for several sources is a modification time that copying can change, so it does not decide a title.

[mbeisser1]

Review summary

Reviewed against the spec (#1924, with the #1741 and #1804 decisions it implements) and the standards (CLAUDE.md, AGENTS.md, writing style, CONTEXT.md, contacts-identities-and-messages.md, http-api.md, ADR 0012, ADR 0002). CI run 37414909306 on fea14d2: success.

Axis	Findings	Fixed	Declined	Deferred
Merge review (main with #1957, then with #1963/#1964)	3	3	0	0
Spec	3	2	1	0
Standards	9	6 (two of them in part)	2 in full, 3 in part	1
Correctness	3	2	1	0
Standards re-review	4	4	0	0
Correctness re-review	1	1 case	1 case	0

• Deferred: A conversation file carries one backup date, so messages from two backups share one #1967, a conversation file carries one backup date, so messages from two backups share one (the SMS Backup & Restore reader's two-file conversations, the XML writer, and an Export Run of a conversation decided by two backups).
• Declined, with the evidence in each thread: the in-import mark for undated copies (asked for by A message's deletion mark is never cleared by a later import, and the first file staged decides it #1741's acceptance criteria); modification times deciding where a source has nothing better (asked for by The conversation file says when the backup was made, so an append knows which backup is newer #1924); a timestamp newtype; export_meta's parameters; the in discriminator in storageUtils.ts; the XML date clearing a mark (an sms-backup-restore copy never carries one).
• Fixes of note: a later backup that clears an Unsent mark gives back the search index row; equal backup dates fall back to the rules for files without one; a Mac read is dated by chat.db and its write-ahead log; an Export Run names a backup date only when every message shares it; the SMS Backup & Restore XML writer writes backup_date; the backup date is stored to the millisecond.

Merges of the base:

• c588bc8: crates/server/server/src/db/staging.rs, the doc comment of promote_deletion_marks (both texts kept). The rename from refactor(export): rename message-crate-pull to message-crate-export #1961 dropped out of the diff.
• f4f5a07: CHANGELOG.md (both Upgrading entries kept). The resolution also passes the backup date to format_utc_timestamp in milliseconds, because fix(server): keep a message's time to the millisecond #1963 changed that function to take milliseconds.

No commits for CI failures. No Spec skip. No user threads.