feat: each message says whether its time has milliseconds, and a whole-second twin is a duplicate

[mbeisser1]

Feature Description

Each message in the conversation file now says whether its source recorded its time to the millisecond or in whole seconds (time_precision: seconds or milliseconds). The server stores the flag, answers it on the API, and an Export Run writes it back out. With the flag, the dedupe can hide a whole-second message as the duplicate of a message from the same source that matches it in everything else and has milliseconds in the same second, so the message is shown once, at its time to the millisecond. A millisecond time that ends in .000 is not taken for a whole second: the flag decides, never the value.

For people who import the same phone from backups that record time differently, for example SMS Backup+ mails timed by Date beside mails timed by X-smssync-date. Asked for by the decision of 2 October 2026 on #1096.

User Story

As a person who imports two backups of one phone, one that records whole seconds and one that records milliseconds, I want each message shown once, at its exact time, so that my conversation has no doubled messages and keeps the order the phone recorded.

Implementation Details

• File format (schema version 11 → 12). IrMessage gains a required time_precision (message_ir::TimePrecision, now serde lowercase). A version-11 file is refused by its version (check_schema_version), never upgraded. one_copy_per_message now returns the precision it keeps with each time, so a whole-second copy that takes a millisecond copy's time (GO SMS Pro PDU beside the XML) takes the millisecond precision too.
• Exporters. The projection already knew the precision (message_time); it is now written. iMazing, OpenExtract, GO SMS Pro PDU-only messages, and SMS Backup+ mails without X-smssync-date write seconds; Apple Messages, WhatsApp, SMS Backup & Restore, GO SMS Pro XML and SMS Backup+ with X-smssync-date write milliseconds. Each exporter's fixture test checks the field; SMS Backup+ gains a test for the Date fallback.
• ir-format. CSV gains a time_precision column; EML and MBOX gain a required X-ME-Time-Precision header (the repo's X-ME- prefix; the issue text said X-MC-*). A blank, missing or unknown value is refused rather than guessed.
• Server. messages.time_precision and staging_messages.time_precision (NOT NULL, CHECK in the two values). The import stages and promotes the flag; the message read path decodes it strictly.
• Dedupe. flag_exact_content_key_dupes now reads time_precision. A new content_key_group_flags sets aside, within each content-key group, every whole-second message whose own source also holds a millisecond message in the group, runs the existing cross-source rule (exact_group_flags) over the rest, and hides each set-aside message under the rest's winner (always shown). The content key stays at whole seconds. A source that holds a message only in whole seconds, or only with milliseconds, keeps every copy as before.
• Export. message-crate-export maps the API's time_precision back into the file.
• Demo seed. Writes milliseconds (every source it imitates records milliseconds).

Tests, each verified to fail without the change it covers:

• dedupe::tests::a_whole_second_message_is_the_duplicate_of_its_millisecond_twin_in_one_source and imports_api::tests::time_precision::a_whole_second_copy_and_a_millisecond_copy_are_shown_once_with_the_milliseconds (imported once from a whole-second file and once from a millisecond file, both orders, shown once with .250): fail with the twin rule disabled.
• dedupe::tests::a_millisecond_time_ending_in_000_is_not_whole_seconds: fails with the twin rule disabled, and also fails under a rule that reads .000 from the time instead of the flag.
• schema_version::tests::refuses_a_version_11_file_by_name: fails with SCHEMA_VERSION at 11.
• imports_api::tests::time_precision::each_precision_is_kept_through_import_the_api_and_an_export_run (seconds and a .000 milliseconds message, through import, GET /v1/messages and GET /v1/exports/{id}/messages): fails when staging drops the flag.
• export::project::tests::a_message_keeps_the_precision_the_server_stored: fails when the export ignores the stored flag.
• ir-format every_format_keeps_whether_a_time_has_milliseconds (JSON, JSONL, CSV, EML, MBOX): fails when the CSV writer or the mail writer drops the flag.

Key Files Changed

• crates/libs/ir/src/{lib.rs,identity.rs,projection.rs,schema_version.rs} - the field, version 12, precision through one_copy_per_message
• crates/libs/ir-format/src/{write.rs,read_csv.rs}, crates/libs/mail/src/{headers.rs,lib.rs,parse.rs} - CSV column and X-ME-Time-Precision
• crates/exporters/sms-backup-restore-exporter/src/read.rs, crates/exporters/imessage-ir-exporter/src/convert.rs - the two exporters that build messages outside the projection
• schema/sql/messages.sql, schema/sql/staging.sql, crates/server/server/src/{models.rs,imports_api/staging.rs,db/staging.rs,db/conversation_messages.rs} - storage and the API
• crates/server/server/src/dedupe.rs - the whole-second twin rule
• crates/libs/api-types/src/lib.rs, crates/libs/export/src/project.rs - the API type and Export
• docs/architecture/contacts-identities-and-messages.md - the dedupe rule and its reason

HTTP API Changes

• Message.time_precision (new, required): seconds or milliseconds (new TimePrecision schema). Every route that answers a Message carries it: GET /v1/messages, GET /v1/conversations/{id}/messages, GET /v1/exports/{id}/messages, and the rest.
• POST /v1/imports/{id}/batches: a message line without time_precision, or a header at schema version 11, is refused with 422 naming the line or the version.
• OpenAPI (docs/src/assets/openapi.json) and web types (web/src/lib/serverApi.types.ts) regenerated.

Database Changes

• No schema changes
• schema/sql/*.sql changed (every database rebuilds empty and re-imports; the fingerprint is computed, nothing to bump)

messages.time_precision and staging_messages.time_precision, TEXT NOT NULL CHECK (time_precision IN ('seconds', 'milliseconds')).

How to Test

1. cargo test -p message-crate-server --lib -- dedupe:: time_precision and cargo test -p message-ir-format -p message-ir -p message-crate-export.
2. Export an SMS Backup+ directory holding one message twice, once with X-smssync-date and once with only Date, in two runs; import both with message-crate-server import (dedupe on) and check GET /v1/messages lists it once, with milliseconds and "time_precision": "milliseconds".
3. Convert a conversation to CSV and to EML; check the time_precision column and the X-ME-Time-Precision header.

Checklist

• Code follows the project's style guidelines
• Self-reviewed the code
• Added unit tests for new functionality
• Added integration tests where applicable
• Existing tests pass locally
• Updated documentation
• ./scripts/check-pr.sh passes and formatter rewrites are committed

Run locally on the head: ./scripts/check-pr.sh (pass), cargo test --workspace (2826 passed, 0 failed), cargo test --manifest-path src-tauri/Cargo.toml (pass), cd web && npm test (2062 passed) and npm run build (pass), ./scripts/check-generated-api-types.sh (match), cd docs && npm run check (0 errors) and npm run build (pass).

Related Issues

Closes #1923

Builds on #1096 (merged as #1963) and #1924 (merged as #1966). The branch was cut from #1966's branch and merged #1963's; both are now on main, and this pull request's diff against main carries none of their commits' changes.

Screenshots / Demo

No screen changes.

Deployment Notes

• Schema version 12: conversation files at version 11 are refused when imported or converted; the backup must be exported again with this build.
• The schema change rebuilds the database empty on first start; messages must be imported again.
• The dedupe rule takes effect wherever the dedupe pass runs (an Import Run created with dedupe: true, message-crate-server import, dedupe-cross-source, reset-demo). The desktop app's push does not ask for dedupe today.
• Known gaps, not handled here: an Apple Messages chat.db from before iOS 11 / macOS 10.13 records whole seconds but is written as milliseconds, because the Apple Messages Reader's protocol does not say which form it read (Apple Messages writes milliseconds for a chat.db that records whole seconds #1970). The desktop app's import never asks for dedupe, so the twin rule does not run through it (The desktop app's import never hides duplicates #1969). SMS Backup & Restore XML written back from the common message cannot carry the flag (documented in common-message.md).
• A whole-second copy and a millisecond copy that both land on .000 share a guid; the stored message says milliseconds whichever came first (promote_time_precision, add_staged_copy_milliseconds).

🤖 Generated with Claude Code

[mbeisser1]

Review of 19ff774: Standards, Spec and Correctness findings on their lines below. Findings with no line in the diff follow in a top-level comment.

[mbeisser1]

Findings with no line in the diff at 19ff774.

F1 · Spec · spec (crates/libs/push/src/http.rs line 301). Spec: "An SMS Backup+ message imported once from a whole-second file and again from a millisecond file is shown once, with the millisecond time." create_import sends only source and mode; CreateImportRequest.dedupe defaults to false, and nothing in src-tauri or push sets it. Through the desktop app, the product's import path, both copies stay shown, and not only on .000: the whole-second copy's guid is keyed at .000, so it differs from the .250 copy's guid and an append stores both. a_whole_second_copy_and_a_millisecond_copy_are_shown_once_with_the_milliseconds passes only because it creates the run with "dedupe": true. Fix: run the within-one-source twin rule on every promote whether or not dedupe is on (it only flags rows inside one source), or have the push client send dedupe: true; add a test that imports with the push library's defaults.

F2 · Spec · judgement (crates/server/server/src/imports_api/staging.rs). A whole-second copy and a millisecond copy that ends in exactly .000 share a guid, so an append keeps whichever came first. If the whole-second copy came first, the message stays seconds though a millisecond copy was imported. Fix: on a guid match where the stored row is seconds and the incoming one is milliseconds, set the stored row to milliseconds; or file an issue.

F3 · Correctness · bug (crates/server/server/src/db/staging.rs line 1338, the promote's guid match). Source A imports a whole-second copy W at X.000 (an SMS Backup+ mail with only Date). A later append of the same source brings the millisecond copy at X.000 (with X-smssync-date). Same guid, so promote keeps the stored row, and time_precision stays seconds for good; the API and every export say seconds for a time the source recorded to the millisecond. Fix: on a guid match, raise the stored row to milliseconds when the staged copy says milliseconds: one UPDATE over _promote_msg_map, like the backup_taken_at update.

[mbeisser1]

Re-review of the fix commits (Standards and Correctness), pinned to f833bc8. Each is fixed in this push; replies follow in each thread.

[mbeisser1]

Answers to the findings in #1968 (comment):

F1 · Spec · spec (crates/libs/push/src/http.rs line 301). … create_import sends only source and mode; CreateImportRequest.dedupe defaults to false …

Deferred to #1969. The twin rule runs wherever the dedupe runs (the server's import, dedupe-cross-source and reset-demo commands, and an Import Run created with "dedupe": true), and #1923 places it in the dedupe ("Today dedupe_cross_source … flags a duplicate only when another source holds it"). That the desktop app never asks for dedupe predates #1923 and holds for the cross-source rule too; choosing between the app asking for it and the server running one-source rules on every promote changes every import, so it is filed rather than decided here.

F2 · Spec · judgement (crates/server/server/src/imports_api/staging.rs). A whole-second copy and a millisecond copy that ends in exactly .000 share a guid, so an append keeps whichever came first …

F3 · Correctness · bug (crates/server/server/src/db/staging.rs line 1338, the promote's guid match). … time_precision stays seconds for good …

Fixed in 6bfdb84, with the time guard in d6f19fb: promote_time_precision in db/staging.rs raises a stored message to milliseconds when its staged copy is milliseconds at the same time, and add_staged_copy_milliseconds does the same for two copies in one import. one_message_held_at_whole_seconds_and_at_000_milliseconds_says_milliseconds (both orders, one import and two) fails without either half; a_whole_second_copy_at_another_time_keeps_seconds covers the guard. The architecture doc states the rule.

[mbeisser1]

Review summary for f833bc8

Merge check before the review: the branch recorded #1966's squash (7a8435e) with git merge -s ours. The squash's tree equals fea14d2, the #1966 branch head this branch had already merged in de31ed4, and origin/main is an ancestor of the head. Every file in git diff origin/main HEAD is #1923 work, and no hunk reverts anything main has. The PR was up to date with main, so no merge of the base was made, and none was needed before merging.

Findings, by axis:

• Standards: 5 in the first review, 6 in the re-review of the fixes. 11 Fixed.
• Spec: 3 (one spec: the desktop app never asks for dedupe; two judgement). 1 Fixed (a .000 guid match keeps seconds), 2 Deferred: The desktop app's import never hides duplicates #1969 (the desktop app's import never hides duplicates) and Apple Messages writes milliseconds for a chat.db that records whole seconds #1970 (Apple Messages writes milliseconds for a chat.db that records whole seconds).
• Correctness: 4 in the first review, 1 in the re-review. 4 Fixed (the stored flag on a .000 guid match, the narrowed claim about which copy shows milliseconds, the XML note, and the time guard on the flag raise), 1 Deferred to Apple Messages writes milliseconds for a chat.db that records whole seconds #1970 (the same chat.db gap the Spec axis found).

Totals: 16 Fixed, 0 Declined, 3 Deferred (#1969, #1970, with two findings sharing #1970).

No commits were needed for CI failures: the ready run on f833bc8 passed. No merge of the base was made. There was a Spec review. No user threads are open.