fix(streams): one clock read per stream record on all four backends by LeeroyHannigan · Pull Request #463 · ExtendDB/extenddb

LeeroyHannigan · 2026-10-23T15:03:34Z

Closes #459.

A stream record stored two creation times from two clock reads. ApproximateCreationDateTime in the record body was SystemTime::now() read when the record was built. The created_at column, which the 24-hour retention sweep filters on, was the column default: NOW() on PostgreSQL, the transaction start time, and strftime('now') at insert time on SQLite. MongoDB read DateTime::now() a second time in the insert. Cassandra read Utc::now() once for the HLC sequence number and again for created_at_ms.

On PostgreSQL a write that waits inside its transaction (a row lock, a long TransactWriteItems) stored a created_at earlier than its own ApproximateCreationDateTime by the length of the wait, so the retention sweep deleted it before records with the same ApproximateCreationDateTime written in a short transaction. Measured on main: a PutItem that waited 2.1 s on a row lock stored created_at 1792765145 against ApproximateCreationDateTime 1792765147.

Change

Each backend takes one wall-clock read per record and derives both values from it.

SQLite and PostgreSQL bind created_at explicitly in the transaction path and in the write_stream_record trait method; the column defaults stay for any other writer. SQLite writes the millisecond text shape the default produces (24 characters), so rows from before and after this change sort together as text.

On PostgreSQL the reading is the database's clock_timestamp(), returned by the statement that already draws nextval('stream_seq'), so created_at stays on the clock that the retention sweep's NOW() cutoff uses and the 24-hour window does not depend on skew between the ExtendDB host and the database.

MongoDB uses one DateTime for the record body and the document.
Cassandra's HybridClock gains generate_with_created_at, returning the sequence number and the wall-clock reading taken by the call. The sequence prefix is that reading or a later clamped value, so created_at_ms <= hlc_ms (the 13-digit prefix of the sequence number) holds for every record. At counter overflow the prefix moves to the later of a fresh reading and the previous prefix plus one, so the prefix stays above every earlier reading and the sequence number sorts after every earlier one. That overflow change is the same line fix(cassandra): keep the HLC forward clamp across counter overflow #502 (fix/465-cassandra-hlc-overflow) carries for Cassandra HLC sequence can regress at counter overflow under a forward clamp #465; this PR needs it for the invariant above to hold on the overflow branch, so whichever of the two lands second is rebased onto the other, and Cassandra HLC sequence can regress at counter overflow under a forward clamp #465 closes through fix(cassandra): keep the HLC forward clamp across counter overflow #502. The TTL worker's TtlStreamPlan takes both values from the same call.

No schema change. Records written before the upgrade keep their default-clock created_at. Their difference from ApproximateCreationDateTime is at most the transaction length, plus on PostgreSQL the skew between the ExtendDB host clock and the database clock, and they age out after 24 hours.

Tests

crates/storage-postgres/tests/stream_record_clock.rs: an outside transaction holds a row lock while a PutItem with stream capture waits 2.1 s; every stored row's created_at second equals its ApproximateCreationDateTime, including a service-identity delete, and the waited record's created_at lies between clock_timestamp() reads taken before the lock is released and after the write returns. Fails on main with the values above; passes here. It now runs in the PostgreSQL storage-level step of integration.yml. Against a PostgreSQL whose clock ran one hour ahead of the host, it passes, and it fails on the clock_timestamp() bounds when the helper reads the host clock instead.
SQLite: in-memory engine read-back in data/tx_helpers.rs over three puts and a service-identity delete; also pins the 24-character text shape.
Cassandra: tests/direct/streams.rs reads back sequence_number, created_at and the record, asserting created_at_ms <= hlc_ms and created_at_ms / 1000 == ApproximateCreationDateTime on every row. Runs in the direct_integration job. tests/ttl_integration.rs asserts the same two properties on the TTL sweep's REMOVE record, which the worker writes from its persisted TtlStreamPlan; that step is continue-on-error in CI (Feat/storage cassandra in tree #338).
MongoDB: a live read-back test in backup_live_tests.rs, #[ignore]d like its neighbours and run with EXTENDDB_TEST_MONGODB_URI (passes against a 7.0 replica set).
HybridClock unit tests: across 1,000 clamped reads, created_at_ms stays at or below the prefix and between wall-clock reads taken around each call. The overflow branch runs on every run from injected readings, including a re-read below the call's own reading, where the prefix stays at or above the reading and the sequence follows the previous one, and overflow under a forward clamp (the last prefix 5 s ahead of the reading, the re-read a further 5 s back), where the prefix moves to the last prefix plus one. On the normal branch created_at_ms equals the prefix.

Gates on the head: cargo fmt, cargo +1.97.0 clippy --all-targets -- -D warnings, cargo +1.88.0 check --all-targets on the storage crates, the workspace unit tests, and the PostgreSQL and Cassandra integration tests above against PostgreSQL 16 and Cassandra 4.1.

Not changed here

The SQLite retention cutoff is RFC 3339 with no fraction on a whole second, so a row in the cutoff's own second sorts before it as text and is deleted up to a second early. This predates the commit and is unchanged by it; it is filed as its own issue.
The write_stream_record trait method writes created_at at whole-second precision, which is all the record carries. It has no live caller; every data-plane write goes through the transaction helpers.

robinnsc

Approving: nothing here blocks. The fix works. On the PostgreSQL release binaries, a PutItem that waited about 2 s on an outside row lock stored created_at 2 s before its ApproximateCreationDateTime on base and the same second on this branch, including with the server time zone set to a non-UTC offset. An upgrade from base to head on the same databases kept the sweep's selection and GetRecords ordering correct across the old and new rows. Every production writer on the four backends now takes both values from one read, and the branch merges cleanly with main and with #543. A few things:

stream_record_clock isn't in the PostgreSQL storage-level step's --test list (integration.yml:310-313), so in CI it takes the skip() path and reports ok. It's the one test that reproduces #459 the way production hits it. The SQLite, MongoDB and Cassandra read-back tests would also pass on base, since there the two old reads were microseconds apart. So adding --test stream_record_clock there might be worth doing in this PR.
In "Not changed here", the row in the cutoff's own second sorts before the cutoff, not after (. sorts before Z), which is why created_at < ? deletes it. #464 has it the right way round. The edge is also a little wider than whole seconds: time trims trailing zeros, so a cutoff with one or two fractional digits (...07.5Z) deletes up to about 99 ms early. The fix #464 proposes covers that too, so it might just be worth a note on the issue.

robinnsc · 2026-10-28T06:48:11Z

+    // One wall-clock read feeds both the record's ApproximateCreationDateTime
+    // and the created_at column. The column default (NOW()) is the transaction
+    // start time, which in a long transaction sits seconds before the record.
+    let created_at = time::OffsetDateTime::now_utc();


created_at now comes from the ExtendDB host's clock, while the retention sweep (stream_engine.rs:497-498) still compares it with PostgreSQL's NOW(). Before, both sides used the database clock. Now retention is offset by the skew between the app host and the database, early if the app clock is behind. Under NTP that's milliseconds and well inside the hourly sweep interval, and it's a better trade than the transaction-wait error this fixes. Still, it might be worth binding the cutoff from now_utc() as the SQLite sweep and the MongoDB defence sweep do, so each backend compares against the clock that wrote the row (prune_metrics is a precedent in this crate). With several instances each running the sweep, that moves the skew to between hosts rather than removing it, so a line in the description noting the clock source changed would also be fine.

Agreed, and the push of 16:35 UTC (da74c30) takes the option you describe. The nextval statement now reads clock_timestamp() in the same round trip, so created_at and ApproximateCreationDateTime both come from the database clock, after the write's locking reads, and the retention sweep compares created_at with NOW() on that same clock. The host clock is no longer involved on PostgreSQL. The stream_record_clock test binds the waited record's created_at between two clock_timestamp() readings, one taken in the outside transaction before it releases the lock and one after the write returns, so a helper that reads the host clock fails it as soon as the database clock differs from the host's (it fails on main, and against a database whose clock was skewed with libfaketime in a local run). The description now says the reading is the database's clock_timestamp(), so the 24-hour window does not depend on skew between the ExtendDB host and the database, and that pre-upgrade rows carry that skew.

robinnsc · 2026-10-28T06:48:11Z

+        // Land on the same-millisecond path, then drive the counter to the
+        // limit so the next call takes the overflow branch.
+        let (first, _) = clock.generate_with_created_at();
+        clock.logical_counter = 999_999;


Small one: if a millisecond ticks between the two calls, the second call takes the now_ms > last_timestamp_ms branch. That also resets the counter and gives 000000 and seq > first, so every assertion passes without the overflow branch running, and nothing checks that it did. Arming it properly probably needs an injectable clock, so it may not be worth it here. Relatedly, in the test above, the normal-branch check can't tell now_ms from last_timestamp_ms because they're equal on that branch. Returning the clamped value instead survives every test, and an (before..=after).contains(&created_at_ms) assertion inside the clamp loop would catch it.

Both points are taken in the 16:35 UTC push (0ab169e and 7caeb94). generate_with_created_at now delegates to generate_from(now_ms, reread), and the overflow tests inject both readings, so the branch runs on every run and a flag asserts that reread was called; no sleep or second wall-clock call remains in them. The clamp-branch test asserts (before..=after).contains(&created_at_ms) inside its loop, so returning the clamped prefix fails it there (the prefix sits 5 s ahead of the window). The overflow branch also changed: the prefix moves to the later of the re-read and the last prefix plus one, because under a forward clamp, or a backwards step during the wait, the re-read can sit below the last prefix, and two tests pin that case.

jcshepherd · 2026-10-28T23:16:17Z

+/// sort together as text in time order.
+pub(crate) fn format_epoch_millis(ms: i64) -> String {
+    let dt = OffsetDateTime::from_unix_timestamp_nanos(i128::from(ms) * 1_000_000)
+        .unwrap_or(OffsetDateTime::UNIX_EPOCH);


For large ms, this will default to 1970-01-01. That seems ... undesirable? Is there a more reasonable default?

Changed in 38176a9: the SQLite formatter now stamps an out-of-range value with the current time, as the PostgreSQL engine does, so the two engines agree and such a row is not dated 1970 and swept at the next retention pass. A SystemTime reading never reaches the branch (the range ends in the year 9999), so this is the fallback for a corrupt value only. A unit test passes i64::MAX and checks for the current time in the column's shape.

jcshepherd · 2026-10-28T23:23:08Z

+            let created_at = time::OffsetDateTime::from_unix_timestamp(
+                record.dynamodb.approximate_creation_date_time,
+            )
+            .unwrap_or_else(|_| time::OffsetDateTime::now_utc());


This is probably similar to what the SQLite engine should be doing.

Yes, and SQLite now does the same (38176a9). This comment also changed (f1708df): it said created_at is never a second clock read, while this fallback is one; it now names the case and that both engines stamp it with the current time.

jcshepherd

Some inconsistencies in the default/fallback logic for timestamps between the SQLite and PostgreSQL engines.

A stream record stored two creation times from two clock reads. The record body's ApproximateCreationDateTime was SystemTime::now() read when the record was built. The created_at column, which the 24-hour retention sweep filters on, was the column default: NOW() on PostgreSQL, the transaction start time, and strftime('now') at insert time on SQLite. MongoDB read DateTime::now() a second time in the insert. Cassandra read Utc::now() once for the HLC sequence number and again for created_at_ms. On PostgreSQL a write that waits inside its transaction (a row lock, a long TransactWriteItems) stored a created_at earlier than its own ApproximateCreationDateTime by the length of the wait, so the retention sweep deleted it before records with the same ApproximateCreationDateTime written in a short transaction. Each backend now takes one wall-clock read per record and derives both values from it. SQLite and PostgreSQL bind created_at explicitly in the transaction path and in the write_stream_record trait method; the column defaults stay for any other writer. SQLite writes the millisecond text shape the default produces (24 characters), so rows from before and after this change sort together as text. MongoDB uses one DateTime for the record and the document. Cassandra's HybridClock gains generate_with_created_at, which returns the sequence number and the reading it was derived from, so created_at_ms <= hlc_ms holds for every record (the clock clamps its physical component upward on a backwards step, never below the reading). Records written before the upgrade keep their default-clock created_at; the difference is at most the transaction length and ages out after 24 hours. Tests. crates/storage-postgres/tests/stream_record_clock.rs holds a row lock from an outside transaction while a PutItem with stream capture waits 2.1 s, then asserts created_at's second equals ApproximateCreationDateTime on every row, including a service-identity delete. On main the waited record stores created_at two seconds before its own timestamp (measured: 1792765145 against 1792765147); with this change both are 1792765147. The same read-back runs on SQLite (in-memory engine, in data/tx_helpers.rs), on Cassandra (tests/direct/streams.rs, which also asserts created_at_ms <= hlc_ms on every stored row), and on MongoDB (an ignored live test in backup_live_tests.rs, run with EXTENDDB_TEST_MONGODB_URI). HybridClock unit tests pin created_at_ms <= hlc_ms across 1,000 clamped reads and across the counter-overflow branch, and created_at_ms equal to the wall-clock read on the normal branch. Closes #459.

stream_record_clock skips when EXTENDDB_TEST_PG_CONNECTION_STRING is unset, and the storage-level step that sets the variable names its test targets one by one without this test, so in CI it finished in 0.00 s on the skip path. The step now names it. The test holds a row lock from an outside transaction while a PutItem with stream capture waits 2.1 s on it, then checks that every stored created_at second equals the record's ApproximateCreationDateTime. With the created_at bind removed from the stream insert, the column falls back to NOW(), the transaction start, and the test fails on that check.

The stream helper took created_at and ApproximateCreationDateTime from the ExtendDB host clock. The retention sweep deletes rows where created_at < NOW() - 24 hours, on the database clock. With the host behind the database by S seconds, every record was deleted S seconds early, and a host more than 24 hours behind lost each record on the next sweep. The helper now reads clock_timestamp() in the statement that already draws nextval('stream_seq'). Both values derive from that reading, so each record keeps one clock read and the write adds no round trip. The retention window is measured on the database clock again, as it was when created_at was the column default, without the default's transaction start offset. The PostgreSQL clock test also reads clock_timestamp() before the outside transaction releases the row and after the write returns. It asserts that the waited record's created_at lies between the two reads.

When the logical counter passes 999,999, HybridClock sleeps a millisecond and re-reads the wall clock for the sequence prefix. The branch assigned the raw re-read. If the clock stepped back during the sleep, the re-read sat below the call's own reading and the returned created_at_ms exceeded the prefix of its own sequence number. Under a forward clamp the re-read sat below the previous prefix and the sequence number went backwards by the clamp width, the regression reported in issue #465; this change fixes that too. The overflow branch now sets the prefix to the later of the re-read and the previous prefix plus one. The prefix stays above every earlier reading, and the sequence number sorts after every earlier one. The doc comment states what the method returns: the wall-clock reading taken by this call, with a sequence prefix equal to that reading or a later clamped value. The clock step moved into generate_from, which takes the reading and a re-read function as arguments. A unit test drives the overflow branch with a re-read ten milliseconds below the reading and asserts created_at_ms <= hlc_ms and that the sequence follows the previous one. With the raw re-read assigned again, the test fails on the first assertion.

created_at_never_exceeds_sequence_prefix checked only created_at_ms <= hlc_ms on the clamp branch, so a clock that returned the clamped prefix in place of its reading passed. Each clamped call now has a wall-clock read on either side of it, and the test asserts that the returned value lies between the two; with the clamped prefix returned, it fails on that assertion. created_at_holds_across_counter_overflow reached the overflow branch only when its two calls fell in the same millisecond. Otherwise the second call took the first branch and every assertion still held. The test now drives the step with injected readings and asserts that the re-read ran, that created_at_ms is the call's own reading, that the prefix is the re-read, and that the counter resets. With the overflow branch skipped, it fails on the re-read assertion.

The direct streams test's comment said its service-identity delete had the TTL worker's shape. That delete goes through delete_item and stream_record_statement, while the TTL worker writes through apply_ttl_delete_effects with the TtlStreamPlan it persisted. The comment now says the delete runs through delete_item, and that the sequence prefix is the reading or a later clamped value. The TTL sweep test now reads sequence_number and created_at beside record_data on the REMOVE record and asserts created_at_ms <= hlc_ms and that ApproximateCreationDateTime is created_at_ms / 1000, the properties the direct test checks for the other writes. A plan that stores a created_at_ms one second past its reading fails the first assertion. A plan that took a second clock read microseconds after the sequence number would fail it only when a millisecond boundary fell between the two reads.

Adds the unit test #465 asks for: the last prefix 5 s ahead of the reading and the re-read a further 5 s back. The prefix moves to the last prefix plus one, the sequence follows the previous one and created_at_ms stays at or below the prefix. Flooring the overflow prefix at the reading plus one fails this test and passes the other ten. Also reword the branch comment, which compared a millisecond prefix with sequence numbers.

format_epoch_millis fell back to 1970-01-01 for a millisecond value the time crate cannot represent, so such a row would have been swept at the next retention pass. The PostgreSQL engine stamps the same case with the current time; the SQLite engine now does the same, and the two engines agree on the fallback. A SystemTime reading never reaches this branch: the range ends in the year 9999. A unit test passes i64::MAX and checks that the result is the current time in the column's shape, not 1970.

The comment said created_at is never a second clock read, while the fallback for a timestamp outside the time crate's range reads the current time. The comment now names that case and that the SQLite engine stamps it the same way.

jcshepherd

Timestamp default issue addressed: approving.

LeeroyHannigan · 2026-11-03T09:45:49Z

@jcshepherd merged main to clear the one conflict, in .github/workflows/integration.yml: main now runs every PostgreSQL integration test with --tests, which includes this PR's stream_record_clock, so the merge takes main's line. The resolution touched no other file. The push dismissed your approval; could you approve it again once CI is green?

…unner #666 removed crate::schema::apply in favour of the migration runner, and the migrations on main seed no settings rows. After the merge of main: - The SQLite stream clock test in data/tx_helpers.rs called the removed function, so the crate's tests did not compile. It now calls crate::migrations::apply_for_tests, as the other SQLite unit tests do. - The SQLite and PostgreSQL stream clock tests pinned the control-plane delay to zero with an UPDATE that matched no row, so CreateTable left the SQLite table CREATING and the test failed with TableNotFound. Both now write the row, as the other tests on main do.

With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn, robinnsc and yesyayen as code owners October 23, 2026 15:03

robinnsc previously approved these changes Oct 28, 2026

View reviewed changes

LeeroyHannigan dismissed robinnsc’s stale review via 019f422 October 28, 2026 16:35

LeeroyHannigan force-pushed the fix/stream-record-single-clock branch 2 times, most recently from bbac4ed to 7ce6717 Compare October 28, 2026 18:00

jcshepherd reviewed Oct 28, 2026

View reviewed changes

jcshepherd requested changes Oct 28, 2026

View reviewed changes

LeeroyHannigan force-pushed the fix/stream-record-single-clock branch from f1708df to 7aebd15 Compare October 29, 2026 21:05

LeeroyHannigan added 9 commits October 30, 2026 09:11

LeeroyHannigan requested review from jcshepherd and robinnsc October 30, 2026 13:01

Merge branch 'main' into fix/stream-record-single-clock

b53aa2c

jcshepherd previously approved these changes Nov 2, 2026

View reviewed changes

navareamol added the v1 Needed for the 1.0 release label Nov 3, 2026

navareamol added this to the 1.0 milestone Nov 3, 2026

Merge branch 'main' into fix/stream-record-single-clock

30c19ee

LeeroyHannigan dismissed jcshepherd’s stale review via 30c19ee November 3, 2026 09:39

LeeroyHannigan requested review from eliseong, hitesh96db and navareamol as code owners November 3, 2026 09:40

This was referenced Nov 3, 2026

CI: the MongoDB catalog checks fail with "did not become primary" when the one rs.initiate call does not take effect #793

Open

ci: wait for mongod before initiating the replica set, and initiate again #799

Open

LeeroyHannigan merged commit ad89459 into main Nov 3, 2026
43 of 45 checks passed

LeeroyHannigan deleted the fix/stream-record-single-clock branch November 3, 2026 15:20

navareamol added the release-blocker Must ship for its milestone's release label Nov 3, 2026

robinnsc mentioned this pull request Nov 4, 2026

feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671

Merged

8 tasks done

Navigation Menu

Repository navigation

fix(streams): one clock read per stream record on all four backends - #463

fix(streams): one clock read per stream record on all four backends#463
LeeroyHannigan merged 12 commits into
mainfrom
fix/stream-record-single-clock

LeeroyHannigan commented Oct 23, 2026 •

edited

Loading

Uh oh!

robinnsc left a comment

Uh oh!

robinnsc Oct 28, 2026

Uh oh!

LeeroyHannigan Oct 29, 2026

Uh oh!

robinnsc Oct 28, 2026

Uh oh!

LeeroyHannigan Oct 29, 2026

Uh oh!

jcshepherd Oct 28, 2026

Uh oh!

LeeroyHannigan Oct 29, 2026

Uh oh!

jcshepherd Oct 28, 2026

Uh oh!

LeeroyHannigan Oct 29, 2026

Uh oh!

jcshepherd left a comment

Uh oh!

jcshepherd left a comment

Uh oh!

LeeroyHannigan commented Nov 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

LeeroyHannigan commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change

Tests

Not changed here

Uh oh!

robinnsc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LeeroyHannigan commented Oct 23, 2026 •

edited

Loading

LeeroyHannigan commented Nov 3, 2026 •

edited

Loading