fix(storage-cassandra): index rows follow the base item under every write path by LeeroyHannigan · Pull Request #424 · ExtendDB/extenddb

LeeroyHannigan · 2026-10-21T15:15:26Z

Closes #414.

What was wrong

On Cassandra, every write commits the base row with a lightweight transaction and then writes the item's secondary-index rows in a separate logged batch, with the batch timestamps left to the coordinator. Measured against Amazon DynamoDB and reproduced on a main server, that produced four defects:

The race in storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414. Two writers to one item commit their base rows in Paxos order, but their index batches can reach the index table in the other order, and the later version's delete of the earlier version's row is undone when the earlier batch lands. 14 of 40 delete-versus-put rounds on an LSI table left the index disagreeing with the base table; the service left none.
Any write that keeps the index key removed the item from every index: UpdateItem on a non-key attribute, PutItem with the same keys, and a transactional Update each emitted a DELETE and an INSERT of one index row in one batch, both at the batch timestamp, and the tombstone won. Every such write did this on both LSIs and GSIs.
An item created by TransactWriteItems could not be updated, overwritten or conditionally deleted afterwards. The commit's version = version + 1 left a PREPARE-created row's version null, and the ordinary write paths fenced a null as 0.
A transactional Delete left the item's index rows in place, and a transactional Put over an existing item left the rows of the image it replaced.

What changes

crates/storage-cassandra only.

version becomes max(previous + 1, now) in microseconds since the epoch, bound as a value in every statement that stores it. It is still the OCC token; a row whose version is null fences on IF version = null.
Every index row a base-row mutation writes carries that version as its write timestamp: the post-LWT batch of put_item and update_item, the secondary-effects batch of delete_item, the transaction commit's batch, and the gsi_pending entry an asynchronous GSI queues (the worker applies the stamp). A later version's index mutation beats an earlier version's whatever order the batches arrive in.
An index row whose primary key is unchanged between the old and new image is rewritten by the INSERT alone (same_index_row), in the sync path and in the GSI worker.
Ordinary deletes without TTL go through the IF version = ? fence and retry a lost race from a fresh read. Their index tombstones carry the deleted row's version + 1, taken before the fence (the clock for a null or counter version), so a create that commits after the delete keeps its index rows. A delete whose read finds no item confirms the absence at LOCAL_SERIAL: it retries on an item stored before the delete began, and otherwise writes nothing.
Index statements carry the version with USING TIMESTAMP; the queue rows and stream records in the same batch keep Cassandra's timestamp. GSI backfill stamps each row with the base row's version. A write that finds a version more than one second ahead of its host's clock logs a warning, and a row at the largest bigint version is refused.
The transaction commit reads the existing image and version under its PREPARE fence at LOCAL_SERIAL, binds a concrete version, and syncs index rows for Put, Update and Delete. COMMITTING recovery syncs index rows from the ledger's two images for rows whose commit fence it applied itself, and for a Put or Update whose live commit landed and whose row still holds the ledger's image.

docs/adr/0027-cassandra-index-row-version-stamps.md records the decision and its consequences; ADR-0020 is amended where it described the version value and the commit statement.

Verification

tests/test_index_rows_track_base_item.py: 13 dual-target cases (writes that keep the index key; transactional Put, Update and Delete; ordinary writes on a transaction-created item; a delete racing a put; a DeleteItem and a transactional Delete racing a conditional create of the same index row). 13 of 13 pass on Cassandra. The 11 earlier cases pass on the service; the two race cases added in review have not been run against it yet.
New direct integration tests: the same-row rule emits one statement and keeps the row; version stamps order index mutations under reversed arrival; the async worker keeps the row across a same-key update; every write path stamps its index row with the stored version; COMMITTING recovery leaves a row a later writer owns alone and repairs a row it commits. Added in review: a delete's tombstone one above the deleted version leaves an index row stamped version + 2 in place, for DeleteItem and a transactional Delete; a delete removes the GSI row a backfill wrote; a queue entry written over a version an hour ahead of the clock is drained; the GSI worker applies two entries for one item in reverse version order; recovery repairs the index rows of a commit that landed before the crash. Each of the tests added in review fails with its fix reverted. The index, delete_item, put_get_item, transaction_ledger and transact_write_items direct tests pass, 60 of 60, on Cassandra 4.1.
Before the merge of main, on ca032d4: 139 direct and 27 TTL integration tests passed, 1,053 passed in the backend-agnostic suite against a Cassandra server, and negative controls failed their tests for the same-row rule in the sync path and in the worker, the strictly increasing version, the batch stamp at occ_write, at the IF NOT EXISTS create and at the transaction commit, and the recovery gate.
fmt, clippy 1.97 -D warnings and the crate's 119 unit tests are clean, and the tests build with the MSRV toolchain, 1.88. No dependency change, so the license notices are unchanged.

Consequences

An unconditional delete on a table without TTL now costs a Paxos round, and on a hot key it can answer TransactionConflictException after nine attempts.
Rows written before this change carry a counter or a null version and move into the microsecond domain on their next write. Index rows written before this change carry coordinator timestamps, which the next write's stamp exceeds.
During a rolling upgrade, a node still on the old binary accepts the new versions but keeps dropping the index rows of the items it writes (defect 2); the next write from an upgraded node restores them.
The ADR records the cases that remain: a create that read its clock before another writer created the item can still land under a delete's tombstones when three writers race; the ExtendDB hosts' clocks must agree; a transactional Delete whose base delete landed before a crash is not repaired by recovery; and the TTL-claimed paths keep their pinned wall-clock batch timestamp because the claim cells must not land above a later claim's Paxos timestamp.
The asynchronous GSI workers still stop at startup under serve (storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407); the worker path here is exercised by the crate's direct integration tests, which hold the worker guard.
The sort-key and hash-only delete arms carry the same fence logic twice, and the stamp is threaded as a positional argument through the index helpers; both are candidates for a follow-up refactor and are left as they are here.

Other issues this fixes

Fixes Cassandra: TransactWriteItems Delete removes no index rows and a key-changing Put leaves the old index row #568: the transactional Put now syncs index rows from the image it replaces, and the transactional Delete removes the item's index rows (commit_single_operation).
Part of Cassandra: a transaction completed by the recovery worker writes no index rows and no stream record #579: recovery of a COMMITTING transaction now syncs index rows from the ledger's two images. Recovery still writes no stream record, which stays open in Cassandra: a transaction completed by the recovery worker writes no index rows and no stream record #579.

…rite path Closes #414. Every Cassandra write commits the base row with a lightweight transaction and writes the item's secondary-index rows in a separate logged batch. The batch timestamps were left to the coordinator, which produced four defects, each measured against Amazon DynamoDB and reproduced on a main server: 1. Two writers to one item commit base rows in Paxos order but their index batches arrive in the other order, and the later version's delete of the earlier version's index row is undone when the earlier batch lands. 14 of 40 delete-versus-put rounds on an LSI table left the index disagreeing with the base table; the service left none. This is the race issue #414 reports. 2. A write that keeps the index key (UpdateItem on a non-key attribute, PutItem with the same keys, transactional Update) emitted a DELETE and an INSERT of one index row in one batch. Both carried the batch timestamp, the tombstone won, and the item left every index. Every such write did this. 3. An item created by TransactWriteItems could not be updated, overwritten or conditionally deleted afterwards: the commit's `version = version + 1` left a PREPARE-created row's version null, and the ordinary write paths fenced a null as 0. 4. A transactional Delete left the item's index rows in place; a transactional Put over an existing item left the rows of the image it replaced. The `version` column becomes max(previous + 1, now) in microseconds since the epoch, bound as a value in every statement that stores it, and every index batch a base-row mutation produces is stamped with it: the post-LWT batch of put_item and update_item, the secondary-effects batch of delete_item, the transaction commit's index batch, and the gsi_pending entry an asynchronous GSI queues, whose context carries the stamp for the worker. A later version's index mutation therefore beats an earlier version's whatever order the batches arrive in. An index row whose primary key is unchanged between the old and new image is rewritten by the INSERT alone. Ordinary deletes without TTL go through the same `IF version = ?` fence as writes and retry a lost race from a fresh read. The transaction commit reads the existing image and version under its PREPARE fence, binds a concrete version, and syncs index rows for Put, Update and Delete. Recovery of a COMMITTING transaction syncs index rows from the ledger's two images, only for rows whose commit fence it applied itself, stamped with the version it stored. A row whose version is null fences on `IF version = null`. No storage backend other than Cassandra changes. Verification: - tests/test_index_rows_track_base_item.py: 11 dual-target cases covering writes that keep the index key, transactional Put, Update and Delete, ordinary writes on a transaction-created item, and a delete racing a put over 25 rounds. 11 of 11 pass on the service and on Cassandra; 9 of 11 fail on main. - Direct integration tests: the same-row rule emits one statement and keeps the row; version stamps order index mutations under reversed arrival; the async worker keeps the row across a same-key update; every write path stamps its index row with the stored version; COMMITTING recovery leaves a row a later writer owns alone and repairs a row it commits. 139 direct and 27 TTL integration tests pass. - Negative controls, each failing its test with the guard removed: same-row rule (sync path and worker), strictly increasing version, the batch stamp at occ_write, at the IF NOT EXISTS create and at the transaction commit, the recovery gate. - Regression against a Cassandra server built from this commit: 1,053 passed in the backend-agnostic suite; every failure is in the suites run-tests runs separately (tests/python, the CLI lifecycle files, the PostgreSQL-only GSI queue file). - Six unit tests on the version rule. fmt, clippy 1.97 -D warnings, MSRV 1.88, and the crate's 58 unit tests are clean. No dependency change. Docs: ADR-0023 records the decision and its consequences (an unconditional delete now costs a Paxos round; rows written before this change move into the microsecond domain on their next write; during a rolling upgrade a node on the old binary still drops the index rows of items it writes until an upgraded node writes them again; the asynchronous GSI workers still stop at startup under serve, issue #407). ADR-0020 is amended where it described the version value and the commit statement.

LeeroyHannigan · 2026-10-26T13:02:25Z

Rebased on main at abf8082; conflicts in crates/storage-cassandra/tests/direct/index.rs and docs/adr/README.md, resolved by keeping both sides' tests and README rows.

main already has ADR 0022 (0022-trust-policy-deny-over-allow.md), so this branch's ADR is now 0023-cassandra-index-row-version-stamps.md. Its title, the README row, the three references in ADR-0020 and the commit message say 0023.

Cassandra lib tests (67) and the direct_integration index and transaction_ledger tests (21, including main's TTL test with an asynchronous GSI) pass against Cassandra 4.1.

robinnsc

This fixes most of what it sets out to on a local Cassandra 4.1, with sync GSIs because async workers don't run under serve (#407). We compared the base table with every index after each round of racing writers. A mixed workload (put, update, delete, conditional, TWI, batch) went from 43/60 bad rounds on main to 0/60. Writes that keep the index key went from 53/60 to 0/60. TWI-created rows no longer wedge: main gave 183 TransactionConflictException, head 0. Rows written by main (counter and null versions) stayed consistent once head wrote them. Put and update latency is unchanged.

The one I'd hold this for is the first inline comment: a delete racing a create of the same key and index key now loses the new item from every index, which main didn't do.

Merge notes for #576 (19 hunks, all in transactions.rs). Two of them go wrong without a conflict marker:

Put arm: main passes None as the index old image, so taking main's side quietly drops the #568 Put fix.
Delete arm: main passes &[] for indexes. Passing &indexes isn't enough on its own, because commit_secondary_effects skips the index sync when new_item is None, so that guard has to go too. Its doc comment saying a delete writes no index rows then needs updating.
Main's created_to_prepare = false has to stay in the COMMIT and recovery statements alongside the bound version = ?. Main's fetch_row_for_transaction and this branch's version read can become one read that feeds both the stream old image and the index old image. Recovery's call to commit_sync_indexes becomes commit_secondary_effects with no stream statement.

Other notes:

An unconditional DeleteItem now goes through the fence and its retry loop: one more LWT (+10-20% p50 here on a single node, likely more on a cluster), a p99 under 16-writer contention about 1.5-2x main's, and it can surface TransactionConflict after 9 attempts on a hot key, where main's plain DELETE never failed. Probably worth a line in the ADR's Consequences.
Index batches, gsi_pending inserts and the stream records in the same batch now carry the ExtendDB host's clock. The worker's gsi_pending delete, GSI backfill and TTL tombstones still use Cassandra's. Under host skew that's where things drift (more in the version.rs comment).
Pre-existing, seen on both builds: rows committed by binaries older than #576 keep created_to_prepare = true, and a later cancelled transaction's rollback deletes the committed item and leaves its GSI row. Also, a PREPARE racing a delete can leave a row holding only prepared_txn_id, after which the key answers TransactionConflict forever and Query on its partition answers 500. And Query answers 500 on a partition whose items were all deleted. Probably separate issues.
The description still names ADR 0022, and ADR-0020's retry section still says 20 retries / 16 ms while the shared loop has 8.

robinnsc · 2026-10-31T08:48:22Z

+                        Some(&old_item),
+                        stream,
+                        sys_delay,
+                        super::version::next_row_version(version),


This stamp is read after the fence LWT returns, but a create of the same key reads its clock before its own LWT (put_get_item.rs:481 before IF item_data = null, and update_item.rs:339). Under Paxos contention the create can apply after this delete with a smaller stamp, and its index insert lands under the tombstone. The item is then in the base table and in no LSI/GSI until its next write. One server, no skew:

Delete racing an attribute_not_exists(pk) create with the same index key: 93/200 shadowed on head, 0/200 on main.

The same through a TWI Delete (transactions.rs:1601): 93/200 vs 0/200.

Moving next_row_version(version) before the fence wasn't enough in a test build: still 8/300, and 73/200 with TWI Delete. What did close the two-writer cases was stamping the tombstone previous + 1 with no clock, taken before the fence, here, at :573 and at transactions.rs:1601: 0/300, 0/200, 0/200. Rows with a null or pre-PR counter version would need to keep the clock, since their index rows carry coordinator time. A three-writer interleaving still got through at about 3% (create reads clock, another writer creates, this deletes, the first create's LWT lands). So the ADR may need to name that case, or the delete could leave a version the create fences on. Might also be worth adding this shape as a test.

Fixed in f776100, as you suggested. The tombstones now carry the deleted row's version + 1, taken before the fence (version::delete_stamp), at both delete arms, at the transactional Delete's commit and in its recovery. A row with a null version, or a counter below 10^15, keeps the clock. GSI backfill wrote its rows at coordinator time, which a version + 1 tombstone would not remove, so the backfill now stamps each row with the base row's version.

Your repro on one Cassandra 4.1 node, with the create resent until it lands for the TransactWriteItems shape:

DeleteItem against an attribute_not_exists(pk) create with the same key and index key: 71/200 shadowed on 12e68a3 (ca032d4 merged with main; all 71 creates that landed after the delete), 0/200 on the fix (47 creates landed after the delete).

TransactWriteItems Delete: 77/200 on 12e68a3 (ca032d4 merged with main), 0/200 on the fix (200 creates landed after the delete each time).

A conditional create, an unconditional create and a delete released together on an absent key, three runs of 300 rounds: 22/900 on 12e68a3 and 22/900 on the fix. This is the three-writer case you describe.

ADR-0027 names the three-writer case and says that closing it needs the delete to leave a version the next create fences on. The new direct test test_a_delete_stamps_its_tombstones_one_above_the_deleted_version writes an index row at version + 2 after a DeleteItem and after a transactional Delete, and the row survives. With the stamp read from the clock, the test fails. The dual-target test for your shape is in tests/test_index_rows_track_base_item.py.

robinnsc · 2026-10-31T08:48:22Z

+  restores them.
+- The index rows a later creator writes after a delete are ordered against the
+  delete's tombstones by the clock, as they were before: the deleted row leaves
+  no version behind. A clock that runs backwards between the delete and the


With concurrency this happens without a clock running backwards or a same-microsecond create (see delete_item.rs:323). On main both batches had coordinator timestamps taken after their LWTs, and the same shape shadowed 0/200. The 200 back-to-back iterations were sequential, so they couldn't reach it. Worth restating once the stamp ordering is settled.

Restated in 4dc647e. The paragraph now explains the race under concurrency: the create reads its clock before its LWT, and the delete used to read its clock after the fence. It gives the measured rates before and after the version + 1 stamp, and the three-writer case that remains goes under Consequences. The ADR is 0027 on this branch, since main has taken 0022 to 0026.

robinnsc · 2026-10-31T08:48:22Z

+                // and any concurrent write to the key commit in one order and the
+                // index rows written afterwards carry that order (`data::version`).
+                // Deleting an absent item writes nothing.
+                let Some(old_item) = old_item_opt else {


On a multi-node cluster this changes what an unconditional delete does when its read misses. The read above uses the driver default consistency, ONE. On main the DELETE was sent whatever the read returned. Here, if a replica that hasn't got the latest write answers "absent", the delete returns 200 and writes nothing, so the item survives. The fence catches a stale image but not a stale absence. Could consider reading at LOCAL_QUORUM here, or keeping a blind tombstone when the read finds nothing. I couldn't reproduce it live (init fails on a fresh 3-node cluster on both builds), so this is from the code.

Changed in 2ff273b and 3abeca9. When the read finds no item, the unconditional delete now reads the row again at LOCAL_SERIAL before it answers without writing. If that read finds an item that a write stored before the delete began, the delete returns a lost race and the retry loop starts over from a fresh read. An item whose version was read from the clock after the delete began belongs to a concurrent write, and the delete is ordered before it. Retrying into those raised the three-writer case from 22/900 to 124/900 on one node. A delete of an item that is present takes the same path as before, with no extra read. I could not reproduce the stale absence on one node either, so this change also follows from the code. The direct delete_item tests pass on Cassandra 4.1.

robinnsc · 2026-10-31T08:48:22Z

+                // and version read here are the ones the commit replaces. The
+                // old image is what tells the index sync which rows to remove.
+                let (existing, version) = self
+                    .fetch_item_and_version_for_transaction(key_info, item, Some(txn_id))


This read (select_by_pk, default consistency ONE) supplies both the old image the index sync removes and the version that's bound, and the commit's IF prepared_txn_id = ? doesn't check either. At RF ≥ 3, a lagging replica would give an older image, and the real old index row would be left behind. Since the row is under this transaction's fence, LOCAL_SERIAL here (and at :1474, :1504, read_version_by_pk) should make it exact.

Changed in 2ff273b. The commit's reads of its Put, Update and Delete rows (fetch_row_for_transaction, called with LOCAL_SERIAL from commit_single_operation) and read_version_by_pk now run at LOCAL_SERIAL. The PREPARE-phase reads keep the default consistency. The transaction_ledger and transact_write_items direct tests pass on Cassandra 4.1. ADR-0027 records the extra Paxos read per operation as a consequence.

robinnsc · 2026-10-31T08:48:22Z

-                Ok(())
+                // The fence does not apply when the row was already committed
+                // (idempotent replay) or a later writer owns it; either way
+                // this pass did not write the row and reports no stamp.


If the live commit's LWT landed and its index batch didn't, because of a crash or an error from commit_sync_indexes that leaves the ledger COMMITTING, this fence never applies on recovery, so that op's index rows are never repaired. The writer it defers to is the one that crashed. A row whose last_committed_txn_timestamp is this transaction's and whose item_data equals the ledger image could be re-synced at its current version; a later writer's stamp would still win. Otherwise the ADR's "recovery syncs index rows" covers only ops the crash reached before their LWT.

Changed in c6e1b65, along the lines you suggested. When the recovery fence does not apply to a Put or Update, recovery reads the row at LOCAL_SERIAL. If the row has no fence, its last_committed_txn_timestamp is this transaction's and its item_data parses to the ledger's committed image, recovery re-syncs the index rows from the ledger's two images at the row's current version. Any other row is left alone. The new test test_committing_recovery_repairs_the_index_rows_of_a_commit_that_landed_before_the_crash sets up a row that holds the committed image while the GSI still has the pre-commit key. Recovery replaces that row with the committed key's row. With the image check forced false, the test fails.

A transactional Delete whose base delete landed before the crash is still not repaired. The row is gone, so recovery cannot read the version its tombstones have to exceed. ADR-0027 now says so, in place of "recovery syncs index rows".

robinnsc · 2026-10-31T08:48:22Z

+/// callers that pinned a timestamp earlier in the same request.
+pub fn next_row_version_at(previous: Option<i64>, now: i64) -> i64 {
+    match previous {
+        Some(previous) => previous.saturating_add(1).max(now),


now is now the ExtendDB host's clock, and previous + 1 carries a future version forward, so one write from a fast host keeps every later stamp of that row ahead. Then:

a re-create from a host with a correct clock lands under the delete's tombstone;

the gsi_pending insert in the same stamped batch outlives the worker's coordinator-time delete (workers.rs:356), so the entry gets re-applied every poll once the workers run.

Could consider clamping to now + bound with a log line, keeping the queue and stream rows off the stamp, and stating a clock-sync requirement. Small one: at i64::MAX this saturates and the OCC token stops changing, so refusing the write there might be safer.

Changed in a31d45d. The index statements now carry the version as their own USING TIMESTAMP, and the stamped batches have no batch timestamp, so the gsi_pending rows and stream records beside them keep Cassandra's timestamp. The new test test_a_queue_entry_written_over_a_version_ahead_of_the_clock_is_drained sets a row's version an hour ahead, writes the item again with an async GSI, and waits for the worker to drain the queue. With the batch stamped as before, the entry is still in gsi_pending after 15 s.

I did not clamp the version. A version below previous + 1 would sort under the row's existing index rows and break the ordering for that row. A write that finds the version more than one second ahead of its host's clock now logs a warning, at most once a minute. ADR-0027 states that the ExtendDB hosts' clocks must agree, and names the re-create you describe: after a delete, a create from a host whose clock reads lower can sit under the tombstone until its clock passes the version.

At i64::MAX, next_row_version and delete_stamp now return an error, so the write is refused instead of storing the same version again.

robinnsc · 2026-10-31T08:48:22Z

+    // table even when the worker applies the entries in the other order.
    let mut batch = BatchQueryBuilder::new().with_consistency(Consistency::LocalQuorum);
+    if let Some(stamp) = context.index_stamp {
+        batch = batch.with_timestamp(stamp);


Nothing seems to observe this stamp: with this line replaced by let _ = stamp;, direct_integration still passed 155/155. A test that applies two entries for one item in reverse version order and checks which index row survives would pin it.

Added in a37aac6: test_the_gsi_worker_resolves_two_entries_for_one_item_in_version_order. The test queues two entries for one item (v1 inserts the GSI row for a, v2 moves it to b) and swaps their payloads between the two queue rows, so the worker applies v2 first and v1 last. It then checks that only b remains. With batch.with_timestamp(stamp) replaced by let _ = stamp, the test fails and a comes back.

robinnsc · 2026-10-31T08:48:22Z

+
+        def put():
+            try:
+                raw_client.put_item(TableName=indexed_table, Item=_item(pk, "1", "new", f"g-{pk}", "v2"))


This race moves lk from "old" to "new", so the delete and the put always touch different index rows, and the put is unconditional. That means it can't reach the same-row, attribute_not_exists create case in delete_item.rs:323. The two threads also share the module-scoped raw_client, and nothing asserts that both orders (put last, delete last) happened. A same-lk variant with a conditional create would cover the gap.

Changed in f776100. The two writers now each have their own non-retrying client, and odd and even rounds hold back the delete and the put in turn. The existing test now asserts that both orders occurred (put last, delete last). The new test_a_delete_racing_a_create_of_the_same_index_row_leaves_the_indexes_agreeing races DeleteItem, and a TransactWriteItems Delete, against an attribute_not_exists(pk) PutItem with the same lk and gk. It checks the LSI and the GSI against the base table in every round. A transaction answers later than a DeleteItem, so against it the create is resent until it succeeds or the delete has answered.

…leted version A delete stamped its index tombstones with next_row_version(version), read from the clock after its fence LWT returned. A create of the same key reads its clock before its own LWT (put_item_if_not_exists, and the create arm of occ_write), and under Paxos contention that LWT can apply after the delete's while its clock reading is the smaller one. The create's index rows then landed under the delete's tombstones, and the item was in the base table and in no index until its next write. On one node, a DeleteItem racing an attribute_not_exists(pk) create with the same key and index key left the item out of both indexes in 71 of 200 rounds. The tombstones now carry the deleted row's version + 1, taken before the fence (version::delete_stamp), in both arms of the single-item delete, in the transactional Delete's commit and in its COMMITTING recovery. Every index row of a row this scheme wrote is stamped with at most the row's version, so version + 1 removes all of them, and a create sent after the write that stored that version reads a larger clock value. A row whose version is null or a counter from before this scheme has index rows at coordinator time and keeps a clock stamp. GSI backfill wrote its rows at coordinator time, above the version of the item it copied, so a version + 1 tombstone would have left a backfilled row behind after the item was deleted. The backfill now stamps each row with the base row's version, per statement with USING TIMESTAMP, and leaves rows with a null or counter version at coordinator time. Tests: an index row stamped version + 2 survives the delete's tombstone, for DeleteItem and for a transactional Delete; a delete removes the GSI row a backfill wrote; unit tests for the stamp; and a dual-target race of a delete (DeleteItem and TransactWriteItems) with a conditional create of the same LSI and GSI rows, with one client per writer.

…rsion stamp The post-LWT batches of put_item, update_item and delete_item and the transaction commit's batch were stamped with the row's version as their batch timestamp, so the gsi_pending rows and stream records in them carried it too. The version is max(previous + 1, now) on the writing host's clock, and previous + 1 carries a version written from a host whose clock runs ahead forward for the rest of the row's history. A gsi_pending row stamped with such a version outlives the worker's delete of the applied entry, which carries Cassandra's timestamp, and the worker applies the entry again on every poll. The index statements now carry the version themselves (USING TIMESTAMP, sync_indexes_at), and the batch has no timestamp of its own, so the queue rows and stream records keep Cassandra's. The TTL-claimed paths keep their pinned batch timestamp and pass no stamp. A write that finds a version more than one second ahead of its host's clock now logs a warning, at most once a minute. A write or delete of a row whose version is the largest bigint is refused: saturating there stored the same version again, and the fence could no longer tell two writes apart. Tests: a queue entry written over a version an hour ahead of the clock is removed by the worker (it stayed in gsi_pending with the batch stamp), and unit tests for the refusal.

… LOCAL_SERIAL An unconditional DeleteItem on a table without TTL reads the row at the driver's default consistency, ONE, and answers without writing when the read finds no item. On a cluster, a replica that has not received the latest write reads as absent, so the delete answered 200 and the item survived. Before the fence, the plain DELETE was sent whatever the read returned. The fence catches a stale image (the version no longer matches) but not a stale absence. The delete now confirms an absence at LOCAL_SERIAL before it answers; a row found there is retried from a fresh read, as a lost race is. The transaction commit reads the row it holds under its PREPARE fence to get the old image whose index rows it removes and the version it advances, and its IF prepared_txn_id = ? checks neither. That read was at ONE too, so a lagging replica could hand it an older image, and the old image's real index rows were left behind. The commit's reads of its Put, Update and Delete rows, and recovery's version read, are now at LOCAL_SERIAL, which sees every committed LWT on the row. select_by_pk_at is select_by_pk with a consistency. This could not be reproduced on a single node, where every read is current; the change follows from the code.

…efore the crash COMMITTING recovery synced index rows only for an operation whose commit fence (IF prepared_txn_id = ?) it applied itself. When the live commit's LWT had landed and its index batch had not, because the process crashed between the two or commit_secondary_effects returned an error that left the ledger COMMITTING, the fence no longer applied on recovery and that operation's index rows were never repaired. For a Put or Update whose fence does not apply, recovery now reads the row at LOCAL_SERIAL. A row with no fence, this transaction's last_committed_txn_timestamp and the ledger's committed image is the live commit's, and recovery re-syncs its index rows from the ledger's two images at the row's current version, which is the stamp the lost batch carried. A row that a later writer has rewritten does not match and is left alone, and that writer's batch carries a higher stamp in any case. A transactional Delete whose base delete landed before the crash is not covered: the row is gone, so recovery cannot read the version its tombstones must exceed. Test: a row that holds the committed image, unfenced, with the GSI still on the pre-commit key, gets the committed key's GSI row from recovery.

…tem in version order Nothing observed the stamp the worker gives the index rows of a queued entry: with that line replaced by a no-op, the direct integration tests still passed. The new test queues two entries for one item, v1 inserting the GSI row for a and v2 moving it to b, swaps their payloads between the two queue rows so the worker applies v2 first, and checks that only b remains. Applied at Cassandra's timestamp, v1's late insert brings a back.

…tamps and recovery ADR-0027 now records that a delete's index tombstones carry the deleted row's version + 1, taken before the fence (the clock for a null or counter version), with the race measurements before and after; that a GSI backfill stamps its rows with the base row's version; that only index statements carry the version, while queue rows and stream records keep Cassandra's timestamp; the LOCAL_SERIAL reads of a delete's absence and of the commit's row; and recovery's re-sync of a Put or Update whose live commit landed. Consequences now cover the unconditional delete's extra Paxos round and its TransactionConflictException after nine attempts, the three-writer case that can still hide a create, the clock-sync requirement between ExtendDB hosts, the rolling-upgrade case that leaves index rows behind, and the transactional Delete that recovery cannot repair. ADR-0020's retry section described only UpdateItem's loop (20 retries, 16 ms window). PutItem and DeleteItem use the shared loop: 8 retries, 4 ms base, 256 ms window.

…er writes 2ff273b made an unconditional delete whose read finds no item read the row again at LOCAL_SERIAL and retry when that read finds one. A LOCAL_SERIAL read completes an LWT in progress on the partition before it answers, so it usually found the item a concurrent create had just written, and the retry deleted it. The tombstones carry the deleted item's version + 1, and a second create that read its clock before its own LWT then landed under them: the item was in the base table and in no index. On one Cassandra 4.1 node this raised the three-writer case (a conditional create, an unconditional create and a delete on an absent key) from 22 to 124 of 900 rounds. The delete now takes the clock before its first read, and the LOCAL_SERIAL read counts the row as present only when it holds an item whose version is below that reading. An item whose version was read after the delete began belongs to a write concurrent with the delete; the delete is ordered before it and answers without writing, as it did before 2ff273b. A null or counter version counts as stored before. An item a write stored and acknowledged before the delete started is still retried and deleted, so the stale absence on a lagging replica stays covered. On the same node: DeleteItem and a transactional Delete racing a conditional create shadow 0 of 200 rounds each, and the three-writer case 22 of 900, the rate before this PR. ADR-0027 now describes the rule and gives the three-writer rate.

LeeroyHannigan · 2026-11-03T12:57:51Z

Thanks for the review. I merged main into the branch (12e68a3) and addressed each point in its own commit.

Merge with #576: the Put arm keeps the #568 old image. The Delete arm passes the indexes, and the commit_secondary_effects guard on a missing new item is gone, with its doc comment updated. created_to_prepare = false stays beside the bound version in COMMIT and recovery, one read feeds both old images, and recovery calls commit_secondary_effects with no stream statement.

Points from the review body:

Delete racing a create of the same key and index key: fixed in f776100 (the inline reply has the numbers). The two-writer shapes went from 71/200 and 77/200 to 0/200.
The unconditional delete's cost: ADR-0027's Consequences now has your numbers (one more LWT, +10 to 20 percent p50, p99 at 1.5 to 2 times main's under 16 writers, TransactionConflictException after 9 attempts).
Host clock on index batches, gsi_pending and stream rows: a31d45d moves the version onto the index statements, so queue rows and stream records keep Cassandra's timestamp. GSI backfill now stamps with the base row's version (f776100). ADR-0027 states the clock-sync requirement. The worker's gsi_pending delete and TTL tombstones keep Cassandra's timestamp.
Pre-existing problems: cassandra: a cancelled TransactWriteItems deletes an item that an earlier transaction committed #659 covers rows committed before fix(cassandra): record stream records for transacted writes #576 keeping created_to_prepare = true; the GSI row the rollback leaves is one more effect of it. Cassandra: a single-item write inside a transaction's PREPARE window is accepted undetected, and a deleted item is recreated by the transaction #578 covers the PREPARE marker that recreates a deleted row; the TransactionConflict wedge and the 500 you saw follow from that marker row. The 500 from a Query on a partition whose items were all deleted reproduces on this branch and is filed as Cassandra: Query answers 500 on a partition whose items were all deleted #795.
ADR number and retry numbers: the PR body now names ADR-0027. ADR-0020 now gives both loops: UpdateItem's 20 retries in a 16 ms window, and the 8 retries in a window of up to 256 ms that PutItem and DeleteItem share (4dc647e).

Other commits: 2ff273b (LOCAL_SERIAL reads for a delete's absence and the commit's row; 3abeca9 limits the absence check to items stored before the delete began, which keeps the three-writer case at 22 of 900 rounds, the rate before this PR), c6e1b65 (recovery re-syncs a commit whose LWT landed), a37aac6 (a worker test that applies two entries in reverse version order).

Tests, all on a local Cassandra 4.1 node: fmt, clippy 1.97 with -D warnings and MSRV 1.88; 119 crate unit tests; the direct integration tests for index, delete_item, put_get_item, transaction_ledger and transact_write_items; 13 of 13 in tests/test_index_rows_track_base_item.py against a server built from this branch. Each new direct test fails with its fix reverted.

jcshepherd

Some minor documentation issues, but I'm not going to hold this up on those.

With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

… backfill test The test that main added in #424 builds an UpdateTableInput and a CreateGsiAction, which this branch extends with warm_throughput and sse_specification. Signed-off-by: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com>

With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 7; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window; once its own worker set is stopping (a lost serving registration) it spawns no resume either, a resume being a job that the workers' stop would not stop. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account <id> <table>`, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file; on PostgreSQL, MongoDB and Cassandra its serving registration is named `extenddb backup remove-target` and released when it exits, and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. A resumed Cassandra restore re-applies at most one batch through the put path, which since #424 rewrites an unchanged index row by its INSERT alone (#742) and stamps every index row with its base row's version; the re-apply test checks each GSI and LSI row's write time against its base row's version after the load and after each re-apply. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main, and its actions are pinned to the commits main pins. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 7 is declared breaking on PostgreSQL and does not apply on SQLite, MongoDB and Cassandra; design 4(e) and the reconciler's module doc say what a lost serving registration does: the reconciler stops with the server's workers, as `registration.rs` says, and the running jobs go on (#755); CONTRIBUTING's worked example becomes migration 8; the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Rebased onto main through PR 8: main merged #335, #666 and #681 under the stack, so `restore_progress` is migration 7 (5 is main's `drop_continuous_backups`, 6 PR 6's), declared not applicable in #681's MongoDB and Cassandra registries as on SQLite, and main's migration tests that count the ids count it; the reconciler's resumes meet #666's restartable worker set (above) and PR 8's job drain, which a restore job joins as a backup job does; `remove-target` releases its registration on exit and is named on every backend that registers; and Cassandra's `claim_restore` no longer calls the catalog check #681 removed. Moved with PRs 6 to 8 onto main at e8ca7df: #424 fixed #742 on main (an unchanged re-put keeps its GSI and LSI rows, through the put path and the GSI worker), so this PR's change to `sync_indexes` and `gsi_apply_index` and its two index tests are dropped, main's tests covering the case; and main's #463 stream-clock live test sits beside this PR's MongoDB restore tests. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn, robinnsc and yesyayen as code owners October 21, 2026 15:15

This was referenced Oct 24, 2026

fix(update): write nothing for a removal-only UpdateItem on a missing key #508

Merged

perf(storage-cassandra): restore copy writes plain INSERTs, not the LWT path #527

Open

LeeroyHannigan force-pushed the fix/cassandra-index-row-ordering branch from 7f349c9 to ca032d4 Compare October 26, 2026 13:02

LeeroyHannigan mentioned this pull request Oct 26, 2026

fix(cassandra): a Scan reads on past rows without an item #612

Open

robinnsc reviewed Oct 31, 2026

View reviewed changes

LeeroyHannigan mentioned this pull request Oct 31, 2026

fix(query): check legacy conditions against the key schema #507

Open

jcshepherd assigned jcshepherd and robinnsc and unassigned jcshepherd Nov 2, 2026

navareamol added the v1 Needed for the 1.0 release label Nov 3, 2026

navareamol added this to the 1.0 milestone Nov 3, 2026

LeeroyHannigan added 8 commits November 3, 2026 10:40

Merge branch 'main' into fix/cassandra-index-row-ordering

12e68a3

LeeroyHannigan requested review from eliseong, hitesh96db and navareamol as code owners November 3, 2026 12:57

LeeroyHannigan requested a review from robinnsc November 3, 2026 12:57

This was referenced Nov 3, 2026

test: tolerate the #414 stale index row in the Cassandra put race tests #801

Closed

Cassandra: unconditional PutItem on one key of a GSI table answers TransactionConflictException under contention #802

Open

navareamol added the release-blocker Must ship for its milestone's release label Nov 3, 2026

jcshepherd self-assigned this Nov 3, 2026

jcshepherd approved these changes Nov 3, 2026

View reviewed changes

jcshepherd enabled auto-merge November 3, 2026 22:32

jcshepherd added this pull request to the merge queue Nov 3, 2026

Merged via the queue into main with commit eaf16e8 Nov 3, 2026
47 of 51 checks passed

jcshepherd deleted the fix/cassandra-index-row-ordering branch November 3, 2026 23:33

robinnsc mentioned this pull request Nov 4, 2026

feat(container): publish a Cassandra-backed ExtendDB image #725

Merged

4 of 5 tasks

LeeroyHannigan mentioned this pull request Nov 4, 2026

fix(transact): write nothing when a removal-only update finds no item #513

Merged

LeeroyHannigan mentioned this pull request Nov 4, 2026

feat: store and report WarmThroughput on tables and global secondary indexes #474

Open

Conversation

LeeroyHannigan commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What changes

Verification

Consequences

Other issues this fixes

Uh oh!

LeeroyHannigan commented Oct 26, 2026

Uh oh!

robinnsc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LeeroyHannigan commented Oct 21, 2026 •

edited

Loading

LeeroyHannigan commented Nov 3, 2026 •

edited

Loading