Repository navigation
storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414
Description
Observed
tests/test_put_item_create_race.py::test_racing_puts_on_an_lsi_table_leave_only_the_final_entry failed once in CI on the Cassandra backend and passed on re-run. Eight writers put the same new key at once, each with its own lk value. The base item ended as writer 0's (lk = l0), and a strongly consistent query on lk-index returned two rows, l0 and l6:
assert ['l0', 'l6'] == ['l0']
Run: https://github.com/ExtendDB/extenddb/actions/runs/39412650879/attempts/1 (job cassandra-pytest, line "tests/test_put_item_create_race.py:320"). Attempt 2 of the same run passed. The PR under test (#411) touches no storage code.
Reading of the code
In crates/storage-cassandra/src/data/update_item.rs, occ_write commits the base row with a lightweight transaction (IF version = ?), then records the secondary effects in a separate LOGGED BATCH: sync_indexes deletes the previous item's index row and inserts the new one. The comment at that point says the LWT commits first and the batch follows.
That gives two writers this interleaving:
- Writer 6 reads no item, its LWT applies at version 1.
- Writer 0 reads no item, its LWT fails, the retry loop re-reads writer 6's item and its LWT applies at version 2.
- Writer 0's batch runs: delete the
l6index row, insert thel0row. - Writer 6's batch runs: insert the
l6index row.
The base row is serialized by the LWT, the index rows are not. Step 4 arrives after step 3 and the l6 row comes back. The row carries writer 6's item image, so a query on the index returns an entry for an item that no longer exists in the base table.
Fix direction
The index mutation needs to carry the base row's ordering. Two options:
- Set the batch write timestamp from the base row version (a monotonic function of
version), so a later version's delete always beats an earlier version's insert under Cassandra's last-write-wins rule, whatever the arrival order. - Store the base version in the index row and condition the index insert on it.
The first keeps the current batch shape. The same window exists for GSI rows written synchronously through sync_indexes; the async GSI path in gsi_queue.rs is separate.
Reproduce
Run tests/test_put_item_create_race.py -k lsi_table against a Cassandra server. The test runs 15 rounds of 8 writers; the window is narrow, so it does not fire on every run.
Activity
fix(storage-cassandra): index rows follow the base item under every w…
jcshepherd commentedon Oct 29, 2026
Fix option 1 has a flaw. A timestamp built only from version would be near 1970. Other code paths write these rows with wall-clock time: the TTL-claimed path (mutation_timestamp = now), transactions, restore, and the GSI worker. A version-based timestamp would lose to all of them, or, if offset far ahead, beat them forever. A safer design:
pick timestamp T before the LWT,
store T in the base row as part of the LWT,
the next writer reads it and uses max(now, T_prev + 1),
the index batch runs USING TIMESTAMP T.
Fix option 2 doesn't work as written. Cassandra can't put an LWT in a batch with statements on other tables. An LWT per index row would also add a Paxos round for every index.
More code shares this path. put_item_if_not_exists has the same split. UpdateItem also calls occ_write (update_item.rs:189). I did not check DeleteItem or transactions.
The async GSI queue likely has the same bug. ready_at is set after the LWT with random jitter (jitter_delay_ms). So writer n's queue entry can be applied after writer n+1's. The GSI race test could flake the same way.
A related gap: if the process dies between the LWT and the batch, the new index row is never written. The old row stays. That deserves its own issue.
LeeroyHannigan commentedon Oct 31, 2026
The GSI test,
test_racing_puts_on_a_gsi_table_leave_only_the_final_entry, fails the same way in CI. It failed in three runs since 28 October, each on a PR that changes no Cassandra code, and passed on each rerun:1de4ce30, 28 October.stale GSI entries (want, got): {'k-...': (['g-k-...-1'], ['g-k-...-1', 'g-k-...-6'])}. Attempt 5 passed.fix(storage-cassandra): index rows follow the base item under every w…