[LeeroyHannigan]

What

tests/test_put_item_create_race.py, on the Cassandra backend only:

• test_racing_puts_on_an_lsi_table_leave_only_the_final_entry records each round whose LSI query returns the final item's row plus other writers' rows for the same key, runs all 15 rounds, and then calls pytest.xfail citing storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414. Any other mismatch fails at once, as before.
• test_racing_puts_on_a_gsi_table_leave_only_the_final_entry calls pytest.xfail citing storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414 when, after its 30 s wait, each key whose GSI entries are wrong has its final entry plus extra entries.
• XFAIL_STALE_INDEX_ROWS is true when an endpoint is set and configured_backend() is cassandra. devtools/run-tests --backend cassandra exports EXTENDDB_TEST_BACKEND.

Why

On Cassandra, each put commits the base row with a lightweight transaction and then writes its index rows in a separate batch. When puts to the same new key race, the row of a replaced item can be written after the put that replaced it deleted it (#414). Both tests then fail on PRs that change no Cassandra code, and pass on rerun:

• LSI: run 39412650879 (['l0', 'l6'] == ['l0']), and fix: validate secondary-index keys on UpdateItem and transaction writes the way the service does #416 in run 41633014419 (rows l0 and l4 for one item).
• GSI: fix(mongodb): retry a write conflict on the stream sequence counter #475, fix(auth): match IAM wildcards in ArnEquals and ArnNotEquals #471 (run 40732321029) and fix(storage): plan an LSI query as an index walk on PostgreSQL and SQLite #432 (run 41046200827), listed on storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414.

#424 (Closes #414) is open. Until it merges, any PR's Cassandra job can fail on these two tests. The comment above XFAIL_STALE_INDEX_ROWS lists what to delete once #414 is fixed.

The file's XFAIL_UNTIL_MONGODB_FIX decorator uses raises=AssertionError. On Cassandra that would also hide a failed put, a broken chain of old images and a missing index entry, so the check is made in the test body and those still fail.

Refs #414

Testing done

• pytest --collect-only tests/test_put_item_create_race.py collects the module's 9 tests.
• Both tests, imported into a scratch module, ran against an in-memory client in seven modes, with the backend set to cassandra, set to sqlite, and with no endpoint:
  • cassandra: a clean index passed. Stale rows xfailed both tests, and the LSI test ran all 15 rounds. A failed put, a missing final row, a row of another key and a duplicated final row each failed. Stale rows followed by a failed put in round 5 xfailed the GSI test and failed the LSI test.
  • sqlite, and no endpoint: a clean index passed, and every mode that changes an index's contents or fails a put failed.
• Not run against a Cassandra server here; the Cassandra CI job runs both tests on this branch.
• No Rust code changed; cargo test --workspace and cargo fmt --check were not run.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)

[jcshepherd]

This may be a work-around for the problem that #424 addresses directly. Recommend holding off on this until PR 424 is merged.

[jcshepherd]

Closing: #424 (merged as eaf16e8) fixes #414 directly, so the xfail is no longer needed.

Evidence:

• Local runs of test_racing_puts_on_an_lsi_table_leave_only_the_final_entry and test_racing_puts_on_a_gsi_table_leave_only_the_final_entry, against a single-node Cassandra 4.1 with a fresh catalog per server:

  Server built from	Runs	Both pass	GSI test fails
  25446c6 (main just before fix(storage-cassandra): index rows follow the base item under every write path #424)	20	2	18
  25ee109 (main now)	45	45	0

  Every pre-fix(storage-cassandra): index rows follow the base item under every write path #424 failure exhibited the race condition described in storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414: the final item's GSI row plus another writer's row for the same key.

• CI on main: both tests have passed in cassandra-pytest on all 7 Cassandra runs since the merge. The last failure on main was run 41732179141, just before fix(storage-cassandra): index rows follow the base item under every write path #424 landed.

• Why: index rows now carry the item version as their write timestamp, and the version only increases. A later put's delete of a replaced row always wins, whatever order the batches arrive in.

Merging this now would result in a regression to #414's race condition not being surfaces as a CI failure.