[LeeroyHannigan]

What

• recover_stale_backfills (crates/storage-cassandra/src/workers.rs) skips a CREATING index whose base table belongs to another account. It skips the index before it parses the index definition. Before, each pass over an account keyspace recovered the stale indexes of every account. For another account's index, the pass dropped and created the index data table in its own keyspace and took a propagation hold there. Then it started a backfill. The backfill failed, and the table stayed.
• poll_gsi_backfill_recovery takes the keyspace to scan as only_keyspace: Option<String>. serve passes None and scans every account keyspace, as before.
• spawn_gsi_backfill_recovery_for_tests takes an account and scans only that account's keyspace. The two tests that start a recovery worker pass their own account.
• recover_stale_gsi_backfills_for_tests runs one pass over one account's keyspace. It returns when the backfills it started end.
• The new test index::recovery_for_one_account_leaves_the_indexes_of_other_accounts_alone adds a GSI to one account's table while no worker runs. Then it checks the index after each of these steps:
  • A worker started for another account passes every second. After 3 s, the index is still CREATING, with no heartbeat and the same data table.
  • A pass over the other account's keyspace creates no data table there.
  • A pass over the owner's keyspace makes the index ACTIVE, with its one row.
• status_index_creation builds the UpdateTable input that adds StatusIndex. It replaces three copies of that input.
• test_ttl_enable_rejects_asynchronous_gsi checks the table name, asynchronously propagated global secondary index and index StatusIndex propagates after 1000 ms in the refusal. Before, it looked for a wording that fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 replaced.
• In .github/workflows/integration-cassandra.yml, the TTL step of cassandra-storage no longer has continue-on-error: true. A failed TTL test now fails the job. The direct step runs after a failed TTL step (if: ${{ !cancelled() }}), so a run reports both suites.

Why

cassandra-integration is not a required check on main. Before it can be one, its jobs must pass on changes that break nothing.

The last 200 runs of the workflow ran from 3 November 13:53 to 5 November 16:43 UTC. In them, index::a_backfill_the_recovery_worker_started_ends_before_the_worker_returns failed 14 of the 175 cassandra-storage jobs. Each time it reported index rows 2 s after the worker returned: 20000 (ACTIVE). In the same binary, test_a_delete_removes_the_index_row_a_backfill_wrote starts a second recovery worker, and that worker scanned every account keyspace. Sometimes its pass reached the first test's index before that index had a heartbeat. Then it ran the backfill under its own token, which the first test never fires. The backfill completed and marked the index ACTIVE. #865 has the measurements, and it lists the tables that the scan created in other accounts' keyspaces.

The TTL step ignored the failure of a test that fails on every run (#566). It was the only failed TTL test in 130 sampled cassandra-storage jobs, between 24 October and 5 November.

Closes #865
Closes #566

Testing done

Against a cassandra:4.1 node of mine, with the heap sizes of the CI service container:

• On main at 38992eb, with 8 account keyspaces and one new GSI, one recovery pass created the index data table in 3 keyspaces. These were the owner's keyspace and 2 others.
• A probe outside this change runs a second worker, with a 1 s scan interval, beside the steps of the stop test. On main, the index was ACTIVE 2 s after the probe's own worker returned, in 3 of 3 runs. With this change, it stayed CREATING in 3 of 3 runs.
• Each of these mutants makes a test fail:
  • The test entry point scans every keyspace. The new test fails with a worker started for another account recovered the index.
  • The pass has no account check. The new test fails with the pass over another account's keyspace created index_... there.
  • The backfill runs outside the token, or the pass starts it with tokio::spawn, outside the JoinSet and the token. These are the two mutants that d8fc8d7 names. For each, the stop test fails with 20000 (ACTIVE).
• The three recovery tests pass.
• On a fresh node, in the order of the CI job, the TTL binary passed 27 and ignored 1. Then the whole direct_integration binary passed 211 of 211, with --test-threads=4.
• Before the last changes to the tests, the index module passed 19 of 19 in each of 9 runs on one node, with --test-threads=4. The 10th run failed 9 tests with connection timeouts. The node ran out of heap (java.lang.OutOfMemoryError) after about 13 runs without a restart. On a fresh node, the whole direct_integration binary passed 211 of 211, with --test-threads=4.
• cargo +1.97.0 fmt --all -- --check: clean. cargo +1.97.0 clippy -p extenddb-storage-cassandra --all-targets --features test-hooks -- -D warnings: clean. Pedantic lints flag no line that this change adds.
• cargo +1.88.0 test --locked -p extenddb-storage-cassandra --features test-hooks --lib --tests --no-run: builds. cargo +1.88.0 test --locked -p extenddb-storage-cassandra --lib: 128 passed.

Not in this change:

• The tables that earlier passes created in other accounts' keyspaces stay, as Cassandra GSI backfill recovery creates index tables in other accounts' keyspaces #865 describes.
• One index can still get two recoveries in its owner's keyspace. Two servers on one cluster can both start one. So can one server, when a second new GSI wakes the worker before the first backfill writes a heartbeat.
• The GSI propagation workers that five index tests start read the queue of every keyspace. None of those tests failed in the 175 jobs.
• Each pass still reads the whole index catalog once per account keyspace.
• The CI service container gives Cassandra a 1 GB heap. At that heap, my node logged java.lang.OutOfMemoryError 4 times during each of the two whole-binary runs above, and no test failed.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.