Cassandra: two of four simultaneous restores of one backup were admitted #869

backup_engine::test_a_restore_is_refused_while_a_target_of_its_backup_is_creating (crates/storage-cassandra/tests/direct/backup_engine.rs) starts four restores of one backup at the same time. It expects one of them to be admitted and the other three to get BackupInUse. Two cassandra-storage jobs admitted two of the four:

• job 126252060053, push to main, 3 November 22:35 UTC
• job 126985540143, fix/435-query-key-size, 5 November 01:46 UTC

Both failed with exactly one is accepted: [...], left: 2, right: 1. In the first job, one of the refused restores got Backup is being used to restore table SimTarget1, and SimTarget1 was one of the two admitted.

register_restore_target in crates/storage-cassandra/src/backup_engine.rs inserts the target row first. Then it registers the restore on the backup with a lightweight transaction conditioned on active_restores = null. A refused registration calls registration_refusal, which clears each registration whose target restore_target_in_progress does not find CREATING with this backup as its source, and then tries again. Two admitted restores mean that the map was empty again between the two registrations. So a registration of a restore that was still running was cleared, or it was never written. I have not reproduced the failure, and I do not know which of the two happened.

Both runs are older than #671, which changed the backup code. The test did not fail in the jobs after #671 that I read, up to 5 November 16:43 UTC.