storage-mongodb: CreateBackup answered TableNotFoundException for a table that DescribeTable reported ACTIVE #892

Three tests in the mongodb-backup-store job create a table and wait until DescribeTable reports it ACTIVE (wait_for_active). Then they call CreateBackup. In two runs, CreateBackup answered TableNotFoundException:

• job 127956530974, issue-580-cassandra-test-contact-point (test: let Cassandra suites use a configurable contact point #586), attempt 2, 6 November 13:18 UTC: tests/test_backup_restore_fidelity.py::test_delete_backup_answers_the_full_description_deleted
• job 127932105408, fix/865-gsi-recovery-account-scope (fix(storage-cassandra): recover a GSI backfill only in its owner's keyspace #868), attempt 1, 6 November 13:57 UTC: tests/test_backup_restore_fidelity.py::test_describe_backup_reports_provisioned_throughput_and_sse and tests/test_abac.py::TestBackupResourceTags::test_tagging_a_backup_is_refused_and_grants_nothing

botocore.errorfactory.TableNotFoundException: An error occurred (TableNotFoundException) when calling the CreateBackup operation: Table not found: restore-featp-e90e6e9ec9

Neither PR changes MongoDB code. Both failures came in pull request runs after #700 merged into main at 12:40 UTC, and main's own run on that commit passed. From 5 November 00:00 UTC to the merge, the job finished 24 times, and no run failed this way. The rerun of the second job passed.

The CreateBackup job that #700 added opens the table's snapshot (crates/engine/src/backup/job.rs:252). On MongoDB, open_table_snapshot_impl (crates/storage-mongodb/src/snapshot_engine.rs) starts a new snapshot session. It reads the tables document with ReadConcern::snapshot(). A missing document gives StorageError::TableNotFound, and a status other than ACTIVE gives StorageError::TableNotActive. create_backup_storage_err_to_dynamo (crates/engine/src/backup.rs) answers TableNotFoundException for both. DescribeTable reads outside that session.

The helper _create_backup in test_backup_restore_fidelity.py sends CreateBackup again after ContinuousBackupsUnavailableException or TableInUseException, and not after TableNotFoundException.

I have not reproduced the failure.