CI: the MongoDB catalog checks fail with "did not become primary" when the one rs.initiate call does not take effect #793

start_mongo in devtools/catalog_checks.py starts mongo:7 with --replSet rs0, waits for ping to answer, calls rs.initiate(...) once without reading its result, then waits 60 s for db.hello().isWritablePrimary. When that one initiate does not take effect, nothing initiates the set again, the member never becomes primary, and the check reports <container> did not become primary.

Seen

The mongodb-catalog job failed this way three times on 2026-11-03, each on the step "Check the current schema" (devtools/check-mongodb-schema), and each passed when the failed jobs were rerun:

• fix: read policy conditions and session tags in name order on all builds #783, run 41570456016.
• fix(streams): one clock read per stream record on all four backends #463, run 41616446928.
• fix(mongodb): retry an unconditional PutItem that loses the create race #431, run 41627572787.

In the #463 failure, the mongod log tail that the report prints ends with an FTDC warning: Unable to retrieve storageStats in $collStats stage :: caused by :: Collection [local.oplog.rs] not found. A successful initiate creates local.oplog.rs, so no initiate had taken effect when the 60 s ran out.

Suggested fix

In the wait loop, when db.hello().isWritablePrimary is not true, read rs.status(). If it fails with NotYetInitialized (code 94), call rs.initiate(...) again. Keep the initiate's output and print it in the failure report, so a later failure shows why the initiate did not take effect.