MongoDB: a write conflict on the stream sequence counter answers 500, and BatchWriteItem can answer 500 after one of its puts committed #437

Summary

On the MongoDB backend, writes to a table with a stream enabled take a sequence number from a per-shard counter inside the write's transaction. When two transactions update that counter at the same time, MongoDB raises WriteConflict with the TransientTransactionError label. next_sequence_number_in_session maps every error to StorageError::Internal, so the transaction's retry loop never sees the transient error and the request answers HTTP 500. A BatchWriteItem that already committed one of its puts can answer 500 for the batch, and a later PutItem returns that committed item as its old image.

Measured

2026-10-21, MongoDB 7.0 single-node replica set, main (ec81e9d), stream enabled with NEW_AND_OLD_IMAGES, client retries off.

• PutItem and DeleteItem churn over 4 keys: 8 threads, 2,016 of each: 485 PutItem and 294 DeleteItem answered InternalServerError; 32 threads: 1,014 PutItem and 510 DeleteItem. Server log: 2,303 lines Error code 112 (WriteConflict): Caused by :: Write conflict during plan execution and yielding is disabled ... labels: {"TransientTransactionError"}.
• BatchWriteItem (the raced key plus a side key) racing PutItem, 16 writers x 30 rounds: 40 of 240 BatchWriteItem and 5 of 240 PutItem answered 500. 16 PutItem calls returned, as their ALL_OLD image, the item of a BatchWriteItem that had answered 500; 13 keys had a write chain that did not cover every write; the stream record count was off by 20. Server log in that window: 45 WriteConflict lines.
• The same churn on a table without a stream, after the fix in fix(mongodb): retry an unconditional PutItem that loses the create race #431: 44,800 puts, 0 errors. The counter is the remaining source.

The service, same shapes on a stream-enabled table (32 writers x 63 rounds, one key): 2,016 of 2,016 PutItem 200; PutItem racing DeleteItem and UpdateItem: every call 200; BatchWriteItem racing PutItem on hash and GSI tables: 2,016 of 2,016 with no UnprocessedItems.

Where

crates/storage-mongodb/src/stream_engine.rs, next_sequence_number_in_session: the find_one_and_update on the shard counter maps its error to StorageError::Internal. Classifying a WriteConflict (code 112) or any error carrying the TransientTransactionError label as the transaction path's transient error lets the existing retry loop with backoff re-run the transaction, which is how the same conflict on the data collection is handled.

Expected

Every write to a stream-enabled table under contention answers 200, the stream holds one record per committed write, and no request answers 500 after part of it committed.