Repository navigation
fix(transact): rank index-key faults and handle the ClientRequestToken as the service does - #853
Open
LeeroyHannigan wants to merge 9 commits into
Open
LeeroyHannigan wants to merge 9 commits into
LeeroyHannigan wants to merge 9 commits into
Conversation
…eason on every backend In a TransactWriteItems request an Update member can give a secondary-index key a value of the wrong type in two ways: a literal operand in the request, or a value computed from the stored item (an attribute reference, the fallback of if_not_exists). The service ranks them differently across members, measured on 2026-10-30 in us-east-1 on two tables, every request sent twice: - a wrong type supplied in the request (a table key, a Put item's index key, an Update's literal operand) cancels with ValidationError at that member and None at every other member, ahead of a failed condition and ahead of a computed wrong type elsewhere; - otherwise an empty index key supplied in the request is a top-level ValidationException, in either member order, ahead of a computed wrong type and ahead of a failed condition elsewhere; - otherwise each member keeps its own outcome, and a computed wrong type is a ValidationError at its member beside them: [ValidationError, ConditionalCheckFailed]. The SQLite, PostgreSQL and MongoDB transacted Update arms carried the computed check as the internal key-type-mismatch reason, which the resolver treats as a request fault, so a computed mismatch masked a failed condition on another member ([ValidationError, None]) and outranked an empty key on another member (a cancellation instead of the ValidationException); on MongoDB, which has no pre-pass, it also outranked a Put item's empty key and a wrong-typed value supplied on a later member. Cassandra already carried it as a plain ValidationError and answered every measured row as the service does. The three arms now carry the computed check as a plain ValidationError. The resolver, its doc comment, the docs row and the test docstring state the rule as measured; the earlier statement that any wrong-typed key masks every other member holds only for a value supplied in the request. The pre-pass is unchanged. Tests: tests/test_transact_index_key_cross_member.py runs the 16 measured two-member transactions and 5 single-request controls with their exact codes, messages and the no-write check on SQLite, PostgreSQL, MongoDB and Cassandra. Before this commit it fails A2, A2r, B1 and B1r on SQLite and PostgreSQL, those four plus C1, C1r, F1 and F2 on MongoDB, and nothing on Cassandra; after it, 21 pass on each. Three resolver unit tests cover the three kinds of fault against each other in both orders.
…ared pass Move the SQLite and PostgreSQL key checks into extenddb-storage as transact_op_key_checks and transact_key_check_pass, and run the pass on all four backends before any member is applied: SQLite and PostgreSQL where they ran their own, MongoDB before its session transaction starts, Cassandra before its ledger entry and PREPARE. resolve_key_faults now lets the first internal key fault in request order decide. PostgreSQL's lock-conflict path resolves its reasons too. is_key_fault had no caller and is removed.
…ross members tests/test_transact_index_key_cross_member.py gains the 19 rows of the 2026-10-31 measurement that put a Put item's empty index key against a wrong type supplied in the request (G rows), against an Update member's empty value (H rows), and an Update member's empty value against a wrong-typed literal or a failed condition (J rows), with the measured codes and messages and the no-write check. On servers built from cef39cf90 the G rows with the Put item first fail on all four backends, and G6r and H1 to H4 also fail on MongoDB and Cassandra; on this branch all 40 cases pass on each backend.
…append and arithmetic by their result An Update's empty index-key literal (SET k = :v, :v an empty string or binary) is reported by the service on the statement, after the literal type check and ahead of the condition, of an action that cannot apply and of a wrong type computed from the item, on UpdateItem and in TransactWriteItems (measured 2026-10-31, 15 requests). UpdateItem now checks it right after the literal type check, in the engine and against the write's own index read on PostgreSQL and SQLite. TransactWriteItems runs a second request-level pass, transact_op_empty_literal_checks, over every member after the first and before any member is applied; the first empty literal in request order answers the top-level ValidationException. An empty value written from the item or from the if_not_exists fallback is still found after the condition. list_append always yields a list and arithmetic always a number, and the service checks both on the statement whatever their operands read, so literal_operand_type types them by their result; an attribute reference and if_not_exists stay values written from the item. The four tests of tests/test_index_key_validation_on_update.py that pinned an empty literal after the condition or after an action that cannot apply now expect the measured answer and are renamed for it. The twi_conflict deadlock test keys its placeholder the way the parser does.
…cs row tests/test_transact_index_key_cross_member.py gains the measured rows for an Update's empty index-key literal on UpdateItem (s1 to s9) and in TransactWriteItems (t1 to t6, K1, K1r), and for list_append that reads the item, alone and beside every partner (L1 to L10, la1, la2). On servers built from cef39cf90 the 75 cases fail 38 on SQLite and PostgreSQL and 43 on MongoDB and Cassandra; on this branch all 75 pass on each backend. s8, s9, K1r and the reversed G and H rows pass on cef39cf90 too and pin both orders. The twi_conflict deadlock test now writes its empty value from the stored item, since an empty literal is found before any row is locked. The validation unit tests satisfy clippy. docs/differences-from-dynamodb.md states the rules as measured and the shared differences with their issues.
…rvice does PR #416 checks secondary-index keys on UpdateItem and in TransactWriteItems, and its review asked that the four backends rank those faults across members the same way. A measurement on the service on 1 November 2026, each request and token sequence sent twice, found three rules that the four backends, with the earlier commits of this branch, still answered differently. Across members the first empty index-key value in request order decides, whether an Update supplies it as a literal or writes it from the item: [Update r SET sidx = e, Update b0 SET bidx = :eb] answers the first member's empty string. The backends, with those commits, answered the second member's empty binary literal, because they checked every member's literals before any member ran. Now they find the first Update member with an empty literal, run only the members before it, put the literal's reason at its position and resolve the reasons as before. Within one statement the literal still comes first, and a member whose own condition fails writes nothing, so its empty if_not_exists fallback is never reported. With the literal on the first member nothing runs. With it later, the members before it run, and on PostgreSQL they lock their rows, so such a request waits on a row another transaction holds, as any member does: until the holder commits or rolls back, or the deadlock detector breaks a cycle. The checks on keys supplied in the request (table key types, a Put item's index keys, an Update's literal types) now run before the ClientRequestToken is checked. A reused token beside such a fault answers the fault, and the token stays unused. A fault found after those checks (an empty literal, an empty or wrong-typed value written from the item, a failed condition) now leaves the token used: another request with it answers IdempotentParameterMismatchException and is not applied, and the same request runs again. Each backend rolls the writes back and then stores the token with its fingerprint prefixed "cancelled:", which no request fingerprint carries, and StoredToken::of tells a replay, a rerun and a mismatch apart. Cassandra no longer deletes the token on these paths. Two empty literals of different kinds in one statement had no fixed order on the service, so the unit test that pinned one order now accepts either. tests/test_transact_index_key_cross_member.py gains U3a, U3b, U3c, U3f and U1a to U1c from that measurement, and the reverse list_append rows L4r, L9r and L10r of the 31 October one (85 cases). The new tests/test_transact_index_key_token.py sends its token sequences (20 cases). On servers of main 25ee109 the two files fail 51 of the 105 cases on SQLite and on PostgreSQL, 62 on MongoDB and 52 on Cassandra. On this branch all 105 pass on every backend, as do the 64 of tests/test_index_key_validation_on_update.py. A twi_conflict test pins that an empty literal on the first member answers without waiting on a later member's lock, and a Cassandra direct test pins the token after a request-level fault, an empty literal and a failed condition. The docs row states the rules and the shared differences, three of them with their issues.
…ogress Amazon DynamoDB keeps the ClientRequestToken of a cancelled TransactWriteItems request used for 10 minutes from the last cancellation. This was measured in us-east-1 on 2026-11-05, with each sequence sent twice. A different request with the token answers IdempotentParameterMismatchException 9 minutes after the cancellation and runs 11 minutes after it. A cancellation of the same request sent again starts the 10 minutes again. Every cancellation after the request-level key checks keeps the token used. These include a failed condition on a ConditionCheck, Delete, Put or Update member, an Update that grows the item past 400 KB, an action that cannot apply, and TransactionConflict. After a TransactionConflict the same request runs again and commits. The service also got two sends of one request at the same moment. It answered one with the request's answer and the other with TransactionInProgressException "Transaction from previous request is still in progress". In some rounds it answered both with the request's answer. When two clients sent a request that commits at the same moment, the service applied it once in each of 20 rounds. The commit before this one keeps the token used after such a cancellation, and left these differences: - Cassandra wrote the cancelled token with no TTL, so the token stayed used with no end. It now writes the cancelled token with a TTL of 600 seconds. The take of a cancelled token by its rerun writes the same TTL. So the token stays used for 10 minutes from the last cancellation, as on the other backends. - SQLite, PostgreSQL and MongoDB kept the created_at of the first cancellation when the same request was cancelled again. The record of the cancelled token now sets created_at to the time of the cancellation. MongoDB also replaces a token document older than the 10-minute read window, which $setOnInsert left in place. - On Cassandra, the token row of a send held the request's fingerprint while the send ran. Another send of the same request could read the row. It then answered as a replay of a write that had not happened, also when the first send was then cancelled. The row now holds running:<fingerprint> while the send runs. The send writes the plain fingerprint when it commits and the cancelled fingerprint when it is cancelled. A send that reads a running row of its own request answers TransactionInProgressException. - A Cassandra send that fails after the reservation leaves the running row, and so does a failed write of the plain fingerprint. The reservation and the take now write the row with a TTL of 600 seconds. So the same request answers TransactionInProgressException, and another request the mismatch, until the row expires 10 minutes after the reservation or the take. Before, such a row held the plain fingerprint with no TTL, and the same request answered as a replay with no end. The plain fingerprint of a committed request still has no TTL, so a committed token stays used with no end, as before. - On Cassandra, a send could lose the reservation of the token to another send. It then compared the stored fingerprint with ==, so it answered the mismatch when the other send was cancelled. It now classifies the row as the read before the reservation does. A take that loses its LWT answers TransactionInProgressException, where it answered a replay. The new StorageError::TransactionInProgress maps to that exception. - On SQLite, PostgreSQL and MongoDB, a failed write of the cancelled token after the rollback answered an internal error. The failure is now logged, and the request answers its cancellation. - The PostgreSQL conflict paths outside the member loop released the token: the token check, the stream record, the index work and the commit. These paths raise no conflict at READ COMMITTED, the default isolation, and can raise one at a stricter isolation. The MongoDB TransactionConflict after the last attempt released the token too. They now record it as cancelled. SQLite unit tests, a PostgreSQL twi_conflict test and MongoDB live tests set created_at back to pin the 10 minutes and their restart. The MongoDB tests also pin the replaced document. Cassandra direct tests read the TTL of a cancelled token row and of a reservation. A twi_conflict test pins the token after a deadlock: another request answers the mismatch, and the same request commits. A Cassandra direct test sends one request twice at once, in 10 rounds of first sends and 10 rounds of a cancelled request. It accepts only the cancellation or TransactionInProgressException. Another writes a running row by hand: the same request answers TransactionInProgressException, and another request answers the mismatch. A SQLite unit test makes the token write fail and expects the cancellation. tests/test_transact_index_key_token.py gets the rows of 5 November: an Update past 400 KB, an action that cannot apply, failed conditions on ConditionCheck, Delete and Put members, and the same request twice. It also marks the ReturnConsumedCapacity row as an expected failure (#514). The docs row states the rules of 5 November, the mismatch message (#760) and #514. It also states PostgreSQL's answer to two sends of one request at once (#851). It no longer states an order between the 400 KB check and the index-key checks, which no measurement shows.
Cassandra marks a committed ClientRequestToken of TransactWriteItems by writing the request's plain fingerprint into its row of idempotency_tokens_by_account. The write is an UPDATE ... IF fingerprint = running:<fingerprint>, and it had no TTL. The table has no default TTL, and the Cassandra cleanup_expired_idempotency_tokens returns 0. The token check reads the row without its created_at. So the token of a committed request stayed used for as long as the row existed. At any time after the commit, the same request sent again answered as a replay. A different request with the token answered IdempotentParameterMismatchException. Amazon DynamoDB keeps the token of a committed request used 9 minutes after the request. It treats the token as new 11 minutes after the request (measured in us-east-1 on 2026-11-05, each sequence sent twice). SQLite, PostgreSQL and MongoDB keep a token for 10 minutes through its created_at. The commit mark now writes with USING TTL 600 (IDEMPOTENCY_TOKEN_TTL_SECONDS), as the reservation, the take and the write of a cancelled token do. So every token row that this release writes expires 10 minutes after its last write. A row that an earlier release wrote has no TTL and stays. A send that fails before the mark leaves the running row of its reservation or its take: that row expires 10 minutes after the write. A direct test commits a Put with a token and reads TTL(fingerprint) of its row as a value from 590 to 600 seconds.
LeeroyHannigan
requested review from
amrith,
c33howard,
eliseong,
hitesh96db,
jcshepherd,
navareamol,
pdf-amzn and
robinnsc
as code owners
November 5, 2026 06:16
Three twi_conflict tests close a deadlock cycle through an outside transaction and expect PostgreSQL to abort the write transaction. PostgreSQL aborts the transaction whose deadlock check finds the cycle, and each waiter checks once, deadlock_timeout after it starts to wait. The write transaction waits first. With the default of 1 second on both sides, the two checks fall a few milliseconds apart. Under load the outside transaction's check can then run first, and PostgreSQL aborts the outside transaction instead. The test then fails. The token test of the conflict path failed that way once in 54 runs of the file. The outside transaction now runs SET LOCAL deadlock_timeout = '5s'. The write transaction's check runs 1 second into its wait, with the cycle closed. The outside transaction gets its lock when the write transaction aborts. SET LOCAL ends with the transaction, so the pooled connection keeps the server default. The role must be allowed to set deadlock_timeout. The CI role postgres is a superuser. At the SQL level, an outside session with a check at 100 ms was the victim, and with 5 seconds the write session was. With the change, the two deadlock tests passed 20 runs, the token test passed 20 runs and the whole file passed 5 runs.
6 of 8 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This PR follows #416. It makes
TransactWriteItemsrank its key faults and theClientRequestTokenas the service does, on all four backends. It also keeps the token of a cancelled request as the service keeps it. The rules, as measured on the service:Putitem, and the type of anUpdate's literal operand. The first such fault in request order decides the answer, and the token stays unused.Updatecan supply that value as a literal or write it from the item. In one statement the literal comes first. A member whose condition fails writes nothing.IdempotentParameterMismatchException, and the backend does not apply it. The same request runs again.Updatepast 400 KB, an action that cannot apply, andTransactionConflict. Another request with the token gets the mismatch 570 seconds after the last cancellation, and the backend applies it 600 seconds after it. A cancellation of the same request sent again starts that time again. A committed token stays used as long, and the same request sent again 11 minutes after its commit runs again.TransactionInProgressException"Transaction from previous request is still in progress", or it waits and gets the answer of that send. The service applies such a request once.The changes:
transact_first_empty_literalinextenddb-storagefinds the firstUpdatemember with an empty literal. Each backend runs only the members before that member and puts the literal's reason at its position.resolve_key_faultsthen decides from the reasons, as before.transact_op_key_checksover every member before it checks the token.cancelled:. No request fingerprint has that prefix.StoredToken::oftells a replay, a rerun and a mismatch apart, and the schema does not change. Cassandra no longer deletes the token on these paths. It writes the cancelled token with a TTL of 600 seconds. So the token stays used for 10 minutes from the last cancellation, as on the other backends.cancelled:row, and on Cassandra arunning:row, as the token of another request. So after a downgrade, the same request answersIdempotentParameterMismatchExceptionuntil the token expires.TransactionConflictafter the last attempt. If that write fails, the backend logs the failure, and the request still gets its cancellation.running:and the fingerprint while a send runs, and the plain fingerprint after the send commits. Another send of the same request that reads a running row getsTransactionInProgressException. So does a send that loses the reservation or the take of the token. Before, such a send got a replay of a write that had not happened, or the mismatch. The newStorageError::TransactionInProgressmaps to that exception. The reservation and the take write the row with a TTL of 600 seconds. So a send that fails after them leaves its running row for 10 minutes.twi_conflicttests close a deadlock cycle through an outside transaction and expect PostgreSQL to abort the write transaction. PostgreSQL aborts the transaction whose deadlock check finds the cycle. Each waiter checks once,deadlock_timeoutafter it starts to wait. With the default on both sides, the check of the outside transaction could run first, so PostgreSQL aborted that transaction instead. One of the tests failed that way once in 54 runs of the file. The outside transaction now setsSET LOCAL deadlock_timeout = '5s', so the check of the write transaction runs first.On PostgreSQL, an empty literal on a later member makes the members before it run, and they lock their rows. If another transaction holds one of those rows, the request waits, as any member does. The wait ends when the holder commits or rolls back, or when the deadlock detector breaks a cycle. The backend sets no
lock_timeouton data transactions. When the literal is on the first member, the backend locks nothing.Why
Closes #852.
jcshepherd's review of #416 on 28 October asked for one ranking of these faults on every backend. #416 merged on 3 November with the ranking it had before that review. The issue gives the measurements and the answers of main at
25ee109bfon each backend. The measurements are from 30 October, 31 October, 1 November, 4 November and 5 November 2026, with each request or token sequence sent twice.Testing done
Rust, on the branch (base: main at
94b5f689b):cargo +1.97.0 fmt --all --check: clean.cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings: clean.cargo +1.97.0 check --locked --workspace --all-targetsandcargo +1.88.0 check --locked --workspace: ok.cargo +1.97.0 test --locked --workspace --exclude extenddb-backup-store --no-fail-fast: 3175 passed, 0 failed, 63 ignored. On the test host,cargo +1.97.0 test --locked -p extenddb-backup-store --no-fail-fastfails 59 of its 151 tests. All of them are filesystem tests, which refuse a backup root under a directory that neither root nor the extenddb user owns.twi_conflictagainst PostgreSQL 16, three runs: 12 passed in each run. A new test cancels a request with a token in a deadlock. Another request with the token then gets the mismatch, and the same request commits. Another new test setscreated_atback to check the 10 minutes and their restart.live_tests::): 44 passed. Two new tests setcreated_atback. They check the restart of the 10 minutes and the replaced document.transact_write_items::tests, three runs: 14 passed in each run. Five tests are new. Three read the TTL of a cancelled token row, of a reservation and of a committed token row. One sends one request twice at once from two tasks, in 10 rounds of first sends and 10 rounds of a cancelled request. It accepts only the cancellation orTransactionInProgressException. It also passed in five runs of its own. In each, 38 or 39 of the 40 answers were the cancellation and the othersTransactionInProgressException. One writes a running row by hand.Wire tests against a local release server per backend. The new rows are in
tests/test_transact_index_key_cross_member.py(85 cases) and the newtests/test_transact_index_key_token.py(27 cases).25ee109bftest_index_key_validation_on_update.py, main at25ee109bfThe xfailed case sends the same items with another
ReturnConsumedCapacity(#514). The skipped case is theUpdatepast 400 KB, which runs on SQLite and PostgreSQL only (#818, #761).On this branch,
tests/test_transaction_operations.py(31) andtests/python/test_transactions.py(16) pass on every backend. The 33 test files that cover index keys and transactions give these results:tests/(26 files)tests/python/(7 files)Two clients sent one request with one token to a Cassandra server at the same moment. There were 10 rounds of first sends and 10 rounds of a cancelled request, in each of two runs. On this branch every answer was the cancellation or
TransactionInProgressException, and the item was never written. Main answeredOKfor that write in 1 to 8 of the 20 answers of a run, over three runs.Mutants, one at a time, and the tests that fail on each. The three that change a Cassandra TTL or the commit mark ran on the head of this branch. The others ran before the rebase onto
94b5f689b, which did not change the code they mutate.StoredToken::ofgives the mismatch for a cancelled token of the same request:a_stored_token_is_a_replay_a_rerun_or_a_mismatch.test_a_token_a_cancelled_request_used.a_failed_token_write_keeps_the_cancellation.a_conflict_cancellation_keeps_the_token_used_and_the_same_request_runs_again.live_a_cancellation_keeps_the_token_used_from_its_last_cancellation.test_a_cancelled_token_row_expires_ten_minutes_after_its_last_cancellationandtest_idempotency_token_after_a_key_fault.test_a_cancelled_token_row_expires_ten_minutes_after_its_last_cancellation.==after a lost reservation, or gives a replay when the take loses:test_two_sends_of_one_request_answer_its_cancellation_or_in_progress.test_two_sends_of_one_request_answer_its_cancellation_or_in_progress.test_a_reserved_token_row_expires_ten_minutes_after_its_reservation.test_idempotency_token_replayandtest_a_committed_token_row_expires_ten_minutes_after_its_commit.test_a_committed_token_row_expires_ten_minutes_after_its_commit.Against Amazon DynamoDB: the three test files of this branch (
tests/test_index_key_validation_on_update.py,tests/test_transact_index_key_cross_member.pyandtests/test_transact_index_key_token.py), run at the head on 5 November 2026 in us-east-1, 176 passed.Known differences
All four backends share the first six. This PR does not change the first five.
Actual: N, where the service lists a set of number types (Arithmetic on an index key: the type mismatch names N where the service names a set of types #702).GetItem,DeleteItem,UpdateItemandTransactGetItemsgetsType mismatch for key <attr> expected: <T> actual: <U>, where the service answers "The provided key element does not match the schema" (GetItem, DeleteItem, UpdateItem and TransactGetItems answer a wrong-typed key with the PutItem message #701).AttributeUpdatesparameter gets the update-expression wording, where the service uses thePutItemwording that names the index (UpdateItem: an empty secondary-index key sent through legacy AttributeUpdates gets the UpdateExpression wording on every backend #423).IdempotentParameterMismatchExceptionhas ExtendDB's message, "The request uses the same client token as a previous, but different, request." The service answers "Specified idempotent token was used with different request parameters within the idempotency window" (TransactionCanceledException and IdempotentParameterMismatchException carrymessage; DynamoDB sendsMessage, and the token mismatch text differs #760, PR fix(server): send Message on the two errors the service sends it on #774). The new token tests compare the error code only.ReturnConsumedCapacity. The same items sent again with anotherReturnConsumedCapacityget a replay after a commit, and run again after a cancellation. The service answers the mismatch (TransactWriteItems token replay ignores ReturnConsumedCapacity and ReturnItemCollectionMetrics #514, PR fix(engine): compare the Return fields on a TransactWriteItems token replay #526).Updatethat grows an item past 400 KB gets the cancellation reason "Item size has exceeded the maximum allowed size". The service and SQLite answer "Item size to update has exceeded the maximum allowed size" (MongoDB applies no item size limit to updates, and Cassandra sizes updates on the finished item against a fixed 400 KiB with the PutItem wording #818, TransactWriteItems: updating a key attribute is a top-level ValidationException, and the item size reason text differs #761).Updatethat grows an item past 400 KB commits, where the service cancels it (MongoDB applies no item size limit to updates, and Cassandra sizes updates on the finished item against a fixed 400 KiB with the PutItem wording #818). The token file skips that case on MongoDB.TransactionConflict(MongoDB: a ClientRequestToken older than 10 minutes answers TransactionConflict until the TTL monitor removes its document #836). The backend then stores the cancelled token of that request over the old document. The same request sent again runs, and another request with the token gets the mismatch for 10 minutes.TransactionInProgressException, and another request the mismatch, until the row expires 10 minutes after the reservation or the take. Before this PR it got a replay of the write that did not happen (Cassandra: a TransactWriteItems sent again after a failure that applied nothing answers as a replay #837).[Update cnt SET n = n + :one]with one token at the same moment.ngrew by 2 in 6 rounds on this branch and in 8 rounds on main. On the service it grew by 1 in each of 20 rounds (PostgreSQL: two sends of one TransactWriteItems with one ClientRequestToken at the same moment can both apply #851).These answers of the service are not measured:
TransactionInProgressExceptionfor 10 minutes.Put50 ms apart both gotOKand wrote the items once, but the first send had very likely finished.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. The change gives the answers that DynamoDB gives, and the token store keeps its schema. An older binary reads a cancelled token as a token that another request holds.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.