Repository navigation
fix(server): read request members in sorted order on every build - #781
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
The MongoDB build depends on bson, which turns on serde_json's preserve_order feature, and Cargo enables it for every crate in that binary. A parsed request object there keeps the order the client wrote its members in; the SQLite, PostgreSQL and Cassandra builds sort them by name. The typed input deserializers stop at the first member that fails, so a body with two faults got a different error on the MongoDB build, and which one depended on the client's member order. The data-plane handler now calls Value::sort_all_objects on the parsed body after authentication, which reads the raw bytes. The call sorts every object in the body when preserve_order is on and does nothing otherwise, so every build reads the members in the sorted order the others already used. Amazon DynamoDB answered both orders of each two-fault body in the new test with the same error (us-east-1, 2026-11-03). The test sends each body in both orders, signed, and passes against the service. Fixes #723
LeeroyHannigan
requested review from
amrith,
c33howard,
eliseong,
hitesh96db,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
November 3, 2026 02:34
The first commit sorts the body after authentication. Open pull requests #587, #549 and #547 edit the lines around the end of the authentication segment, so there the call conflicts with all three. This commit moves it to immediately after the parse, where it merges cleanly with the nine open pull requests that edit handler.rs. On a preserve_order build an unauthenticated request now pays for the sort, as it already pays for the parse. Without preserve_order the call does nothing. The comment now says which builds have preserve_order: any build that compiles the MongoDB backend crate, which includes a build of the whole workspace, not only the MongoDB binary.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds with
serde_json'spreserve_orderfeature, the MongoDB build among them, answered some request bodies with two faults differently from other builds, and their answer depended on the order the client wrote the members in.Mechanism
bson2.15.0 declaresserde_jsonwith thepreserve_orderfeature, and Cargo enables it for every crate in a build that compiles the MongoDB backend crate: a--features mongodbbinary, and also any build of the whole workspace, such as CI'scargo build --release. Aserde_json::Mapthere keeps insertion order; in a build of theextenddbpackage without the MongoDB backend it is aBTreeMap, sorted by key. The typed input deserializers (for exampleQueryInputinhandle_query) visit the members of the parsedValuein the map's order and stop at the first one that fails.Change
handler.rscallsValue::sort_all_objectson the parsed body. Withpreserve_orderit sorts every object in the body by key; without it the call does nothing, because its body is compiled only with that feature. Every build then reads the members in sorted order, as builds withoutpreserve_orderalready did. The call runs immediately after the parse, before authentication, so on apreserve_orderbuild an unauthenticated request pays for the sort as it already pays for the parse. After authentication it conflicted with #587, #549 and #547, which edit the lines around the end of the authentication segment; beside the parse it merges cleanly with the nine open pull requests that edithandler.rs.Measured on Amazon DynamoDB
us-east-1, 3 November 2026, raw signed requests: 24 Query and Scan bodies with two faults, each sent in both member orders. Every pair got the same answer in both orders. The three the new test sends:
Limit: 0.0,:ais{}ValidationException:1 validation error detected: Value '0' at 'limit' failed to satisfy constraint: Member must have value greater than or equal to 1Limit: "0",:ais{}SerializationException:STRING_VALUE cannot be converted to Integer:bis{},:ais{"X": "1"}ValidationException:ExpressionAttributeValues contains invalid value: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes for key :bOn these bodies the builds now agree with each other, not with the service: ExtendDB answers the
ExpressionAttributeValuesfault where DynamoDB answers theLimitfault, and reports:awhere DynamoDB names:b(#778). DynamoDB also truncates a fractionalLimit(#720, #563) and answers an unknown type descriptor as an empty value (#629).Other changes on a
preserve_orderbuildKeyorItemin name order, as the other builds do, instead of the first attribute the client wrote. Neither reads the key schema; Per-partition throttling takes the first attribute of Key or Item as the partition key #779 covers that.TransactItems. A retry that reuses aClientRequestTokenacross the upgrade, while the first request's token is still stored, getsIdempotentParameterMismatchExceptionunless the first request listed its members in name order.Tests
tests/test_request_member_order.py, dual-target, posts each body in the table in both member orders, signed with the client's credentials, and requires the two answers to be equal.preserve_orderforced on incrates/server/Cargo.toml(a local mutant, not committed): without thehandler.rscall, 3 failed; with it, 3 passed.attribute_value or expression_param or request_member_order or test_query or test_scan or test_batch or test_transact or test_put_item or test_update_item or test_delete_item or test_get_itemselects gave the same results except for the 3 new tests. The 15 failures on both builds need authorization, which dev mode does not enforce.cargo +1.97.0 fmt --all -- --checkandcargo +1.97.0 clippy -p extenddb-server --all-targets -- -D warnings: clean.cargo +1.88.0 test --locked -p extenddb-server: 250 passed.In CI the new file runs against the MongoDB build and against the PostgreSQL integration server, which is built from the workspace root and so has
preserve_ordertoo.Not in this change
IAM policy documents from the management API and the console are parsed outside
handler.rsand keep their written order on apreserve_orderbuild (#780).Fixes #723