Skip to content

fix(server): read request members in sorted order on every build - #781

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/723-sort-request-keys
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/723-sort-request-keys

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Builds with serde_json's preserve_order feature, the MongoDB build among them, answered some request bodies with two faults differently from other builds, and their answer depended on the order the client wrote the members in.

Mechanism

bson 2.15.0 declares serde_json with the preserve_order feature, and Cargo enables it for every crate in a build that compiles the MongoDB backend crate: a --features mongodb binary, and also any build of the whole workspace, such as CI's cargo build --release. A serde_json::Map there keeps insertion order; in a build of the extenddb package without the MongoDB backend it is a BTreeMap, sorted by key. The typed input deserializers (for example QueryInput in handle_query) visit the members of the parsed Value in the map's order and stop at the first one that fails.

Change

handler.rs calls Value::sort_all_objects on the parsed body. With preserve_order it sorts every object in the body by key; without it the call does nothing, because its body is compiled only with that feature. Every build then reads the members in sorted order, as builds without preserve_order already did. The call runs immediately after the parse, before authentication, so on a preserve_order build an unauthenticated request pays for the sort as it already pays for the parse. After authentication it conflicted with #587, #549 and #547, which edit the lines around the end of the authentication segment; beside the parse it merges cleanly with the nine open pull requests that edit handler.rs.

Measured on Amazon DynamoDB

us-east-1, 3 November 2026, raw signed requests: 24 Query and Scan bodies with two faults, each sent in both member orders. Every pair got the same answer in both orders. The three the new test sends:

Body DynamoDB, both orders
Query, Limit: 0.0, :a is {} ValidationException: 1 validation error detected: Value '0' at 'limit' failed to satisfy constraint: Member must have value greater than or equal to 1
Query, Limit: "0", :a is {} SerializationException: STRING_VALUE cannot be converted to Integer
Scan, :b is {}, :a is {"X": "1"} ValidationException: ExpressionAttributeValues contains invalid value: Supplied AttributeValue is empty, must contain exactly one of the supported datatypes for key :b

On these bodies the builds now agree with each other, not with the service: ExtendDB answers the ExpressionAttributeValues fault where DynamoDB answers the Limit fault, and reports :a where DynamoDB names :b (#778). DynamoDB also truncates a fractional Limit (#720, #563) and answers an unknown type descriptor as an empty value (#629).

Other changes on a preserve_order build

  • Per-partition throttling picks the bucket for GetItem, PutItem, DeleteItem and UpdateItem by the first attribute of Key or Item in name order, as the other builds do, instead of the first attribute the client wrote. Neither reads the key schema; Per-partition throttling takes the first attribute of Key or Item as the partition key #779 covers that.
  • The TransactWriteItems idempotency fingerprint is computed over the sorted TransactItems. A retry that reuses a ClientRequestToken across the upgrade, while the first request's token is still stored, gets IdempotentParameterMismatchException unless the first request listed its members in name order.

Tests

tests/test_request_member_order.py, dual-target, posts each body in the table in both member orders, signed with the client's credentials, and requires the two answers to be equal.

  • Amazon DynamoDB: 3 passed.
  • Dev-mode SQLite builds with preserve_order forced on in crates/server/Cargo.toml (a local mutant, not committed): without the handler.rs call, 3 failed; with it, 3 passed.
  • On the same two builds, the 695 tests that attribute_value or expression_param or request_member_order or test_query or test_scan or test_batch or test_transact or test_put_item or test_update_item or test_delete_item or test_get_item selects gave the same results except for the 3 new tests. The 15 failures on both builds need authorization, which dev mode does not enforce.
  • cargo +1.97.0 fmt --all -- --check and cargo +1.97.0 clippy -p extenddb-server --all-targets -- -D warnings: clean. cargo +1.88.0 test --locked -p extenddb-server: 250 passed.

In CI the new file runs against the MongoDB build and against the PostgreSQL integration server, which is built from the workspace root and so has preserve_order too.

Not in this change

IAM policy documents from the management API and the console are parsed outside handler.rs and keep their written order on a preserve_order build (#780).

Fixes #723

The MongoDB build depends on bson, which turns on serde_json's
preserve_order feature, and Cargo enables it for every crate in that
binary. A parsed request object there keeps the order the client wrote
its members in; the SQLite, PostgreSQL and Cassandra builds sort them by
name. The typed input deserializers stop at the first member that fails,
so a body with two faults got a different error on the MongoDB build,
and which one depended on the client's member order.

The data-plane handler now calls Value::sort_all_objects on the parsed
body after authentication, which reads the raw bytes. The call sorts
every object in the body when preserve_order is on and does nothing
otherwise, so every build reads the members in the sorted order the
others already used.

Amazon DynamoDB answered both orders of each two-fault body in the new
test with the same error (us-east-1, 2026-11-03). The test sends each
body in both orders, signed, and passes against the service.

Fixes #723
The first commit sorts the body after authentication. Open pull
requests #587, #549 and #547 edit the lines around the end of the
authentication segment, so there the call conflicts with all three.
This commit moves it to immediately after the parse, where it merges
cleanly with the nine open pull requests that edit handler.rs.

On a preserve_order build an unauthenticated request now pays for the
sort, as it already pays for the parse. Without preserve_order the call
does nothing.

The comment now says which builds have preserve_order: any build that
compiles the MongoDB backend crate, which includes a build of the whole
workspace, not only the MongoDB binary.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MongoDB build: bson turns on serde_json preserve_order, so a request body with two faults reports a different first error than on the other backends

1 participant