Repository navigation
fix(scan): report every Limit, Segment and TotalSegments range fault in the service's two forms - #850
Open
LeeroyHannigan wants to merge 3 commits into
Open
LeeroyHannigan wants to merge 3 commits into
LeeroyHannigan wants to merge 3 commits into
Conversation
…in the service's two forms The Scan range check reported the first fault in the order Limit, Segment, TotalSegments, with the previous release's texts for the two segment fields (Value '-1' at 'segment', and "The parameter TotalSegments should be greater than or equal to 1" for a value below 1). A float or an integer outside i64 in any of the three answered SerializationException. The service reports every range fault in one message. When each field present is an integer inside the int32 range it uses "Value at '<Field>' failed to satisfy constraint: ..." with the field capitalised and no value echoed, in the order Limit, Segment, TotalSegments. When any of the three is a float or outside int32, or an ExpressionAttributeValues entry is not one AttributeValue (empty, an unknown type descriptor, two descriptors), it truncates or saturates the value and uses "Value '<n>' at '<field>' ..." with the field in lower camel case and limit last. A float or an out-of-range integer that satisfies the constraint is accepted: Limit 1.5 returns one item and Segment 1.5 returns part of the table. The request-time check now builds that message from the raw body, and the three fields deserialize from any JSON number, truncated toward zero and saturated to int32, so the handler uses the value the check validated. Query answers the older form for a Limit of 0 beside an entry that is not one AttributeValue, as measured on Query. The two assertions in tests/test_query_scan.py that pinned the previous segment texts now assert the measured text. Measured against the service on 2026-10-25 (09:23, 09:25, 09:28, 10:38, 10:55 and 12:56 UTC), each request twice.
…members
Measured against the service on 2026-10-30 at 12:40 UTC, each request
sent twice, as a raw signed Scan:
- A fault in each of Limit, Segment and TotalSegments in the older form
(Limit 0.0, Segment -1, TotalSegments 1000001, and the same with
TotalSegments 1000001.0 and Limit 0) answers "3 validation errors
detected: Value '1000001' at 'totalSegments' ...; Value '-1' at
'segment' ...; Value '0' at 'limit' ...". The older form lists the
faults in the reverse of the newer form's order, which agrees with the
two-fault rows measured on 2026-10-25. The check put segment ahead of
totalSegments.
- Limit 0 beside an ExpressionAttributeValues entry {"L": [{}]} or
{"M": {"k": {"X": "1"}}} answers the older form, Value '0' at 'limit'.
The check read only the top level of each entry and answered the
newer form. Query reads the entries with the same function, so its
answer to Limit 0 beside these entries changes too.
- Limit -1.5 answers Value '-1', and Limit -0.5 and Limit -0 answer
Value '0'. The check already answered these as measured.
The tests send each measured row. The rows no test sent before were
measured on 2026-10-31 at 12:36 UTC, each request sent twice, and the
tests cite that measurement: Limit 0 beside a readable entry of each of
the ten types answers the newer form, Segment 0 with TotalSegments 1
returns the whole table, a Query with Limit -2147483649 beside an empty
entry echoes -2147483648 in the older form, and a Query with Limit 0
beside the two nested entries or the lower-case descriptor answers the
older form. The combinations test, which said its rows were not sent,
is renamed for what it checks: every fault in one message, and nothing
reported for a field inside its bounds; its eleven rows were measured
the same day.
Measured against the service on 2026-10-31 between 23:54 and 23:56 UTC, each request sent twice, as a raw signed Scan or Query: - Limit 0 answers the older form, Value '0' at 'limit', beside an empty value or one with an unknown type descriptor in ExclusiveStartKey or in a ScanFilter AttributeValueList on Scan, and beside an empty value in ExclusiveStartKey or in a KeyConditions or QueryFilter AttributeValueList on Query. Segment -1 with TotalSegments 2 beside an empty ExclusiveStartKey value answers Value '-1' at 'segment'. The check read only ExpressionAttributeValues and answered the newer form for these. - Limit 0 beside a readable value keeps the newer form: a number as the value of an S key in ExclusiveStartKey, an N entry in a ScanFilter list and an S entry in a KeyConditions list. The older-form check now reads ExclusiveStartKey and the AttributeValueList entries of ScanFilter on Scan and of KeyConditions and QueryFilter on Query, with the rule it applies to ExpressionAttributeValues entries. It reads the L and M members of those values the same way, which is not measured. range_tests.rs and query_key_values/tests.rs have the measured rows, and tests/python/test_scan_range_constraints.py sends them as raw signed requests: seven Scan and four Query tests.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
November 5, 2026 04:31
8 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #562 (
fix/scan-number-check). This branch adds three commits on top of it.validate_rangesincrates/engine/src/scan_request.rsreadsLimit,SegmentandTotalSegmentsfrom the raw body and builds one message from every fault. It uses the older form when any of the three is a float or outside int32. It also uses the older form when a value in the request is an object that is not one AttributeValue. Such a value is empty, has an unknown or lower-case type descriptor, or has more than one descriptor. That holds at the top level and inside anLorMmember. The values it reads are theExpressionAttributeValuesentries, theExclusiveStartKeyvalues and theAttributeValueListentries ofScanFilter. The older form saturates or truncates each value, writes the field in lower camel case and lists the faults as totalSegments, segment, limit.ScanInputreadsLimit,SegmentandTotalSegmentsfrom any JSON number, truncated toward zero and saturated to int32 (saturated_int32incrates/core/src/types/query.rs). The handler uses the value the check read. The check accepts a float or an out-of-range integer that satisfies the constraint.Query answers the older form for a
Limitbelow 1 beside a value that is not one AttributeValue, with the echoed value saturated to int32 (crates/engine/src/query_key_values.rs). On Query the values read are theExpressionAttributeValuesentries, theExclusiveStartKeyvalues and theAttributeValueListentries ofKeyConditionsandQueryFilter.The first commit makes these changes with the older form in the order segment, totalSegments, limit and an entry read only at its top level. The second commit, after the 2026-10-30 and 2026-10-31 measurements, reverses the older form's order to totalSegments, segment, limit and reads
LandMmembers. It adds the tests forSegment: 0, TotalSegments: 1, the negative fractions, the ten readable entry types and the Query clamp. The third commit, after the measurement of 2026-10-31 between 23:54 and 23:56 UTC, readsExclusiveStartKeyand the condition lists for the older form. Before it, the check readExpressionAttributeValuesalone and answered the newer form beside an unreadable key value or list entry.Why
Fixes #563.
Measured against DynamoDB (us-east-1) on 2026-10-25 at 09:23, 09:25, 09:28, 10:38, 10:55 and 12:56 UTC, and on 2026-10-30 at 12:40 UTC. Measured again on 2026-10-31 at 12:36 UTC and between 23:54 and 23:56 UTC, and on 2026-11-05 between 00:54 and 00:55 UTC. The probe sent each request twice, as a raw signed Scan or Query.
When every
Limit,SegmentandTotalSegmentspresent is an integer inside the int32 range, the service reports all range faults in one message. The order is Limit, Segment, TotalSegments, asValue at '<Field>' failed to satisfy constraint: ...with the field capitalised and no value echoed. When any of the three is a float or outside int32, the service truncates or saturates the values. It then reportsValue '<n>' at '<field>' failed to satisfy constraint: ...with the field in lower camel case, in the order totalSegments, segment, limit. Beside a value that is not one AttributeValue,Limit: 0answers the same older form,Value '0' at 'limit' ..., on Scan and on Query. Such a value is empty, has an unknown or lower-case type descriptor, or has two descriptors, at the top level or inside anLorMmember. That holds for anExpressionAttributeValuesentry, for anExclusiveStartKeyvalue and for anAttributeValueListentry ofScanFilteron Scan and ofKeyConditionsorQueryFilteron Query. The probe measured{}and, on Scan,{"X": "1"}in the key and the lists on 2026-10-31, and six shapes there on 2026-11-05:{},{"X": "1"},{"S": "x", "N": "1"},{"s": "x"},{"L": [{}]}and{"M": {"k": {"X": "1"}}}. The 2026-11-05 run sent 151 requests. They put each shape in the key, inKeyConditionsand in the filter lists besideLimit0, -1 and -2147483648, and -2147483649 on Query. On Scan they also put each shape in the key besideSegment: -1withTotalSegments: 2and besideSegment: 0withTotalSegments: 1000001. They put each shape in theScanFilterlist besideSegment: -1withTotalSegments: 2. All 151 answer the older form, and 145 answer as this branch does. The six withLimit: 0andAttributesToGet: []beside a key value addValue '[]' at 'attributesToGet' failed to satisfy constraint: Member must have length greater than or equal to 1after theLimitclause. The next branch adds that clause.Segment: -1withTotalSegments: 2beside an emptyExclusiveStartKeyvalue answersValue '-1' at 'segment' .... A readable value of the wrong type for the key ({"N": "1"}for anSkey) keeps the newer form. The service truncates a negative fraction toward zero and echoes-1.5as-1, and-0.5and-0as0. The service accepts a float or an out-of-range integer that satisfies the constraint, andSegment: 0withTotalSegments: 1. Query takes the older form for aLimitbelow 1 beside a value that is not one AttributeValue, with the echoed value saturated to int32.Each table row is one request. The second column is the service's answer, which is also this branch's answer. The last two columns are the answers of SQLite servers built from origin/main at 817b240 and from #562's head at d676c6e, this branch's base. origin/main has since moved to b186d1a. A SQLite server of 25ee109 answers the 103 requests of the measurement set the same way as 817b240 on every row these tables hold. The three commits after 25ee109 change no file of these checks. The rows measured on 2026-10-31 are in the last table. The tables shorten each message to
...after the field name or the value. The full texts are in the tests. The probe measured the rows marked 2026-10-30 or 2026-10-31 on that date, and the others on 2026-10-25.Limit: 0Segment: -1Segment: -1, TotalSegments: 2Segment: 1000000, TotalSegments: 1000000TotalSegments: 0Segment: 0, TotalSegments: 0Segment: 0, TotalSegments: -1TotalSegments: 1000001Limit: 0, Segment: -1, TotalSegments: 1000001Limit: 0, TotalSegments: 0Segment: -1, TotalSegments: 1000001Segment: 1000000, TotalSegments: 0Limit: -2147483648Limit: -2147483649Segment: 0, TotalSegments: 2147483648Limit: 0.00.0, expected i640.0, expected i64Limit: -2147483649, Segment: -1TotalSegments: 1000001, Limit: 0.00.0, expected i640.0, expected i64Limit: 0.0, Segment: -1, TotalSegments: 1000001(2026-10-30)0.0, expected i640.0, expected i64Limit: 0, Segment: -1, TotalSegments: 1000001.0(2026-10-30)1000001.0, expected i641000001.0, expected i64Limit: -1.5(2026-10-30)-1.5, expected i64-1.5, expected i64Limit: -0.5(2026-10-30)-0.5, expected i64-0.5, expected i64Limit: -0(2026-10-30)-0.0, expected i64-0.0, expected i64Limit: 0beside{}Limit: 0beside{"X": "1"}Limit: 0beside{"S": "x", "N": "1"}Limit: 0beside{"NS": []}Limit: 0beside{"NS": ["1", "1"]}Limit: 0beside{"L": [{}]}, with a filter that reads another entry (2026-10-30)Limit: 0beside{"M": {"k": {"X": "1"}}}, the same filter (2026-10-30)Limit: 0beside{"s": "x"}, the same filter (2026-10-30)Limit: 0beside one readable entry of each type (S,N,B,SS,NS,BS,M,L,NULL,BOOL), no expression (2026-10-31, ten rows)KeyConditionExpression: "pk = :p")Limit: 0beside{}Limit: 0beside{"X": "1"}Limit: 0beside{"S": "x", "N": "1"}Limit: 0beside{"NS": []}Limit: 0beside{"NS": ["1", "1"]}Limit: -2147483649beside{}(2026-10-31)Limit: 0beside{"L": [{}]}(2026-10-31)Limit: 0beside{"M": {"k": {"X": "1"}}}(2026-10-31)Limit: 0beside{"s": "x"}(2026-10-31)Limit: 2147483648Limit: 92233720368547758089223372036854775808, expected i649223372036854775808, expected i64Limit: 1.51.5, expected i641.5, expected i64Limit: 1.01.0, expected i641.0, expected i64Limit: 1e2100.0, expected i64100.0, expected i64Segment: 1.5with aTotalSegments1.5, expected i641.5, expected i64Segment: 0, TotalSegments: 1, ConsistentRead: true(2026-10-31)ExclusiveStartKeyor a condition list (2026-10-31, 23:54 to 23:56 UTC)Limit: 0,ExclusiveStartKey{"pk": {}, "sk": {"S": "x"}}Limit: 0,ExclusiveStartKey{"pk": {"X": "1"}, "sk": {"S": "x"}}Limit: 0,ScanFilterAttributeValueList[{}]Limit: 0,ScanFilterAttributeValueList[{"X": "1"}]Segment: -1, TotalSegments: 2,ExclusiveStartKey{"pk": {}, "sk": {"S": "x"}}Limit: 0,ExclusiveStartKey{"pk": {"N": "1"}, "sk": {"S": "x"}}(a number for anSkey)Limit: 0,ScanFilterAttributeValueList[{"N": "1"}]Limit: 0,ExclusiveStartKey{"pk": {}, "sk": {"S": "x"}}Limit: 0,QueryFilterAttributeValueList[{}]Limit: 0,KeyConditionsAttributeValueList[{}]Limit: 0,KeyConditionsAttributeValueList[{"S": "p0"}]Testing done
crates/engine/src/scan_request/range_tests.rshas every row of the tables: ten tests, every row measured. Among them are the three-fault older-form rows, theLandMmember rows, the negative fractions, and one loop over the ten readable entry types besideLimit: 0. One test holds every combination the probe sent to the service on 2026-10-31 at 12:36 UTC (eleven requests). One holds theExclusiveStartKeyandScanFilterrows of 2026-10-31 between 23:54 and 23:56 UTC.crates/core/src/types/query.rstests the saturation function, with-1.5,-0.5and-0among its values, and the deserializer.crates/engine/src/query_key_values/tests.rshas the Query rows, with theL,Mand lower-case entries and theExclusiveStartKey,KeyConditionsandQueryFilterrows.tests/python/test_scan_range_constraints.pyhas 58 tests that send each row as a raw signed request and compare the message byte for byte. On main (25ee109bf), 55 of the 58 fail. The three that pass sendLimit: 2147483648andSegment: 0, TotalSegments: 1, which main accepts, and the QueryKeyConditionslist with{"S": "p0"}. On the base, 45 fail. The 13 that pass cover rows the base already answers like the service. Those areLimit: 0,Limit: -2147483648,Limit: -2147483649,Limit: 2147483648,Segment: 0, TotalSegments: 1, andLimit: 0beside an empty or duplicate NS on Scan and on Query. The readable key and list values and theSegmentfault beside an empty key value pass there too. On the first commit of this branch, 14 fail. Those are the two three-fault rows, theLandMmember rows on Scan and on Query, and the eight unreadable key and list rows. On the second commit, 8 fail: the eight unreadable key and list rows.tests/test_query_scan.pythat pinnedValue '1000000' at 'segment'andValue '1000001' at 'totalSegments'now assert the service's text.Gates on the head:
cargo +1.97.0 fmt --all --check,cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings,cargo +1.97.0 check --locked --workspace --all-targets,cargo +1.88.0 check --locked --workspace, andcargo +1.97.0 test --locked -p extenddb-core -p extenddb-engine --lib(521 and 219 tests). The 58 wire tests pass on SQLite and PostgreSQL (pgvector/pgvector:pg16) servers built from this head. On the same servers, the 103 requests of the measurement set answer as measured, except 23 rows. Those are the 17 rows the next branch covers and the 6 known differences it leaves open. The 151 requests of 2026-11-05 answer as measured on the same servers, except the six withAttributesToGet: [].Not measured:
Limit: 1.5withSegment: -1). The code uses the older form, as the issue's rule says for any float or out-of-int32 field.ExpressionAttributeValuesentry that is not one AttributeValue. The probe measured onlyLimit: 0there. Beside the key and theScanFilterlist it measuredSegment: -1withTotalSegments: 2, and beside the keySegment: 0withTotalSegments: 1000001. The code uses the older form for every fault in the message.TotalSegmentsbelow 1 in the older form (TotalSegments: 0.0). The code reportsValue '0' at 'totalSegments' ... greater than or equal to 1.Limit: 0beside{"B": "!!!"}(not base64). The service answersSerializationException: Base64 encoded length is expected a multiple of 4 bytes but found: 3. main, the base and this branch answerSerializationException: Start of structure or map found where not expected: invalid base64: Invalid symbol 33, offset 0.The exception type matches and the text does not. The text comes from the AttributeValue deserializer, which this branch does not touch, so it waits for its own issue.Limit,SegmentorTotalSegmentsof the wrong JSON type (a string, a boolean, an array, an object) answersSerializationException, as before, with a different text:invalid type: string "5", expected i64becomesinvalid type: string "5", expected a JSON number. No measurement covers the service's text for these, and no test pins either text.Limitat0and-2147483649beside anExpressionAttributeValuesentry that is not one AttributeValue. It measured0,-1,-2147483648and-2147483649beside such a value in the key,KeyConditionsorQueryFilter, and0beside a readable one. A floatLimiton Query, and aLimitoutside int32 beside a readable entry, do not change here and are not measured.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.