Repository navigation
Query and Scan: a page stops before the item that takes it past 1 MB; the service keeps that item and ends the page on it #815
Description
ExtendDB ends a Query or Scan page before the item whose size would take the page's evaluated bytes past 1 MB (1,048,576 bytes), and that item comes back on the next page. The service keeps the item that crosses 1 MB, ends the page on it, and returns that item's key as LastEvaluatedKey. On a partition of three 380 KB items, the service's first page holds all three, and ExtendDB's holds two.
Measured against Amazon DynamoDB in us-east-1, one request per page: the classic Query and Scan on 4 November 2026, and the same partition through ExecuteStatement on 2 November 2026. Table pk S HASH, sk S RANGE; partition a holds a/1, a/2 and a/3, each with a blob of 380,000 bytes (380,010 bytes per item); partition b holds one small item. The classic requests set ReturnConsumedCapacity to TOTAL. ExtendDB answers for the first two rows are from main eaf16e89e (SQLite, debug build). The ConsistentRead and Scan rows are from a SQLite debug build of the #411 branch on main 74c03690e, which does not change Query or Scan, and that build gives the same Query pages as main.
| Request | DynamoDB | ExtendDB (SQLite) |
|---|---|---|
Query pk = :p with :p = a, no Limit |
page 1: a/1, a/2, a/3, LastEvaluatedKey a/3, 139.5 capacity units; page 2: no items, 0.5 |
page 1: a/1, a/2, LastEvaluatedKey a/2, 93.0 capacity units; page 2: a/3, no LastEvaluatedKey, 46.5 |
the same Query with ProjectionExpression pk, sk |
the same two pages and units | the same two pages and units |
the same Query with ConsistentRead |
page 1: the three items, LastEvaluatedKey a/3, 279.0; page 2: no items, 1.0 |
page 1: a/1, a/2, LastEvaluatedKey a/2, 186.0; page 2: a/3, 93.0 |
Scan, no Limit |
page 1: b/1, a/1, a/2, a/3, LastEvaluatedKey a/3, 141.0; page 2: no items, 0.5 |
page 1: a/1, a/2, LastEvaluatedKey a/2, 93.0; page 2: a/3, b/1, no LastEvaluatedKey, 46.5 |
ExecuteStatement SELECT * FROM "<t>" WHERE pk = 'a', no Limit |
page 1: a/1, a/2, a/3, LastEvaluatedKey a/3 and a NextToken; page 2: no items |
main has no PartiQL; the #411 branch gives the service's pages |
A PartiQL page that reads several partitions applies the same rule against one 1 MB budget: with one 380 KB item in each of partitions a, b and c, pk IN ['a', 'b', 'c'] returns all three on the first page with LastEvaluatedKey c/1 (measured on 3 November 2026). A PartiQL page of whole-item reads by full primary key works the other way and stops before the item that would take it past 1 MB: on a hash-only table with three 380 KB items, pk IN ['a', 'b', 'c', 'd'] returns a and b, then c and d (measured on 2 November 2026).
Where the answer comes from
crates/engine/src/read_helpers.rs. PostReadAccumulator::push refuses the item when self.scanned_count > 0 && self.evaluated_bytes + item_bytes > PAGE_BYTE_BUDGET (line 152, with PAGE_BYTE_BUDGET = 1,048,576 at line 76), and read_page then ends the page with the last evaluated item's key. handle_query (crates/engine/src/query.rs, line 523) and handle_scan (crates/engine/src/scan.rs, line 473) both read their pages through read_page.
Not measured
Query and Scan with Limit, and with a FilterExpression that passes no item; Queries and Scans of secondary indexes; the PostgreSQL, MongoDB and Cassandra backends.
Related
#411 reads each PartiQL partition through handle_query. To match the service, it reads the item that crosses 1 MB with one more Query (see the page rules documented in crates/engine/src/partiql/select/steps.rs on that branch), so a change to read_page changes that path as well.
Activity