[LeeroyHannigan]

Closes #366. Merge after #428: the LSI paragraph in docs/design/04-component-storage.md is written against the section 7 text #428 introduces (the **... Query:** headings, explicit ASC and DESC, the hash-only and no-base-sort-key paragraphs). Merged on top of #428 it replaces #428's LSI paragraph; merged first, #428 meets the same hunk when it is rebased and needs the three edits below. The conflict is one hunk, plus two #428 sentences that merge cleanly but describe the LSI cursor this change removes:

1. Conflict hunk in section 7: take this branch's LSI Query: paragraph and SQL block in place of docs(design): describe the query cursor as the row comparison the code emits #428's LSI Query: paragraph; keep docs(design): describe the query cursor as the row comparison the code emits #428's Hash-only index: paragraph and the "On a base table without a sort key" paragraph after it unchanged.
2. Section 7 intro: replace "For the Scan, base table Query, GSI and hash-only index shapes below, the ordering index serves the page in order with no sort step; the LSI shape is described after them." with "For every shape below, the ordering index can serve the page in order with no sort step."
3. Section 5.1: replace "so a GSI cursor is a row comparison the ordering index serves in one seek (the LSI case is in section 7)." with "so a GSI or LSI cursor is a row comparison the ordering index serves in one seek."

What the service does

Measured 2026-10-21 on DynamoDB with a (pk, sk) table and an LSI on (pk, lsk), 9 items in three groups of equal lsk: a Query on the LSI returns the items ordered by lsk, the items inside a group in one deterministic order (the same across two tables), and a reverse Query is the exact mirror of the forward one. Page boundaries and LastEvaluatedKey ({lsk, pk, sk}) follow that order. The order inside a group is not byte order of the base sort key; that difference is #425.

What ExtendDB did

Every index table whose index has a sort key has an ordering index on (pk, sk, base_pk, base_sk) on PostgreSQL and SQLite, with one sk column per RANGE attribute. A GSI Query orders by its RANGE columns in key-schema order, then base_pk and base_sk ((sk, base_pk, base_sk) with one RANGE attribute), the index without the constant pk, so the database walks the index and stops at the Limit. An LSI Query ordered by (sk, base_sk) with a cursor over the same columns. The planners cannot tell that base_pk is constant inside an LSI partition, so the index served sk only. PostgreSQL added an Incremental Sort over the index scan (a Sort when lsk = :l left one value); SQLite added a temporary B-tree for the last ORDER BY term and seeked a resumed page on sk alone. On a 50,000-item partition with 50 items per lsk value, a Limit 100 page read 151 rows and 154 buffers on PostgreSQL to return 101; a page resumed inside a group read 130 rows and 133 buffers.

Changes

• crates/storage-postgres/src/data/query_scan.rs: order_by_columns puts base_pk between the index sort key columns and the base sort key for every index with a sort key (the is_lsi parameter is gone from it and from build_pagination_where); an LSI cursor binds the queried partition key as base_pk; cursor unit tests updated.
• crates/storage-postgres/src/data/query.rs: PaginationBinds::BaseSkOnly removed (no remaining user).
• crates/storage-sqlite/src/data/query_scan.rs: the same column change in query_order_columns; append_query_pagination takes the queried partition key and binds it for an LSI; the statement builder moves from query_impl into query_statement with no behaviour change, so tests can run EXPLAIN QUERY PLAN on the statement a Query sends; cursor unit tests updated plus a bind test; new plan_tests module in crates/storage-sqlite/src/data/query_scan/plan_tests.rs.
• crates/storage-postgres/tests/lsi_query_plan.rs: new storage-level plan test. It reads the statement the engine sent from pg_stat_activity and asserts that the generic plan and the custom plan are an index walk: no Sort or Incremental Sort node, no Filter under the index scan, for a first page and a resumed page with the partition alone, with lsk = :l and with lsk BETWEEN, in both directions. It checks the page rows and that a start key naming another partition returns the same page. It skips below PostgreSQL 16 (EXPLAIN (GENERIC_PLAN)).
• crates/storage-postgres/tests/lsi_query_plan/multi_range_gsi.rs: the same plan check for a GSI with RANGE attributes a, b and c, a module of lsi_query_plan so the CI step runs it (see Rebase on fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429).
• .github/workflows/integration.yml: --test lsi_query_plan added to the PostgreSQL storage-level step (pgvector/pgvector:pg16).
• docs/design/04-component-storage.md: the LSI Query paragraph in section 7, with the statement, the bind, and the sort step the ordering index avoids.

The cursor token format and the LastEvaluatedKey shape do not change.

Verification

Servers built from the pre-rebase head e9d30451e, PostgreSQL 16.15 and SQLite in memory. Plans from the servers' own statements: PostgreSQL EXPLAIN (ANALYZE, BUFFERS) EXECUTE and EXPLAIN (GENERIC_PLAN); SQLite EXPLAIN QUERY PLAN. The rebase changed no LSI statement, and these rows were not repeated on 03d9c72e3.

Check	PostgreSQL	SQLite
LSI pages, 21 request shapes (first, resumed, FilterExpression, =, >, <, <=, >=, begins_with, BETWEEN, engine chunks), both directions	Index Scan or Index Scan Backward on the ordering index, cursor in the Index Cond, no sort node, custom and generic plan; rows read equal the LIMIT (was Incremental Sort or Sort, up to 130 rows for LIMIT 98)	SEARCH ... USING INDEX idx_order_*, no temporary B-tree, resumed seek on (sk_s, base_pk, base_sk_s) (was sk_s only plus a temporary B-tree); 1,196 VM opcodes per LIMIT 98 page, was 2,892
GSI, base table Query and Scan plans	unchanged	unchanged
Pages of the 52,000-item table before and after (every item key and LastEvaluatedKey)	identical	identical
9-item probe against the service output: items per page, group order, Count, LastEvaluatedKey keys	match on every query; order inside a group differs (#425)	same
LSI cases in test_item_operations.py, test_table_operations.py, test_put_item_create_race.py	3 passed	3 passed
key-type matrix (pk, lsk, sk in N, B, S combinations; full, paged, BETWEEN, COUNT queries both directions)	0 problems	0 problems, output identical to PostgreSQL

The rows below are for the rebased head. The servers (SQLite and PostgreSQL 16) and the PostgreSQL tests ran on 03d9c72e3, the rebased commit before the SQLite plan tests moved into their own file and the multi-RANGE case gained its row check; this head keeps that commit's non-test code and PostgreSQL tests. The SQLite plan_tests and cursor unit test rows ran on 16a1b92f1:

Check	PostgreSQL	SQLite
tests/test_query_scan.py + tests/test_multipart_gsi.py	126 passed	126 passed
tests/test_gsi_multi_range_query.py, tests/test_read_paging.py, tests/test_vector_index_query_scan.py	90 passed, 1 skipped (the case for a backend that refuses a condition past the first RANGE attribute)	90 passed, 1 skipped
tests/python/test_query_scan.py	27 passed	27 passed
crates/storage-postgres/tests/lsi_query_plan.rs (20 LSI plans, 32 multi-RANGE GSI plans)	2 passed	n/a
plan_tests (12 LSI statements, 4 GSI statements, 16 multi-RANGE GSI statements, the rows of the multi-RANGE GSI pages, LSI paging both directions)	n/a	4 passed
cursor unit tests	9 passed	9 passed

Negative controls

On the pre-rebase head e9d30451e: each mutation applied to the committed tree, the tests run, the file restored with git checkout, crates/ clean afterwards. The cursor unit test counts are that head's; main has since added one test per backend.

Mutation	Fails	Passes
PostgreSQL: base_pk filtered out of the LSI ORDER BY (all shapes)	lsi_query_plans_as_an_index_walk_in_both_directions: first page, forward=true, generic plan: the plan sorts the rows the index scan reads (Incremental Sort)	8 cursor unit tests
PostgreSQL: base_pk filtered out of the LSI ORDER BY only when the key condition has a sort key condition	same test: lsk = :l, first page, forward=true, generic plan: the plan sorts the rows the index scan reads (Sort, Sort Key: base_sk_s)	8 cursor unit tests
PostgreSQL: base_pk replaced by its bound value in the LSI cursor comparison	same test: resumed page, forward=true, generic plan: the plan filters rows after the index scan instead of seeking on them (Index Cond sk_s >= $2, Filter on the row comparison)	8 cursor unit tests
PostgreSQL: the start key's partition key bound instead of the queried one	same test: resumed page, forward=true: a start key naming partition q moved the page boundary
SQLite: base_pk filtered out of the LSI ORDER BY (all shapes)	plan_tests::lsi_query_walks_the_ordering_index_without_a_sort: LSI pk only forward=true resumed=false: the plan sorts in a temporary B-tree	10 (GSI plan test, paging test, cursor unit tests)
SQLite: base_pk filtered out of the LSI ORDER BY only with a sort key condition	same test: LSI lsk = :l forward=true resumed=false: the plan sorts in a temporary B-tree	10
SQLite: base_pk removed from the LSI cursor columns and bind	lsi_cursor_compares_the_ordering_index_columns, order_by_columns_match_the_cursor_columns_for_every_shape, lsi_cursor_binds_the_queried_partition_key_as_base_pk	8, the paging test among them: the rows do not change
SQLite: the start key's partition key bound instead of the queried one	lsi_cursor_binds_the_queried_partition_key_as_base_pk: left ["l", "other", "s"], right ["l", "p", "s"]	10

On 03d9c72e3, with the ORDER BY of an index with several RANGE attributes cut to its first RANGE column and the cursor untouched: plan_tests::multi_range_gsi_query_walks_the_ordering_index_without_a_sort fails with multi-RANGE GSI h only forward=true resumed=false: the plan sorts in a temporary B-tree (USE TEMP B-TREE FOR LAST 2 TERMS OF ORDER BY), and multi_range_gsi_query_plans_as_an_index_walk_in_both_directions fails on the rows of the first page ((a0, b1, c1, ..) where (a0, b0, c0, ..) was expected); the other plan tests pass.

On 16a1b92f1, with SQLite's query_statement applying the sort key condition to the first RANGE column instead of the one it names: plan_tests::multi_range_gsi_query_walks_the_ordering_index_without_a_sort fails on rows (multi-RANGE GSI a = :a AND b BETWEEN forward=true: rows, no items where 60 were expected); the other 167 SQLite --lib tests pass.

Gates

On 16a1b92f1: cargo +1.97.0 fmt --all --check clean; cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings clean; cargo test -p extenddb-storage-sqlite --lib: 168 passed, 1 ignored. On 03d9c72e3, whose non-test code and PostgreSQL tests this head keeps: cargo +1.97.0 check --locked --workspace --all-targets and cargo +1.88.0 check --locked --workspace build; cargo test -p extenddb-storage-postgres against PostgreSQL 16.15: lib 63, backup_restore 30, catalog_converge 5, doc_version_literals 3, gsi_online_build 12, gsi_stale_writer 1, key_collation 3, lsi_query_plan 2, migrate_data_db_name 2, put_create_race 9, stream_records_page_order 1, stream_shard_ownership 6, stream_shard_retention 1, table_snapshot 81 (2 ignored), twi_conflict 8, vector_control_plane 42, all passed. No dependency change. CI passed all 35 checks on 16a1b92f1.

Related behaviour

• An LSI ExclusiveStartKey whose partition key differs from the queried partition: the service returns HTTP 400 ValidationException, The provided starting key is outside query boundaries based on provided conditions. ExtendDB accepts it and resumes inside the queried partition, on main and on this branch; the storage layer does not read the start key's partition key, so the refusal belongs in request validation (Query accepts an ExclusiveStartKey from another partition and returns an empty page #440).
• The order of items that share an index sort key value: the service's order is not byte order of the base sort key (SQL backends: LSI Query orders items that share an index sort key differently from the service #425).
• A Query whose Limit lands on the last item of a partition, with no sort key condition: the service returns that page with a LastEvaluatedKey on some calls and without one on others (Query on an index returns a LastEvaluatedKey on the page that holds the last item; the service does so for base tables only #434, measured 2026-10-23); ExtendDB returns the key and then an empty page.
• Cassandra clusters an index table's rows on (sk, base_pk, base_sk) and resumes an LSI page with up to three statements over that order (sk = :s AND base_pk = :p AND base_sk > :b, sk = :s AND base_pk > :p, sk > :s), so its rows come back in clustering order.

Rebase on #429

Rebased on main at f49c5f82 (#429) on 1 November. #429 orders a Query on a GSI with several RANGE attributes over every RANGE column: on PostgreSQL order_by_columns and build_pagination_where, and on SQLite query_order_columns and append_query_pagination, take the queried key schema's RANGE attributes as a slice in place of the first one, and keep is_lsi. This change removes is_lsi from the same functions, so both query_scan.rs files conflicted. The resolved functions take #429's slice and drop is_lsi: an index with a sort key orders by its RANGE columns in key-schema order, then base_pk, then the base sort key, which is the column order of its ordering index after pk. An LSI cursor still binds the queried partition key as base_pk. GSI columns are the same as on main, with one RANGE attribute or several; only the LSI columns differ from main, as they did before the rebase. On SQLite, query_statement now holds #429's statement code, which applies each RANGE condition to the column of the attribute it names. .github/workflows/integration.yml, query.rs and the design doc merged without a conflict.

The plan tests now cover a GSI with RANGE attributes a, b and c: crates/storage-postgres/tests/lsi_query_plan/multi_range_gsi.rs, a module of the lsi_query_plan test so the CI step runs it, and a fourth plan_tests case on SQLite. Each runs the partition alone, an equality on a, an equality on a with b BETWEEN, and equalities on all three, first and resumed pages in both directions, and asserts a walk of the ordering index with no sort step. Both also check the rows of the pages: the PostgreSQL test of every page it plans, the SQLite case of every page of each shape read with Limit 7 in both directions. Their plan checks pass on main too, because main already orders such a GSI over every RANGE column. The SQLite plan tests are in their own file, crates/storage-sqlite/src/data/query_scan/plan_tests.rs, so query_scan.rs is 971 lines here (883 on main).

With the plan tests applied to main alone (on SQLite with the statement builder moved into query_statement, which the tests call), lsi_query_plans_as_an_index_walk_in_both_directions fails on the first page with an Incremental Sort over the index scan (presorted key sk_s), and lsi_query_walks_the_ordering_index_without_a_sort fails with USE TEMP B-TREE FOR LAST TERM OF ORDER BY; the GSI cases and the LSI paging test pass there. The second Verification table, the Gates and the last two Negative controls paragraphs are the rebased head's, on the commits they name; the first Verification table and the mutation table are the pre-rebase head's.

[LeeroyHannigan]

Rebased on main at abf8082; conflicts in .github/workflows/integration.yml, resolved by keeping both test entries: the PostgreSQL storage-level step runs gsi_online_build, gsi_stale_writer and lsi_query_plan, and its comment describes both.

The earlier merge of main is folded into the one commit. crates/storage-postgres (lsi_query_plan included, on PostgreSQL 16) and crates/storage-sqlite tests pass.