Repository navigation
fix(auth): bind LeadingKeys for a refused Query's partition operand - #511
Conversation
…y condition A Query whose partition key condition uses a non-equality operator, and a legacy KeyConditions entry with LE or BEGINS_WITH on the partition key, are shapes the engine rejects. Authorization extracted no dynamodb:LeadingKeys value for them, so a LeadingKeys condition was evaluated against no value and the shape's ValidationException was what every principal saw. Live DynamoDB binds LeadingKeys to the single operand the partition key's condition names and evaluates the policy before it rejects the shape. Measured 2026-10-21 for `pk > :v`, legacy `pk LE` and legacy `pk BEGINS_WITH` with the operand alice: ValidationException to the table owner and to a principal fenced to alice, AccessDeniedException to a principal fenced to nobody. Authorization now reads that operand. The expression path takes the single operand of the clause naming the partition key when no equality pins it, and the legacy path takes the partition attribute's one AttributeValueList entry whatever the operator. A condition that does not parse, one that names no partition clause, and a partition clause carrying no single operand (BETWEEN, IN) still carry no LeadingKeys, which is what the service answers for those. The equality shapes the engine runs keep the value they had.
…eadable The partition operand of a non-equality clause is bound only when no clause names the partition key by equality and exactly one clause names it at all. When a key condition such as `pk = :u AND pk > :v` carries a `:u` whose value cannot be read as the key's type, the second clause's operand was bound in its place. The key now stays absent there, as on main. Two non-equality partition clauses bind nothing, since which operand the service binds for them is not measured, and a legacy ComparisonOperator that is not one of the service's single-operand operators binds nothing. key_condition_clause_operands replaces key_condition_inequality_operands and reports every single-operand clause with whether it is an equality.
robinnsc
left a comment
There was a problem hiding this comment.
Looks good to me. On a local SQLite server, with an owner, principals fenced to alice and to nobody under ForAllValues, and ForAnyValue / Null-guarded variants, every shape the engine serves (base table, LSI, GSI, two-HASH GSI, legacy EQ, about 700 generated expressions) gave the same items and answers on head as on main, and the bound value always matched the partition the engine read. The three measured rows now answer as described: the owner and the alice principal get ValidationException and the nobody principal gets AccessDeniedException. tests/test_fgac_leading_keys.py is 31/31 on head and fails exactly those three rows on main. Query latency was the same within noise, though equality queries now parse their clauses twice.
A few smaller things:
- Heads-up on #429: this PR and #429 only conflict textually at the end of the
key_condition.rstest module, but there's a behavioural overlap too. #429's engine accepts multi-attribute clause orders thatequality_pk_valuestill resolves throughresolve_pk_sk's positional swap, so authorization finds no key for some of them. I've left the details on #429 since that's where the change is, but whichever of the two lands second might want to bind the equality on the first HASH attribute by name. - The description says an undefined
#nameor placeholder still carries no LeadingKeys. That holds when the#nameis the partition, but withpk > :vplus an undefined#qon another clause, an unused:w, a reserved word, the size limit, orKeyConditionExpressionplusKeyConditions, the fenced-away principal now gets AccessDeniedException where main gave ValidationException. Nothing is served, and main already does the same for the equality versions of those shapes, so it might just be a wording tweak in the description.
| equality: false, | ||
| }) | ||
| } | ||
| RawClause::Between(..) => None, |
There was a problem hiding this comment.
Since BETWEEN drops out here, pk BETWEEN :a AND :c AND pk > :b leaves one partition clause and binds :b, so a principal fenced away from :b gets AccessDeniedException where main gave ValidationException. pk > :a AND pk > :b still binds nothing, which is what the comment at leading_keys.rs:281 describes. The engine refuses both, so it's only the error code, but it might be worth returning BETWEEN as a partition clause with no operand so the "exactly one" check counts it.
| ] { | ||
| let input = json!({ | ||
| "TableName": "t", | ||
| "KeyConditions": {"pk": {"ComparisonOperator": "BETWEEN", |
There was a problem hiding this comment.
nit: BETWEEN is turned away by the operator list before the value-count check, so as far as I can tell nothing reaches list.len() != 1. Dropping that check passes all 52 leading_keys tests. Could consider a case with LE (or EQ) and two values.
| #[test] | ||
| fn clause_operands_reject_what_the_parser_rejects() { | ||
| for expr in [ | ||
| "pk :: :v", |
There was a problem hiding this comment.
nit: pk :: :v fails in the tokenizer (empty value placeholder ':'), so the let Ok(tokens) = ... else { continue } below skips it and this case checks nothing. Might be worth a syntax error that tokenizes, or asserting the tokenizer error for this one.
…fication On a key schema with several HASH or several RANGE attributes the engine assigns every key condition clause to its key attribute by name, so it serves clause orders such as `g2 = :g AND g1 = :v AND gs = :x` and `b > :lo AND a = :a AND h = :v`. Authorization still read the partition equality with `resolve_pk_sk`, found no partition key in those orders and left dynamodb:LeadingKeys absent, so a ForAllValues:StringEquals fence let the query read any partition of the index. `KeyCondition::resolve_key_schema_with` holds the choice between the multi-attribute classifier and `resolve_pk_sk`. The engine calls it, and so does `equality_pk_value`, which `query_leading_keys` calls for a condition with an equality on the partition attribute. When a clause compares the first HASH attribute by equality, authorization binds that attribute's value for every clause order the engine serves, and a condition the classifier refuses leaves the key absent. For `g1 = :a` on HASH `g1`, `g2` that gives the answer the service gives every principal, ValidationException (measured 2026-10-21). A single clause on the partition attribute under an operator the engine rejects still binds that clause's operand, as it does on main since #511. tests/test_fgac_leading_keys.py adds the three-clause orders from the review on a GSI with HASH `g1`, `g2` and RANGE `gs` and a GSI with HASH `h` and RANGE `a`, `b`, and expects ValidationException for the missing-g2 shape from the principal fenced away from the value. That shape was the only entry in the list of codes that still differed from the service, so the list is removed. It and tests/test_ttl.py read the backend through `configured_backend()` in tests/conftest.py, so this file can expect the MongoDB and Cassandra refusal of a condition on `b`.
run-rust-integration ran the storage-level tests after the wire-level Rust suite, 7.1 of its 27.4 minutes in the merge group for #511 on 30 Oct. They drive the storage layer against the job's PostgreSQL and need no server, so run-postgres-storage-tests runs them with the same image, environment and cargo command, without waiting for the release build. The integration summary job requires it.
…service does Closes #427. A Query whose KeyConditionExpression has no equality clause on the HASH attribute was refused with a parser sentence of ExtendDB's own. The service chooses the sentence from the key schema. Measured on 2026-10-23, every row repeated with identical answers: - Single HASH attribute, any clause names it (so the clause on it is a non-equality): "Query key condition not supported". Rows: pk > :x, pk BETWEEN :x AND :y, begins_with(pk, :x), pk > :x AND sk = :y, sk = :x AND pk > :y, zz = :x AND pk > :y, and the GSI forms. - Single HASH attribute, no clause names it: "Query condition missed key schema element: <HASH>". Rows: sk > :x, zz > :x, sk > :x AND zz > :y, zz = :x AND yy = :y, and the GSI forms naming the index HASH attribute. - Several key attributes (two HASH attributes, or one HASH with several RANGE attributes): "HASH key attributes h1, h2 must have equality conditions specified in the query", naming every HASH attribute, for every operator and for an N value on the HASH attribute. - Two conditions on one attribute: "Invalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key". - An undefined #name or :value in the key condition is refused first, with the "Invalid KeyConditionExpression: " prefix. The rows include pk = :x with no values. - Single HASH attribute, a value of the wrong type: pk > :n with an N value on an S key answers "One or more parameter values were invalid: Condition parameter type does not match schema type", and begins_with(pk, :n) on an N key answers "Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: N". - Legacy KeyConditions is desugared with every reference defined and an equality clause, or refused there; no change. Measured on 2026-11-01, two samples each, on a single HASH attribute schema: two conditions on one attribute answer the one condition per key sentence whatever their value types (pk > :x AND pk < :n). A wrong-typed value is reported only in the clause on the HASH attribute: pk > :n AND sk > :y, sk = :x AND pk > :n and pk BETWEEN :n AND :m answer the type sentence, and pk > :x AND sk > :n answers "Query key condition not supported". With no clause on the HASH attribute no value is checked: sk > :n answers "Query condition missed key schema element: pk", and on an index with HASH gsi1_pk and RANGE SK, PK > :n (PK the table's HASH attribute) and SK > :n answer "Query condition missed key schema element: gsi1_pk". Ordering, measured: unused values, filter syntax, projection syntax, the Select conflicts and undefined key references are reported before this refusal; an ExclusiveStartKey fault and an oversized key value after it. An unused value and an invalid FilterExpression are also reported before the type sentence for pk > :n (measured again on 2026-10-31). The refusal is in crates/core/src/expression/key_condition/no_hash_equality.rs, with KeyCondition::lacks_hash_equality, refuse_two_conditions_on_one_attribute, operands_naming and refuse_without_hash_equality, and KeyCondition::validate_references is in key_condition/references.rs. The parser now records one or two clauses with no equality instead of refusing them, so the engine can choose the sentence. handle_query skips the classification from #429 for a key condition with no equality clause, and on a single HASH attribute schema for an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through that classification as before. After validate_select_projection, handle_query calls the engine's refuse_without_hash_equality. It checks the references of every key condition and refuses the ones the classification skipped. It refuses two conditions on one attribute first. Then, on a single HASH attribute schema and only when a clause names the HASH attribute, it runs the type check and the begins_with check from #429 on that clause, against that attribute's definition. Then it refuses with the key schema's wording. On a multi-attribute key schema no value is checked, because the service answers h > :n with the HASH sentence. LeadingKeys keeps the binding from #511. query_kce_without_a_partition_key_equality_binds_only_a_partition_operand asserts the values it binds for the shapes the parser now records, and the_engine_still_refuses_every_shape_whose_operand_is_now_bound asserts that the core refusal answers "Query key condition not supported" for each shape it binds. Tests: unit tests in no_hash_equality/tests.rs, references.rs and crates/engine/src/query/no_hash_equality_tests.rs. The wire file tests/python/test_query_lone_non_equality.py has 81 rows. 67 of them passed against the service as a file; the other 14 repeat requests measured on 2026-10-23 (427-8c, row 6, and rows 5a and 5b with the N value) and on 2026-11-01 (the ten rows of TestValueTypes, two of them on the file's single-idx in place of the measured index). All 81 pass on SQLite and PostgreSQL 16 builds of this commit, and a SQLite build of main at f49c5f8 fails 74 of them. Related behaviour, measured: "<>", NOT, OR, IN and contains in a key condition get "Invalid operator used in KeyConditionExpression" on the service; pk = :x AND z > :x gets "missed key schema element: sk" on the service and 200 here; the Select ALL_PROJECTED_ATTRIBUTES text lacks the "1 validation error detected: " prefix here. A key condition with an equality clause and a HASH attribute without one (sk = :x on a table, g1 = :x on an index with HASH g1, g2) is refused by the key schema checks handle_query runs before it parses the FilterExpression. The service reports an unused value or an invalid FilterExpression first (measured 2026-10-31).
…service does Closes #427. A Query whose KeyConditionExpression has no equality clause on the HASH attribute was refused with a parser sentence of ExtendDB's own. The service chooses the sentence from the key schema. Measured on 2026-10-23, every row repeated with identical answers: - Single HASH attribute, any clause names it (so the clause on it is a non-equality): "Query key condition not supported". Rows: pk > :x, pk BETWEEN :x AND :y, begins_with(pk, :x), pk > :x AND sk = :y, sk = :x AND pk > :y, zz = :x AND pk > :y, and the GSI forms. - Single HASH attribute, no clause names it: "Query condition missed key schema element: <HASH>". Rows: sk > :x, zz > :x, sk > :x AND zz > :y, zz = :x AND yy = :y, and the GSI forms naming the index HASH attribute. - Several key attributes (two HASH attributes, or one HASH with several RANGE attributes): "HASH key attributes h1, h2 must have equality conditions specified in the query", naming every HASH attribute, for every operator and for an N value on the HASH attribute. - Two conditions on one attribute: "Invalid KeyConditionExpression: KeyConditionExpressions must only contain one condition per key". - An undefined #name or :value in the key condition is refused first, with the "Invalid KeyConditionExpression: " prefix. The rows include pk = :x with no values. - Single HASH attribute, a value of the wrong type: pk > :n with an N value on an S key answers "One or more parameter values were invalid: Condition parameter type does not match schema type", and begins_with(pk, :n) on an N key answers "Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: N". - Legacy KeyConditions is desugared with every reference defined and an equality clause, or refused there; no change. Measured on 2026-11-01, two samples each, on a single HASH attribute schema: two conditions on one attribute answer the one condition per key sentence whatever their value types (pk > :x AND pk < :n). A wrong-typed value is reported only in the clause on the HASH attribute: pk > :n AND sk > :y, sk = :x AND pk > :n and pk BETWEEN :n AND :m answer the type sentence, and pk > :x AND sk > :n answers "Query key condition not supported". With no clause on the HASH attribute no value is checked: sk > :n answers "Query condition missed key schema element: pk", and on an index with HASH gsi1_pk and RANGE SK, PK > :n (PK the table's HASH attribute) and SK > :n answer "Query condition missed key schema element: gsi1_pk". On a table with HASH pk (N) and RANGE sk (S), begins_with(pk, :n) AND sk > :x and sk > :x AND begins_with(pk, :n) answer the begins_with operand type sentence, as begins_with(pk, :n) alone does. Ordering, measured: unused values, filter syntax, projection syntax, the Select conflicts and undefined key references are reported before this refusal; an ExclusiveStartKey fault and an oversized key value after it. An unused value and an invalid FilterExpression are also reported before the type sentence for pk > :n (measured again on 2026-10-31). The refusal is in crates/core/src/expression/key_condition/no_hash_equality.rs, with KeyCondition::lacks_hash_equality, refuse_two_conditions_on_one_attribute, operands_naming and refuse_without_hash_equality, and KeyCondition::validate_references is in key_condition/references.rs. The parser now records one or two clauses with no equality instead of refusing them, so the engine can choose the sentence. When only the second of two such clauses is a begins_with, the parser keeps that clause whole as the sort key condition, which the begins_with check reads. handle_query skips the classification from #429 for a key condition with no equality clause, and on a single HASH attribute schema for an equality on another attribute paired with a non-equality on the HASH attribute (sk = :x AND pk > :y). Every other key condition goes through that classification as before. After validate_select_projection, handle_query calls the engine's refuse_without_hash_equality. It checks the references of every key condition and refuses the ones the classification skipped. It refuses two conditions on one attribute first. Then, on a single HASH attribute schema and only when a clause names the HASH attribute, it runs the type check and the begins_with check from #429 on that clause, against that attribute's definition. Then it refuses with the key schema's wording. On a multi-attribute key schema no value is checked, because the service answers h > :n with the HASH sentence. LeadingKeys keeps the binding from #511. query_kce_without_a_partition_key_equality_binds_only_a_partition_operand asserts the values it binds for the shapes the parser now records, and the_engine_still_refuses_every_shape_whose_operand_is_now_bound asserts that the core refusal answers "Query key condition not supported" for each shape it binds. Tests: unit tests in no_hash_equality/tests.rs, references.rs and crates/engine/src/query/no_hash_equality_tests.rs. The wire file tests/python/test_query_lone_non_equality.py has 83 rows. 67 of them passed against the service as a file; the other 16 repeat requests measured on 2026-10-23 (427-8c, row 6, and rows 5a and 5b with the N value) and on 2026-11-01 (the ten rows of TestValueTypes, two of them on the file's single-idx in place of the measured index, and the two begins_with rows on a table with HASH pk (N) and RANGE sk). All 83 pass on SQLite and PostgreSQL 16 builds of this commit, and a SQLite build of main at f49c5f8 fails 76 of them. Related behaviour, measured: "<>", NOT, OR, IN and contains in a key condition get "Invalid operator used in KeyConditionExpression" on the service; pk = :x AND z > :x gets "missed key schema element: sk" on the service and 200 here; the Select ALL_PROJECTED_ATTRIBUTES text lacks the "1 validation error detected: " prefix here. A key condition with an equality clause and a HASH attribute without one (sk = :x on a table, g1 = :x on an index with HASH g1, g2) is refused by the key schema checks handle_query runs before it parses the FilterExpression. The service reports an unused value or an invalid FilterExpression first (measured 2026-10-31).
What this fixes
Closes #448.
A
Querywhose partition key condition uses a non-equality operator, and a legacyKeyConditionsentry using
LEorBEGINS_WITHon the partition key, are shapes the engine refuses. Authorizationread no
dynamodb:LeadingKeysvalue for them, so a policy carrying aLeadingKeyscondition wasevaluated against no value and every principal saw the shape's
ValidationException. A principalfenced away from the value the request named got
ValidationExceptionwhere DynamoDB answersAccessDeniedException.What the service does
Measured against live DynamoDB on 2026-10-21 at 23:14 UTC, us-east-1, on a table with partition key
pk(S) and sort keysk(S), with three principals: the table owner, whose policy carries noLeadingKeyscondition; principal A, alloweddynamodb:*on the table underForAllValues:StringEquals dynamodb:LeadingKeys ["alice","galice","10"]; and principal Z, the sameAllow with the single value
["nobody"]. Every request namedaliceas the partition operand.KeyConditionExpression: "pk > :v"Query key condition not supportedKeyConditions: {pk: {ComparisonOperator: LE, AttributeValueList: [alice]}}KeyConditions: {pk: {ComparisonOperator: BEGINS_WITH, AttributeValueList: [alice]}}A was allowed
aliceand got the shape error; Z was not and was denied, so the policy was evaluatedagainst a
LeadingKeysvalue ofalicebefore the shape was refused. The owner's row shows thatbinding the key changes nothing for a policy with no condition on it.
Four other refused shapes in the same run returned
ValidationExceptionto all three principals: asyntax error (
"pk :: :v"), a condition on the sort key alone ("sk = :s"), an undefinedExpressionAttributeNamesreference ("#p = :v"), and a query on a two-attribute-partition globalsecondary index naming one of the two. The first three name no partition operand that can be read.
The change
crates/server/src/leading_keys.rsreads the clauses that name the queried partition attribute.When one of them is an equality, that equality decides the key, as the engine reads it; when its
value cannot be read (an N value that is not a number, an undefined placeholder) the key stays
absent, as on main, and no other clause's operand is bound in its place. When no clause names the
partition attribute by equality and exactly one clause names it, under
<,<=,>,>=orbegins_with, that clause's operand is bound. Two non-equality clauses on the partition attributebind nothing. On the legacy path the partition attribute's entry is read when its
ComparisonOperatoris one of the service's single-operand operators and itsAttributeValueListholds exactly one value.
crates/core/src/expression/key_condition.rsgrowskey_condition_clause_operands, which returnsthe attribute path and single operand of each clause and whether it is an equality. It runs the
clause loop and the pre-classification checks
parse_key_conditionalready ran, now shared throughparse_clauses, so both functions refuse the same key conditions.Everything that carried no
LeadingKeysbefore still carries none: an expression that does nottokenize or parse, an undefined
#nameor placeholder, a condition naming no partition clause, anunknown
IndexName, a table with no key info. ABETWEENon the partition key, and legacyBETWEENand
IN, name more than one value and no record says which the service binds, so they are left asthey are.
Tests
tests/test_fgac_leading_keys.pyalready sent all seven refused shapes as all three principals andcompared them with the service's rows. It accepted either error code for the three shapes above;
it now requires the code the service returned. The run on main fails those three cases and passes at
this head.
crates/server/src/leading_keys/tests.rscovers the three measured shapes, the operators that followthe same rule, an N operand (normalized, as on the equality path for a
Query), a non-equalitycondition on a queried index's own partition key, a name reference that resolves and one that does
not, an unknown placeholder, an unreadable partition equality next to another partition clause, two
non-equality partition clauses, unknown legacy operators, and the shapes that stay absent. One test
asserts that the engine's own parser still refuses every single-clause shape whose operand is now
bound. The two-clause shapes it binds (
pk > :v AND sk = :s,pk > :v AND sk < :s) are refused bythe engine when it resolves the partition attribute, which a wire test checks.
tests/test_fgac_leading_keys.pyalso coverspk = :u AND pk > :vwhere:ucannot be read, andchecks that
aliceis not bound in its place.crates/core/src/expression/key_condition.rscovers the clause list for thirteen expressions and therefusals it inherits.
Not measured
The three principals of that run were deleted at the end of it and no IAM principal can be created on this
host, so no row covers these:
<,<=,>=andbegins_with(pk, :v)in aKeyConditionExpression, legacyNE,LT,GT,GE,CONTAINSandNOT_CONTAINS, and anon-equality partition condition combined with a sort key condition, two non-equality conditions on
the partition key, and a legacy
ComparisonOperatorthat is not one of the service's operators.Each single-clause shape names one partition operand, as the three measured shapes do, and is bound
the same way; the two-clause and unknown-operator shapes bind nothing.
The two-attribute-partition index shape is a separate difference in the other direction: the service
returned
ValidationExceptionto all three principals and ExtendDB binds the index's first partitionattribute and denies Z. This branch does not change that. The test records it as a divergence rather
than asserting the service's code.
Merges
Of the 66 pull requests open on 2026-10-24, six change a file this branch changes. #484 and #429
conflict in
crates/core/src/expression/key_condition.rs: each adds to a region this branch alsoadds to, and neither rewrites the other's lines. #484's duplicate-key check belongs at the tail of
parse_clauses, where both callers get it. #399 and #507 (stacked on #399) conflict in thepub useblock of
crates/core/src/expression/mod.rs; the merge keeps both sets of exports. #508 mergesclean.
#479 merges clean textually and collides in behaviour. It moves the refusal of a key condition with no
equality out of
parse_key_conditionintoKeyCondition::refuse_without_hash_equality, which theengine calls with the key schema. On the merged tree this fix still binds
aliceforpk > :v, andtwo tests need updating: #479's
query_kce_without_a_partition_key_equality_leaves_the_key_undetermined, which asserts no key isbound for
pk > :v, and this branch's assertion thatparse_key_conditionrefuses that shape, whichhas to move to the engine's call. Whichever lands second carries both edits.